--- title: 'Container Orchestration' source: 'https://academia.sh/en/courses/orchestration' language: en updated: '2026-09-13T05:08:25+00:00' license: 'CC BY-SA 4.0' lessons: 30 outcomes: - "Explains the reconciliation loop and the cluster components' duties with measurements" - 'Deploys and scales an application with declarative objects' - 'Ties resource, security, and deployment policies to numbers' --- # Container Orchestration Delegating to a system the decision of how many replicas run where and when: the limits of manual management, the duties of control plane and node components, the reconciliation loop that continuously compares desired state against actual state, the core objects reaching from the smallest deployment unit to replica sets and declarative version management, stable in-cluster addressing and external access, configuration, secret and persistent storage objects, the effect of resource requests and limits on the scheduler's decision, placement constraints, eviction and priorities, metric-based autoscaling, deployment patterns, health checks, authorization and network policies, templating, resource type extension, encoding operational knowledge into a controller, and multi-cluster operation. ## Cluster Architecture The decision moving from human to system: where manual container management runs out, the control plane splitting into the API server, scheduler, controller manager, and state store, the duties of the node agent, runtime, and network proxy on the node side, the reconciliation loop that continuously compares desired state against actual state measured in delay and work, and the cost of the decision to run the cluster yourself. ### The Need for Orchestration Source: https://academia.sh/en/courses/orchestration/need-for-orchestration Manual management is scanned across machine and service counts: decision count grows linearly while the information examined per decision grows as a product, and failure-detection delay ties to round duration. The threshold for delegation is computed in days, and the number of copies the system misplaces when one of the six information items a decision depends on goes unwritten is counted. All of the previous course's measurements ran on a single machine: the output became portable, its identity was derived from its content, and its isolation and authority were counted item by item. One question remained — a human being still decided how many copies of that output would run, on which machine, and when. This lesson measures that decision itself: how it grows with scale when given manually, what a system needs to know when the decision is delegated to it, and where delegating fails to pay off. **Orchestration** is the work of delegating placement, replica count, and post-failure restart decisions for workloads running across multiple machines to a system. This course's unit of measure is **the cost of delegation**: every delegated decision requires the information it depends on to be produced continuously and on time. The measurement runs, once again, on the **regional measurement network**: **fictional** software that collects, verifies, and bills water-meter readings and opens field work orders; the nightly batch job is part of the same network. What runs here is not a real cluster but a **model** built with `node`; the numbers are the model's numbers, and the model's assumptions are written below. **CA1.** The network consists of four services and each service has three copies; machine count and service count are scanned separately. **CA2.** The manual step cost is fixed: placing one copy takes 4 steps, preparing one machine takes 6 steps, visually inspecting one copy takes 20 seconds. **CA3.** Failures come from a 32-bit generator we wrote ourselves; the seed `20260401` appears in the output, and the daily failure probability per copy is 0.15. **CA4.** A human inspects the copies in sequence, making rounds; two scenarios are run — a continuous watch and a single eight-hour shift. **CA5.** Under delegated management, preparation per machine is 9 steps instead of 6 (the node agent is also installed), the reporting period is 10 seconds, and the reconciliation period is 15 seconds. **CA6.** The placement constraint is real: collector copies can run only on machines with a physical connection to the meter network; four of the sixteen machines qualify. #### Manual Management Grows at Two Different Speeds ```js // scale.mjs — scans manual management across machine and service counts. This is a model; a // real cluster does not run. Time is in seconds, a "step" is one human operation. export const generator = (seed) => { let s = seed >>> 0; // our own generator: 32-bit, seed visible return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const A_COPY = 4, A_MACHINE = 6, D_TIME = 20, DAY = 86400, SHIFT = 8 * 3600; export function manual({ M, S, r, failureRate, seed, continuous }) { const N = S * r, rnd = generator(seed), T = N * D_TIME; // T: the duration of one inspection round const failures = []; for (let i = 0; i < N; i++) if (rnd() < failureRate) failures.push({ i, t: rnd() * DAY }); let delay = 0; for (const a of failures) { let t = a.t; if (!continuous && t % DAY >= SHIFT) t = Math.floor(t / DAY) * DAY + DAY; // outside the shift const position = a.i * D_TIME; // the copy's position within the round let d = Math.ceil((t - position) / T) * T + position; if (d < t) d += T; if (!continuous && d % DAY >= SHIFT) d = Math.floor(d / DAY) * DAY + DAY + position; delay += d - a.t; } const decision = N + S + failures.length; // placement + how many copies + re-place return { N, decision, scan: (N + failures.length) * M, roundDuration: T, steps: A_MACHINE * M + A_COPY * N + A_COPY * failures.length, failureCount: failures.length, avgDelay: failures.length ? delay / failures.length : 0 }; } if (import.meta.url === `file://${process.argv[1]}`) { // the second block imports this file console.log("M S copies decisions scan steps round(min) failures delay(min) delay-single-shift"); for (const [M, S] of [[1,4],[2,4],[4,6],[8,8],[16,12],[32,16]]) { const g = { M, S, r: 3, failureRate: 0.15, seed: 20260401 }; const k = manual({ ...g, continuous: true }), v = manual({ ...g, continuous: false }); console.log(`${String(M).padStart(2)} ${String(S).padStart(2)} ${String(k.N).padStart(6)}` + `${String(k.decision).padStart(7)}${String(k.scan).padStart(7)}${String(k.steps).padStart(6)}` + `${(k.roundDuration/60).toFixed(1).padStart(9)}${String(k.failureCount).padStart(7)}` + `${(k.avgDelay/60).toFixed(1).padStart(13)}${(v.avgDelay/60).toFixed(1).padStart(21)}`); } } ``` ``` M S copies decisions scan steps round(min) failures delay(min) delay-single-shift 1 4 12 21 17 74 4.0 5 1.2 394.8 2 4 12 21 34 80 4.0 5 1.2 394.8 4 6 18 29 92 116 6.0 5 2.4 394.8 8 8 24 38 240 168 8.0 6 3.6 330.2 16 12 36 56 704 272 12.0 8 6.5 363.5 32 16 48 74 1856 424 16.0 10 7.7 324.5 ``` Two columns grow at two different speeds. **Decision count is linear**: from 21 to 74, three and a half times. Manual steps are linear too: from 74 to 424. Looking at these two numbers alone, someone could conclude that manual management scales without trouble, because copy count also grew by exactly a factor of four. The **scan column**, over the same scale, rises from 17 to 1,856 — one hundred and nine times. Scan is how many machines' state gets examined while making one placement decision. Decision count grows with copy count; the information examined per decision grows with machine count; the product of the two does not grow at the same speed as decision count itself. What makes manual management expensive is not the decision count but the information-gathering work underneath every decision. The third measure is detection delay. Under a continuous watch, delay rises from 1.2 minutes to 7.7 minutes and looks linear: one inspection round lengthens with copy count, and average wait runs about half a round. The single-shift column says something entirely different — from 394.8 to 324.5 minutes, meaning it does not grow with scale; it even falls. This is not a computation error but a change in the dominant term: under a single shift, the source of the delay is not round duration but the night itself. The measurement network's nightly batch job starts at 02:30; a collector copy that goes down around then stays down until the shift opens, and adding more copies does not lengthen this wait. #### The Threshold for Delegation ```js // threshold.mjs — compares manual management against delegated management on the same items. import { manual, generator } from "./scale.mjs"; const SETUP = 120, REPORT = 9, A_NODE = 9, A_COPY = 4, DAY = 86400; const P = 10, U = 15; // reporting and reconciliation period (seconds) function delegated({ M, S, r, failureRate, seed }) { const N = S * r, rnd = generator(seed), failures = []; for (let i = 0; i < N; i++) if (rnd() < failureRate) failures.push({ i, t: rnd() * DAY }); let g = 0; for (const a of failures) g += Math.ceil(Math.ceil(a.t / P) * P / U) * U - a.t; return { setup: SETUP + REPORT * S + A_NODE * M, failureCount: failures.length, avgDelay: failures.length ? g / failures.length : 0 }; } console.log("copies manual-setup manual/day deleg-setup threshold(day) manual-delay(min) deleg-delay(sec)"); for (const [M, S] of [[1,2],[2,4],[4,6],[8,8],[16,12],[32,16]]) { const g = { M, S, r: 3, failureRate: 0.15, seed: 20260401 }; const e = manual({ ...g, continuous: true }), d = delegated(g); const eSetup = e.steps - A_COPY * e.failureCount, eDay = A_COPY * e.failureCount; console.log(`${String(e.N).padStart(6)}${String(eSetup).padStart(14)}${String(eDay).padStart(12)}` + `${String(d.setup).padStart(13)}${Math.ceil((d.setup - eSetup) / eDay).toString().padStart(16)}` + `${(e.avgDelay/60).toFixed(1).padStart(19)}${d.avgDelay.toFixed(1).padStart(18)}`); } console.log("\nfailure rate failures/day manual/day deleg-setup threshold(day) [M=8, S=8, 24 copies]"); for (const o of [0, 0.02, 0.05, 0.10, 0.15, 0.30]) { const g = { M: 8, S: 8, r: 3, failureRate: o, seed: 20260401 }; const e = manual({ ...g, continuous: true }), d = delegated(g); const eSetup = e.steps - A_COPY * e.failureCount, eDay = A_COPY * e.failureCount; console.log(`${o.toFixed(2).padStart(11)}${String(e.failureCount).padStart(11)}` + `${String(eDay).padStart(10)}${String(d.setup).padStart(11)}` + `${(eDay ? String(Math.ceil((d.setup - eSetup) / eDay)) : "infinite").padStart(11)}`); } // Unwritten constraint: collector copies can run only on machines connected to the meter network. const machines = Array.from({ length: 16 }, (_, i) => ({ name: i, meterNetwork: i % 5 === 0 })); const copies = [...Array(6).fill("collector"), ...Array(4).fill("verifier"), ...Array(3).fill("billing"), ...Array(3).fill("work-order")]; for (const written of [false, true]) { const load = machines.map(() => 0), placement = []; for (const k of copies) { const fit = machines.filter((m) => !(written && k === "collector") || m.meterNetwork); const s = fit.reduce((a, b) => (load[a.name] <= load[b.name] ? a : b)); load[s.name]++; placement.push({ k, m: s }); } const wrong = placement.filter((y) => y.k === "collector" && !y.m.meterNetwork).length; console.log(`constraint reported ${written ? "written " : "unwritten"}: copies ${placement.length}` + ` wrong placement ${wrong} fix steps ${wrong * A_COPY}` + ` report items ${written ? 5 : 4}`); } ``` ``` copies manual-setup manual/day deleg-setup threshold(day) manual-delay(min) deleg-delay(sec) 6 30 4 147 30 1.0 2.0 12 60 20 174 6 1.2 8.1 18 96 20 210 6 2.4 8.1 24 144 24 264 5 3.6 8.7 36 240 32 372 5 6.5 9.1 48 384 40 552 5 7.7 8.0 failure rate failures/day manual/day deleg-setup threshold(day) [M=8, S=8, 24 copies] 0.00 0 0 264 infinite 0.02 2 8 264 15 0.05 3 12 264 10 0.10 6 24 264 5 0.15 6 24 264 5 0.30 10 40 264 3 constraint reported unwritten: copies 16 wrong placement 4 fix steps 16 report items 4 constraint reported written : copies 16 wrong placement 0 fix steps 0 report items 5 ``` The setup cost of delegating is higher than manual setup at every scale: 147 against 30 steps at 6 copies, 552 against 384 at 48 copies. Delegating is always a loss on day one. The difference shows up in the daily item: under manual management, failures are handled by hand every day; under delegated management, the daily manual step count is zero. The threshold column gives the day on which the two costs cross. This threshold drops fast with scale. In a six-copy setup, manual management stays cheaper for thirty days; at twelve copies that falls to six days, and above twenty-four it falls to five. Thirty days is a real answer for a single-service trial environment or a one-month pilot — at that scale, delegating is unnecessary work. Five days is not an answer; the measurement network's four services and twelve copies sit in the region where delegating the decision pays for itself within a week. The delay columns show a sharper split than the threshold does: 1.0–7.7 minutes under manual management, 2.0–9.1 seconds under delegated management. Under delegated management, this number does not grow with scale, because the source of the delay is not scanning but two fixed periods — the reporting period and the reconciliation period. As scale grows, manual management's real loss accumulates not in step count but here. #### The Six Items the System Needs to Know The delegated decision can be written in one sentence: which copy runs on which machine, and what happens when it goes down. For the system to make this decision, it must continuously know the following items. | Information item | Source | If it goes stale | |---|---|---| | machine list | node registration | placement onto a machine that no longer exists | | machine capacity | node report | placement onto a machine whose capacity is full | | machine health | periodic report | assuming a copy is running on a dead machine | | desired copy count | human-written entry | too few or too many copies | | copies' actual state | report from the node | detecting a dropped copy late | | placement constraints | human-written entry | placement onto a machine that cannot run it | Four of the six items are measurements the system collects itself; two are items a human must write. This split is also a split in cost. The four collected items produce a continuous reporting load: every machine must report its capacity, its health, and the state of the copies on it in every period, and delegated management's nine-second detection delay is exactly the cost of that period. Widening the period reduces the load and widens the delay. The two written items, by contrast, are one-time, but if left unmaintained they go quietly wrong — once the desired copy count is written and forgotten, the system keeps working toward that number and no one suspects it. The cost of the last item sits in the output's final two lines. While the constraint was unwritten, the placer put four of the six collector copies on machines without a meter-network connection; those four copies start but cannot pull readings. Once the constraint was added to the report, wrong placement dropped to zero and the report's item count rose from four to five. One written item prevented four wrong placements — this is exactly the cost of delegation, and it is paid up front. #### Three Cases Where Delegating Does Not Pay Off The first is below the threshold: in a six-copy setup, delegating is a loss for thirty days, and if the setup will not live thirty days, it never pays off at all. The second is a low failure rate, and the second scan gives it a number. In the same twenty-four-copy setup, the threshold is three days at a failure rate of 0.30; it rises to ten days at 0.05, to fifteen days at 0.02, and becomes infinite once the rate hits zero — a setup with no failures at all never earns delegating back. Delegating the decision makes failure cheaper, not the absence of failure; the entire gain comes from the failure item, and the gain shrinks as that item shrinks. The third, and the most expensive, is that the information the decision depends on is not writable. The constraint experiment was the easy version of this: the constraint could be written in one line. There are constraints that cannot be written — a machine being physically reachable by the field crew, a service having to sit in a specific building under a contract with a specific customer. As long as these items never enter a report, the system stays unaware of them, and every item it is unaware of produces a class of wrong decisions. Under manual management, this information sits in a person's head and looks like it costs nothing; once delegated, it shows up as a cost for the first time. The decision to delegate is therefore as much a writability decision as a scale decision: how much of the information feeding the decision can actually go into a file. #### Summary - Under manual management, decision count grew linearly (21 → 74) while information examined per decision grew as a product (17 → 1,856); what was expensive was not the decision but the information-gathering work underneath it. - Failure-detection delay grew linearly with round duration under a continuous watch (1.2 → 7.7 minutes); under a single shift it stayed in the 325–395-minute range and was independent of scale, because the dominant term was the night itself. - Delegating is a loss at every scale on day one; the threshold is 30 days at 6 copies, 6 days at 12 copies, and 5 days above 24 copies. - Under delegated management, detection delay stayed in the 2–9 second range and did not grow with scale; its source was not scanning but two fixed periods. - The placement decision depends on six information items; when the constraint item went unwritten, four of six collector copies were placed on machines that could not run them, and one written item brought that to zero. #### Next Step Delegating the decision requires a system in exchange, and that system has to collect and store the six information items, produce the decision, and apply the result. None of this work sits in a single part. The next lesson splits the party that takes on the decision — the control plane — into its parts: the API server, the scheduler, the controller manager, and the state store. What gets measured is how many components a request passes through end to end, which information each component reads and writes, and, when one component goes down, which decisions stop and which keep going. ### Control Plane Components Source: https://academia.sh/en/courses/orchestration/control-plane-components The party that takes on the decision is split into four parts, and a single request is traced end to end: how many components it passes through, which information each component reads and writes, and how many reads and writes land on the state store are counted. Then each component is brought down one at a time, and which decisions stop and which keep going is measured. The previous lesson counted that the placement decision depends on six information items, and that four of these items must be collected continuously. The party that collects, stores, produces the decision, and applies the result is not a single part. This lesson splits that party — the **control plane** — into four components and measures two things: a request's end-to-end trace, and which decisions stop when one component goes down. The components are named by their duties: the **API server** (the single write gate, plus validation), the **state store** (the durable home for every item), the **scheduler** (the placement decision), the **controller manager** (the work of closing the gap between desired and actual). The node side stands here as a single collective part; the next lesson splits it into its own parts. The measurement runs, once again, on the **regional measurement network**: **fictional** software that collects and verifies water-meter readings. What runs is not a real cluster but a **model** built with `node`. **CA7.** The schema is our own: three key prefixes — `service/`, `copy/`, `node/`. **CA8.** The control plane is four components; the node side is modeled as a single collective part. **CA9.** The API server is the single write gate and has no state of its own; every write lands on the state store. **CA10.** A read is a single operation that scans one prefix; the number of items scanned counts as read load. **CA11.** The starting state is two services (3 collector copies, 2 verifier copies) and three nodes; node capacity is four. **CA12.** When one component goes down, only that component stops; the network and the nodes stay up. A copy's death is also recorded in the ledger by passing through the API server. #### Four Components, One Write Gate ```js // plane.mjs — the control plane's four components and the state store. Model; not a real cluster. // The schema is our own: service/, copy/, node/. export function create(initial) { const data = new Map(Object.entries(initial)); const trace = []; let rev = 0, active = "-"; const d = { data, trace, get rev() { return rev; }, read(prefix) { const c = [...data].filter(([a]) => a.startsWith(prefix)); trace.push({ b: active, i: "read", a: prefix + "*", n: c.length }); return c; }, write(a, v) { data.set(a, v); rev++; trace.push({ b: active, i: "write", a, n: 1 }); }, remove(a) { data.delete(a); rev++; trace.push({ b: active, i: "remove", a, n: 1 }); }, component(name) { active = name; }, }; return d; } // 1) API server: the single write gate. Validates the schema, writes to the store. Has no state of its own. export const apiServer = (d, request) => { d.component("API server"); if (!request.key.match(/^(service|copy|node)\//)) return "out of schema"; request.value === null ? d.remove(request.key) : d.write(request.key, request.value); return "written"; }; // 2) Controller manager: compares desired copy count against actual copy count. export const controllerManager = (d) => { d.component("controller manager"); let work = 0; const services = d.read("service/"), all = d.read("copy/"); for (const [a, v] of all) if (v.status === "dead") { d.remove(a); work++; } // clean up dead records for (const [a, s] of services) { const name = a.slice(8), prefix = `copy/${name}-`; const k = all.filter(([, v]) => v.service === name && v.status !== "dead"); let n = Math.max(0, ...all.filter(([b]) => b.startsWith(prefix)).map(([b]) => +b.slice(prefix.length))); for (let i = k.length; i < s.desired; i++, work++) d.write(prefix + ++n, { service: name, node: null, status: "pending" }); for (let i = k.length; i > s.desired; i--, work++) d.remove(k[i - 1][0]); } return work; }; // 3) Scheduler: picks the emptiest node with enough capacity for nodeless copies. export const scheduler = (d) => { d.component("scheduler"); let work = 0; const nodes = d.read("node/").filter(([, v]) => v.health === "alive"); const copies = d.read("copy/"); const load = new Map(nodes.map(([a]) => [a, copies.filter(([, v]) => v.node === a).length])); for (const [a, k] of copies.filter(([, v]) => v.node === null && v.status === "pending")) { const eligible = nodes.filter(([n, v]) => load.get(n) < v.capacity); if (!eligible.length) continue; // unsolved: no room const s = eligible.reduce((x, y) => (load.get(x[0]) <= load.get(y[0]) ? x : y))[0]; load.set(s, load.get(s) + 1); d.write(a, { ...k, node: s }); work++; } return work; }; // 4) The node agents' collective stand-in: starts an assigned copy and reports its status. export const nodes = (d) => { d.component("node agents"); let work = 0; for (const [a, k] of d.read("copy/")) if (k.node && k.status === "pending") { d.write(a, { ...k, status: "running" }); work++; } return work; }; ``` The point worth noticing is that the components **never call one another**. The controller manager does not notify the scheduler; it writes the copy object into the ledger without a node and leaves it there. The scheduler notifies no one either; it reads nodeless copies, picks a node, writes to the ledger. The four components' shared language is a single place — the state store — and this choice also determines how they behave when something goes down. The **revision** counter the store increments on every write is that language's timestamp: only that number tells a component which moment's ledger it is looking at. #### The End-to-End Trace of a Request ```js // trace.mjs — the end-to-end trace of a single request: which component reads what, writes what. import { create, apiServer, controllerManager, scheduler, nodes } from "./plane.mjs"; export const initial = () => { const b = { "service/collector": { desired: 3 }, "service/verifier": { desired: 2 } }; for (let i = 1; i <= 3; i++) b[`copy/collector-${i}`] = { service: "collector", node: `m${i}`, status: "running" }; for (let i = 1; i <= 2; i++) b[`copy/verifier-${i}`] = { service: "verifier", node: `m${i}`, status: "running" }; for (let i = 1; i <= 3; i++) b[`node/m${i}`] = { capacity: 4, health: "alive" }; return b; }; if (import.meta.url === `file://${process.argv[1]}`) { // the third block imports this file const d = create(initial()); apiServer(d, { key: "service/collector", value: { desired: 5 } }); let round = 0, work; do { work = controllerManager(d) + scheduler(d) + nodes(d); round++; } while (work > 0 && round < 8); let reads = 0, writes = 0; console.log("request: collector service's desired copy count 3 -> 5"); console.log("# component op key items rd wr"); d.trace.forEach((e, i) => { e.i === "read" ? (reads += e.n) : writes++; console.log(`${String(i + 1).padStart(2)} ${e.b.padEnd(23)} ${e.i.padEnd(7)} ` + `${e.a.padEnd(26)} ${String(e.n).padStart(5)} ${String(reads).padStart(4)} ${String(writes).padStart(4)}`); }); const components = new Set(d.trace.map((e) => e.b)); console.log(`\ncomponents visited ${components.size} trace steps ${d.trace.length} store reads ${reads}` + ` store writes ${writes} revision 0 -> ${d.rev} round ${round}`); } ``` ``` request: collector service's desired copy count 3 -> 5 # component op key items rd wr 1 API server write service/collector 1 0 1 2 controller manager read service/* 2 2 1 3 controller manager read copy/* 5 7 1 4 controller manager write copy/collector-4 1 7 2 5 controller manager write copy/collector-5 1 7 3 6 scheduler read node/* 3 10 3 7 scheduler read copy/* 7 17 3 8 scheduler write copy/collector-4 1 17 4 9 scheduler write copy/collector-5 1 17 5 10 node agents read copy/* 7 24 5 11 node agents write copy/collector-4 1 24 6 12 node agents write copy/collector-5 1 24 7 13 controller manager read service/* 2 26 7 14 controller manager read copy/* 7 33 7 15 scheduler read node/* 3 36 7 16 scheduler read copy/* 7 43 7 17 node agents read copy/* 7 50 7 components visited 4 trace steps 17 store reads 50 store writes 7 revision 0 -> 7 round 2 ``` A human wrote a single item: the collector service's desired copy count, 3 instead of 5. The counterpart of that was **four components, seventeen trace steps, 50 item reads, and 7 writes**. The write count is the most instructive number: against the human's 1 write, the plane produced 6 more writes — creating two copy objects, assigning both to a node, reporting both as running. **One item gets written three times**, because each component fills in a separate field of the same object. The copy object's journey is visible in the rows: at step 4 it is born nodeless and `pending`, at step 9 it gets a node, at step 12 it becomes `running`. Three components touch the same key in sequence, and none of them waits for the other; each one only looks at the ledger and does its own work. The second round — steps 13 through 17 — carries this lesson's most expensive number: **0 writes, 26 item reads**. After the gap closed, every component still read the ledger from the top and found no work to do. This is the price the cluster pays for "doing nothing," and it never stops; even on a quiet night for the measurement network, the controller manager and the scheduler keep scanning the ledger. Read load grows with item count, not with work done. #### When One Component Goes Down ```js // matrix.mjs — when one component goes down, which decisions stop, which keep going. import { create, apiServer, controllerManager, scheduler, nodes } from "./plane.mjs"; import { initial } from "./trace.mjs"; const EVENTS = [["service/collector", { desired: 5 }], // scale request ["copy/collector-2", { service: "collector", node: "m2", status: "dead" }], // node report ["node/m4", { capacity: 4, health: "alive" }]]; // new node function run(down) { const d = create(initial()); if (down === "state store") { d.read = () => []; d.write = () => {}; d.remove = () => {}; } for (const [a, v] of EVENTS) if (down !== "API server") apiServer(d, { key: a, value: v }); const reported = d.data.get("copy/collector-2")?.status === "dead"; // was the death recorded let round = 0, work; do { work = (down === "controller manager" ? 0 : controllerManager(d)) + (down === "scheduler" ? 0 : scheduler(d)) + nodes(d); round++; } while (work > 0 && round < 8); const k = [...d.data].filter(([a]) => a.startsWith("copy/collector")); const count = (t) => k.filter(([, v]) => v.status === t).length; const outcome = (done, half) => (done ? "done" : half ? "half" : "halted"); const believed = count("running"), actual = believed - (reported ? 0 : 1); // collector-2 had died return [down, outcome(believed === 5, count("pending") > 0), reported ? outcome(count("dead") === 0 && believed === 5, count("dead") === 0) : "unknown", d.data.has("node/m4") ? "done" : "halted", down === "API server" || down === "state store" ? "halted" : "done", "continues", `${believed}/${actual}`, d.trace.filter((e) => e.i === "read").reduce((t, e) => t + e.n, 0), d.trace.filter((e) => e.i !== "read").length]; } console.log("down component scale failure node reads traffic believed/actual rd wr"); for (const b of ["-", "API server", "state store", "scheduler", "controller manager"]) { const r = run(b); console.log(`${r[0].padEnd(23)} ${r[1].padEnd(8)} ${r[2].padEnd(9)} ${r[3].padEnd(8)}` + ` ${r[4].padEnd(8)} ${r[5].padEnd(11)} ${r[6].padStart(11)} ${String(r[7]).padStart(5)}` + ` ${String(r[8]).padStart(3)}`); } ``` ``` down component scale failure node reads traffic believed/actual rd wr - done done done done continues 5/5 52 13 API server halted unknown halted halted continues 3/2 20 0 state store halted unknown halted halted continues 3/2 0 0 scheduler half half done done continues 2/2 30 7 controller manager halted halted done done continues 2/2 14 3 ``` **The traffic column reads `continues` on every row.** Even if the entire control plane goes down, running copies keep serving; meter readings keep getting collected, verification keeps running, the nightly batch job still starts. The cluster does not become "unable to work" — it becomes **unable to decide**. This distinction holds for the whole course: the control plane does not do the work, it tells the parties that do the work what to do. The four failure scenarios produce three distinct failure types. **When the API server or the state store goes down,** no write lands on the ledger at all. The scale request is lost, the new node's registration is lost, a read request gets no answer. The heaviest row is the last one: `unknown`. Because a copy's death also has to pass through the API server, the report never gets recorded, and the ledger still counts that copy as running — believed 3, actual 2. Even after repair, this gap does not close on its own; what was lost was not a decision but an **event**, and the event itself does not come back. The only way to relearn it is for the node to report the same information again. **When the scheduler goes down,** the decision gets made but no place gets found. The controller manager sees the missing copies and creates their objects; the objects pile up without a node. The scale request stays `half`: the ledger holds five copies, two of which run nowhere. Repairing this case is cheap — once the scheduler comes back, it reads the accumulated copies and distributes them, because **the work is waiting in the ledger**. **When the controller manager goes down,** the information is correct, but nothing acts on it. The dead copy's report lands on the ledger, the ledger knows it is dead, but no one creates a replacement. The cluster knows what is wrong and does not fix it. This case, too, closes on repair. The distinction comes down to one sentence: **not knowing and knowing but not acting are separate failures.** The second closes on its own once the component comes back, because the gap is still on the ledger. The first does not close, because the information needed to close it was never written. The last two columns give the cost side of this. With everything up, the ledger took 52 item reads and 13 writes; once the scheduler went down, reads fell to 30, and once the controller manager went down, to 14. **Most of the store's load comes from the decision-making components**, not from requests. A single human request produced 1 write; in the same span, the decision components read 52 items. #### Summary - A single human write (desired copies 3 → 5) passed through four components and produced 17 trace steps, 50 item reads, and 7 store writes; the copy object was written three times by three separate components. - Components do not call one another, only the state store; the verification round after the gap closed took 0 writes and 26 item reads — read load grows with item count, not with work done. - Even with the whole control plane down, running copies kept serving; the ability the cluster lost was not working but deciding. - When the API server or the state store went down, the death report was never recorded, and the ledger kept counting the copy as running: believed 3, actual 2. - When the scheduler went down, the scale request stayed half-done (5 copies on the ledger, 2 nodeless); when the controller manager went down, the information stayed correct but nothing acted on it — both close on their own with repair, an unrecorded report does not. - Store reads were 52 with everything up, 30 with the scheduler down, and 14 with the controller manager down; the source of the load was the decision-making components, not requests. #### Next Step All of this lesson's measurements sat on one side of the ledger: how the decision gets produced and where it gets written. On the ledger's other side stood an assumption — that nodes correctly report what they are running. The last scenario showed a moment where that assumption broke: once a report could not be recorded, the cluster kept believing an old fact. The next lesson splits the node side into its parts — the **node agent, the runtime, the network proxy** — and measures the report itself: what gets reported, how often, with how much delay, and, when a report is lost, for how long the cluster keeps believing the old fact. ### Node Components Source: https://academia.sh/en/courses/orchestration/node-components The information flow on the node side is measured: what the node agent reports, how often, how much of the report has actually changed. Scanning the liveness threshold counts how long the cluster believes the old fact, how many wrong dead declarations happen, and how many requests are lost; the network proxy's rule count is shown to grow with service count, while rule change spreads with node count. The previous lesson's last scenario was where a single assumption broke: once a report could not be recorded, the ledger kept counting a dead copy as running. That assumption — that nodes correctly and promptly report what they are running — sits underneath every decision the control plane makes. This lesson splits the node side into its parts and measures the report itself. There are three parts. The **node agent** is the node's face toward the ledger: it reads the copies assigned to it, requests that they run, and reports the result back. The **runtime** is the part that actually runs the output; its isolation and resource limits were measured in the previous course, and here it stands only as the party that returns status to the node agent. The **network proxy** holds, on the node, the rule set that translates a service name into the copies currently running. The measurement runs, once again, on the **regional measurement network**, and it is **fictional**; what runs is not a real cluster but a **model** built with `node`. **CA13.** The node side is three parts: the node agent, the runtime, the network proxy. **CA14.** The node agent reports once every 10 seconds; a report carries 3 fixed items per node (health, total and used capacity) and 2 items per copy (status, resource usage). **CA15.** The share of items that actually change in one period is 0.02. **CA16.** The liveness threshold is E: a node with E seconds since its last report is considered unknown, and the copies on it are declared dead and re-created. **CA17.** In one day there are 4 permanent node failures and 30 temporary reporting outages (5–90 seconds); we wrote the generator ourselves, and the seed `20260403` appears in the output. **CA18.** The network proxy reads the ledger every 5 seconds; the rule count per node is the sum of the service entry and the copy routing rules, and the request rate per copy is 80 per second. #### What the Node Agent Reports ```js // node.mjs — the information the node side reports and its load at cluster scale. Model. export const P = 10; // reporting period (seconds) const FIXED = 3; // health, total capacity, used capacity export const items = (copies) => FIXED + 2 * copies; // per copy: status + resource usage export const CHANGE_RATE = 0.02; // the share of items that actually change in one period if (import.meta.url === `file://${process.argv[1]}`) { console.log("nodes copies reports/node items/sec daily items writes/sec changed-only/sec"); for (const M of [1, 3, 10, 30, 100, 300]) { const copies = 8, k = items(copies), itemsSec = (M * k) / P; console.log(`${String(M).padStart(5)}${String(copies).padStart(8)}${String(k).padStart(16)}` + `${itemsSec.toFixed(1).padStart(11)}${(itemsSec * 86400).toFixed(0).padStart(14)}` + `${(M / P).toFixed(1).padStart(12)}${(itemsSec * CHANGE_RATE).toFixed(2).padStart(19)}`); } } ``` ``` nodes copies reports/node items/sec daily items writes/sec changed-only/sec 1 8 19 1.9 164160 0.1 0.04 3 8 19 5.7 492480 0.3 0.11 10 8 19 19.0 1641600 1.0 0.38 30 8 19 57.0 4924800 3.0 1.14 100 8 19 190.0 16416000 10.0 3.80 300 8 19 570.0 49248000 30.0 11.40 ``` The content of a report is small: a node with eight copies sends 19 items every period. What creates the load is not the content but the **frequency**. In a three-hundred-node cluster, 570 items get reported per second, 49 million per day; to do this, nodes produce 30 writes per second. The last column gives the real number. Of the 570 items reported, only **11.4** have actually changed — two percent. The remaining ninety-eight percent is the cost of saying that nothing changed. This measure is the node side's counterpart to the previous lesson's verification round: there, the control plane read 26 items for 0 writes; here, nodes send 558 items because nothing changed at all. The place where cutting this load looks easy is a trap. Reporting only what changed would divide the item count by fifty, but the cluster's liveness decision rests on the **existence** of the report, not its **content**. A silent node and a node saying "nothing changed" look identical on the ledger; the only thing that tells them apart is whether the report arrived at all. This is why a report gets sent even when it is empty, and most of the reporting load is paid **not for the information itself, but for the proof of it**. The **scope** of the report is a separate limit. The node agent asks the runtime whether the copy is up or not and passes along the answer it gets; it does not ask whether the copy is **doing its job**. On the measurement network, a collector copy could be up and processing no meter readings at all — the process is alive, resource usage is low, the report says `running`. On the ledger this copy is healthy, the network proxy sends it traffic, and the controller manager does nothing because it sees no missing copy. All three components work correctly, and the outcome is wrong, because **the item the decision depends on is not in the report**. Widening the scope — asking the copy to report its own state separately — adds this item, but brings a separate question along with it: who produces that report, how often, and by looking at what. #### How Long the Cluster Believes the Old Fact ```js // stale-truth.mjs — once a report is lost, how long the cluster believes the old fact. import { P } from "./node.mjs"; const generator = (seed) => { let s = seed >>> 0; // our own generator; seed visible return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const DAY = 86400, COPIES = 3, REQUEST = 80, V = 5; // copies per node, requests/sec per copy, const rnd = generator(20260403); // V: the network proxy's store-read period const failure = Array.from({ length: 4 }, () => rnd() * DAY); // permanent node failure const outage = Array.from({ length: 30 }, () => ({ t: rnd() * DAY, d: 5 + rnd() * 85 })); console.log(`longest outage ${Math.max(...outage.map((k) => k.d)).toFixed(1)} sec` + ` reporting period ${P} sec proxy read period ${V} sec`); console.log("threshold avg-detect worst wrong-dead-decl wasted-copies lost-requests"); for (const E of [20, 30, 40, 60, 90, 100, 120, 300]) { const g = failure.map((t) => Math.floor(t / P) * P + E - t); // duration of the stale fact const wrong = outage.filter((k) => Math.floor(k.t / P) * P + E < k.t + k.d).length; const lost = g.reduce((s, x) => s + (x + V) * COPIES * REQUEST, 0); console.log(`${String(E).padStart(9)}${(g.reduce((a, b) => a + b) / g.length).toFixed(1).padStart(15)}` + `${Math.max(...g).toFixed(1).padStart(9)}${String(wrong).padStart(18)}` + `${String(wrong * COPIES).padStart(15)}${lost.toFixed(0).padStart(15)}`); } ``` ``` longest outage 84.6 sec reporting period 10 sec proxy read period 5 sec threshold avg-detect worst wrong-dead-decl wasted-copies lost-requests 20 17.3 19.9 26 78 21408 30 27.3 29.9 21 63 31008 40 37.3 39.9 19 57 40608 60 57.3 59.9 13 39 59808 90 87.3 89.9 2 6 88608 100 97.3 99.9 0 0 98208 120 117.3 119.9 0 0 117408 300 297.3 299.9 0 0 290208 ``` The first column is how long the cluster is wrong. Whatever the threshold, the cluster learns of a failure **at least** E minus one period, **at most** E seconds later: at a 40-second threshold, it counts a dead node as alive for an average of 37.3 seconds. This duration cannot be zeroed out, because there is no such thing as a failure reporting itself — the cluster's only evidence is a report that does not arrive, and deciding when something that has not arrived has failed to arrive requires waiting. The second and third columns are the cost of shortening the wait. Once the threshold is lowered to 20 seconds, **twenty-six** of thirty temporary outages were mistaken for permanent failures; the cluster wastefully declared 78 copies dead and opened replacements — even though the old copies had not died, the node came back a few seconds later. This number falls as the threshold grows: 13 at 60 seconds, 2 at 90 seconds, zero at 100 seconds. Where the zero falls is not a coincidence. The longest outage is 84.6 seconds and the reporting period is 10 seconds; wrong declarations end at the first step where the threshold exceeds the sum of the two. **The right threshold is derived not from the cluster's own measurements but from the network's behavior** — from the longest tail of a distribution the cluster neither knows nor measures. If outages one day run 130 seconds instead of 84.6, the 100-second threshold quietly starts producing wrong declarations again, and the cluster has no alarm for this. The last column is the other half of the payment. Because the network proxy's rule only updates once a copy is declared dead, requests arriving before then get sent to a copy that is not running. At the 100-second threshold that zeroes out wrong declarations, **98,208 requests** went to dead copies over the course of a day; at the 20-second threshold that number was 21,408. The choice between the two ends is not a setting but a **split between two wrongs**: a fast-acting cluster kills healthy copies, a patient cluster sends traffic to dead ones. No threshold zeroes out both, and the node holds no information that could zero out both at once. On the measurement network, the counterpart of this is concrete — 98 thousand meter-reading requests go unanswered, and the nightly batch job starts with incomplete data. #### Where the Reporting Delay Accumulates Between a copy going down and that fact landing on the ledger and turning into a decision, there are four separate waits, and each one imposes its own period. ```js // delay.mjs — one event's reporting delay, split into four parts. Each stage adds its own wait. import { P } from "./node.mjs"; const generator = (seed) => { let s = seed >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const rnd = generator(20260404), N = 20000; const C = 2, WRITE = 0.05, U = 15; // runtime poll, store write, reconciliation period const stage = [["runtime poll", C], ["node agent report", P], ["API server + store write", 0], ["controller manager round", U]]; let total = new Array(N).fill(0); console.log("stage period avg(sec) worst(sec) cumulative avg(sec)"); for (const [name, period] of stage) { const share = total.map(() => (period ? rnd() * period : WRITE)); total = total.map((x, i) => x + share[i]); console.log(`${name.padEnd(29)}${(period || WRITE).toFixed(2).padStart(8)}` + `${(share.reduce((a, b) => a + b) / N).toFixed(2).padStart(10)}` + `${Math.max(...share).toFixed(2).padStart(13)}` + `${(total.reduce((a, b) => a + b) / N).toFixed(2).padStart(21)}`); } console.log(`events ${N} worst end-to-end ${Math.max(...total).toFixed(2)} sec` + ` stage count ${stage.length}`); ``` ``` stage period avg(sec) worst(sec) cumulative avg(sec) runtime poll 2.00 1.00 2.00 1.00 node agent report 10.00 4.99 10.00 5.99 API server + store write 0.05 0.05 0.05 6.04 controller manager round 15.00 7.49 15.00 13.53 events 20000 worst end-to-end 26.36 sec stage count 4 ``` The average end-to-end delay is 13.53 seconds, the worst is 26.36 seconds. The distribution is instructive: 1.00 second of the delay comes from the runtime's poll, 4.99 seconds from the node agent's reporting period, 0.05 seconds from the write operation, and **7.49 seconds from the controller manager's round**. So more than half the delay sits not on the node side but in the control plane's own period. This split determines which setting actually helps. Halving the node agent's reporting period lowers the average by 2.5 seconds and doubles the reporting load; halving the controller manager's round period lowers it by 3.75 seconds and puts no load at all on the nodes, but doubles the ledger's read load. **The same delay is sold at two separate places and two separate prices**; which setting is cheap depends on where the cluster is running tight. #### The Network Proxy's Rule Count ```js // proxy.mjs — how the network proxy's rule count grows with service count; where change spreads. const M = 30, r = 5, V = 5, CHANGE = 200; // node, copies/service, read period, copies/day const rule = (S) => S * (1 + r); // per node: service entry + copy routing rules console.log(`nodes ${M} copies/service ${r} proxy read period ${V} sec`); console.log("service copies rules/node cluster-rules single-change items-read/sec daily-writes"); for (const S of [2, 4, 8, 16, 32, 64]) { console.log(`${String(S).padStart(7)}${String(S * r).padStart(8)}${String(rule(S)).padStart(13)}` + `${String(M * rule(S)).padStart(15)}${String(M).padStart(16)}` + `${((M * S * r) / V).toFixed(0).padStart(17)}${String(CHANGE * M).padStart(14)}`); } ``` ``` nodes 30 copies/service 5 proxy read period 5 sec service copies rules/node cluster-rules single-change items-read/sec daily-writes 2 10 12 360 30 60 6000 4 20 24 720 30 120 6000 8 40 48 1440 30 240 6000 16 80 96 2880 30 480 6000 32 160 192 5760 30 960 6000 64 320 384 11520 30 1920 6000 ``` Two columns grow along two separate axes. **Rule count grows with service count**: 12 rules per node at two services, 384 at sixty-four. The cluster-wide total is this multiplied by node count and reaches 11,520 rules, because the same rule set sits separately on each of the thirty nodes. The proxy's ledger-reading load grows along the same axis too: from 60 items per second to 1,920. **The spread of a change, however, is independent of service count.** One copy being born or dying, whatever the service count, produces thirty rule writes across thirty nodes — because every node needs to be able to reach that service. Two hundred copy changes per day come out to 6,000 rule writes, and that number stays the same even if service count grows thirty-two times over. The distinction matters operationally: adding a new service grows the **static** cost (rule count, memory, read load); adding a node, or changing copies often, grows the **flowing** cost (rule writes, the propagation window). The lost-request count from the previous section also accumulates within this window: once a copy is declared dead, updating the rule is delayed further by the proxy's read period. This five seconds closes separately on each of the thirty nodes, at different moments; a rule change does not reach the cluster at a single instant but spreads across a window. #### Summary - A node with eight copies reports 19 items every period; across 300 nodes this comes to 49 million items a day, and only 11.4 items per second of that has actually changed — the remaining ninety-eight percent is paid not for information but for proof that the node is still up. - An event turning into a decision passes through four waits and takes an average of 13.53 seconds, at worst 26.36; 7.49 seconds of that comes not from the node but from the controller manager's round period. - The cluster learns of a failure at least the threshold minus one period, at most the full threshold, later; at a 40-second threshold it counts a dead node as alive for an average of 37.3 seconds, and this duration cannot be zeroed out. - Once the threshold was lowered to 20 seconds, 26 of 30 temporary outages were mistaken for permanent failures and 78 copies were wastefully opened; wrong declarations zeroed out only at 100 seconds, once the threshold exceeded the sum of the longest outage (84.6 sec) and one reporting period. - At the threshold that zeroed out wrong declarations, requests sent to dead copies rose to 98,208; at the 20-second threshold it was 21,408 — the choice between the two ends is a split between two wrongs. - The network proxy's rule count grows with service count (12 → 384 per node, 11,520 cluster-wide), while the spread of a change is independent of service count: one copy change produces 30 rule writes across 30 nodes, 6,000 a day. #### Next Step Two lessons measured two sides separately: the control plane produces the decision, the nodes report the state. What connects the two has not yet been named — the continuous comparison of desired state against actual state, and closing the gap. The next lesson is this topic's center, and it genuinely writes the **reconciliation loop** at cluster scale: the convergence round, work per round, and the question this lesson leaves behind — as reporting delay grows, how many copies does the loop open unnecessarily, how many does it delete for nothing, and what relationship holds between the loop's period and the reporting delay. ### The Reconciliation Loop Source: https://academia.sh/en/courses/orchestration/reconciliation-loop The continuous comparison of desired state against actual state is written as a loop running at cluster scale, and report delay is scanned: how many unneeded copies the system opens, how many copies it deletes for nothing, and how wide the oscillation runs. The relationship between loop period and report delay is tested across fifty-six cells, and the missing information item is shown to be the system's own history. This topic's first three lessons measured two sides separately. Now the thing that connects them gets written: the continuous comparison of desired state against actual state, and closing the gap — the **reconciliation loop**. The loop itself is not new — a reconciler running on a single machine was examined earlier: it read the desired state from a file and the actual state from disk, touched only the differing item, and did zero work on its second run. That measurement is taken here as an input and is not repeated. At cluster scale, exactly one thing changes. Actual state no longer gets read from disk; it comes from **reports** arriving from the nodes, carrying the delay the previous lesson measured. This entire lesson counts the cost of that one difference: how many wrong corrections do correct decisions, made by looking at a delayed picture, produce. The measurement runs, once again, on the **regional measurement network**, and it is **fictional**; what runs is a **model** built with `node`, not a real cluster. **CA19.** Desired state is a single item: the collector service's copy count is six, and it comes from a human-written entry. **CA20.** Actual state comes from node reports; the ledger sits G seconds behind the real world. **CA21.** A created copy becomes running B seconds later, and the fact that it is running lands on the ledger G seconds after that; a deleted copy also disappears from the ledger G seconds later. **CA22.** The loop period is U; every round the ledger gets read, the gap gets computed, a correction gets written. **CA23.** Events are scripted, there is no randomness: two copies die at second 300; the run lasts 900 seconds. **CA24.** Two controllers are compared — one that looks only at the ledger, and one that also counts its own pending actions. **CA25.** A delete command is given to whatever the ledger shows; a command given to a copy that is already dead counts as work but does not change the live copy count. #### The Reconciliation Loop at Cluster Scale ```js // reconcile.mjs — the reconciliation loop at cluster scale. Desired state comes from the report, // actual state comes from delayed reports. Model; a real cluster does not run. No randomness, events are scripted. export const DEATHS = [[300, 2]]; // scripted event: 2 copies die at second 300 export function run({ desired = 6, U = 15, G = 0, B = 5, duration = 900, intent = false }) { const actual = []; // {birth, ready, death} let created = 0, commands = 0, deleted = 0, work = 0, round = 0, overshoot = 0, lastDrift = 0; const alive = (t) => actual.filter((r) => r.ready <= t && (r.death === null || r.death > t)); const seenAt = (t) => actual.filter((r) => // the ledger is G seconds behind r.ready + G <= t && (r.death === null || r.death + G > t)); for (let t = 0; t <= duration; t++) { for (const [at, count] of DEATHS) // scripted death: the oldest copies drop if (t === at) alive(t).slice(0, count).forEach((r) => { r.death = t; }); if (t % U === 0) { round++; const seen = seenAt(t); const pending = intent ? actual.filter((r) => r.death === null && r.ready + G > t).length : 0; const diff = desired - seen.length - pending; // intent: count its own pending actions if (diff > 0) { for (let i = 0; i < diff; i++) actual.push({ birth: t, ready: t + B, death: null }); created += diff; work += diff; } else if (diff < 0) { commands += -diff; work += -diff; // delete command: given to what the ledger shows seen.slice(0, -diff).forEach((r) => { if (r.death === null) { r.death = t; deleted++; } }); } } const c = alive(t).length; overshoot = Math.max(overshoot, c - desired); if (c !== desired) lastDrift = t; } const required = desired + DEATHS.reduce((s, k) => s + k[1], 0); return { round, workRound: work / round, created, unneeded: created - required, commands, deleted, overshoot, stable: lastDrift === duration ? null : lastDrift + 1 }; } const row = (lead, r) => `${lead}${r.workRound.toFixed(2).padStart(12)}${String(r.created).padStart(11)}` + `${String(r.unneeded).padStart(10)}${String(r.commands).padStart(17)}${String(r.deleted).padStart(9)}` + `${String(r.overshoot).padStart(11)}${(r.stable === null ? "none" : r.stable).toString().padStart(13)}`; if (import.meta.url === `file://${process.argv[1]}`) { console.log("U=15 sec loop, B=5 sec startup, desired 6 copies, 2 deaths at sec 300, 900 sec run"); for (const intent of [false, true]) { console.log(`\ncontroller: ${intent ? "ledger + its own pending actions" : "ledger only"}`); console.log("G(sec) work/round created unneeded delete commands deleted overshoot stable(sec)"); for (const G of [0, 5, 10, 20, 30, 60]) console.log(row(String(G).padStart(6), run({ G, intent }))); } } ``` ``` U=15 sec loop, B=5 sec startup, desired 6 copies, 2 deaths at sec 300, 900 sec run controller: ledger only G(sec) work/round created unneeded delete commands deleted overshoot stable(sec) 0 0.13 8 0 0 0 0 305 5 0.13 8 0 0 0 0 320 10 0.13 8 0 0 0 0 320 20 0.52 16 8 16 8 6 375 30 0.92 24 16 32 16 12 405 60 2.75 40 32 128 32 24 495 controller: ledger + its own pending actions G(sec) work/round created unneeded delete commands deleted overshoot stable(sec) 0 0.13 8 0 0 0 0 305 5 0.13 8 0 0 0 0 320 10 0.13 8 0 0 0 0 320 20 0.13 8 0 0 0 0 335 30 0.13 8 0 0 0 0 335 60 0.13 8 0 0 0 0 365 ``` The top table's first three rows are familiar. At a delay of zero, five, or ten seconds, the loop created eight copies — six for the initial setup, two to replace the dead ones — deleted none for nothing, and produced no overshoot at all. Work per round is 0.13: almost all of the sixty-one rounds did zero work. The inefficiency measured on a single machine holds at cluster scale too. At the fourth row, the loop's behavior changes. At a twenty-second report delay, the loop created sixteen copies; eight of them were unneeded. Eight copies got deleted for nothing, and the live copy count rose from six to twelve. At a sixty-second delay, the table falls apart: forty copies were created, thirty-two of them unneeded; a hundred twenty-eight delete commands were issued; overshoot hit twenty-four, meaning **thirty copies** ran at once at one point. What is worth noticing in these rows is that the controller never made a mistake. Every round it read the ledger, computed the gap correctly, and wrote exactly the correction needed to close it. The rule was right, the arithmetic was right, the input was **stale**. The copies were created, but because they had not been reported yet, the controller could not see them on the next round and **saw the same shortfall again**. The source of the wrong correction is not the decision rule but the age of the picture the decision rests on. The gap between delete commands and copies actually deleted is the same blindness's other face. At the sixty-second delay, 128 delete commands were issued, and only 32 of them landed on a live copy; the remaining 96 commands went to copies that were already dead. The system sees not only what it created, but also **what it deleted**, with a delay. On the measurement network, the counterpart of this is concrete: the six-copy collector service ran with thirty copies at one point, its database behind it took five times the connections, and the same meter-reading batches got pulled by more than one copy. The oscillation is not a display problem, it is real load. And this load leaves no trace at all in the controller's final line, which says "six copies wanted, six copies present"; once the oscillation closes, all that remains is a correct final state. #### The Relationship Between Loop Period and Report Delay ```js // relation.mjs — the relationship between loop period U and report delay G, with numbers. import { run } from "./reconcile.mjs"; const B = 5, DESIRED = 6; const predict = (G, U) => DESIRED * (Math.ceil((G + B) / U) - 1); // the relation to be tested console.log("G=30 sec fixed, B=5 sec; scanning the loop period"); console.log("U(sec) ceil((G+B)/U) predicted overshoot measured overshoot unneeded deleted work/round"); for (const U of [5, 10, 15, 20, 30, 35, 60]) { const r = run({ U, G: 30, B }); console.log(`${String(U).padStart(6)}${String(Math.ceil(35 / U)).padStart(15)}` + `${String(predict(30, U)).padStart(21)}${String(r.overshoot).padStart(20)}` + `${String(r.unneeded).padStart(10)}${String(r.deleted).padStart(9)}` + `${r.workRound.toFixed(2).padStart(12)}`); } console.log("\novershoot table (row: report delay G, column: loop period U)"); const U_VALUES = [5, 10, 15, 20, 30, 45, 60, 90]; console.log("G\\U " + U_VALUES.map((u) => String(u).padStart(5)).join("")); let mismatches = 0, cells = 0; for (const G of [0, 10, 20, 30, 45, 60, 90]) { const s = U_VALUES.map((U) => { const a = run({ U, G, B }).overshoot; cells++; if (a !== predict(G, U)) mismatches++; return String(a).padStart(5); }); console.log(String(G).padStart(3) + " " + s.join("")); } console.log(`\nrelation: overshoot = desired x (ceil((G+B)/U) - 1); ${cells} cells, ${mismatches} mismatched`); ``` ``` G=30 sec fixed, B=5 sec; scanning the loop period U(sec) ceil((G+B)/U) predicted overshoot measured overshoot unneeded deleted work/round 5 7 36 36 48 48 1.90 10 4 18 18 24 24 1.14 15 3 12 12 16 16 0.92 20 2 6 6 8 8 0.70 30 2 6 6 8 8 0.77 35 1 0 0 0 0 0.31 60 1 0 0 0 0 0.50 overshoot table (row: report delay G, column: loop period U) G\U 5 10 15 20 30 45 60 90 0 0 0 0 0 0 0 0 0 10 12 6 0 0 0 0 0 0 20 24 12 6 6 0 0 0 0 30 36 18 12 6 6 0 0 0 45 54 24 18 12 6 6 0 0 60 72 36 24 18 12 6 6 0 90 108 54 36 24 18 12 6 6 relation: overshoot = desired x (ceil((G+B)/U) - 1); 56 cells, 0 mismatched ``` The top table holds one delay (30 seconds) fixed and scans the loop period, and it shows the unexpected direction: **speeding up the loop makes the damage worse.** At a five-second period, overshoot is thirty-six and unneeded copies are forty-eight; at a thirty-five-second period, both are zero. A loop that reconciles often reads the same stale picture more times while the delay stays fixed, and tries to close the same shortfall again on every reading. The bottom table gives the relationship. Overshoot has this value in every cell measured: **overshoot = desired copies × (⌈(G + B) / U⌉ − 1)** None of the fifty-six cells deviate. The relationship reads in a direct way: `(G + B) / U` is **how many rounds** run in the time between a copy being created and it appearing on the ledger. The first of those rounds closes the real shortfall; each of the rest closes the same shortfall one more time. Once the number of blind rounds is one — that is, `U ≥ G + B` — overshoot zeroes out, and the table's zero region sits exactly above this diagonal. This is the sibling of the previous lesson's liveness-threshold rule. There, the threshold had to exceed the longest temporary outage plus one reporting period; here, the loop period has to exceed the report delay plus the startup time. In both cases, the cluster's setting depends on a duration the cluster does not measure itself, and in both cases a wrong setting carries no alarm — the oscillation runs silently. The relationship itself therefore carries operational value for this reason: it is possible to compute it knowing only three durations, without ever observing the overshoot. G is not a free setting here; the previous lesson measured it. The four-stage reporting chain averaged 13.53 seconds, worst case 26.36 seconds. With a five-second startup time, `G + B` comes out to 18.5 on average, 31.4 seconds in the worst case. A fifteen-second loop period sits below both of these numbers — meaning a cluster running with the measured delays sits in the oscillation zone **at its default setting**, with nothing broken. The table's `U=15, G=20` row is not an edge case; it is an ordinary day. The loop's base cost also shows up here. Work per round is 0.13, and most rounds do zero work, but a zero-work round is not free: every round reads the ledger from the top. The previous lesson's verification round counted this read at 26 items. The loop's cost grows not with the corrections it makes but with **the ratio of item count to period**; shrinking the period grows both the read load and — if the delay stays fixed — the overshoot. #### The Missing Item Is the System's Own History The bottom table of the first code block shows the second controller, and the result is sharp: the controller that counts its own pending actions created eight copies at every delay value, zero unneeded, zero deletions, 0.13 work per round. Overshoot is zero on every row. The added information item is a single line, and it has nothing to do with the outside world: **a record of what I created in the last few rounds.** The controller no longer says "I see four copies on the ledger instead of six"; it says "I see four copies on the ledger, and I created two of them two rounds ago myself — they may not have shown up yet." This is where the course's measure lands: the item added so that a delegated decision can be made correctly is, most of the time, not a measurement but **the system's memory of its own actions** — and this item comes close to free, because it asks no one for a report. It comes close to free, but it does not solve everything. Comparing the two tables' last column shows this: even for the controller with an intent record, the moment it turns stable grows with the delay — from 305 seconds to 365. A replacement for the two dead copies cannot open until the death shows up on the ledger. The distinction comes down to one sentence: **delay postpones noticing, an unrecorded intent multiplies the work.** These are two separate problems; the second closes with one line, the first does not — it can only be reduced, expensively, by shrinking the reporting period and the network delay — and, as the previous lesson showed, the bill for shrinking those periods comes back as reporting load per node. #### Summary - At report delays of zero, 5, and 10 seconds, the loop created 8 copies, deleted none for nothing, and almost all of the 61 rounds did zero work; the inefficiency measured on a single machine holds at cluster scale too. - At a 20-second delay, 8 unneeded copies got opened and 8 copies got deleted for nothing, with the live count rising from 6 to 12; at 60 seconds, 32 unneeded copies, 128 delete commands, and an oscillation of 30 live copies resulted. - Only 32 of 128 delete commands landed on a live copy; the system sees both what it created and what it deleted with a delay. - Overshoot came out to `desired × (⌈(G+B)/U⌉ − 1)` across every scan, with zero deviation across 56 cells. The boundary where overshoot zeroes out is `U ≥ G + B` — the loop period exceeding the report delay plus the startup time. - Speeding up the loop made the damage worse: at a 30-second delay, overshoot was 36 at a 5-second period and 0 at a 35-second period. - The controller that counted its own pending actions produced 0 unneeded copies and 0 wasted deletions at every delay; the moment it turned stable still rose from 305 to 365 seconds — delay postpones noticing, an unrecorded intent multiplies the work. #### Next Step This lesson measured the loop itself: what it reads, how often it runs, how much wrong work it does by looking at a delayed picture. The topic's final lesson returns to an operational decision — whether you will run this plane yourself, or delegate its operation to someone else. What gets measured is the two options compared on the same items: how many operational items get delegated, which items remain, the version-upgrade window, the loss of access to the control plane, and how reversible the decision is. ### Local and Managed Clusters Source: https://academia.sh/en/courses/orchestration/local-and-managed-clusters Running the cluster yourself is compared against delegating its operation on the same items: delegated and remaining operational items, monthly hours, the version-upgrade window, the loss of access to the control plane, and exit cost. The loss of access is shown with a number for how it takes away the lever in the previous lesson's oscillation relation. This topic measured what a cluster is, which components it is built from, and how it reconciles. One operational decision remains: whether you will run this plane yourself, or leave its operation to a party that takes it over. The two options are referred to here **as types** — **the cluster you operate yourself** and **the cluster with delegated operations** — and are compared on the same items. This decision is one layer above this course's measure. The previous lessons counted the cost of delegating the placement and replica-count decision to a system; this lesson counts the cost of delegating **the operation of that system itself**. The questions are the same: which items get delegated, what information and control get lost in exchange, and is the decision reversible. The measurement runs, once again, on the **regional measurement network**, and it is **fictional**; what runs are **models** built with `node`. **CA26.** Operation is represented by twenty-one items; each item's monthly hours, owner, and whether it is rebuilt on exit are written down. **CA27.** For shared items, half the hours go to the provider. **CA28.** For items rebuilt on exit, the initial setup load is four times the monthly hours. **CA29.** A release ships every four months, its support lasts twelve months; the run is thirty-six months. **CA30.** Under self-run operation, upgrades happen according to the team's capacity; under delegated operation, they happen the month support ends, and the calendar belongs to the provider. **CA31.** The load per upgrade is 8 control-plane hours plus 7 node hours under self-run operation, and only 7 node hours under delegated operation. **CA32.** Report delay and startup time are taken from the previous lessons: average 13.53 and worst 26.36 seconds, startup time 5 seconds. #### The Same Items, Two Owners ```js // items.mjs — the cluster operation's items. [name, monthly hours, who, rebuilt on exit?] // "who" points to the same name in both options: the party taking over operations, shared, or you. export const ITEMS = [ ["control plane setup", 6, "provider", true], ["control plane version upgrades", 8, "provider", true], ["state store backup", 5, "provider", true], ["state store recovery", 4, "provider", true], ["API server certificates", 2, "provider", true], ["control plane redundancy", 5, "provider", true], ["control plane metric collection", 3, "provider", true], ["node provisioning", 6, "shared", false], ["node version upgrades", 7, "shared", false], ["node OS patching", 5, "shared", false], ["cluster autoscaling rules", 3, "shared", false], ["network configuration", 4, "shared", false], ["storage attachment", 3, "shared", false], ["incident response", 6, "shared", false], ["workload declarations", 9, "you", false], ["resource requests and limits", 5, "you", false], ["authorization rules", 4, "you", false], ["network policies", 3, "you", false], ["observability and alerting", 8, "you", false], ["release rollout", 7, "you", false], ["cost tracking", 3, "you", false], ]; const SHARE = 0.5; // for a shared item, half the hours go to the provider if (import.meta.url === `file://${process.argv[1]}`) { const byOwner = (k) => ITEMS.filter(([, , x]) => x === k); const hours = (l) => l.reduce((t, [, s]) => t + s, 0); const total = hours(ITEMS); const delegated = hours(byOwner("provider")) + hours(byOwner("shared")) * SHARE; console.log("item total provider shared you"); console.log(`count ${String(ITEMS.length).padStart(5)}${String(byOwner("provider").length).padStart(10)}` + `${String(byOwner("shared").length).padStart(9)}${String(byOwner("you").length).padStart(6)}`); console.log(`hours ${String(total).padStart(5)}${String(hours(byOwner("provider"))).padStart(10)}` + `${String(hours(byOwner("shared"))).padStart(9)}${String(hours(byOwner("you"))).padStart(6)}`); console.log(`\ncluster you operate : items ${ITEMS.length} monthly hours ${total}`); console.log(`operations delegated : items ${byOwner("shared").length + byOwner("you").length}` + ` monthly hours ${total - delegated} delegated ${delegated} share ${(100 * delegated / total).toFixed(1)}%`); const exitItems = ITEMS.filter(([, , , c]) => c); console.log(`exit cost : items to rebuild ${exitItems.length}` + ` monthly hours ${hours(exitItems)} initial setup ${hours(exitItems) * 4} hours (four months' equivalent)`); } ``` ``` item total provider shared you count 21 7 7 7 hours 106 33 34 39 cluster you operate : items 21 monthly hours 106 operations delegated : items 14 monthly hours 56 delegated 50 share 47.2% exit cost : items to rebuild 7 monthly hours 33 initial setup 132 hours (four months' equivalent) ``` Seven of the twenty-one items get delegated, seven get shared, seven stay with you no matter what. On the hours side, the delegated share is 50 of the monthly 106 hours — **47.2 percent**. Item count falls from 21 to 14, monthly load from 106 hours to 56. The most commonly misread part of this table is the **remaining** column. Delegating provides a reduction of close to half, but **more than half stands unchanged**: workload declarations, resource requests, authorization rules, network policies, observability, release rollout, and cost tracking do not shrink under any delegation, because all of these are items of **your application**, not the cluster. The delegation decision touches none of these items, and assuming it does is the most common wrong guess about post-delegation team load. The declarations that need to be written for the measurement network's four services and nightly batch job are independent of who operates the cluster. The middle seven items — the shared column — belong to **exactly no one** in either option, and this is where the surprises come from. Node provisioning, node version upgrades, network configuration, and incident response stay half yours even after delegating; if the delegation contract does not write down the boundary of these items, which half of the monthly 34 hours falls to whom becomes clear only at the first incident. On the measurement network, the counterpart of this is: if the collector's node cannot take a copy because its capacity is full, both sides go looking separately for whether the problem is in the cluster or in the workload declaration. The last line gives the decision's reversibility. The seven delegated items must be rebuilt on exit; against a 33-hour monthly maintenance load, their initial setup is 132 hours. Exit cost is measured **in item count**: seven items, and none of these seven show up in day-to-day work, because as long as they are delegated, no one looks at them. The decision is reversible, but the cost of reversing it is a cost not paid on the day the delegation decision was made. #### The Version-Upgrade Window ```js // window.mjs — a 36-month version-upgrade window. A release ships every 4 months, support lasts 12. const MONTHS = 36, INTERVAL = 4, SUPPORT = 12, HOURS_PLANE = 8, HOURS_NODE = 7; const lastRelease = (m) => Math.floor(m / INTERVAL) * INTERVAL; function run({ name, team, mandatory }) { // team: your own capacity (months); mandatory: the provider's let s = 0, upgrades = 0, unsupported = 0, oldest = 0, hours = 0; for (let m = 1; m <= MONTHS; m++) { if (mandatory ? m === s + SUPPORT : m % team === 0 && lastRelease(m) > s) { s = lastRelease(m); upgrades++; hours += mandatory ? HOURS_NODE : HOURS_PLANE + HOURS_NODE; } const age = m - s; oldest = Math.max(oldest, age); if (age > SUPPORT) unsupported++; } return [name, upgrades, unsupported, oldest, hours, mandatory ? "provider" : "you"]; } const row = (r) => console.log(`${String(r[0]).padEnd(27)}${String(r[1]).padStart(10)}` + `${String(r[2]).padStart(16)}${String(r[3]).padStart(14)}${String(r[4]).padStart(7)} ${r[5]}`); row(["option", "upgrades", "unsupported mo", "oldest (mo)", "hours", "calendar"]); for (const team of [4, 6, 12, 18, 24]) row(run({ name: `self-run (team ${team} mo)`, team })); row(run({ name: "operations delegated", mandatory: true })); ``` ``` option upgrades unsupported mo oldest (mo) hours calendar self-run (team 4 mo) 9 0 3 135 you self-run (team 6 mo) 6 0 7 90 you self-run (team 12 mo) 3 0 11 45 you self-run (team 18 mo) 2 12 19 30 you self-run (team 24 mo) 1 11 23 15 you operations delegated 3 0 11 21 provider ``` Six rows describe the same thirty-six months. Under self-run operation, upgrade frequency depends on the team's capacity, and the results are linear: a team upgrading every four months spends 9 upgrades and 135 hours, and the release is never more than 3 months old. A team upgrading every twelve months spends 3 upgrades, 45 hours, and the release ages at most 11 months. Neither of these two rows is a wrong answer; both are a **choice**, and you are the one making it. The lower bound shows up in the fifth and sixth rows. A team that can only upgrade every eighteen months spends **twelve of the thirty-six months unsupported** and ages the release up to nineteen months. Self-run operation gives you the calendar, but keeping to it is also your job; a calendar not kept turns into unsupported months, and there is no alarm reported to anyone for this. The delegated row sits in an interesting spot: 3 upgrades, 0 unsupported months, an oldest release of 11 months — the **same freshness** as a team upgrading every twelve months, but 21 hours instead of 45. The difference is not in the hours; it is in the last column. The party that decides **when** an upgrade happens changes. The measurement network's billing period falls in the first week of the month, and the nightly batch job starts every night at 02:30; if a mandatory upgrade window falls on either of these two calendars, the only thing you can do to avoid the collision is talk to the provider. Under self-run operation, this same collision is a scheduling problem; under delegated operation, it is a conversation, and the two are resolved on two different timescales. #### The Loss of Access to the Control Plane ```js // access.mjs — access to setting and observability items, and its effect on the previous lessons' relation. const SETTINGS = [ // [item, when you run it yourself, when operations are delegated] ["reporting period (sets G)", "full", "none"], ["liveness threshold", "full", "none"], ["loop period (U)", "full", "none"], ["proxy read period", "full", "partial"], ["scheduler priorities", "full", "partial"], ["API server audit log", "full", "partial"], ["direct state-store reads", "full", "none"], ["control plane logs", "full", "partial"], ["controller manager metrics", "full", "none"], ["startup time (B)", "full", "full"], ["desired copy count", "full", "full"], ["node type and count", "full", "full"], ]; const count = (col, level) => SETTINGS.filter((a) => a[col] === level).length; console.log("access full partial none items"); for (const [col, name] of [[1, "cluster you operate"], [2, "operations delegated"]]) console.log(`${name.padEnd(21)}${String(count(col, "full")).padStart(6)}${String(count(col, "partial")).padStart(9)}` + `${String(count(col, "none")).padStart(7)}${String(SETTINGS.length).padStart(8)}`); // The relation tested in the previous lesson: overshoot = desired x (ceil((G+B)/U) - 1) const DESIRED = 6, B = 5; const overshoot = (G, U) => DESIRED * (Math.ceil((G + B) / U) - 1); console.log("\novershoot with the measured report delay (desired 6 copies, B=5 sec)"); console.log("G(sec) U selectable (35) U fixed (15) B that zeroes overshoot feasible?"); for (const G of [13.53, 26.36]) { const neededB = 15 - G; console.log(`${G.toFixed(2).padStart(6)}${String(overshoot(G, 35)).padStart(20)}` + `${String(overshoot(G, 15)).padStart(15)}${neededB.toFixed(2).padStart(24)} sec` + `${(neededB > 0 ? "very hard" : "impossible").padStart(14)}`); } ``` ``` access full partial none items cluster you operate 12 0 0 12 operations delegated 3 4 5 12 overshoot with the measured report delay (desired 6 copies, B=5 sec) G(sec) U selectable (35) U fixed (15) B that zeroes overshoot feasible? 13.53 0 6 1.47 sec very hard 26.36 0 12 -11.36 sec impossible ``` The top table gives the size of the loss: of twelve setting and observability items, all of them are fully accessible on the cluster you operate yourself, while on the cluster with delegated operations only three are fully accessible; four are partial, **five are none at all**. The five inaccessible items were not picked at random — they are exactly the items this topic's three lessons measured: the reporting period, the liveness threshold, the loop period, direct state-store reads, and the controller manager's metrics. The bottom table turns this into a number. The previous lesson tested, across fifty-six cells, that overshoot equals `desired × (⌈(G+B)/U⌉ − 1)`. With the measured report delay — average 13.53, worst 26.36 seconds — when the loop period is selectable it can be pulled to 35 seconds and overshoot is **zero in both cases**. Once the loop period is pinned to the provider's default, overshoot comes out to 6 at the average delay, 12 at the worst delay. The one lever left is the startup time, and the last two columns show that it is not enough either: zeroing out overshoot requires the startup time to fall to 1.47 seconds, and at the worst delay, the required value is negative 11.36 seconds — **impossible**. Delegating here did not delegate an hourly item; it **took away the setting** the previous lesson found. The observability side of the loss is quieter than the settings side. This topic's second lesson traced a single request, counting that it passed through four components and produced 50 item reads and 7 writes. That measurement requires direct state-store reads. Once access is gone, so is the measurement: the question of why a request was not applied now gets answered **from the outside**, by looking at results, not from inside the plane. Looking at results does not give a wrong answer, it gives a **late** one; the previous lesson's oscillation is exactly this kind of thing — an error that converges to the correct number in the end and is invisible along the way. The distinction is this: most delegated items are **work**, and load falls when they are delegated. But some are **information and control**; delegating them does not reduce load, it only changes whose hands they are in. The second class's cost shows up not in monthly hours but in **the number of options you have left when a problem appears**. The decision is therefore not one-directional. The 50 hours gained is 6.25 hours per person per month on an eight-person team, and on a setup that needs none of the five lost setting items, this trade is a clear win. The same trade, on a delay-sensitive workload — a cluster that needs to tune its loop period — buys, for 50 hours, an oscillation that cannot be closed. What needs to be measured is not the hours but **which items need to stay with you**; this list changes from setup to setup, and if it is not written before the delegation contract is signed, it cannot be written afterward. #### Summary - 7 of 21 operational items get delegated, 7 get shared, 7 stay with you no matter what; 50 of the monthly 106 hours (47.2 percent) get delegated, and item count falls from 21 to 14. - The remaining items are items of the application, not the cluster — declarations, resource requests, authorization, network policy, observability, release rollout — and none of them shrink under any delegation. - Exit cost is 7 items: a monthly 33-hour maintenance load, a 132-hour rebuild; for as long as the delegation lasts, these are items no one looks at. - Over thirty-six months, delegated operation gave 3 upgrades, 0 unsupported months, and an oldest release of 11 months; keeping the same freshness with self-run operation took 45 hours, against 21 hours under delegation, but the calendar passed to the provider. - Under self-run operation the calendar stays with you, but a calendar not kept writes a cost: with eighteen-month capacity, 12 of thirty-six months went unsupported, and the release aged up to 19 months. - Of twelve setting and observability items, only 3 are fully accessible on the delegated cluster; the 5 inaccessible items are exactly what this topic measured, and once the loop period is pinned, overshoot goes from 0 to 6 (12 at the worst delay); the one lever left demands an impossible startup time. #### Next Step Across this topic, what a cluster is, which parts it is built from, what those parts report, how the loop reconciles, and who operates it were all measured. One thing was never defined: **the thing being reconciled itself**. Across four lessons, items like "desired copy count," "placement constraint," and "service" appeared, and all of them stood as a field in the model's own small schema. What actually gets declared to the cluster — what the smallest deployable unit of a workload is, which object copies get counted by, which copies a name resolves to, where configuration and secrets get written — was never defined. The next topic builds these objects and tests each one by the same measure: which decision does this object carry, and what information does it need in order to carry it. ## Core Objects Deploying with declarative objects: what the smallest deployment unit holds together, declarative version management with the replica set and version object, stable in-cluster addressing, the routing and balancing layer for external access, rolling out configuration and secret objects, the persistent volume claim and driver interface, stateful applications that need a stable identity, one-shot and recurring tasks, and multi-tenant resource separation with namespaces and quotas. ### Pod Source: https://academia.sh/en/courses/orchestration/pod Measuring the smallest deployment unit declared to the cluster: what keeping three things together buys, the scale unit of packaging two processes into one pod versus two pods, and the difference in failure propagation, resource accounting, and network round trips. The cluster is up, and the reconciliation loop is turning: desired state sits in one place, actual state is measured in another, and the difference between them is converted into a correction. This loop was measured — how many rounds it takes to converge, how much work it does per round, how many wrong corrections it produces with delayed information. But the thing the loop operates on was never defined. **What is declared to the cluster?** Desired state is not a number; it is an object, and that object's boundaries determine what the reconciler treats as a single unit and what it corrects separately. This topic's first decision starts right there: what is the **smallest piece** that can be declared to the cluster? The answer is not a single container. The smallest deployment unit is a set of containers that declares that it will be moved together; this unit is called a pod. This lesson counts what a pod holds together and compares two packaging decisions across four measurements. What is measured is a model built with `node`; no real cluster runs. **CO1 — the regional measurement network runs on a cluster in this course:** this is the fictional software that collects, verifies, and bills meter readings. **CO2 — the processes inside a pod share three things:** the same network identity, the same local filesystem, the same lifetime. **CO3 — every exchange between two processes in separate pods adds one name resolution and one network round trip**, and inside the same pod this count is zero. **CO4 — the reading collector needs six replicas, the log transformer two**; the verifier and the local cache need four each, billing and the work order three each. **CO5 — if a process inside a pod crashes, the whole pod restarts** and goes two rounds without serving. **CO6 — the resource request and limit are declared at the pod level**; the processes inside it are not accounted separately. #### Where the Boundary Is Drawn A log transformer runs alongside the reading collector: it reads the raw lines the collector produces, formats them, and emits them. Are these two processes packaged in the same pod, or in separate pods? The question looks like a matter of preference; the model below reduces it to four numbers. ```js // pod.mjs — measuring two packaging decisions (MODEL; no real cluster runs). // CO1-CO6 assumptions are embedded in the code; every number below comes from this file. export const PROCESSES = { 'reading-collector': { replicas: 6, sharedFile: true, exchanges: 120 }, 'log-transformer': { replicas: 2, sharedFile: true, exchanges: 120 }, 'verifier': { replicas: 4, sharedFile: false, exchanges: 400 }, 'local-cache': { replicas: 4, sharedFile: false, exchanges: 400 }, 'billing': { replicas: 3, sharedFile: false, exchanges: 12 }, 'work-order': { replicas: 3, sharedFile: false, exchanges: 12 }, }; export const TRIPS = 2; // CO3: name resolution + network round trip export const SHARED = ['network identity', 'local filesystem', 'lifetime']; // Two packaging decisions keep the same process under the same load; only the boundary changes. export function pack(a, b) { const A = PROCESSES[a], B = PROCESSES[b]; const max = Math.max(A.replicas, B.replicas), total = A.replicas + B.replicas; const exchanges = Math.min(A.exchanges, B.exchanges); return { exchanges, together: { pods: max, procs: 2 * max, extra: 2 * max - total, scaleUnit: 1, failures: 2, accounting: max, procAccounting: 0, trips: 0, extraNotify: 0 }, apart: { pods: total, procs: total, extra: 0, scaleUnit: 2, failures: 1, accounting: total, procAccounting: total, trips: TRIPS * exchanges, extraNotify: SHARED.length }, }; } // Devolved decision: should these two processes travel in the same pod? export const INFO = ['shared file', 'exchange frequency', 'replica ratio']; const THRESHOLD = 100; export function decide(k) { const missing = INFO.filter((x) => k[x] === undefined); if (missing.length) return { result: 'cannot decide', note: `missing: ${missing.join(', ')}` }; const tight = k['shared file'] || k['exchange frequency'] >= THRESHOLD; if (k['replica ratio'] !== 1 && tight) return { result: 'conflict', note: 'two criteria point opposite ways' }; if (k['replica ratio'] !== 1) return { result: 'separate pods', note: 'replica ratio is not 1' }; if (tight) return { result: 'same pod', note: 'tightly coupled, ratio 1' }; return { result: 'separate pods', note: 'sparse exchange' }; } export function record(a, b) { const A = PROCESSES[a], B = PROCESSES[b]; return { 'shared file': A.sharedFile && B.sharedFile, 'exchange frequency': Math.min(A.exchanges, B.exchanges), 'replica ratio': Math.max(A.replicas, B.replicas) / Math.min(A.replicas, B.replicas) }; } export const PAIRS = [ ['reading-collector', 'log-transformer', 'collector+transformer'], ['verifier', 'local-cache', 'verifier+cache'], ['billing', 'work-order', 'billing+work-order'], ]; if (import.meta.url === `file://${process.argv[1]}`) { const s = (x, n) => String(x).padStart(n); console.log(`${'pair'.padEnd(23)}${'boundary'.padEnd(12)}${s('pods', 7)}${s('procs', 6)}` + `${s('extra', 6)}${s('scale', 6)}${s('fail', 6)}${s('acct', 6)}${s('trips', 6)}` + `${s('notify', 9)}`); for (const [a, b, ad] of PAIRS) { const p = pack(a, b); for (const [boundary, r] of [['one pod', p.together], ['two pods', p.apart]]) { console.log(`${ad.padEnd(23)}${boundary.padEnd(12)}` + `${s(r.pods, 7)}${s(r.procs, 6)}${s(r.extra, 6)}${s(r.scaleUnit, 6)}` + `${s(r.failures, 6)}${s(r.procAccounting, 6)}${s(r.trips, 6)}${s(r.extraNotify, 9)}`); } } console.log(`\nshared item (one pod): ${SHARED.join(', ')} = ${SHARED.length}`); console.log(`in two pods each one turns into a separate notification item: ${SHARED.length}`); console.log(`\n${'pair'.padEnd(28)}${'decision'.padEnd(16)}reason`); for (const [a, b, ad] of PAIRS) { const r = decide(record(a, b)); console.log(`${ad.padEnd(28)}${r.result.padEnd(16)}${r.note}`); } const missingK = { ...record('billing', 'work-order') }; delete missingK['replica ratio']; const r = decide(missingK); console.log(`${'billing+work-order (blind)'.padEnd(28)}${r.result.padEnd(16)}${r.note}`); // The cost of information: if 'replica ratio' is unknown, the system still has to pack. let wrong = 0, extraProcs = 0; for (const [a, b] of PAIRS) { const k = record(a, b); const full = decide(k); const blindK = { ...k }; delete blindK['replica ratio']; const blind = blindK['shared file'] || blindK['exchange frequency'] >= THRESHOLD ? 'same pod' : 'separate pods'; const correct = full.result === 'same pod' ? 'same pod' : 'separate pods'; if (blind !== correct) { wrong++; extraProcs += pack(a, b).together.extra; } } console.log(`\nif replica ratio is unknown: ${wrong} of ${PAIRS.length} pairs mispacked,` + ` ${extraProcs} unnecessary process instances`); } ``` ``` pair boundary pods procs extra scale fail acct trips notify collector+transformer one pod 6 12 4 1 2 0 0 0 collector+transformer two pods 8 8 0 2 1 8 240 3 verifier+cache one pod 4 8 0 1 2 0 0 0 verifier+cache two pods 8 8 0 2 1 8 800 3 billing+work-order one pod 3 6 0 1 2 0 0 0 billing+work-order two pods 6 6 0 2 1 6 24 3 shared item (one pod): network identity, local filesystem, lifetime = 3 in two pods each one turns into a separate notification item: 3 pair decision reason collector+transformer conflict two criteria point opposite ways verifier+cache same pod tightly coupled, ratio 1 billing+work-order separate pods sparse exchange billing+work-order (blind) cannot decide missing: replica ratio if replica ratio is unknown: 1 of 3 pairs mispacked, 4 unnecessary process instances ``` The first pair shows why the pod boundary is not a matter of preference. The collector wants six replicas, the transformer two. In one pod the scale unit is **one**: six pods open, twelve process instances run, and four of them serve no load at all. Four extra process instances are the direct cost of the scale unit being one — the transformer has ridden along on the collector's scale even though it did not need to. In two pods these four instances disappear, and in exchange two hundred forty network round trips appear: each of the hundred twenty exchanges pays one name resolution and one network round trip. The second pair gives the reverse of the same accounting. The verifier and the local cache have equal replica counts, so in one pod the extra process instances are **zero**; separating them buys nothing, and eight hundred network round trips are paid instead. Tightly coupled helper processes that scale at the same rate belong inside the pod boundary; this is the reason the boundary exists. The third pair shows the third case: equal scale, but sparse exchange. Twenty-four round trips are cheap; separating is reason enough to bring failure propagation down from two to one and to gain per-process resource accounting. The resource accounting column is a direct consequence of **CO6**. In one pod there are six accounting entries but zero per-process accounting: the pod's request and limit are the sum of the two processes, and which one consumes how much never shows up in the declaration. In two pods there are eight entries and eight per-process accounts. This difference feeds into the scaling decision — a system that does not know what is growing scales both of them at once. The last row counts the notification cost. In one pod the three shared items are free because the boundary itself provides them. Separating turns all three into a notification item: the network identity into a service object, the local filesystem into a transfer format, and the shared lifetime into a guarantee that no longer exists. The compose file on a single machine also gathers several processes into one file; the difference is that togetherness there means *being on the same machine*, while in a pod it means *moving together, dying together, and sharing a single identity*. The compose file's startup waves and readiness-check measurement were done there; they are not repeated here, they become an input to the comparison. #### The Cost of the Same Lifetime Scale, round-trip, and accounting counts can all be read from a fixed table. The third shared item — the same lifetime — only becomes visible over time. The second run applies the same failure sequence to both packagings. ```js // lifetime.mjs — the cost of the same lifetime: the same failure sequence applied to two packagings (MODEL). import { PROCESSES } from './pod.mjs'; const SEED = 20260801, ROUNDS = 24, REPAIR = 2, RATE = 0.35; // The generator is hand-written; 32-bit integer operations, seed is visible. const make = (t) => () => { t ^= t << 13; t ^= t >>> 17; t ^= t << 5; return (t >>> 0) / 2 ** 32; }; // Required process instances: 6 collectors + 2 transformers. The failure sequence is generated once. const NEEDED = [ ...Array.from({ length: PROCESSES['reading-collector'].replicas }, (_, i) => ['collector', i]), ...Array.from({ length: PROCESSES['log-transformer'].replicas }, (_, i) => ['transformer', i]), ]; const rnd = make(SEED); const FAILURES = Array.from({ length: ROUNDS }, () => (rnd() < RATE ? NEEDED[Math.floor(rnd() * NEEDED.length)] : null)); // One pod: pod i carries both collector i and transformer i; if one falls, both fall. // Two pods: each process instance is in its own pod; a failure only takes it down. function run(together) { const down = {}; // process instance -> down until when let fell = 0, neighbor = 0; const missingRounds = { collector: 0, transformer: 0 }; for (let t = 0; t < ROUNDS; t++) { if (FAILURES[t]) { const [role, i] = FAILURES[t]; // Pod i, at the one-pod boundary, carries both collector i and transformer i. CO5. for (const [r, k] of together ? [['collector', i], ['transformer', i]] : [[role, i]]) { const key = `${r}#${k}`; if (!NEEDED.some(([x, y]) => x === r && y === k)) continue; if ((down[key] ?? -1) < t) { fell++; if (r !== role) neighbor++; } down[key] = t + REPAIR; } } for (const role2 of ['collector', 'transformer']) { const ad = role2 === 'collector' ? 'reading-collector' : 'log-transformer'; const up = Array.from({ length: PROCESSES[ad].replicas }, (_, k) => `${role2}#${k}`) .filter((a) => (down[a] ?? -1) < t).length; if (up < PROCESSES[ad].replicas) missingRounds[role2]++; } } return { fell, neighbor, ...missingRounds }; } const s = (x, n) => String(x).padStart(n); console.log(`seed ${SEED}, ${ROUNDS} rounds, ${FAILURES.filter(Boolean).length} failures,` + ` repair ${REPAIR} rounds`); console.log(`${'boundary'.padEnd(12)}${s('fell', 7)}${s('from neighbor', 16)}` + `${s('collector short', 17)}${s('transformer short', 19)}`); for (const [ad, b] of [['one pod', true], ['two pods', false]]) { const r = run(b); console.log(`${ad.padEnd(12)}${s(r.fell, 7)}${s(r.neighbor, 16)}${s(`${r.collector} rounds`, 17)}` + `${s(`${r.transformer} rounds`, 19)}`); } ``` ``` seed 20260801, 24 rounds, 11 failures, repair 2 rounds boundary fell from neighbor collector short transformer short one pod 13 3 22 rounds 7 rounds two pods 10 0 20 rounds 4 rounds ``` Eleven failures produce thirteen and ten drops across the two packagings. The gap of three and the number three in the third column say the same thing: **in one pod, three process instances fell because of a failure that had nothing to do with them.** In none of these three drops was there a problem in the transformer's or the collector's own code; its neighbor crashed, the pod restarted, and it went down with it. At the two-pod boundary this column is zero — the failure never crosses the boundary drawn by the declaration. The last column is how this shows up in service. The transformer ran with a missing replica in seven of twenty-four rounds in one pod, and in four when separated. A gap of three rounds looks small; what actually grows is not the ratio but the **cause**: nearly half of the transformer's missing rounds come not from its own failure but from its neighbor's, and this share grows with the number of processes placed in the pod. The same lifetime is declared as a convenience and paid for as a source of outages. #### The Devolved Decision and What It Does Not Know The decision that moves from human to system in this lesson is: **which processes travel as a single unit.** Once the decision is made, placement, restart, and scaling all operate on this unit; the system never looks inside the pod. The information items the decision depends on are named in the model, and there are three of them: shared file use, exchange frequency, replica ratio. All three have a source — the first two come from measurement, the third from a load declaration. The cost of information is most visible in the third item. The replica ratio is not a measurement but a **guess**: will the two processes' future load grow in the same direction? If this guess is not written down, the system still has to pack, and the model's last line states what happens: one of three pairs is mispacked, four unnecessary process instances appear. The source of the error is not a bug but a **gap**; a rule that does not know the replica ratio takes the option that looks cheap and puts the collector and the transformer in the same pod. The first pair's decision, though, is not resolved even with information: because the two criteria point in opposite directions, the result comes out as `conflict`. The shared file and frequent exchange argue for togetherness, while the threefold replica ratio argues for separation. This is a case where devolving **does not pay off**: the system applies the rule, the rule produces no answer, and the decision returns to a human. An answer only emerges once one of the inputs changes — either the four extra process instances are accepted, or the shared file is converted into a transfer and two hundred forty round trips are paid. Devolving the decision forces a change in one of the quantities the decision depends on. #### Summary - A pod is the smallest deployment unit that can be declared to the cluster, and it keeps three things together: the same network identity, the same local filesystem, the same lifetime. Separating turns these three into three separate notification items. - If two processes asking for six and two replicas are packed into one pod, the scale unit drops to one and four unnecessary process instances appear; if they are separated, these four instances disappear and two hundred forty network round trips are added. - For the pair with an equal replica ratio, separating gains nothing and costs eight hundred round trips; for the pair that scales equally but exchanges rarely, failure propagation drops from two to one in exchange for twenty-four round trips. - Resource accounting is at the pod level: one pod keeps six entries, per-process accounting is zero, and what is growing never shows up in the declaration. - On the same failure sequence, one pod produced thirteen drops and two pods produced ten; the entire difference is neighbor failure, and the transformer ran with a missing replica for seven rounds instead of four. - The devolved decision rests on three information items; when the replica ratio is missing, one of three pairs is mispacked and four unnecessary process instances appear, and when the criteria conflict the rule produces no answer. #### Next Step This lesson drew the boundary of a single pod and took the number of replicas it needs as a given: six, four, three. Yet that number is itself something declared to the cluster, and the moment it is declared a new question arises — when the actual replica count drifts from what is desired, who corrects it, how fast, and in what order? Moreover, while the desired replica count stays fixed, **what the replica contains** can change; moving to a new version requires declaring how many extra and how many missing replicas are allowed at the same time. The next lesson writes both objects with `node` and counts the rollout round by round: how many rounds it takes to complete, how many rounds it runs below capacity, and how many rounds a rollback takes. ### Replica Sets and Version Objects Source: https://academia.sh/en/courses/orchestration/replica-sets-and-version-objects Two measures of declarative version management: the controller's work per round reconciling the desired replica count against the actual one, and the effect of rollout settings — how many extra, how many missing replicas — on rollout rounds, below-capacity rounds, and the rollback window. The previous lesson drew the pod boundary and took needing six replicas as a given. That assumption alone does nothing: if a pod goes down, nothing replaces it; if a node is lost, the pods on it are lost too. For the number six to have any effect, it must be **declared** somewhere, and a controller must continuously compare that declaration against reality. The object that carries this declaration is the **replica set**: a single pod template and a single number. The number is not enough. The template itself also changes — a new version ships, and the content of the six pods has to change. The object that manages this is the **version object**; it holds two replica sets and shifts the desired count between them step by step. This lesson writes both objects with `node` and counts the rollout round by round. What is measured is a model; no real cluster runs. **CO7 — the desired replica count for the reading collector is six.** **CO8 — the only thing the controller knows is the desired count and the actual count**; the inside of the pods is invisible to it. **CO9 — a newly created pod becomes ready after two rounds**; removal finishes in the same round. **CO10 — the version object holds two replica sets** and splits the desired count between them. **CO11 — the rollout setting is two numbers:** the maximum extra pods allowed, the maximum missing ready replicas allowed. **CO12 — the new version turns out broken at the end of round four**; rollback runs the same settings in reverse. #### The Desired Count and the Actual Count The controller's job is one line: take the gap, produce an action in whatever direction its sign points. The run below carries this one line through four events — cold start, two pods crashing, the desired count changing, and two pods being added by hand from outside. ```js // replica.mjs — replica set controller: reconciles desired vs. actual replica count (MODEL). export const READY_ROUNDS = 2; // CO9: a new pod is ready in 2 rounds export const set = (name, desired) => ({ name, desired, ready: 0, pending: [] }); export const total = (k) => k.ready + k.pending.length; export function advance(k) { // pending pods become ready k.pending = k.pending.map((x) => x - 1); const done = k.pending.filter((x) => x <= 0).length; k.ready += done; k.pending = k.pending.filter((x) => x > 0); return done; } export const create = (k, n) => { for (let i = 0; i < n; i++) k.pending.push(READY_ROUNDS); }; export function remove(k, n) { // pending pods go first, then ready ones let left = n; while (left > 0 && k.pending.length) { k.pending.pop(); left--; } const d = Math.min(left, k.ready); k.ready -= d; return n - (left - d); } // Controller: turns the gap into an action. All it knows is the desired and actual count. CO8. export function reconcile(k) { const gap = k.desired - total(k); if (gap > 0) { create(k, gap); return `+${gap} create`; } if (gap < 0) { remove(k, -gap); return `${gap} remove`; } return 'no gap'; } if (import.meta.url === `file://${process.argv[1]}`) { const K = set('reading-collector', 6); // CO7 const EVENT = { 4: ['two pods crashed', (k) => (k.ready -= 2)], 8: ['desired became 8', (k) => (k.desired = 8)], 11: ['two pods added by hand', (k) => create(k, 2)] }; const s = (x, n) => String(x).padStart(n); let created = 0, removed = 0, shortRounds = 0; console.log(`${'round'.padEnd(6)}${'event'.padEnd(25)}${s('desired', 8)}${s('ready', 6)}` + `${s('pending', 9)} action`); for (let t = 1; t <= 14; t++) { const o = EVENT[t]; if (o) o[1](K); advance(K); const before = total(K); const action = reconcile(K); const d = total(K) - before; if (d > 0) created += d; else removed += -d; if (K.ready < K.desired) shortRounds++; console.log(`${String(t).padEnd(6)}${(o ? o[0] : '').padEnd(25)}${s(K.desired, 8)}` + `${s(K.ready, 6)}${s(K.pending.length, 9)} ${action}`); } console.log(`\nover 14 rounds: ${created} created, ${removed} removed, ${shortRounds} rounds` + ` with ready count below desired`); } ``` ``` round event desired ready pending action 1 6 0 6 +6 create 2 6 0 6 no gap 3 6 6 0 no gap 4 two pods crashed 6 4 2 +2 create 5 6 4 2 no gap 6 6 6 0 no gap 7 6 6 0 no gap 8 desired became 8 8 6 2 +2 create 9 8 6 2 no gap 10 8 8 0 no gap 11 two pods added by hand 8 8 0 -2 remove 12 8 8 0 no gap 13 8 8 0 no gap 14 8 8 0 no gap over 14 rounds: 10 created, 2 removed, 6 rounds with ready count below desired ``` The four events come from four separate causes, but the controller's response has one shape. Six pods on cold start, two on crash, two when the desired count rises; in every case the rule applied is the same, and in none of them is the **reason** for the drop ever asked. The controller does not distinguish a crash from a manual deletion, because under **CO8** both reduce to the same number. This is devolving's most concrete payoff: four separate procedures are not written for four different scenarios. Round eleven is the other side of this. The two pods added from outside are deleted in the same round; a change made outside the declaration does not outlive the next reconciliation round. Manual work done in the cluster is not permanent, and the only evidence of this is the `-2 remove` line in the table. The cost of the numbers shows up in rounds two and five: in the two rounds marked `no gap`, genuinely no work is done, but the ready replica count is still below the desired one. The total measure is six rounds — in six of fourteen rounds the cluster ran with short capacity. All of this comes from **CO9**, that is, from readiness taking two rounds; the controller cannot close the gap in the round it sees it, it can only issue the order to close it. The capacity shortfall is the delay between a decision being made and its effect becoming visible. The `no gap` on ten rounds of the table is also a result. The same rule runs again every round and does nothing when the gap is closed; this is a property of the reconciliation loop that does not change at cluster scale either. The controller is not a procedure that fires once and finishes, it is a comparison that runs continuously and produces no work in most rounds. Work per round is nonzero in four of fourteen rounds. #### The Rollout's Settings When the replica count stays fixed but the template changes, the work stops being reconciliation and turns into a **rollout**. The version object manages two replica sets at once and obeys two numbers: at most how many extra pods can exist at once, at most how many replicas can be missing at once. ```js // rollout.mjs — version object: staged rollout and rollback between two replica sets (MODEL). import { set, total, advance, create, remove } from './replica.mjs'; const N = 6; // CO7: desired replica count // surplus: max extra pods allowed at once; shortfall: max missing ready replicas allowed. CO11. function rollout(surplus, shortfall, badRound = null) { const old = set('version-1', N), fresh = set('version-2', 0); old.ready = N; const created = { 'version-1': 0, 'version-2': 0 }; let t = 0, belowRounds = 0, peak = N, removedCount = 0, backRound = null, stall = 0, dir = 'forward'; while (t < 80) { t++; advance(old); advance(fresh); if (badRound && t === badRound && dir === 'forward') { dir = 'back'; backRound = t - 1; } const target = dir === 'forward' ? fresh : old, source = dir === 'forward' ? old : fresh; const room = Math.min(N + surplus - (total(old) + total(fresh)), N - total(target)); if (room > 0) { create(target, room); created[target.name] += room; } const drop = Math.min(old.ready + fresh.ready - (N - shortfall), total(source)); if (drop > 0) { remove(source, drop); removedCount += drop; } peak = Math.max(peak, total(old) + total(fresh)); if (old.ready + fresh.ready < N) belowRounds++; stall = room <= 0 && drop <= 0 && !old.pending.length && !fresh.pending.length ? stall + 1 : 0; if (stall >= 2) return { round: null, belowRounds, peak, created, removedCount, backRound }; if (total(source) === 0 && target.ready >= N) break; } return { round: t, belowRounds, peak, created, removedCount, backRound }; } const SETTING = [[0, 0], [0, 1], [1, 0], [1, 1], [2, 2], [6, 0]]; const s = (x, n) => String(x).padStart(n); console.log(`desired replicas ${N}, ready in 2 rounds`); console.log(`${s('surplus', 8)}${s('shortfall', 10)}${s('round', 11)}${s('below capacity', 16)}` + `${s('peak pods', 13)}${s('created', 9)}${s('removed', 9)}`); for (const [f, e] of SETTING) { const r = rollout(f, e), o = r.created['version-1'] + r.created['version-2']; console.log(`${s(f, 8)}${s(e, 10)}${s(r.round ?? 'stalls', 11)}${s(`${r.belowRounds} rounds`, 16)}` + `${s(r.peak, 13)}${s(o, 9)}${s(r.removedCount, 9)}`); } console.log(`\nif the new version turns out broken at the end of round 4 (CO12):`); console.log(`${s('surplus', 8)}${s('shortfall', 10)}${s('forward rounds', 15)}${s('back rounds', 12)}` + `${s('total', 8)}${s('below capacity', 16)}${s('wasted pods', 13)}`); for (const [f, e] of SETTING.slice(1)) { const r = rollout(f, e, 5); if (r.backRound === null) { console.log(`${s(f, 8)}${s(e, 10)} rollout finished before round 4: no old replica to roll back to`); continue; } console.log(`${s(f, 8)}${s(e, 10)}${s(r.backRound, 15)}${s(r.round - r.backRound, 12)}` + `${s(r.round, 8)}${s(`${r.belowRounds} rounds`, 16)}${s(r.created['version-2'], 13)}`); } ``` ``` desired replicas 6, ready in 2 rounds surplus shortfall round below capacity peak pods created removed 0 0 stalls 0 rounds 6 0 0 0 1 19 18 rounds 6 6 6 1 0 18 0 rounds 7 6 6 1 1 10 9 rounds 7 6 6 2 2 6 5 rounds 8 6 6 6 0 3 0 rounds 12 6 6 if the new version turns out broken at the end of round 4 (CO12): surplus shortfall forward rounds back rounds total below capacity wasted pods 0 1 4 6 10 9 rounds 1 1 0 4 5 9 0 rounds 2 1 1 4 6 10 9 rounds 3 2 2 4 6 10 9 rounds 6 6 0 rollout finished before round 4: no old replica to roll back to ``` The first row gives the sharpest sense of what the two numbers mean: if both are zero, the rollout **does not start**. There is no room for an extra pod and no permission to run short a replica; the version object cannot take a single step without violating the rules, and the model sees the stall and exits. That a declarative object needs slack in at least one direction to make progress is this table's first result. The next two rows create and delete the same number of pods yet produce diametrically opposite costs. Allowing one shortfall and no surplus stretches the rollout to nineteen rounds, and in eighteen of them the cluster runs with short capacity; peak pod count stays at six, so no extra resource is spent at all. Allowing one surplus and no shortfall takes eighteen rounds, **never** drops below capacity, and in exchange asks for only one pod's worth of extra resource. The round gap between the two settings is one; the capacity gap is eighteen rounds. Being willing to give extra resource is the price of getting through the same span without interruption. The bottom two rows show where speed comes from. When one surplus and one shortfall are allowed together, the round count drops from eighteen to ten; at two and two, to six. Speed is fed by both settings, but below-capacity rounds come along with it. The last row is the tip of this curve: when six surplus pods are allowed, the rollout finishes in three rounds and capacity never drops — peak pod count is twelve, meaning resources temporarily double. This setting is the numerical equivalent of a blue-green rollout. The second table makes that three-round speed pay its hidden cost. When the new version turns out broken at the end of round four, rollback with one surplus and zero shortfall takes five rounds, capacity never drops, and two pods go to waste. With the two-two setting rollback takes six rounds and six pods go to waste — the new version was fully installed within four rounds. With the six-surplus setting there is no such thing as rollback: the rollout finished before round four, and the old replica set has already been zeroed out. **The rollback window is the same thing as the rollout's duration.** Every setting that speeds up the rollout shortens this window, and past a certain point closes it. #### The Devolved Decision and What It Does Not Know There are two devolved decisions in this lesson: **how many replicas run** and **how fast to move to the new template.** For the first, the only information item the system needs to know is the desired count — this is why it is cheap and answers four different events with one rule. The second needs two items: surplus and shortfall. These two numbers are not a preference, they are the mutual price of two resources; one is temporary capacity, the other temporary disruption. The cost of the two items is not equal either. The desired count is written once and rarely changes; it needs no measurement, does not age, and when wrong its result is directly visible. Rollout settings, though, rest on an estimate that must be reassessed at every release: how long the new version will take to come up, and if it is broken, how long that will take to become clear. Because the second estimate cannot be measured, the settings are usually written once and stay the same for every release — yet as the table shows, the same setting can entirely close the rollback window on a version that comes up fast. What the system **does not know** stands in the same table. The version object sees that new replicas are ready, not that they are working correctly; under **CO8** the inside of the pod is closed to it. The breakage arrives from somewhere at the end of round four, and in the model this is an assumption. Because of this information item the system cannot measure, six pods are built for nothing under the two-two setting, and under the six-surplus setting there is nowhere to roll back to because no old version is left at all. The wrong decision here is not an arithmetic error: speed is proceeding without waiting for a piece of information the system cannot see. #### Summary - The replica set controller applies one rule: create if the gap is positive, remove if it is negative. Cold start, crash, the desired count changing, and manual addition are all handled by the same procedure. - The two pods added by hand outside the declaration were deleted in the same round; manual work done in the cluster does not outlive the next reconciliation round. - In six of fourteen rounds the ready replica count stayed below desired; the source of this is readiness taking two rounds, that is, the delay between a decision and its effect. - If both rollout numbers are zero, the rollout never starts; allowing one shortfall spends eighteen of nineteen rounds below capacity, and allowing one surplus does the same work in eighteen rounds without ever dropping capacity. - Two surplus and two shortfall bring the rollout down to six rounds, six surplus to three; in the latter, peak pod count rises to twelve, that is, it doubles. - The rollback window is the same as the rollout's duration: breakage noticed in round four wastes two pods under the one-zero setting and six pods under the two-two setting; a rollout that finishes in three rounds leaves no old replica to roll back to. #### Next Step This lesson measured keeping six pods up and changing what they contain. In both runs pods were continually created and deleted: ten creations and two deletions in fourteen rounds, six creations and six deletions in the rollout. Every new pod is a new pod, and the previous one's address went with it. In that case, which one will another component sending it requests connect to? A replica set guarantees six pods but not **which six**. The next lesson measures the object that fills this gap: a name that stays fixed on top while the replica set changes underneath it, the round that resolving that name adds to a request, and the number of requests that go to a replica that is not ready. ### Service Objects Source: https://academia.sh/en/courses/orchestration/services Two measures of stable in-cluster addressing: how many times the name changes and how many times the list changes while eleven pods come and go, the number of requests that reach a not-ready endpoint with and without a readiness check, and the trade-off between the round added by name resolution's cache lifetime and requests that reach a stale endpoint. The previous lesson measured keeping six replicas up, and in measuring it made something visible: a replica set guarantees that there will be **six pods**, not **which six**. In the fourteen-round run ten pods were created, and in the rollout six were deleted and six created. Every new pod is a new pod; the previous one's address went with it. When the verifier wants to send a request to the reading collector, who will it connect to? The object that fills this gap is the **service object**. What it holds is not a piece of software; it is a name and the endpoint list behind that name. The list is derived from the replica set and updates itself as pods come and go. In M16/K06, **service** was a separately deployed and separately owned piece of software; the service object here is a name standing in front of that software's replicas — the same word, a different object. This lesson models the name and the list with `node` and counts both; no real cluster runs. **CO13 — a service object holds an endpoint list tied to a name**, and the list is derived from the pods in a replica set. **CO14 — resolving the name to a list adds one round to a request**, and the result can be cached for a lifetime. **CO15 — requests are distributed round-robin over the endpoint list.** **CO16 — a new pod becomes ready two rounds later**; without a readiness check it enters the list in the round it is created. **CO17 — a request to an endpoint that is not ready or has been dropped from the declaration fails.** **CO18 — the run lasts twenty-four rounds**, twelve requests arrive per round, and pod churn is produced by a seeded generator. #### The List That Changes, the Name That Does Not The run below applies the same churn schedule twice. The only thing that changes is where the endpoint list is derived from: all pods, or only the ready ones. ```js // service.mjs — service object: the name stays fixed while the endpoint list changes (MODEL). export const SEED = 20260803, ROUNDS = 24, REQUESTS = 12, N = 6, READY = 2, CHURN = 0.45; // The generator is hand-written; 32-bit integer operations, seed is visible. export const make = (t) => () => { t ^= t << 13; t ^= t >>> 17; t ^= t << 5; return (t >>> 0) / 2 ** 32; }; // The churn schedule is generated once; both runs lose the same pods in the same rounds. const rnd = make(SEED); export const SCHEDULE = Array.from({ length: ROUNDS }, () => (rnd() < CHURN ? Math.floor(rnd() * N) : null)); export const SERVICE_NAME = 'reading-collector.measurement'; // the one thing that never changes // The replica set holds N pods every round; churned slot gets a new pod (CO13). export function run(check) { let counter = N, name = SERVICE_NAME; const slot = Array.from({ length: N }, (_, i) => ({ name: `pod-${i}`, readyAt: 0 })); let requests = 0, failed = 0, churns = 0, nameChanges = 0, listChanges = 0, cursor = 0; let previous = ''; const endpointCounts = [], load = []; for (let t = 0; t < ROUNDS; t++) { const j = SCHEDULE[t]; if (j !== null) { slot[j] = { name: `pod-${counter++}`, readyAt: t + READY }; churns++; } // With the readiness check the list only draws from ready pods; without it, from all of them. const endpoints = check ? slot.filter((k) => k.readyAt <= t) : [...slot]; const signature = endpoints.map((k) => k.name).sort().join(','); if (t && signature !== previous) listChanges++; if (name !== SERVICE_NAME) nameChanges++; // no path ever touches the name previous = signature; endpointCounts.push(endpoints.length); load.push(endpoints.length ? Math.ceil(REQUESTS / endpoints.length) : REQUESTS); for (let i = 0; i < REQUESTS; i++) { requests++; if (!endpoints.length) { failed++; continue; } const target = endpoints[cursor++ % endpoints.length]; // CO15: round-robin distribution if (target.readyAt > t) failed++; // CO17 } } return { requests, failed, churns, nameChanges, listChanges, endpointMin: Math.min(...endpointCounts), endpointMax: Math.max(...endpointCounts), loadMax: Math.max(...load) }; } if (import.meta.url === `file://${process.argv[1]}`) { const s = (x, n) => String(x).padStart(n); console.log(`seed ${SEED}, ${ROUNDS} rounds, ${REQUESTS} requests per round, ${N} replicas,` + ` readiness ${READY} rounds, service name '${SERVICE_NAME}'`); console.log(`${'check'.padEnd(14)}${s('requests', 9)}${s('failed', 8)}${s('rate', 8)}` + `${s('endpoint count', 16)}${s('max per endpoint', 19)}${s('list changes', 15)}` + `${s('name changes', 14)}`); for (const [label, d] of [['off', false], ['readiness', true]]) { const r = run(d); console.log(`${label.padEnd(14)}${s(r.requests, 9)}${s(r.failed, 8)}` + `${s(`%${(100 * r.failed / r.requests).toFixed(1)}`, 8)}` + `${s(`${r.endpointMin}-${r.endpointMax}`, 16)}${s(r.loadMax, 19)}${s(r.listChanges, 15)}` + `${s(r.nameChanges, 14)}`); } console.log(`\n${ROUNDS} rounds, ${run(true).churns} pods came and went;` + ` the caller used the same name throughout`); } ``` ``` seed 20260803, 24 rounds, 12 requests per round, 6 replicas, readiness 2 rounds, service name 'reading-collector.measurement' check requests failed rate endpoint count max per endpoint list changes name changes off 288 44 %15.3 6-6 2 10 0 readiness 288 0 %0.0 4-6 3 13 0 24 rounds, 11 pods came and went; the caller used the same name throughout ``` The last line is this lesson's whole rationale. In twenty-four rounds eleven pods came and went; the endpoint list changed thirteen times; **the name changed zero times.** When the count of change and the count of constancy stand side by side, what a service object does becomes visible: it moves a source of volatility to a place the caller never sees. The only thing written in the verifier's code is the name `reading-collector.measurement`, and that name is unaffected by any event in the replica set. The list itself is not declared either, it is **derived**. The only thing written to the service object is which pods count as belonging to this name; which pods exist at any given moment comes from a measurement. This distinction is the same one that ran through the previous two lessons' information items: the desired replica count is a declaration and does not age, the endpoint list is a measurement and has to be regenerated every round. Whether an object carries a declaration or a measurement determines how often it can be wrong. The failed-request column, though, says the name **is not enough**. Without the readiness check the endpoint list always has six pods — the count is right, the content is wrong. A newly created pod cannot serve for two rounds, yet it sits in the list, and round-robin distribution sends it a request in turn. The result is forty-four of two hundred eighty-eight requests, that is, one in every seven. When the readiness check is turned on this number drops to **zero**; the list only shows pods that can serve. The cost sits in the two columns right next to it. In the run with the check, the endpoint count drops to four at times and the maximum load on one endpoint rises from two to three. The check does not save requests, it **redistributes** them: a request that would have gone to a not-ready pod is added to the ready ones instead. The list-change count rising by three over ten is due to the same cause — a pod now enters and leaves the list twice: it enters when ready, not when created, and leaves when it drops. A stable name is bought at the cost of the list underneath it changing more often. The source of the zero is not luck but a separation. The name sits in an object distinct from the replica set; no path that creates a pod, deletes a pod, or counts readiness ever touches that object. Without the service object, this zero would be a thirteen instead: every change in the endpoint list would mean the caller's own address list had to change too, and keeping that list current would be a component in the verifier's own code. The name exists precisely to remove that component. #### The Cost of Resolving the Name The name is fixed, the list is variable. The caller has to turn the name into a list before sending a request, and **CO14** says this costs one round. It is possible to pay this round on every request, and it is possible to pay it once and keep the result; the space between the two is measured below. ```js // resolution.mjs — whether the name is resolved fresh every time or read from a cache (MODEL). import { SCHEDULE, N, READY, ROUNDS, REQUESTS } from './service.mjs'; // Same churn schedule; each round's live set and ready-endpoint list are extracted once. function timeline() { let counter = N; const slot = Array.from({ length: N }, (_, i) => ({ name: `pod-${i}`, readyAt: 0 })); const alive = [], readyList = []; for (let t = 0; t < ROUNDS; t++) { const j = SCHEDULE[t]; if (j !== null) slot[j] = { name: `pod-${counter++}`, readyAt: t + READY }; alive.push(new Set(slot.map((k) => k.name))); readyList.push(slot.filter((k) => k.readyAt <= t).map((k) => k.name)); } return { alive, readyList }; } // CO14: every resolution adds one round to the request; the result may be cached for a lifetime. function cache(lifetime) { const { alive, readyList } = timeline(); let resolves = 0, requests = 0, stale = 0, cursor = 0, list = [], last = -Infinity; for (let t = 0; t < ROUNDS; t++) { for (let i = 0; i < REQUESTS; i++) { if (lifetime === 0 || t - last >= lifetime) { list = readyList[t]; last = t; resolves++; } requests++; const target = list[cursor++ % list.length]; if (!alive[t].has(target)) stale++; // request to an endpoint dropped from the declaration } } return { resolves, requests, stale }; } const s = (x, n) => String(x).padStart(n); console.log(`${ROUNDS} rounds x ${REQUESTS} requests = ${ROUNDS * REQUESTS} requests`); console.log(`${s('cache lifetime', 14)}${s('resolves', 11)}${s('extra rounds', 14)}` + `${s('extra rounds per request', 26)}${s('stale-endpoint requests', 25)}${s('rate', 8)}`); for (const lifetime of [0, 2, 4, 8, ROUNDS]) { const r = cache(lifetime); console.log(`${s(lifetime === 0 ? 'none' : `${lifetime} rounds`, 14)}${s(r.resolves, 11)}` + `${s(r.resolves, 14)}${s((r.resolves / r.requests).toFixed(2), 26)}${s(r.stale, 25)}` + `${s(`%${(100 * r.stale / r.requests).toFixed(1)}`, 8)}`); } ``` ``` 24 rounds x 12 requests = 288 requests cache lifetime resolves extra rounds extra rounds per request stale-endpoint requests rate none 288 288 1.00 0 %0.0 2 rounds 12 12 0.04 12 %4.2 4 rounds 6 6 0.02 35 %12.2 8 rounds 3 3 0.01 72 %25.0 24 rounds 1 1 0.00 216 %75.0 ``` The table should be read from both ends. At the top there is no caching at all: every request resolves the name again, two hundred eighty-eight requests pay two hundred eighty-eight extra rounds, and stale-endpoint requests are zero. At the bottom the name is resolved once: extra rounds drop to one, and in exchange two hundred sixteen requests — three quarters of them — go to pods that no longer exist. The three rows in between are the curve between these two ends, and the curve is steep: when the lifetime goes from two to four, extra rounds drop to six and stale requests rise from twelve to thirty-five. What this table says is this course's measurement axis itself. The endpoint list is an **information item**, and every information item has two prices: the cost of acquiring it and the cost of it going stale. At zero lifetime the first price is paid, in the last row the second. The seventy-five-percent share of stale-endpoint requests arises in a cluster where no component has failed — it is only that the list in the caller's hands is twenty-four rounds behind the cluster's actual state. The middle of the curve also says something. A two-round lifetime cuts resolution load from two hundred eighty-eight to twelve, that is, to a twenty-fourth, while keeping stale requests at four and a half percent. A four-round lifetime only halves resolution again, while tripling stale requests. At neither end do gain and loss grow at the same rate: the first step gains a lot and pays little, the later steps gain little and pay a lot. This asymmetry shows that the lifetime is a number to be chosen, and that neither end is correct. #### The Devolved Decision and What It Does Not Know The devolved decision here is clear: **which pod a request goes to.** This decision used to be written in the caller's configuration; now it is read from the service object's list. The decision rests on three information items: which pods exist, which are ready, when the list was read. The first comes free from the replica set. The second requires a check, and its cost is written in the first table: the maximum load per endpoint rises from two to three. The third also costs a round, and it sits in the second table's first column. What the system does not know is **when the list goes stale**. The model resolves this approximately with a lifetime number; the lifetime is an estimate of how often pods churn. If churn is more frequent than the estimate, stale-endpoint requests rise; if it is sparser, unnecessary resolution rounds are paid. In this run, with eleven churns over twenty-four rounds, even a two-round lifetime produces twelve stale requests; had the churn frequency been half as much, the same lifetime would have been cheaper. The source of the wrong decision here is not an arithmetic error, it is **not knowing how long a number stays true.** When the three results are counted together, the table comes out like this. In the run with the check and no caching, all two hundred eighty-eight routing decisions are correct. In the run without the check, forty-four are wrong, and the source of the error is a single missing information item: whether the pod is ready. **Failure to decide** never appeared in this run, because the ready-endpoint count dropped to four at the lowest and never to zero. If churn frequency rose and every pod ended up waiting to become ready at the same time, the list would go empty, and the service object would hold a name with nowhere to route to; in that case the decision is neither right nor wrong, it is not made at all. #### Summary - A service object is not a piece of software, it is a name and the endpoint list behind it; the list is derived from the replica set. M16/K06's **service** was a separately deployed piece of software; the object here is the name in front of that piece's replicas. - In twenty-four rounds eleven pods came and went and the endpoint list changed thirteen times; the name changed zero times. The only thing written in the caller's code is the name. - Without a readiness check, forty-four of two hundred eighty-eight requests went to a not-ready pod; with the check on, this number dropped to zero. - The check's cost is not lost requests but shifted load: the endpoint count drops to four, the maximum load on one endpoint rises from two to three, and the list changes thirteen times instead of ten. - Name resolution adds one round to a request; with no caching at all, two hundred eighty-eight extra rounds are paid, and with the name resolved once and kept, extra rounds drop to one but seventy-five percent of requests go to endpoints that no longer exist. - These two tables show an information item's two prices: the cost of acquiring it and the cost of it going stale; choosing the lifetime is an estimate of how often pods churn. #### Next Step Every request in this lesson came from inside the cluster: the verifier called the reading collector in the same cluster. Both name resolution and round-robin distribution ran inside that boundary. Yet a request to the regional measurement network's billing interface arrives from outside, from a client that knows nothing about the cluster. How many layers does that request have to cross, and how many rules have to be written at each layer, before it reaches a service object? The next lesson counts the layers of a single incoming external request: the round each layer adds, the relationship between rule count and service count for path-based versus name-based routing, and how many endpoints and how many paths the externally exposed surface consists of. ### External Access Source: https://academia.sh/en/courses/orchestration/external-access Two measures of the path opening to the outside of the cluster: the round added and the number of rules scanned by the four layers — external balancer, ingress rule, service object, and pod — and the difference between path-based and name-based routing in external hosts, identity items, and total surface. Every request in the previous lesson came from inside the cluster. The verifier called the reading collector in the same cluster by its name; both name resolution and round-robin distribution ran inside that boundary. A request arriving at the regional measurement network's billing interface, though, is external — from a client that does not know the cluster, cannot resolve the service name, and does not see the endpoint list. For this request to reach a pod, the distance between them has to be paved with objects. The object that paves this is the **ingress rule**: a list of matches that ties an external name and path to an internal service object. It is not enough on its own; the rule also needs an external host it listens on and a load balancer standing behind that host. This lesson counts that chain layer by layer and compares two routing formats. The configuration examples are written with our own small schema; what is measured is a model, no real cluster runs. **CO19 — the regional measurement network's externally exposed interfaces number up to twelve**, all of them fictional. **CO20 — an incoming external request passes through four layers:** external load balancer, ingress rule, service object, pod. **CO21 — each layer adds one round**; the balancer and the ingress rule also scan their own lists in order. **CO22 — in path-based routing all services sit behind a single external host**, in name-based routing each service has its own external host. **CO23 — each external host needs one identity item and one balancer entry.** **CO24 — the rule list is scanned in order and the first match wins**; the longest-prefix rule is not applied automatically. #### Four Layers and the Externally Exposed Surface The model below first counts a single request's layers, then how much surface each of two routing formats exposes to the outside at three different service counts. ```js // external-access.mjs — layers of an incoming external request and the exposed surface (MODEL). // The ingress rule's schema is defined here: each rule ties one (host, path) pair to a service. export const SCHEMA = { host: 'string', path: 'string', service: 'string', port: 'number' }; export const SERVICES = ['billing', 'work-order', 'measurement-report', 'meter-record', 'subscriber-portal', 'outage-notice', 'tariff-lookup', 'balance-lookup', 'collections', 'fault-report', 'contract', 'consumption-history']; // CO19: externally exposed fictional interfaces export function rules(n, format) { return SERVICES.slice(0, n).map((s) => (format === 'name' ? { host: `${s}-external`, path: '/', service: s, port: 8080 } : { host: 'measurement-external', path: `/${s}`, service: s, port: 8080 })); } // Externally exposed surface: how many hosts, how many paths, one identity item per host (CO22, CO23). export function surface(n, format) { const r = rules(n, format); const hosts = new Set(r.map((x) => x.host)).size; const paths = new Set(r.map((x) => `${x.host}${x.path}`)).size; return { hosts, rules: r.length, paths, identities: hosts, total: hosts + paths + hosts }; } // CO20-CO21: four layers, each one round; the number of rules scanned depends on the layer. export function layers(n, format) { const y = surface(n, format); return [ ['external balancer', 1, (y.hosts + 1) / 2], ['ingress rule', 1, format === 'name' ? 1 : (y.rules + 1) / 2], ['service object', 1, 1], ['pod', 1, 0], ]; } if (import.meta.url === `file://${process.argv[1]}`) { const s = (x, n) => String(x).padStart(n); const N = 6; console.log(`${N} external services, rule schema: ${Object.keys(SCHEMA).join(', ')}`); console.log(`${'layer'.padEnd(19)}${s('round', 5)}${s('path-based scanned', 20)}` + `${s('name-based scanned', 20)}`); const pathL = layers(N, 'path'), nameL = layers(N, 'name'); const sum = (k, i) => k.reduce((a, x) => a + x[i], 0); for (let i = 0; i < pathL.length; i++) { console.log(`${pathL[i][0].padEnd(19)}${s(pathL[i][1], 5)}${s(pathL[i][2], 20)}` + `${s(nameL[i][2], 20)}`); } console.log(`${'total'.padEnd(19)}${s(sum(pathL, 1), 5)}${s(sum(pathL, 2), 20)}` + `${s(sum(nameL, 2), 20)}`); console.log(`\n${'services'.padEnd(9)}${'format'.padEnd(12)}${s('hosts', 8)}${s('rules', 7)}` + `${s('exposed paths', 15)}${s('identity items', 16)}${s('surface items', 15)}`); for (const n of [3, 6, 12]) { for (const f of ['path', 'name']) { const y = surface(n, f); console.log(`${String(n).padEnd(9)}${(f === 'path' ? 'path-based' : 'name-based').padEnd(12)}` + `${s(y.hosts, 8)}${s(y.rules, 7)}${s(y.paths, 15)}${s(y.identities, 16)}${s(y.total, 15)}`); } } } ``` ``` 6 external services, rule schema: host, path, service, port layer round path-based scanned name-based scanned external balancer 1 1 3.5 ingress rule 1 3.5 1 service object 1 1 1 pod 1 0 0 total 4 5.5 5.5 services format hosts rules exposed paths identity items surface items 3 path-based 1 3 3 1 5 3 name-based 3 3 3 3 9 6 path-based 1 6 6 1 8 6 name-based 6 6 6 6 18 12 path-based 1 12 12 1 14 12 name-based 12 12 12 12 36 ``` The first table says two things. The first is the layer count: an incoming external request passes four stops before reaching a pod, and all four cost one round. These four rounds are a fixed price the client pays; compared against the round count an in-cluster call pays, the difference is the price of being outside the cluster. How much these four rounds amount to can only be understood through comparison. The in-cluster call from the previous lesson cost two rounds: one name-resolution round and the request itself. An incoming external request pays four rounds, that is, twice as much; the two rounds in between belong to the balancer and the ingress rule. If these two layers were removed and pods were exposed directly, the gap would close — but then the exposed surface would be as large as the pod count, and pod addresses would change as pods churned. The two layers are the price of carrying the stability the previous two lessons built out to the outside. The second is more interesting: **the total rules scanned is the same in both formats** — five and a half. In path-based routing the balancer holds one entry and the ingress rule scans six rules; in name-based routing the balancer scans six entries and the ingress rule matches on one rule. The difference is not a saving, it is a **relocation**: the same work moves from one layer to the other. This shows why decisions like "name-based is faster," made without measurement, have no ground to stand on. The real difference is in the second table, and it is not rounds but **surface**. At three services the gap between the two formats is five to nine; at six services, eight to eighteen; at twelve services, fourteen to thirty-six. In path-based format the external host count is independent of the service count, always one; in name-based format it equals the service count. Because identity items are tied to hosts, they grow at the same rate. When twelve services are exposed, path-based format requires managing fourteen items, name-based thirty-six — about 2.6 times as many. That rule count is equal in both formats is this table's second result. **Rule count equals service count and is independent of routing format.** What changes is how many separate external hosts the rules are spread across. This is why "how many rules will be written" is not a criterion for choosing a format; "how many hosts will have to be maintained" is the criterion. #### The Rule's Order Knowing the rule count does not show that routing will be correct. **CO24** says the rule list is scanned in order and the first match wins; once nested paths combine with this rule, order stops being a detail. ```js // routing.mjs — the effect of rule order on the routing decision (MODEL). import { SCHEMA } from './external-access.mjs'; // Same schema, with nested paths: a subpath should go to its own service. const RULES = [ { host: 'measurement-external', path: '/billing', service: 'billing', port: 8080 }, { host: 'measurement-external', path: '/billing/statement', service: 'billing-statement', port: 8080 }, { host: 'measurement-external', path: '/work-order', service: 'work-order', port: 8080 }, { host: 'measurement-external', path: '/work-order/urgent', service: 'urgent-work-order', port: 8080 }, { host: 'measurement-external', path: '/report', service: 'measurement-report', port: 8080 }, { host: 'measurement-external', path: '/report/monthly', service: 'monthly-report', port: 8080 }, ]; // Expected result: the longest prefix wins. This expectation is not written in the rule. const REQUESTS = [ ['/billing', 'billing'], ['/billing/statement', 'billing-statement'], ['/billing/statement/2026', 'billing-statement'], ['/work-order', 'work-order'], ['/work-order/urgent', 'urgent-work-order'], ['/work-order/urgent/12', 'urgent-work-order'], ['/report', 'measurement-report'], ['/report/monthly', 'monthly-report'], ['/report/monthly/03', 'monthly-report'], ['/report/daily', 'measurement-report'], ['/subscriber', null], ['/', null], ]; // CO24: the rule list is scanned in order, the first match wins. function match(rules, path) { for (let i = 0; i < rules.length; i++) { const k = rules[i]; if (path === k.path || path.startsWith(`${k.path}/`)) return { service: k.service, scanned: i + 1 }; } return { service: null, scanned: rules.length }; } const ORDERS = { 'declaration order': RULES, 'specificity order': [...RULES].sort((a, b) => b.path.length - a.path.length), }; const s = (x, n) => String(x).padStart(n); console.log(`${RULES.length} rules, ${REQUESTS.length} requests, schema: ${Object.keys(SCHEMA).join(', ')}`); console.log(`${'rule order'.padEnd(18)}${s('correct', 7)}${s('wrong service', 15)}` + `${s('rule not found', 18)}${s('avg scanned', 18)}`); for (const [label, rules] of Object.entries(ORDERS)) { let correct = 0, wrong = 0, notFound = 0, scanned = 0; for (const [path, expected] of REQUESTS) { const r = match(rules, path); scanned += r.scanned; if (r.service === null) notFound++; if (r.service === expected) correct++; else if (r.service !== null) wrong++; } console.log(`${label.padEnd(18)}${s(correct, 7)}${s(wrong, 15)}${s(notFound, 18)}` + `${s((scanned / REQUESTS.length).toFixed(2), 18)}`); } console.log(`\nrequests that went wrong in declaration order:`); for (const [path, expected] of REQUESTS) { const r = match(ORDERS['declaration order'], path); if (r.service !== expected && r.service !== null) { console.log(` ${path.padEnd(20)} -> ${String(r.service).padEnd(18)} (expected ${expected},` + ` matched at rule ${r.scanned})`); } } ``` ``` 6 rules, 12 requests, schema: host, path, service, port rule order correct wrong service rule not found avg scanned declaration order 6 6 2 3.67 specificity order 12 0 2 3.75 requests that went wrong in declaration order: /billing/statement -> billing (expected billing-statement, matched at rule 1) /billing/statement/2026 -> billing (expected billing-statement, matched at rule 1) /work-order/urgent -> work-order (expected urgent-work-order, matched at rule 3) /work-order/urgent/12 -> work-order (expected urgent-work-order, matched at rule 3) /report/monthly -> measurement-report (expected monthly-report, matched at rule 5) /report/monthly/03 -> measurement-report (expected monthly-report, matched at rule 5) ``` The same six rules, the same twelve requests, two different orders: ten correct or four correct. Reading the list as written in declaration order, nothing looks wrong — every service has its own line, and every line is correct. What is wrong is **the relationship between the lines**, and this relationship is not written down anywhere in the file. That a subpath should go to its own service is an expectation; the rule list does not know this expectation, only the order. The last column of the listing below shows how the error occurs: all six requests match the **more general rule**, and the match happens at rule one, three, and five. The more specific rule sits right underneath in the list and is never read. This is why the average rules scanned is **lower** in the wrong order — three point sixty-seven against three point seventy-five. The wrong routing is achieved by doing less work; the fast answer is not the correct answer. This run also says, in hindsight, what the name-based format buys. When each service has its own external host, that host's rule list has a single line; there is no such thing as one rule containing another, and the six misroutings here **cannot happen**. This is what the thirty-six surface items buy: the routing decision stops depending on rule order. The choice between the two formats is made between the number of objects to manage and the existence of a whole class of wrong decisions. In both rows the count of unmatched requests is two, and these two requests are not an error. There is no rule in the exposed surface for `/subscriber` and `/`, so the ingress rule cannot make a decision and rejects the request. This is what it means for the exposed surface to be countable: a path that is not on the surface cannot enter the cluster by any route. #### The Devolved Decision and What It Does Not Know The devolved decision in this lesson is **which internal service an incoming external request goes to.** This decision could also have been made at the client — a separate address for every interface, in the client's configuration. The ingress rule carries that decision into the cluster, and in exchange asks for four information items: external name, path, target service name, target port. All four items are declarations; they need no measurement and do not age. The cost is not in the declaration itself but in its **count** and its **order**. The second table showed how the count grows: twelve services amount to thirty-six surface items in name-based format, and twelve of them are identity items; each is an object that expires and has to be renewed. In path-based format the same work is done with fourteen items, and there is a single identity item. Here the price of devolving is not a delay, it is **the number of objects to manage.** For a known set of requests, three results come out as follows. Ten of twelve requests match a rule, two do not. In specificity order there are ten correct decisions, zero wrong decisions, and two **failures to decide**; in declaration order there are four correct, six wrong, and again two failures to decide. That the failure-to-decide count stays the same across both orders is meaningful: those two requests are rejected because they have no counterpart on the surface, and this is a result independent of order. The only thing that changes is how many of the ten matching requests go to the right place. What the system does not know was seen in the second run: **which rule contains which.** The ingress rule does not infer the containment relationship between paths, it reads the list in order. This missing piece of information sent six requests to the wrong service, and none of them produced an error record — the requests were answered, just from the wrong place. The correct order works once it is written, but when a rule is added later, the same information is needed again. Devolving a decision forces the relationship it depends on to be **re-declared with every change.** A declaration is not text written once and forgotten, it is an ordering that has to be revisited every time the rule set grows; a relationship visible to the eye at six rules becomes invisible at twelve. #### Summary - An incoming external request passes through four layers — external balancer, ingress rule, service object, pod — and each layer adds one round; the total is four rounds. - Path-based and name-based routing scan the same total number of rules (five and a half); the difference is not a saving, it is work moving from one layer to the other. - Rule count equals service count and is independent of format; what changes is how many separate external hosts the rules spread across. - The exposed surface is five to nine at three services, eight to eighteen at six, fourteen to thirty-six at twelve; in name-based format the external host count and identity item count grow together with the service count. - The same six rules sent six of twelve requests to the wrong service in declaration order; in specificity order all of them went to the right place. The wrong matches occurred by scanning fewer rules, that is, faster. - The ingress rule asks for four declaration items and none of them age; the one thing it does not know is the containment relationship between rules, and this relationship has to be re-declared with every new rule. #### Next Step This lesson brought the request to the pod's door. The process running inside, though, has needs beyond the request that arrives at the door: which data store to connect to, which thresholds to operate with, which key to authenticate with. If these values are written into the image, three environments mean three separate images; if they are kept outside, they mean a new object declared to the cluster. The next lesson measures these objects: when a change made to the same setting set passed in two different forms reaches the running pod, how many keys travel in which form, and how many subjects within the cluster can see values that need to stay secret. ### Configuration and Secret Objects Source: https://academia.sh/en/courses/orchestration/configuration-and-secret-objects Two measures of passing application settings through the cluster: how many key-rounds late a change is reflected and how many rounds of outage it produces when the same fourteen keys are passed as environment variables versus a mounted file, and how many subjects within the cluster can read a secret object. The previous lesson brought the request to the pod's door. The process running inside, though, has needs beyond the request arriving at the door: which record store to connect to, which threshold to validate with, which key to sign invoices with. If these values are written into the image, three environments mean three separate images — and then the image's portability, the property measured in M22/K02, disappears. If they are kept outside, they mean a new object declared to the cluster. There are two of these objects: the **configuration object**, which carries open values, and the **secret object**, which carries values that must stay hidden. Both do the same job; what separates them is an expectation about the visibility of the values they carry, and the second half of this lesson counts how well that expectation holds. M22/K01's **leak-path scan** is not repeated here; the question there was where a value leaks to, the question here is how many subjects in the cluster are authorized to read it. What is measured is a model; the values are fictional. **CO25 — the verifier reads fourteen setting keys**, four of which must be kept secret. **CO26 — a setting passed as an environment variable is read once when the process starts**; even if the object changes, the running process keeps seeing the old value. **CO27 — a setting mounted as a file reaches the running pod three rounds later**, and because the process re-reads the file, no restart is needed. **CO28 — a restart takes three rounds**, and the pod serves nothing during those three rounds. **CO29 — a pod that binds an object, the node agent on that pod's node, and two components of the control plane can read the object**; the last two cannot be closed off by an authorization rule. **CO30 — three changes occur over a twenty-four-round run:** an open setting in round five, a secret setting in round twelve, an open setting again in round nineteen. #### The Same Setting, Two Forms The run below passes the same fourteen keys and the same three changes through four arrangements. The measured quantity is **stale key-rounds**: every key, for every round, where the value the running pod sees differs from the value in the object counts as one. ```js // config.mjs — the same setting set passed in two forms (MODEL). Values are fictional. export const ROUNDS = 24, PROPAGATION = 3, RESTART = 2; // CO27, CO28 export const KEYS = [ ['collector-address', false], ['verification-threshold', false], ['log-level', false], ['retry-count', false], ['timeout-rounds', false], ['batch-size', false], ['region-code', false], ['measurement-unit', false], ['report-interval', false], ['cache-lifetime', false], ['record-store-password', true], ['billing-signing-key', true], ['work-order-token', true], ['fault-notification-key', true], ]; // CO25: 14 keys, 4 secret // Change schedule: which key's value changes in which round. const CHANGE = { 5: 'verification-threshold', 12: 'billing-signing-key', 19: 'timeout-rounds' }; // CO26: an env variable is read once when the process starts; a file reflects PROPAGATION rounds later. export function run(format, restartOnChange) { const object = {}, visible = {}; for (const [a] of KEYS) { object[a] = 'mock-value-0'; visible[a] = 'mock-value-0'; } const pending = []; let stale = 0, restarts = 0, outage = 0, outageEnd = -1; for (let t = 0; t < ROUNDS; t++) { const key = CHANGE[t]; if (key) { object[key] = `mock-value-${t}`; const f = format(key); if (f === 'file') pending.push({ key, at: t + PROPAGATION }); else if (restartOnChange) { // env format: only reflected by a restart restarts++; outageEnd = t + RESTART; for (const [x] of KEYS) pending.push({ key: x, at: t + RESTART }); } } for (let i = pending.length - 1; i >= 0; i--) { // a reflected declaration leaves the queue if (pending[i].at <= t) { visible[pending[i].key] = object[pending[i].key]; pending.splice(i, 1); } } stale += KEYS.filter(([a]) => visible[a] !== object[a]).length; if (t <= outageEnd) outage++; } const remaining = KEYS.filter(([a]) => visible[a] !== object[a]).length; return { stale, restarts, outage, remaining }; } if (import.meta.url === `file://${process.argv[1]}`) { const s = (x, n) => String(x).padStart(n); const secret = KEYS.filter(([, g]) => g).length; console.log(`${KEYS.length} keys (${secret} secret), ${ROUNDS} rounds, ${Object.keys(CHANGE).length}` + ` changes; propagation ${PROPAGATION} rounds, restart ${RESTART} rounds`); const FORMATS = [ ['all env, no restart', () => 'env', false, `${KEYS.length}/0`], ['all env, with restart', () => 'env', true, `${KEYS.length}/0`], ['all file', () => 'file', false, `0/${KEYS.length}`], ['mixed', (a) => (KEYS.find(([x]) => x === a)[1] ? 'file' : 'env'), true, `${KEYS.length - secret}/${secret}`], ]; console.log(`${'format'.padEnd(22)}${s('env/file', 13)}${s('stale key-rounds', 19)}` + `${s('restarts', 18)}${s('outage rounds', 15)}${s('stale at end', 14)}`); for (const [ad, f, yb, split] of FORMATS) { const r = run(f, yb); console.log(`${ad.padEnd(22)}${s(split, 13)}${s(r.stale, 19)}${s(r.restarts, 18)}` + `${s(r.outage, 15)}${s(r.remaining, 14)}`); } } ``` ``` 14 keys (4 secret), 24 rounds, 3 changes; propagation 3 rounds, restart 2 rounds format env/file stale key-rounds restarts outage rounds stale at end all env, no restart 14/0 36 0 0 3 all env, with restart 14/0 6 3 9 0 all file 0/14 9 0 0 0 mixed 10/4 7 2 6 0 ``` The first row is the environment-variable format's silent result. Nothing fails, no outage occurs — and at the end of the run three keys are still at the wrong value. Thirty-six stale key-rounds are a drift nobody notices: the object changed, the declaration was updated, and the pod kept running with the old value. This is the most dangerous case in a declarative system, because the gap between desired state and actual state **is not where the reconciler is looking**; the replica count is correct, the pods are up, only the value inside them has gone stale. The second row closes this gap and presents the bill. If the change triggers a restart, stale key-rounds drop from thirty-six to six, and stale-at-end is zero — but three restarts and **nine rounds of outage** appear. Nine of twenty-four rounds, nearly forty percent, are out of service because of a one-line setting change. The source of this cost is **CO26**: an environment variable is given to the process once, at the start, and there is no way to update it within the process's lifetime. The third row lowers both measures at once: nine stale key-rounds, zero restarts, zero outage. A setting mounted as a file reaches the running pod with a three-round propagation delay, and the process stays up because it re-reads the file. Three changes each produced three rounds of staleness; nine in total. That is three more than the six in the second row — but in exchange, the nine rounds of outage vanish. The fourth row measures a common split: the four secret keys as a file, the ten open keys as environment variables. The result sits between the two — seven stale key-rounds, two restarts, six rounds of outage — and it is **exactly worse than the third row.** The mixed arrangement pays the cost of both formats: outage for the open settings, propagation delay for the secret ones. The reason for this split is not reflection speed, it is the visibility measured in the next section; but looking at this table alone, nobody can **say** the mixed arrangement is inherently better. The lesson common to all four rows concerns **where the drift is measured**. The replica set controller compares the desired count against the actual count and closes the gap; setting values sit outside that comparison. The gap between the value in the object and the value in the running process is nowhere any controller is looking, which is why nobody notices anything across twenty-four rounds in the first row. In a declarative system, every field that is not reconciled is a field that can drift silently. #### The Secret Object's Visibility The reason for keeping secret keys in a separate object type is visibility. This section turns that reason into a number: with the same authorization schema, in the same cluster, how many subjects can read a secret object. ```js // visibility.mjs — how many subjects in the cluster can read a secret object (MODEL). import { KEYS } from './config.mjs'; // Our own small authorization schema: a rule grants a subject a verb over an object type and name pattern. export const AUTH_SCHEMA = { subject: 'string', type: 'string|*', name: 'string|*', verb: 'list' }; const PODS = { // which pod runs on which node 'verifier-pod': 'node-1', 'billing-pod': 'node-2', 'collector-pod': 'node-3', }; const BOUND_BY = { // which pods bind which object 'billing-signing-key': ['verifier-pod', 'billing-pod'], 'verification-threshold': ['verifier-pod', 'billing-pod', 'collector-pod'], }; const CONTROL_PLANE = ['state-store', 'api-server']; // both carry both objects const ROLES = { broad: [ { subject: 'operate-role', type: '*', name: '*', verb: ['read', 'write'] }, { subject: 'observe-role', type: '*', name: '*', verb: ['read'] }, { subject: 'release-subject', type: '*', name: '*', verb: ['read', 'write'] }, { subject: 'backup-subject', type: '*', name: '*', verb: ['read'] }, { subject: 'audit-subject', type: '*', name: '*', verb: ['read'] }, ], least: [ { subject: 'operate-role', type: 'config', name: '*', verb: ['read', 'write'] }, { subject: 'observe-role', type: 'config', name: '*', verb: ['read'] }, { subject: 'release-subject', type: 'secret', name: 'billing-signing-key', verb: ['write'] }, { subject: 'backup-subject', type: 'config', name: '*', verb: ['read'] }, { subject: 'audit-subject', type: 'config', name: '*', verb: ['read'] }, ], }; export function canRead(object, type, policy) { const pods = BOUND_BY[object] ?? []; const agents = [...new Set(pods.map((k) => `agent-${PODS[k]}`))]; const roles = ROLES[policy] .filter((r) => (r.type === '*' || r.type === type) && (r.name === '*' || r.name === object) && r.verb.includes('read')) .map((r) => r.subject); return { pods, roles, agents, plane: CONTROL_PLANE }; } if (import.meta.url === `file://${process.argv[1]}`) { const s = (x, n) => String(x).padStart(n); const secret = KEYS.filter(([, g]) => g).length; console.log(`${secret} of ${KEYS.length} keys are in the secret object, ${KEYS.length - secret}` + ` are in the config object; authorization schema: ${Object.keys(AUTH_SCHEMA).join(', ')}`); console.log(`${'object'.padEnd(24)}${'type'.padEnd(14)}${'policy'.padEnd(9)}${s('pods', 8)}` + `${s('roles', 7)}${s('node agents', 14)}${s('control plane', 16)}${s('total', 8)}`); for (const [object, type] of [['billing-signing-key', 'secret'], ['verification-threshold', 'config']]) { for (const p of ['broad', 'least']) { const r = canRead(object, type, p); const total = r.pods.length + r.roles.length + r.agents.length + r.plane.length; console.log(`${object.padEnd(24)}${type.padEnd(14)}${p.padEnd(9)}${s(r.pods.length, 8)}` + `${s(r.roles.length, 7)}${s(r.agents.length, 14)}${s(r.plane.length, 16)}${s(total, 8)}`); } } const least = canRead('billing-signing-key', 'secret', 'least'); console.log(`\nunder least privilege, those who can read the secret object: ${[...least.pods, ...least.roles, ...least.agents, ...least.plane].join(', ')}`); console.log(`${least.agents.length + least.plane.length} of them cannot be removed by an authorization rule`); } ``` ``` 4 of 14 keys are in the secret object, 10 are in the config object; authorization schema: subject, type, name, verb object type policy pods roles node agents control plane total billing-signing-key secret broad 2 5 2 2 11 billing-signing-key secret least 2 0 2 2 6 verification-threshold config broad 3 5 3 2 13 verification-threshold config least 3 4 3 2 12 under least privilege, those who can read the secret object: verifier-pod, billing-pod, agent-node-1, agent-node-2, state-store, api-server 4 of them cannot be removed by an authorization rule ``` The first and third rows should be read together. Under the broad authorization policy, eleven subjects can read the secret object and thirteen can read the open configuration object. The gap of two comes from the secret being bound by one fewer pod — the role count is five in both. **No role narrowed because the object's type is secret.** This is this section's first result: putting a value into a secret object does not, by itself, change who can read it; what changes it is the authorization rule, and that rule has to be written separately. The second row shows what happens once the rule is written. Under the least-privilege policy, the role count that can read the secret object drops from five to zero; the total falls from eleven to six. Under the same policy the configuration object stays at twelve, because reading the open settings is the job of four roles. This is where the real difference between the two object types is born — not in the type, but in the rule written for that type. The last two rows say where the narrowing stops. Even under least privilege there are six readers, and four of them **cannot be removed by an authorization rule**: the node agents on the two nodes where the pods binding the secret run, and two components of the control plane. The node agent has to see the value because it sets up the pod's filesystem; the state store and the API server see it because they store and distribute the object. The moment a secret object is written to the cluster, these four readers are born, and no role definition reduces them. The count does not drop to zero, it **drops to four.** #### The Devolved Decision and What It Does Not Know The devolved decision here is **how the setting reaches the pod.** This decision used to belong to whoever produced the image: the value was written into the image, and a separate image was built for each environment. Once the decision moves to the cluster, three information items the system needs to know are born: which key travels in which form, how quickly a change needs to be reflected, and which key counts as secret. All three are declarations and need no measurement; their cost is not in writing them, it is that **nothing warns you when they are written wrong.** What the system does not know is **when** the process reads the setting. The model turned this into an assumption with **CO26** and **CO27**; in reality the only place that knows this is the process's own code, and the cluster cannot look inside it. The cost of this ignorance sits in the first table's first row: thirty-six stale key-rounds and three wrong values at the end, without producing a single error record. The restart decision is born from the same ignorance too — the nine rounds of outage that would not be needed if the process re-read the file are paid because whether it does is unknown. The second table shows a case where devolving **does not pay off**. Moving the secret value into a separate object type brought visibility down from thirteen to eleven; the real reduction was done by the authorization rule, and that rule was written by a human. The system does not know which role genuinely needs which secret; knowing this would require observing every subject's access to every object and dropping what goes unused. Because it is not known, the default stays broad, and five roles carry read authorization they do not need. #### Summary - The same fourteen keys were passed in four arrangements. In the environment-variable format, if no restart happens, thirty-six stale key-rounds appear and three keys are still at the wrong value at the end of the run; no error record is produced. - Restarting on change brings stale key-rounds down to six but produces three restarts and outage in nine of twenty-four rounds. - Mounting as a file gives nine stale key-rounds, zero restarts, and zero outage; the propagation delay is three rounds per change. - The mixed arrangement, secret as file and open as environment variable, produces seven stale key-rounds and six rounds of outage — it pays the cost of both formats and is worse than the file format alone. - Putting a value into a secret object does not narrow visibility by itself: under the broad policy five roles can read both object types; what narrows it is the authorization rule written separately. - Under the least-privilege policy, the secret object's readers drop from eleven to six, but not to zero: the node agents on two nodes and two components of the control plane cannot be removed by an authorization rule. #### Next Step Configuration and secret objects reached the pod through a filesystem area, and that area was born with the pod and died with the pod. This is the correct behavior for settings: the value stays in the object, and nothing is lost when the pod goes away. For data the application **produces**, though, the same behavior means data loss. When the record store's pod restarts, the meter readings must not go with it; yet everything measured in this topic so far — the pods the replica set deletes, the new version installed in a rollout, the endpoints that come and go with churn — assumed that a pod's lifetime is short. The next lesson takes up the place where that assumption breaks: how an area of data that outlives the pod is declared to the cluster, and the new information items that declaration carries. ### Storage and Volumes Source: https://academia.sh/en/courses/orchestration/storage-and-volumes Matching a persistent volume claim against cluster supply: the volume pool the cluster offers against the size, access mode, and performance class the application declares, unmet requests counted by reason, the number of nodes the access mode eliminates from placement, and the wrong binding that appears when the inventory declaration goes stale. Settings moved outside the pod: configuration and secrets stand as separate objects, and the pod reads them at run time. The same cannot be said for data. Every byte the pod writes is still tied to the pod's lifetime; when the pod is re-created it starts with a fresh writable layer, and yesterday's working state is not there. On a single machine, the solution to this problem was already measured — moving data out of the writable layer, into a named volume or a directory referenced by an absolute path — and what was counted there was the bytes remaining after the container was deleted. In the cluster, that number is taken as solved; the question asked is a different one. If a pod is to find the same bytes no matter which node it opens on, the storage holding those bytes has to be **reachable from that node**. This is a decision the application cannot make on its own: the application knows what it wants, not what the cluster offers. The devolution here happens through a declaration split in two. The application writes a **persistent volume claim**; the cluster declares a **volume pool**; what matches the two is a **storage driver interface**. This lesson counts three things: how many requests go unmet and why, how many options the access mode eliminates from placement, and how many bindings turn out wrong once the inventory declaration goes stale. #### Request, Supply, and Matching Request and supply have to use the same vocabulary. The application declares three items: how much space, which access mode, which performance class. The cluster declares four items per volume: size, class, the modes it supports, and topology. The matching rule is nothing more than three comparisons; what is hard is not the comparison, it is both sides keeping their declarations current. - **CO31** — The cluster, volume pool, node topology, and inventory declaration are a model built in `node`; no real cluster runs. The numbers below are the model's numbers. - **CO32** — The measurement network is fictional. Ten persistent volume claims come from the network's services and its nightly batch job; the names are drawn from that fiction. - **CO33** — The performance classes are `fast`, `standard`, `archive`, and `ultra-fast`; the access modes are single-writer, multi-reader, multi-writer. - **CO34** — The cluster is six nodes and two regions. A volume is either bound to a single node or open to an entire region; there is no third topology. - **CO35** — In the staleness sweep the pool has 24 volumes, the run is 10 rounds, and four requests arrive per round. Each round two volumes are bound by another claimant outside the cluster, and one volume is freed. - **CO36** — Randomness is produced by a linear congruential generator, seed `20260801`. Three inventory ages run over the same pool and the same request sequence. ```js // measure/claim-supply.mjs — matching a persistent volume claim against cluster supply and the placement constraint. // MODEL: the cluster, volume pool, node topology, and inventory declaration are a model built in // node; no real cluster runs. The numbers below are the model's numbers. const NODES = [["n1", "region-a"], ["n2", "region-a"], ["n3", "region-a"], ["n4", "region-b"], ["n5", "region-b"], ["n6", "region-b"]]; // request: name, size(Gi), access mode, perf class -> the 3 items the application declares const REQUESTS = [ ["collector-state", 20, "single-writer", "fast"], ["verifier-cache", 5, "single-writer", "standard"], ["billing-data", 200, "single-writer", "fast"], ["work-order-data", 50, "single-writer", "standard"], ["tariff-table", 2, "multi-reader", "standard"], ["nightly-output", 100, "multi-writer", "standard"], ["archive-2025", 500, "multi-reader", "archive"], ["raw-measurement", 300, "single-writer", "fast"], ["report-share", 40, "multi-writer", "fast"], ["closing-ledger", 30, "single-writer", "ultra-fast"]]; // supply: name, size(Gi), class, supported modes, region, bound node (null if region-wide) const SUPPLY = [ ["v01", 30, "fast", ["single-writer"], "region-a", "n1"], ["v02", 8, "standard", ["single-writer"], "region-a", "n2"], ["v03", 250, "fast", ["single-writer"], "region-b", "n4"], ["v04", 60, "standard", ["single-writer"], "region-b", "n5"], ["v05", 4, "standard", ["single-writer", "multi-reader"], "region-a", null], ["v06", 120, "standard", ["single-writer"], "region-a", null], ["v07", 400, "archive", ["single-writer", "multi-reader"], "region-b", null], ["v08", 20, "fast", ["single-writer"], "region-a", "n3"], ["v09", 512, "standard", ["single-writer", "multi-reader"], "region-b", null], ["v10", 50, "fast", ["single-writer", "multi-reader", "multi-writer"], "region-a", null]]; const fits = (t, b) => b[1] >= t[1] && b[2] === t[3] && b[3].includes(t[2]); function match(rule) { // rule: first-fit | best-fit const free = new Set(SUPPLY.map((b) => b[0])), result = []; for (const t of REQUESTS) { const candidates = SUPPLY.filter((b) => free.has(b[0]) && fits(t, b)); if (candidates.length) { const s = rule === "first-fit" ? candidates[0] : candidates.reduce((a, b) => (b[1] < a[1] ? b : a)); free.delete(s[0]); result.push([t, s, null]); continue; } const hasClass = SUPPLY.some((b) => free.has(b[0]) && b[2] === t[3]); const hasMode = SUPPLY.some((b) => free.has(b[0]) && b[2] === t[3] && b[3].includes(t[2])); result.push([t, null, !hasClass ? "no class" : !hasMode ? "no mode" : "size too small"]); } return result; } const A = match("best-fit"), reason = {}; console.log(`${"request".padEnd(22)}${"Gi".padStart(5)}${"mode".padStart(15)}${"class".padStart(12)}${"volume".padStart(8)}${"result".padStart(16)}`); for (const [t, b, n] of A) { if (n) reason[n] = (reason[n] || 0) + 1; console.log(`${t[0].padEnd(22)}${String(t[1]).padStart(5)}${t[2].padStart(15)}${t[3].padStart(12)}` + `${(b ? b[0] : "-").padStart(8)}${(b ? "bound" : n).padStart(16)}`); } const bound = A.filter(([, b]) => b); console.log(`${REQUESTS.length} requests, ${bound.length} bound, ${REQUESTS.length - bound.length} unmet: ` + Object.entries(reason).map(([k, v]) => `${k} ${v}`).join(", ")); for (const k of ["first-fit", "best-fit"]) { const r = match(k).filter(([, b]) => b); console.log(`rule ${k.padEnd(11)} bound ${r.length} overallocated ${r.reduce((a, [t, b]) => a + b[1] - t[1], 0)} Gi`); } const eligible = (b) => NODES.filter(([d, bo]) => (b[5] ? d === b[5] : bo === b[4])).map(([d]) => d); console.log(`\n${"request".padEnd(22)}${"mode".padStart(15)}${"volume topology".padStart(20)}${"eligible nodes".padStart(15)}${"excluded".padStart(9)}`); let eligibleTotal = 0; for (const [t, b] of bound) { const u = eligible(b); eligibleTotal += u.length; console.log(`${t[0].padEnd(22)}${t[2].padStart(15)}${(b[5] ? `node ${b[5]}` : `region ${b[4]}`).padStart(20)}` + `${u.join(",").padStart(15)}${String(NODES.length - u.length).padStart(9)}`); } console.log(`${bound.length * NODES.length} placement options with no constraint, ${eligibleTotal} remained; excluded ${bound.length * NODES.length - eligibleTotal}`); for (const k of ["single-writer", "multi-reader", "multi-writer"]) { const g = bound.filter(([t]) => t[2] === k); if (!g.length) continue; console.log(`mode ${k.padEnd(14)} requests ${g.length} avg eligible nodes per request ` + `${(g.reduce((a, [, b]) => a + eligible(b).length, 0) / g.length).toFixed(2)}`); } // --- inventory staleness: the decision is made with a stale declaration --- const SEED = 20260801; let s = SEED % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const POOL = 24, ROUNDS = 10, ROUND_REQUESTS = 4, EXTERNAL = 2; const pool = Array.from({ length: POOL }, (_, i) => [`h${String(i + 1).padStart(2, "0")}`, [10, 40, 120, 400][Math.floor(random() * 4)], ["fast", "standard", "archive"][Math.floor(random() * 3)]]); const incoming = []; for (let t = 0; t < ROUNDS; t += 1) for (let i = 0; i < ROUND_REQUESTS; i += 1) incoming.push([t, [10, 40, 120][Math.floor(random() * 3)], ["fast", "standard", "archive"][Math.floor(random() * 3)]]); function run(age) { s = SEED % 2147483647; for (let i = 0; i < 3 * POOL + 2 * incoming.length; i += 1) random(); const occupied = new Set(), history = [], queue = []; let correct = 0, wrong = 0, undecided = 0, waitTotal = 0; for (let t = 0; t < ROUNDS; t += 1) { history.push(new Set(occupied)); const inventory = age === 0 ? occupied : history[Math.max(0, t - age)]; queue.push(...incoming.filter((g) => g[0] === t)); for (const request of queue.splice(0, queue.length)) { const candidate = pool.find((b) => !inventory.has(b[0]) && b[1] >= request[1] && b[2] === request[2]); if (!candidate) { undecided += 1; queue.push(request); continue; } if (occupied.has(candidate[0])) { wrong += 1; queue.push(request); continue; } occupied.add(candidate[0]); correct += 1; waitTotal += t - request[0]; } for (let i = 0; i < EXTERNAL; i += 1) { // another claimant binds a volume from the pool const free = pool.filter((b) => !occupied.has(b[0])); if (free.length) occupied.add(free[Math.floor(random() * free.length)][0]); } const d = [...occupied]; if (d.length) occupied.delete(d[Math.floor(random() * d.length)]); } return { correct, wrong, undecided, waitTotal, pending: queue.length }; } console.log(`\n${"inventory age"}${"correct".padStart(8)}${"wrong bind".padStart(16)}${"undecided".padStart(16)}` + `${"pending".padStart(10)}${"avg wait rounds".padStart(19)}${"declarations per round".padStart(23)}`); for (const age of [0, 1, 3]) { const r = run(age); console.log(`${String(age).padEnd(13)}${String(r.correct).padStart(8)}${String(r.wrong).padStart(16)}` + `${String(r.undecided).padStart(16)}${String(r.pending).padStart(10)}${(r.waitTotal / Math.max(1, r.correct)).toFixed(2).padStart(19)}` + `${String(Math.round(POOL * 3 / (age + 1))).padStart(23)}`); } console.log(`information items: ${REQUESTS.length} requests x 3 + ${SUPPLY.length} volumes x 4 + ${NODES.length} nodes x 1 = ` + `${REQUESTS.length * 3 + SUPPLY.length * 4 + NODES.length} items (seed ${SEED})`); ``` ``` request Gi mode class volume result collector-state 20 single-writer fast v08 bound verifier-cache 5 single-writer standard v02 bound billing-data 200 single-writer fast v03 bound work-order-data 50 single-writer standard v04 bound tariff-table 2 multi-reader standard v05 bound nightly-output 100 multi-writer standard - no mode archive-2025 500 multi-reader archive - size too small raw-measurement 300 single-writer fast - size too small report-share 40 multi-writer fast v10 bound closing-ledger 30 single-writer ultra-fast - no class 10 requests, 6 bound, 4 unmet: no mode 1, size too small 2, no class 1 rule first-fit bound 6 overallocated 85 Gi rule best-fit bound 6 overallocated 75 Gi request mode volume topology eligible nodes excluded collector-state single-writer node n3 n3 5 verifier-cache single-writer node n2 n2 5 billing-data single-writer node n4 n4 5 work-order-data single-writer node n5 n5 5 tariff-table multi-reader region region-a n1,n2,n3 3 report-share multi-writer region region-a n1,n2,n3 3 36 placement options with no constraint, 10 remained; excluded 26 mode single-writer requests 4 avg eligible nodes per request 1.00 mode multi-reader requests 1 avg eligible nodes per request 3.00 mode multi-writer requests 1 avg eligible nodes per request 3.00 inventory age correct wrong bind undecided pending avg wait rounds declarations per round 0 21 0 80 19 0.33 72 1 15 86 44 25 1.93 36 3 10 134 33 30 3.20 18 information items: 10 requests x 3 + 10 volumes x 4 + 6 nodes x 1 = 76 items (seed 20260801) ``` #### The Unmet Request: Four Rejections, Three Reasons **Six of ten requests were bound, four went unmet**, and the rejection reasons are not all in the same class. One request said **no mode**: a multi-writer volume is requested for the nightly batch job's output, the pool has a volume of that size and class, but none of them supports that mode. Two requests said **size too small**: in the archive class there is only a 400-Gi volume, and 500 Gi was requested; in the fast class the largest volume left is 50 Gi, and 300 Gi was requested. One request said **no class**: not a single volume declares a class called `ultra-fast`. These three reasons do not call for the same work, and separating them is the first payoff of devolving. A size rejection is a capacity decision: a larger volume gets added to the pool. A mode rejection is an **interface** decision; no amount of growing fixes it, because offering multi-writer mode is a property of the storage itself. A class rejection can be a **vocabulary** error: the application has declared a class name the cluster does not recognize. Once the decision is devolved, the rejection reason has to be produced by the machine, or the request just says "not bound." The pair of second-block rows gives the cost of the selection rule. Choosing the first fitting volume versus choosing the smallest one that fits **does not change the count met**, six in both; what changes is waste: eighty-five Gi against seventy-five Gi overallocated. A rule that consumes large volumes early ends up unable to meet large requests that arrive later. The selection rule does not change today's decision, it changes **tomorrow's met rate**, and the system does not know what requests tomorrow will bring. #### Access Mode: Nodes Eliminated From Placement The second table shows that the storage decision is really a **placement** decision. For the six bound requests, there were 36 placement options in the six-node cluster; mode and topology together **eliminated 26 of them**, leaving 10 options. The elimination splits sharply by mode. The four requests in single-writer mode have volumes bound to a specific node; **a single eligible node** is left for each, and five are eliminated. The two requests in multi-reader and multi-writer mode have volumes open to the whole region; three nodes are eligible for them, and three are eliminated. The average eligible node count is 1.00 in single-writer mode and 3.00 in the others. The consequence is that placement's first question is no longer "which node has room." The moment a pod requests a volume in single-writer mode, the node is already chosen; the only question left is whether that node has enough processor and memory, and if not, placement stays **unsolvable** with no other node to fall back to. The devolved decision ties two systems together: the storage match narrows the scheduler's selection space, and the scheduler has to know about that narrowing. If it does not, it puts the pod on an ineligible node, the volume does not bind, and the pod does not open. In the measurement network the distinction turns into a readable rule: the tariff table is declared in multi-reader mode because every verifier replica reads it, and three nodes stay eligible; the work order data is held by a single writer, so single-writer mode nails it to one node. #### The Inventory's Age and the Wrong Binding The third table gives the devolution's real cost. The matching decision rests on a **declared inventory**, and past a certain moment that inventory no longer tells the truth. At zero inventory age — that is, when the pool's actual occupancy is read at the moment of decision — **there is no wrong binding**: 21 requests bound correctly, average wait 0.33 rounds. When the age rises to one round, correct bindings fall to 15 and **86 wrong bindings** appear: the system picks a volume that looks free in the inventory, discovers at binding time that the volume is occupied, and the request returns to the queue and retries next round. At an age of three rounds, correct bindings fall to 10 and wrong bindings rise to 134; average wait climbs from 0.33 to 3.20 rounds, and the number of requests still pending at the end of the run rises from 19 to 30. The undecided column moves in the opposite direction, and this is deceptive at first glance: it falls from 80 to 33. The number is not improving — because the pool looks **emptier than it actually is** in the stale inventory, the system stops saying "no suitable volume" and starts picking the wrong volume instead. In other words, delayed information turns one class of decision into another: it produces a wrong "found it" in place of an honest "I do not know." The second is more expensive, because a binding attempt is made, it fails, and a round is spent. The last column is the other end of the cost. Keeping the age at zero requires 72 declaration items per round — re-reading three items for each of the pool's 24 volumes every round. At an age of three this number falls to 18, declaration load drops to a quarter, and in exchange wrong bindings rise from zero to 134. For this lesson, the devolution's price can be stated in one line: if the **76 information items** the matching decision rests on (10 requests × 3, 10 volumes × 4, 6 nodes × 1) are not produced continuously and on time, the result degrades even though the decision mechanism itself has not changed. What the system does not know is who bound the volume and when; because it does not know, what it does is attempt to bind an occupied volume. #### Summary - Request and supply are separate declarations: the application declares three items, the cluster declares four per volume; the storage driver interface does the matching. - Six of ten requests were met, four were not, and the reasons fall into separate classes: size 2, mode 1, class 1. Size is a capacity decision, mode is an interface decision, class is a vocabulary error. - The selection rule does not change the number of requests met (6 in both), it changes how much space is overallocated: first-fit 85 Gi, best-fit 75 Gi. - Access mode is a placement constraint: 26 of 36 options were eliminated. In single-writer mode an average of 1.00 eligible node per request remains, in the multi-node modes 3.00. - As inventory age rises 0 → 1 → 3 rounds, correct bindings go 21 → 15 → 10, wrong bindings 0 → 86 → 134, average wait 0.33 → 1.93 → 3.20 rounds. Declaration load falls to 72 → 36 → 18 items per round. - Delayed information turns "failure to decide" (80 → 33) into a "wrong decision": cheap not-knowing is replaced by expensive wrong binding. #### Next Step Persistent data found its place in the cluster: a request is declared, supply is matched, and the pod opens on the node where the volume is reachable. Everything measured was between **a single request and a single volume**. Yet the verifier feeding the nightly batch job is not one replica; it runs three, and each replica has its own data. A replica set treats replicas as interchangeable and does not keep their names permanent either — so when a replica restarts, what will tell it which volume to bind to, and when the scale is turned down, whose data is left orphaned? The next lesson compares an object that gives out a stable identity against an unordered replica set in the same sequence of events, and counts: how many replicas returned to the same volume on restart, which replica left on scale-down, what round cost a stable startup pays, and how cluster membership and data ownership break when order is not followed. ### Stateful Applications Source: https://academia.sh/en/courses/orchestration/stateful-applications Measuring stable identity and persistent storage together: a stateful set and an unordered replica set are compared over the same sequence of six events, the persistence of the replica name and volume binding is counted, the round cost of ordered startup and shutdown is surveyed, and how cluster membership and data ownership break when order is not followed is modeled. A persistent volume claim makes a pod's data outlive the pod. That is enough for the bond between a single claim and a single volume. The measurement network's verifier, though, is not one replica: it runs five, each replica has its own 40-Gi share of data, and the replicas cannot serve that data interchangeably. A replica owns a range of meters; that range's data sits in its own volume. A replica set builds the exact opposite assumption. For it, replicas are **interchangeable**; their names are not persistent across a run, because nothing is supposed to be tied to the name. The moment a volume tied to a name appears, this assumption breaks: if a re-created replica's name has changed, no information is left to say which volume it should bind to. This lesson names the devolved decision as follows: **which replica owns which data, and where it binds when re-created**. The system that makes this decision needs to know three items per replica — sequence number, volume binding, readiness. Three things are counted: how many orphaned volumes the two objects leave behind over the same sequence of events, what round cost ordered startup pays, and what breaks when order is not followed. #### The Same Sequence of Events, Two Objects The two objects pass through the same six events: setup, node failure, re-creation, scale-down, full restart, scale-up. The comparison only becomes meaningful at the end of this sequence; on any single event the two look the same. - **CO37** — Replicas, volumes, cluster membership, and shard ownership are a model built in `node`; no real cluster runs. - **CO38** — The measurement network is fictional. The verifier runs five replicas, each has its own 40-Gi volume, and they are placed on three nodes in sequence. - **CO39** — In a stateful set the replica name is derived from the sequence number, and the volume claim is tied to the name; in a replica set the name is regenerated on every creation, and every new name spawns a new volume claim. - **CO40** — On scale-down a stateful set removes the highest sequence number; in a replica set the departing replica is chosen by the generator. The removed replica's volume is not deleted, it is left orphaned. - **CO41** — Readiness times are given by the generator, between 1 and 3 rounds. In an ordered startup each replica waits for the previous one's readiness; shutdown runs in reverse order, one round per replica. - **CO42** — Membership model: sequence zero is the seed; a replica that opens while the seed is not up sets up its own membership, and there is no merging afterward. Shard ownership is distributed within each membership by `shard mod member count`; shard `p`'s data physically sits in the volume of sequence `p mod 5`. Seed `20260807`. ```js // measure/identity.mjs — a stateful set and an unordered replica set in the same sequence of events. // MODEL: replicas, volumes, membership, and shard ownership are a model built in node; no real // cluster runs. The numbers below are the model's numbers. const N = 5, NODES = ["n1", "n2", "n3"], SHARDS = 12, SEED = 20260807; let s = SEED % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const suffix = () => Array.from({ length: 4 }, () => "abcdefghijklmnopqrstuvwxyz"[Math.floor(random() * 26)]).join(""); function run(kind) { const volumes = new Map(); // volume name -> owner replica name (null if empty) let replicas = [], log = []; const create = (seq) => { const name = kind === "ordered" ? `rep-${seq}` : `rep-${suffix()}`; const v = `data-${name}`, existed = volumes.has(v); volumes.set(v, name); return { seq, name, vol: v, existed, node: NODES[seq % NODES.length] }; }; const remove = (k) => { volumes.set(k.vol, null); replicas = replicas.filter((x) => x !== k); }; const event = (label, fresh) => { const orphaned = [...volumes.values()].filter((v) => v === null).length; log.push([label, fresh.length, fresh.filter((k) => k.existed).length, replicas.length, orphaned]); }; let y = Array.from({ length: N }, (_, i) => create(i)); replicas = [...y]; event("initial setup", y); for (const k of replicas.filter((k) => k.node === "n2")) remove(k); event("node n2 failure", []); const missing = [0, 1, 2, 3, 4].filter((i) => !replicas.some((k) => k.seq === i)); y = missing.map((i) => create(i)); replicas.push(...y); event("re-creation", y); for (let i = 0; i < 2; i += 1) { // scale 5 -> 3 const candidate = kind === "ordered" ? replicas.reduce((a, b) => (b.seq > a.seq ? b : a)) : replicas[Math.floor(random() * replicas.length)]; remove(candidate); } const remainingSeq = replicas.map((k) => k.seq).sort(); event(`scale 5->3 (leaving ${[0, 1, 2, 3, 4].filter((i) => !remainingSeq.includes(i)).join(",")})`, []); const old = [...replicas]; for (const k of old) remove(k); y = old.map((k) => create(k.seq)); replicas = [...y]; event("full restart", y); const empty = [0, 1, 2, 3, 4].filter((i) => !replicas.some((k) => k.seq === i)).slice(0, 2); y = empty.map((i) => create(i)); replicas.push(...y); event("scale 3->5", y); return { log, orphaned: [...volumes.values()].filter((v) => v === null).length, volumes: volumes.size }; } console.log(`${"event".padEnd(26)}${"object".padStart(13)}${"created".padStart(11)}${"returned to old volume".padStart(24)}${"open replicas".padStart(15)}${"orphaned volumes".padStart(18)}`); for (const kind of ["ordered", "unordered"]) { const r = run(kind); for (const [e, created, returned, open, orphaned] of r.log) console.log(`${e.padEnd(26)}${(kind === "ordered" ? "stateful set" : "replica set").padStart(13)}` + `${String(created).padStart(11)}${String(returned).padStart(24)}${String(open).padStart(15)}${String(orphaned).padStart(18)}`); console.log(` -> ${kind}: ${r.volumes} volumes created in total, ${r.orphaned} orphaned (${r.orphaned * 40} Gi unclaimed)`); } // --- round cost of ordered startup and shutdown --- console.log(`\n${"replicas".padEnd(9)}${"readiness rounds".padStart(20)}${"ordered startup".padStart(17)}${"parallel".padStart(10)}${"ordered shutdown".padStart(18)}${"parallel".padStart(10)}${"difference".padStart(12)}`); s = SEED % 2147483647; for (const n of [3, 5, 9]) { const ready = Array.from({ length: n }, () => 1 + Math.floor(random() * 3)); const seqSum = ready.reduce((a, b) => a + b, 0), parallelMax = Math.max(...ready); console.log(`${String(n).padEnd(9)}${ready.join(",").padStart(20)}${String(seqSum).padStart(17)}${String(parallelMax).padStart(10)}` + `${String(n).padStart(18)}${String(1).padStart(10)}${String(seqSum + n - parallelMax - 1).padStart(12)}`); } // --- when order is not followed: cluster membership and data ownership --- function membership(order) { const groups = []; for (const i of order) { const seeded = groups.find((g) => g.includes(0)); if (i === 0 || !seeded) groups.push([i]); else seeded.push(i); } const claims = Array.from({ length: SHARDS }, () => []); for (const g of groups) for (let p = 0; p < SHARDS; p += 1) claims[p].push(g[p % g.length]); return { membership: groups.length, dual: claims.filter((a) => a.length > 1).length, wrong: claims.reduce((a, l, p) => a + l.filter((k) => k !== p % N).length, 0), noData: claims.filter((a, p) => !a.includes(p % N)).length, }; } console.log(`\n${"startup order".padEnd(18)}${"membership".padStart(11)}${"dual-claimed".padStart(15)}${"wrong claims".padStart(14)}${"served without data".padStart(21)}`); const orders = [[0, 1, 2, 3, 4]]; for (let i = 0; i < 3; i += 1) { const p = [0, 1, 2, 3, 4]; for (let j = p.length - 1; j > 0; j -= 1) { const k = Math.floor(random() * (j + 1)); [p[j], p[k]] = [p[k], p[j]]; } orders.push(p); } for (const order of orders) { const r = membership(order); console.log(`${order.join(",").padEnd(18)}${String(r.membership).padStart(11)}${String(r.dual).padStart(15)}` + `${String(r.wrong).padStart(14)}${String(r.noData).padStart(21)}`); } console.log(`${SHARDS} shards, ${N} replicas, seed ${SEED}; information items: 3 per replica (sequence, volume binding, readiness) = ${N * 3}`); ``` ``` event object created returned to old volume open replicas orphaned volumes initial setup stateful set 5 0 5 0 node n2 failure stateful set 0 0 3 2 re-creation stateful set 2 2 5 0 scale 5->3 (leaving 3,4) stateful set 0 0 3 2 full restart stateful set 3 3 3 2 scale 3->5 stateful set 2 2 5 0 -> ordered: 5 volumes created in total, 0 orphaned (0 Gi unclaimed) initial setup replica set 5 0 5 0 node n2 failure replica set 0 0 3 2 re-creation replica set 2 0 5 2 scale 5->3 (leaving 0,1) replica set 0 0 3 4 full restart replica set 3 0 3 7 scale 3->5 replica set 2 0 5 7 -> unordered: 12 volumes created in total, 7 orphaned (280 Gi unclaimed) replicas readiness rounds ordered startup parallel ordered shutdown parallel difference 3 2,1,1 4 2 3 1 4 5 1,2,3,3,1 10 3 5 1 11 9 1,1,1,1,2,3,1,1,1 12 3 9 1 17 startup order membership dual-claimed wrong claims served without data 0,1,2,3,4 1 0 0 0 2,1,0,3,4 3 12 30 6 1,4,3,0,2 4 12 38 2 0,1,2,4,3 1 0 4 4 12 shards, 5 replicas, seed 20260807; information items: 3 per replica (sequence, volume binding, readiness) = 15 ``` #### Is the Name Persistent, Does the Volume Come Back The first two rows of the first table are the same for both objects: five replicas were set up, `n2` went down, two replicas were lost, two volumes were left orphaned. The split starts on the **re-creation** row. Because the stateful set sees two missing sequence numbers, it regenerates the same two names; since the name is the same, the volume claim is the same too, and **both replicas return to their old volume**, orphan count drops to zero. The replica set generates two new names; the new name spawns a new claim, the count of replicas returning to their old volume is **zero**, and the two orphaned volumes stay right where they are. The identity of the departing replicas on scale-down is also different. In the stateful set the departing replicas are `3` and `4` — the two highest sequences, and this rule does not change from run to run. In the replica set the departing replicas turned out to be `0` and `1`; the selection comes from whatever order the generator gives, and a different run would send a different pair. The difference is not just which replica leaves: in the stateful set, which data the scale-down sets aside is **known in advance**, and when scale-up comes the same two replicas return to the same two volumes. In the table, this is the number `2` in the last row. The last two rows of the sequence give the summary. Across the stateful set's run **5 volumes** were created and **0** ended up orphaned. The replica set created **12 volumes** over the same events and **7 were left orphaned** — 280 Gi of data sits in volumes no replica is bound to. This data is not lost; what is lost is the **name** that would reach it. When identity is not persistent, persistent storage itself becomes unreachable: persistence and identity are not problems that can be solved separately. The seven orphaned volumes carry a second cost too. There is no lifecycle that deletes them — when a replica leaves, the volume stays, because the data in it may be valuable. The decision falls to a human, and that human has to know which night, which meter range, a name like `data-rep-qxlm` belongs to. Because the name is not persistent, this information sits nowhere; the only option left is to open the volume and look inside. In the stateful set this question is never asked, because the name `data-rep-3` already carries the answer. #### The Round Cost of Ordered Startup The second table gives the price paid in exchange for a stable identity. A stateful set opens replicas one at a time; each replica waits for the previous one to pass its readiness check. Shutdown runs in reverse order, again one at a time. For five replicas, readiness times are 1, 2, 3, 3, 1 rounds. Ordered startup takes their sum, **10 rounds**; if all of them opened at once, the slowest replica's time, **3 rounds**, would be enough. Shutdown is 5 rounds ordered, 1 round parallel. The total gap between the two ends is **11 rounds**. For nine replicas, startup rises to 12 rounds and shutdown to 9, a gap of **17 rounds**. As scale grows the gap grows with it: parallel cost stays tied to the slowest replica while ordered cost grows linearly with replica count. This cost is an information cost. For the system to enforce the order, it has to answer the question "is the previous replica really ready" at every step, and that answer comes from a check. If the check is relaxed, rounds are gained, but the order guarantee is lost too — the next table counts what that means. There is also a case where this cost buys nothing, and it needs to be named. Order is paid for because there is a **formation dependency** among replicas: one is the seed, the others join it. In an application where replicas know nothing about each other, no such dependency exists; in that case the 11 rounds paid for five replicas buy nothing at all, they only stretch out the release window. In the measurement network the distinction is clear: the verifier's replicas form a shared membership, the reading collector's replicas do not — one needs a stateful set, the other does not. #### When Order Breaks: Membership and Ownership The third table separates two distinct breakdowns. In the correct order — `0,1,2,3,4` — a single membership forms, there are no dual-claimed shards, no wrong claims, no shards served without data. In the `2,1,0,3,4` order the seed opens third. The two replicas that open before it, finding no seed, set up their own memberships: **three separate memberships**. Because each membership distributes the 12 shards according to its own member count, **all 12 shards end up dual-claimed**, and 30 claims fall to the wrong replica. Six shards are served by replicas that do not hold their data. In the `1,4,3,0,2` order the split rises to four, and wrong claims climb to 38. The last row is the sneakiest case and deserves to be read on its own. In the `0,1,2,4,3` order the seed opens first, and **membership does not split** — a single membership, zero dual claims. Despite this, four shards fall to a replica that does not hold their data, because ownership is distributed **by join order**, and `4` and `3` have swapped places. So an order violation produces two different failures: the visible one is membership splitting, the invisible one is ownership drifting. In the second case the cluster looks healthy, queries are answered, only the answer to four shards comes from the wrong replica. The devolution's cost adds up in one line here: the decision rests on three items per replica — sequence number, volume binding, readiness; **15 items** for five replicas. If all of these items are produced correctly and on time, orphaned volumes are 0, membership is 1, wrong claims are 0. If the sequence-number item is missing, 7 orphaned volumes result; if the readiness item is missing, split membership and drifting ownership appear. #### Summary - Over the same six events, the stateful set creates 5 volumes and leaves 0 orphaned; the replica set creates 12 volumes and leaves 7 orphaned (280 Gi of unclaimed data). - Name persistence is a precondition for volume binding: on re-creation, 2 of 2 replicas returned to their old volume in the stateful set, and 0 did in the replica set. - On scale-down the stateful set removes the highest sequence (3 and 4), and on scale-up the same two replicas return to the same volumes; in the replica set the departing pair (0 and 1) depends on the run, and there is no return. - The cost of ordered startup is in rounds: 10 rounds against 3 for five replicas, 12 against 3 for nine; combined with shutdown, the total gap is 11 and 17 rounds. - An order violation produces two separate failures: when the seed opens late, membership splits into 3–4 and all 12 shards end up dual-claimed; when the seed opens early but the join order is out of place, membership stays single but 4 shards fall to a replica without their data. - The decision rests on three items per replica (sequence, volume binding, readiness); 15 items for five replicas. #### Next Step The difference between a stateful set and a replica set was measured, and both share one common assumption: pods **do not finish**. The replica count is brought to the desired value and held there; if a pod terminates, this is a failure, and it is restarted. The measurement network's nightly batch job does not fit this assumption. That job starts, verifies the day's readings, produces the billing entry, and **finishes** — finishing is the expected behavior, not a failure. Who decides it is finished, and by looking at what; how many times is it retried if it is not finished; if it has to start at the same time every night, who guards that time? The next lesson builds one-shot and recurring tasks with two separate objects and counts: how many attempts a completion guarantee costs, what result the parallelism and completion-count settings give, what happens on a missed run, and how three policies diverge from each other when a new run arrives before the previous one has finished. ### Jobs and Scheduled Jobs Source: https://academia.sh/en/courses/orchestration/jobs-and-scheduled-jobs Modeling tasks expected to finish with a cluster object: a job object's completion guarantee is counted against the attempt limit, the round and empty-slot results of parallelism and completion-count settings are surveyed, a missed run on a scheduled job is measured, and the skip, queue, and run-parallel policies for overlap are compared separately. A replica set and a stateful set share the same assumption: pods do not finish. The desired replica count is a target, and if a pod terminates this is a drift, which the reconciler re-creates. The measurement network's nightly batch job does not fit this assumption. That job starts every night, verifies the day's readings across twelve meter sets, produces the billing entry, and **finishes**. Finishing is the expected behavior; if it were given to a replica set, the reconciler would restart it endlessly every night. This is why a second object type is needed: the **job**. A job's target is not the number of running pods, it is the **number of completed runs**. Termination is not drift, it is the target itself. Adding a schedule on top produces a **cron job**: an object that produces job objects at specific times. The devolved decision here has three parts: whether the job is finished, whether it is retried if not, and when the next run happens and what happens if the previous one is still going. Four things are counted: how much completion the attempt limit buys, what round count and what waste parallelism gives, what each of the three overlap policies sacrifices, and whether a missed run can be made up. #### Work Expected to Finish The reconciler runs the same loop in both objects — desired state, actual state, gap, correction — but "desired state" is a different thing. In a replica set, desired state is about a **moment**: three pods should be running right now. In a job, desired state is about a **history**: twelve slices should have completed. The first case is checked continuously, the second is satisfied once and never breaks again. - **CO43** — The job, attempt counter, parallelism slots, and nightly schedule are a model built in `node`; no real cluster runs. - **CO44** — The measurement network is fictional. The nightly batch job runs for 30 nights, and each run is split into twelve meter sets (slices). - **CO45** — An attempt's failure probability is 0.5; the model has a single failure class (input file not ready). The attempt limit is swept at 1, 3, and 6. - **CO46** — Each slice takes one round; parallelism is the number of concurrent slots. If a slice falls to a second attempt, one round is added to the total time. - **CO47** — The cron job's interval is 24 hours. Run durations come from the generator: roughly a quarter of the time 30–54 hours, otherwise 3–7 hours. - **CO48** — The controller is down for three nights. The startup window is swept at 0, 1, and 3 nights. Seed `20260813`. ```js // measure/jobs.mjs — a job object's completion guarantee and a cron job's overlap policies. // MODEL: the job object, attempt counter, parallelism slots, and nightly schedule are a model built // in node; no real cluster runs. The numbers below are the model's numbers. const NIGHTS = 30, SLICES = 12, INTERVAL = 24, SEED = 20260813, FAIL_PROB = 0.5; let s = SEED % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; // --- 1. completion guarantee: attempt limit --- console.log(`${"attempt limit".padEnd(14)}${"completed nights".padStart(17)}${"failed nights".padStart(15)}` + `${"total attempts".padStart(16)}${"wasted attempts".padStart(17)}${"leftover pods".padStart(15)}`); for (const limit of [1, 3, 6]) { s = SEED % 2147483647; let completed = 0, attempts = 0, wasted = 0; for (let g = 0; g < NIGHTS; g += 1) { let d = 0, ok = false; while (d < limit && !ok) { d += 1; attempts += 1; if (random() >= FAIL_PROB) ok = true; else wasted += 1; } if (ok) completed += 1; } console.log(`${String(limit).padEnd(14)}${String(completed).padStart(17)}${String(NIGHTS - completed).padStart(15)}` + `${String(attempts).padStart(16)}${String(wasted).padStart(17)}${String(wasted).padStart(15)}`); } // --- 2. parallelism and completion count --- console.log(`\n${"parallelism".padEnd(12)}${"completions".padStart(13)}${"rounds".padStart(8)}${"pod-rounds".padStart(12)}` + `${"empty slots".padStart(13)}${"one slice needs 2 tries".padStart(25)}${"increase".padStart(10)}`); for (const p of [1, 3, 5, 6, 12]) { const rounds = Math.ceil(SLICES / p), delayed = rounds + 1; console.log(`${String(p).padEnd(12)}${String(SLICES).padStart(13)}${String(rounds).padStart(8)}${String(SLICES).padStart(12)}` + `${String(p * rounds - SLICES).padStart(13)}${String(delayed).padStart(25)}${`%${Math.round((delayed / rounds - 1) * 100)}`.padStart(10)}`); } // --- 3. overlap: three policies --- s = SEED % 2147483647; const DURATION = Array.from({ length: NIGHTS }, () => (random() < 0.25 ? 30 + Math.floor(random() * 25) : 3 + Math.floor(random() * 5))); console.log(`\n${NIGHTS} nights' run durations (hours): ${DURATION.join(",")}`); function policy(label) { const start = new Array(NIGHTS).fill(null), end = new Array(NIGHTS).fill(null); let finish = 0, started = 0, skipped = 0, delay = 0; for (let g = 0; g < NIGHTS; g += 1) { const t = g * INTERVAL; if (label === "skip" && finish > t) { skipped += 1; continue; } const b = label === "queue" ? Math.max(t, finish) : t; start[g] = b; end[g] = b + DURATION[g]; finish = end[g]; delay += b - t; started += 1; } let maxConcurrent = 0, maxQueue = 0; for (let g = 0; g < NIGHTS; g += 1) { const t = g * INTERVAL; maxConcurrent = Math.max(maxConcurrent, start.filter((b, i) => b !== null && b <= t && end[i] > t).length); maxQueue = Math.max(maxQueue, start.filter((b, i) => b !== null && b > t && i * INTERVAL <= t).length); } const last = NIGHTS * INTERVAL; return { started, skipped, delay, maxConcurrent, maxQueue, carryover: start.filter((b) => b !== null && b >= last).length, podHours: start.reduce((a, b, i) => a + (b === null ? 0 : end[i] - b), 0) }; } const P = {}; console.log(`${"policy".padEnd(14)}${"started".padStart(9)}${"skipped".padStart(9)}${"max concurrent".padStart(17)}` + `${"max queue".padStart(12)}${"avg delay (hours)".padStart(20)}${"pod-hours".padStart(12)}`); for (const label of ["skip", "queue", "run parallel"]) { const r = policy(label); P[label] = r; console.log(`${label.padEnd(14)}${String(r.started).padStart(9)}${String(r.skipped).padStart(9)}${String(r.maxConcurrent).padStart(17)}` + `${String(r.maxQueue).padStart(12)}${(r.delay / r.started).toFixed(2).padStart(20)}${String(r.podHours).padStart(12)}`); } console.log(`unprocessed nights: skip ${NIGHTS - P["skip"].started}, queue ${NIGHTS - P["queue"].started}, ` + `run parallel ${NIGHTS - P["run parallel"].started}; runs carrying past the horizon ${P["queue"].carryover}`); // --- 4. missed run: the controller is down from night 10 to 12 --- const MISSED = 3; console.log(`\n${"startup window".padEnd(16)}${"made up".padStart(11)}${"dropped".padStart(11)}` + `${"ran under skip".padStart(17)}${"ran under queue".padStart(18)}${"ran under parallel".padStart(21)}`); for (const window of [0, 1, 3]) { const madeUp = Math.min(window, MISSED); console.log(`${String(window).padEnd(16)}${String(madeUp).padStart(11)}${String(MISSED - madeUp).padStart(11)}` + `${String(Math.min(madeUp, 1)).padStart(17)}${String(madeUp).padStart(18)}${String(madeUp).padStart(21)}`); } console.log(`information items: 4 per night (scheduled time, run duration, attempt counter, active-run count) = ${NIGHTS * 4} (seed ${SEED})`); ``` ``` attempt limit completed nights failed nights total attempts wasted attempts leftover pods 1 19 11 30 11 11 3 27 3 49 22 22 6 29 1 54 25 25 parallelism completions rounds pod-rounds empty slots one slice needs 2 tries increase 1 12 12 12 0 13 %8 3 12 4 12 0 5 %25 5 12 3 12 3 4 %33 6 12 2 12 0 3 %50 12 12 1 12 0 2 %100 30 nights' run durations (hours): 5,4,7,5,6,5,5,4,7,31,3,3,5,3,50,4,51,6,5,3,5,7,47,5,3,37,6,5,3,5 policy started skipped max concurrent max queue avg delay (hours) pod-hours skip 25 5 1 0 0.00 266 queue 30 0 1 2 4.23 335 run parallel 30 0 2 0 0.00 335 unprocessed nights: skip 5, queue 0, run parallel 0; runs carrying past the horizon 0 startup window made up dropped ran under skip ran under queue ran under parallel 0 0 3 0 0 0 1 1 2 1 1 1 3 3 0 1 3 3 information items: 4 per night (scheduled time, run duration, attempt counter, active-run count) = 120 (seed 20260813) ``` #### The Completion Guarantee: Attempt Limit With a single attempt the job completes **19** of 30 nights, and 11 nights fail. When the attempt limit rises to three, completed nights rise to 27; at six, to 29. The guarantee can be bought, but not linearly: the first increase gains 8 nights, the second only 2. The cost is in the other columns. Total attempts rise from 30 to 49, then to 54; wasted attempts from 11 to 22, then to 25. So raising the limit from 3 to 6 gains 2 nights while adding 3 wasted attempts — one and a half wasted runs for every night gained. These runs consume resources and eat into the cron job's window too. The last column is a silent cost. A failed attempt's pod is not deleted; it is kept so its log can be read. At a limit of six, **25 leftover pods** remain, and nobody collects them. The job needs to declare a retention period, or by the end of thirty nights the cluster fills up with failed pods that have long lost their meaning. What "failed night" means also has to be stated separately. At a limit of 1, eleven nights' data was never verified; the job declares this in a status field, but **declaring is not processing**. The devolved decision is only the "retry" decision; the owner of the "attempts exhausted, data stayed incomplete" decision is still outside. #### Parallelism and Completion Count The second table shows what the two settings give together. The completion count is fixed at 12 — all twelve slices have to be processed. As parallelism rises from 1 to 12, round count drops from 12 to 1, and **the total pod-rounds never changes**: 12 in every case. Parallelism does not reduce total work, it only spreads it out over time. Two things change. The first is empty slots: at a parallelism of 5, in the last of three rounds only 12 of 15 slots are filled, and **3 slots sit empty**. Parallelism values that do not divide the completion count produce waste. The second, and more important, is the last two columns: if a slice falls to a second attempt, total time stretches by one round, and this one round's **share** grows together with parallelism. At a parallelism of 1 the increase is 8%, at 12 it is **100%**. At full parallelism, the work equals the duration of the slowest slice; a single slow slice doubles the total time. When completion count is not declared, the object switches to a different mode: pods drain a work queue and terminate themselves once it empties. The round count stays the same, but **the finish decision sits inside the pod**. The cluster only sees the pod terminate; it does not know how many slices were processed. This is where the devolved decision is taken back here: if the completion count is declared, the cluster gives the guarantee; if not, the application does. #### Overlap: Three Policies The third table is the nightly job's real question. Among the thirty nights' run durations, five nights run 31, 37, 47, 50, and 51 hours — longer than the 24-hour interval. A new run arrives before the previous one has finished. The **skip** policy does not start a conflicting run: 25 runs started, **5 nights were skipped**. Concurrency stays at 1, there is no queue, no delay, and resource consumption is lowest at 266 pod-hours. The cost is direct data coverage: five nights' readings were never verified, and this policy does not bring them back. The **queue** policy skips no night at all: 30 runs, 0 unprocessed nights. In exchange runs start an average of **4.23 hours** late from their scheduled time, and at one point two runs wait in the queue at once. Concurrency is still 1 — the single-runner guarantee is preserved. The critical point is that the delay **does not accumulate**: because long nights are spread out, the queue empties, and no run carries past the thirtieth night. If long runs came back to back, this column would grow without bound. The **run parallel** policy also starts all 30 runs, with zero delay, but concurrency **rises to 2**. Two runs read the same meter sets and write the same billing entry at the same time. From the cluster's point of view there is no problem — both pods are healthy; the problem is in the application's data layer, and the cluster cannot see it. On the resource side, the same 335 pod-hours are spent as under queue. What all three policies have in common is this: none of them **knows** the run duration. The policy is chosen before the run starts; the duration is learned after it. Because the system does not know, one sacrifices data coverage, one sacrifices timing, and one sacrifices the singleton guarantee. The correct choice for the measurement network can be read from the table. The billing entry cannot tolerate a missing night, so skip's 5 nights are unacceptable. Two runs writing the same night at once is also unacceptable, because the billing entry is written additively, and two runs would add the same readings twice. What is left is queue, and its cost is visible in the table: an average delay of 4.23 hours, a peak queue of two runs. This cost can only be paid as long as the queue empties; if long nights arrive back to back, the delay accumulates, and the nightly job spills over into the day at some point. So the choice of policy is not a setting made once and left alone, it is a decision that has to be revisited as the run-duration distribution changes. #### The Missed Run The last table measures the case where the controller is down for three nights. With a startup window of 0 nights, **none** of the three missed runs are made up; with a window of 1 night, one runs and two are dropped; with a window of 3 nights, all three go into the queue. Being made up does not mean running. At window 3, three runs become eligible at the same moment, and the overlap policy kicks in right there: under the **skip** policy only one runs and two drop again; under queue, all three run one after another; under run parallel, all three start at once. So the two settings can cancel each other out — if a wide window is paired with a strict overlap policy, catch-up is open on paper and closed in practice. This lesson's devolution balance sheet is this: the decision rests on four items per night — scheduled time, run duration, attempt counter, active-run count; **120 items** for thirty nights. Three of these come from measurement; one (run duration) is only known after the run finishes. This is the one item the system does not know, and the entire difference between the three policies comes from it. #### Summary - A job's target is not the number of running pods but the number of completed runs; termination is not drift, it is the target itself. - As the attempt limit rises 1 → 3 → 6, completed nights go 19 → 27 → 29 and wasted attempts 11 → 22 → 25. The last increase gains 2 nights and adds 3 wasted runs; 25 leftover pods remain. - Parallelism does not change total work (12 pod-rounds in every case), it brings rounds down from 12 to 1; at a parallelism of 5, 3 empty slots appear, and the cost of a single slow slice rises from 8% to 100%. - If the completion count is not declared, the guarantee moves from the cluster to the application: the cluster only sees termination, it does not know how many slices were processed. - The three overlap policies sacrifice three separate things: skip leaves 5 nights unprocessed (266 pod-hours), queue delays by an average of 4.23 hours, run parallel raises concurrency to 2 and lets two writers loose on the same data (335 pod-hours). - With a startup window of 0 → 1 → 3 nights, catch-up is 0 → 1 → 3; but how many of the made-up runs actually run is decided by the overlap policy (1 under skip). #### Next Step The nightly job became a cluster object: its finish is defined, its attempts are limited, its schedule is declared. Every number measured was on a single application's own account — 335 pod-hours, two concurrent runs, twelve parallel slices. Yet the cluster does not belong to the measurement network alone. Two other applications run on the same nodes, and the question of what happens to their requests when the nightly job opens twelve slices at once was never asked. Pod, replica set, version object, service object, volume, job — all of them sit in a single naming space and all of them eat from the same resource pool. The next lesson gathers three tenants into a single cluster and counts: how many requests one tenant starves another out of without a quota, how many requests are rejected once a quota is set, how many units of the cluster the sum of the quotas leaves idle, and how default request values shift these three numbers. ### Namespaces and Quotas Source: https://academia.sh/en/courses/orchestration/namespaces-and-quotas Building multi-tenant resource separation with numbers: while three tenants share a cluster, how one tenant starves the others without a quota is measured, the number of requests rejected once a quota is set is compared against the capacity the quota leaves idle, and the effect of the default request value applied to undeclared pods on acceptance, overallocation, and node pressure is surveyed. The nightly job became a cluster object and can open twelve slices at once. Every number measured so far was on a single application's own account; that the cluster also belongs to others was never taken into account. Two other applications run on the same six nodes, and the objects of all three sit in the same naming space, the same resource pool. The **namespace** is the object that divides this space. The word is familiar: in the runtime measurements, `namespace` was the **visibility mapping** that determines which processes, which network, and which filesystem the kernel shows a process; here the same word describes a **partition** that divides the object names and resource accounting inside the cluster — one narrows what a process sees, the other separates a tenant's naming and resource space. The two are different objects at different layers. A namespace alone separates only names; it does not separate resources. The object that separates resources is the **quota**. This lesson names the devolved decision as follows: **whether a request is accepted**. Once the decision moves from a human to the cluster, the cluster needs to know three items per request — which tenant, how much it wants, how much it actually uses. The third item is in no declaration; it only comes from measurement, and this lesson's entire drift comes from there. #### Three Tenants, One Cluster The three measured cases use the same request sequence: no quota, quota, and quota with a default applied to undeclared pods. The sequence's order does not change, because the order itself is one of the things measured. - **CO49** — Cluster capacity, tenants, request sequence, and quota accounting are a model built in `node`; no real cluster runs. - **CO50** — The cluster is 96 processor units (6 nodes × 16). A single dimension is measured; memory is not counted separately because it works the same arithmetic. - **CO51** — Three tenants are fictional: measurement network, field service, reporting. Sixty requests arrive; requests 11–30 are the measurement network's nightly burst, the rest round-robin. - **CO52** — Request sizes come from the generator by tenant: measurement network 1–6, field service 1–4, reporting 1–2 units. - **CO53** — Roughly a third of requests declare no resource request. Actual usage is 0.4–1.1 times the declared value. - **CO54** — Quota accounting looks at the **declared** request, not actual usage. Seed `20260819`. ```js // measure/quota.mjs — three tenants sharing a cluster: no quota, with quota, and with a default request. // MODEL: cluster capacity, tenants, request sequence, and quota accounting are a model built in // node; no real cluster runs. The numbers below are the model's numbers. const CAPACITY = 96, SEED = 20260819; // 6 nodes x 16 processor units const TENANTS = ["measurement-network", "field-service", "reporting"]; let s = SEED % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const REQUESTS = []; // order: round-robin -> burst -> round-robin for (let i = 0; i < 60; i += 1) { const k = i >= 10 && i < 30 ? TENANTS[0] : TENANTS[i % 3]; const requested = 1 + Math.floor(random() * (k === "reporting" ? 2 : k === "field-service" ? 4 : 6)); REQUESTS.push({ k, requested, declared: random() >= 0.3, actual: Math.max(1, Math.round(requested * (0.4 + random() * 0.7))) }); } const count = (f) => TENANTS.map((k) => REQUESTS.filter((i) => i.k === k && f(i)).length); // --- 1. no quota: first come, first served --- let remaining = CAPACITY; const accepted = new Set(); for (const [n, i] of REQUESTS.entries()) if (i.requested <= remaining) { remaining -= i.requested; accepted.add(n); } console.log(`${"tenant".padEnd(20)}${"incoming".padStart(11)}${"accepted".padStart(10)}${"rejected".padStart(10)}` + `${"accept rate".padStart(13)}${"consumed".padStart(11)}${"cluster share".padStart(15)}`); for (const k of TENANTS) { const g = REQUESTS.filter((i) => i.k === k), a = g.filter((i) => accepted.has(REQUESTS.indexOf(i))); const t = a.reduce((x, i) => x + i.requested, 0); console.log(`${k.padEnd(20)}${String(g.length).padStart(11)}${String(a.length).padStart(10)}${String(g.length - a.length).padStart(10)}` + `${`%${Math.round(a.length / g.length * 100)}`.padStart(13)}${String(t).padStart(11)}${`%${Math.round(t / CAPACITY * 100)}`.padStart(15)}`); } console.log(`no quota: ${accepted.size} requests accepted, ${REQUESTS.length - accepted.size} rejected, idle capacity ${remaining}`); // --- 2. quota: two splits --- function quota(limits, defaultValue = null) { const usage = Object.fromEntries(TENANTS.map((k) => [k, 0])); const acceptedK = Object.fromEntries(TENANTS.map((k) => [k, 0])); const quotaRejected = Object.fromEntries(TENANTS.map((k) => [k, 0])); let free = CAPACITY, actual = 0, charged = 0, pressure = 0; for (const i of REQUESTS) { const amount = i.declared ? i.requested : (defaultValue === null ? i.requested : defaultValue); if (usage[i.k] + amount > limits[i.k] || amount > free) { quotaRejected[i.k] += 1; continue; } usage[i.k] += amount; free -= amount; acceptedK[i.k] += 1; charged += amount; actual += i.actual; if (i.actual > amount) pressure += 1; } return { usage, acceptedK, quotaRejected, free, actual, charged, pressure }; } const SPLITS = { equal: { "measurement-network": 32, "field-service": 32, reporting: 32 }, weighted: { "measurement-network": 48, "field-service": 32, reporting: 16 } }; console.log(`\n${"split".padEnd(10)}${"tenant".padEnd(20)}${"quota".padStart(6)}${"accepted".padStart(10)}${"quota rejections".padStart(18)}` + `${"used".padStart(6)}${"quota headroom".padStart(16)}`); for (const [label, limits] of Object.entries(SPLITS)) { const r = quota(limits); for (const k of TENANTS) console.log(`${label.padEnd(10)}${k.padEnd(20)}${String(limits[k]).padStart(6)}${String(r.acceptedK[k]).padStart(10)}` + `${String(r.quotaRejected[k]).padStart(18)}${String(r.usage[k]).padStart(6)}${String(limits[k] - r.usage[k]).padStart(16)}`); const totalRejected = TENANTS.reduce((a, k) => a + r.quotaRejected[k], 0); console.log(` -> ${label}: accepted ${REQUESTS.length - totalRejected}, rejected ${totalRejected}, idle cluster capacity ${r.free} ` + `(quota total ${Object.values(limits).reduce((a, b) => a + b, 0)})`); } // --- 3. default request value --- console.log(`\n${"default request".padEnd(17)}${"accepted".padStart(9)}${"charged to quota".padStart(18)}${"actual usage".padStart(15)}` + `${"overallocation".padStart(16)}${"node pressure".padStart(15)}${"idle capacity".padStart(15)}`); for (const v of [0, 1, 2, 4, null]) { const r = quota(SPLITS.weighted, v); const acceptedT = TENANTS.reduce((a, k) => a + r.acceptedK[k], 0); console.log(`${(v === null ? "declaration required" : String(v)).padEnd(17)}${String(acceptedT).padStart(9)}${String(r.charged).padStart(18)}` + `${String(r.actual).padStart(15)}${String(r.charged - r.actual).padStart(16)}${String(r.pressure).padStart(15)}${String(r.free).padStart(15)}`); } console.log(`${REQUESTS.length} requests, ${count((i) => !i.declared).reduce((a, b) => a + b, 0)} do not declare a request`); console.log(`information items: 3 per request (tenant, request, actual usage) = ${REQUESTS.length * 3}, seed ${SEED}`); ``` ``` tenant incoming accepted rejected accept rate consumed cluster share measurement-network 34 26 8 %76 77 %80 field-service 13 5 8 %38 13 %14 reporting 13 4 9 %31 6 %6 no quota: 35 requests accepted, 25 rejected, idle capacity 0 split tenant quota accepted quota rejections used quota headroom equal measurement-network 32 13 21 32 0 equal field-service 32 12 1 30 2 equal reporting 32 13 0 21 11 -> equal: accepted 38, rejected 22, idle cluster capacity 13 (quota total 96) weighted measurement-network 48 17 17 48 0 weighted field-service 32 12 1 30 2 weighted reporting 16 10 3 16 0 -> weighted: accepted 39, rejected 21, idle cluster capacity 2 (quota total 96) default request accepted charged to quota actual usage overallocation node pressure idle capacity 0 53 79 92 -13 22 17 1 47 88 77 11 11 8 2 41 94 67 27 2 2 4 32 96 52 44 0 0 declaration required 39 94 64 30 0 2 60 requests, 22 do not declare a request information items: 3 per request (tenant, request, actual usage) = 180, seed 20260819 ``` #### Without a Quota: the Starved Tenant In a quota-free cluster the rule is one line: accept if there is room. The result is in the first table. The measurement network sent 34 requests, **26 were accepted** — an accept rate of 76% — and it held **80%** of the cluster. Field service got 5 of 13 requests through (38%), reporting 4 of 13 (31%). Reporting's cluster share is **6%**. The cluster did not sit idle; capacity was fully exhausted and 25 requests were rejected. The cause of starvation is not request size, it is **order**. The measurement network's nightly burst flows uninterrupted from request 11 to request 30, and it consumes most of the capacity in that stretch; the small reporting requests that come after are left bidding for scraps. Reporting's requests are the smallest (1–2 units) and still get the lowest accept rate. So under quota-free sharing, share is distributed not by need or courtesy but by **timing**. One tenant raising its own pod count is a direct effect on another's requests, and this effect is declared nowhere. What happens to the 25 rejected requests also has to be stated separately. The request is not rejected, it **waits**: the pod object is declared, it just has no room on any node. Reporting's user does not see this as an error, they see it as slowness, and they look for the cause in their own namespace. This is the most expensive part of quota-free sharing — one tenant's problem is born from another tenant's behavior, and it sits somewhere neither side can see. A namespace does not change this table. Even if the three tenants' objects sit in separate namespaces, they share the same 96 units; a namespace only lets the name `nightly-job` be used three times and lets authorization be granted per partition. A second object is needed for resource separation. #### Once a Quota Is Set: Two Costs The second table shows together what the quota buys and what it makes you pay in exchange. Under the **equal split** (32/32/32), the measurement network hits its quota: it gets **21 quota rejections** and stays at 32 units. In exchange reporting gets all 13 of its requests through, without a single rejection. Starvation is over. The cost is in the last column: reporting **never uses 11 units** of its quota, and **13 units sit idle** across the cluster. While the measurement network's 21 requests are rejected, 13 units sit empty in the cluster — this is exactly the quota's waste. Total acceptance rises from 35 under no quota to 38, so the quota did not lower throughput; but idle capacity rose from zero to 13. Under the **weighted split** (48/32/16), waste falls: idle capacity is 2 units, total acceptance 39. The measurement network's rejections fall from 21 to 17. The cost has shifted — reporting now takes 3 rejections. Weighting requires knowing demand; this split is only correct because who would want how much was measured in advance. If demand changes — say reporting grows during monthly close — the same split produces waste in the opposite direction. What a quota counts is not processor alone. The same accounting can be kept separately for memory, for the total size of persistent volume claims, and for object counts — how many service objects, how many secret objects, how many persistent volume claims a namespace can hold at most. Count items protect something different from the processor item: not capacity, but the control plane's own load. A tenant declaring thousands of secret objects fills no node, but it fills the state store and the reconciliation loop. The measured table tracks a single item; the others work by the same arithmetic. Reading the two rows together shows the quota's real nature: a quota is not a **capacity allocation**, it is an **upper bound**. Unused quota does not pass to anyone; the idle 13 units are not opened up to the measurement network, they just sit there. The devolved decision is partly taken back here: the cluster makes the "accept this request" decision, but the "who gets how much" decision still belongs to the human who wrote the quota, and that human does not know next week's demand. #### The Default Request Value The third table asks where the number quota accounting rests on comes from. **22 of 60** requests declare no resource request. A quota has to see a number, so a default is applied to undeclared pods. There are two separate numbers here, and they must not be confused. The **resource request** is the accounting and placement number: the quota sums it, and the scheduler looks for that much room on a node. The **limit** is the ceiling enforced at run time; the pod cannot exceed it. A quota can count both separately, and a separate default can be defined for each. The sweep below changes the request default, because that is the number that decides acceptance; the limit default's effect shows up somewhere else — if the limit is set low, the pod clears the quota, finds room on a node, and then hits its own ceiling, and work slows down or terminates. When the two are not considered together, the quota table looks clean while unexplained terminations pile up on the application side. At a default of **0**, these pods eat nothing from the quota: acceptance jumps to 53, the total charged to the quota stays at 79, and the cluster thinks it has 17 units idle. Actual usage, though, is **92** — 13 units more than declared, and the 22 pods are using most of what they never declared. The cluster is comfortable in its accounting ledger while its nodes are under pressure. This is the quota's most dangerous setting: the rule looks like it is being enforced, and it is not. As the default grows, the table turns the other way. At **2** units node pressure falls to 2, at **4** to **0**; acceptance drops from 53 to 32, and overallocation rises to 44 units — 44 of the 96 units charged to the quota are never used. Safety was bought, and in exchange nearly half the cluster was spent on paper. The last row gives the third path: making declaration mandatory. Acceptance 39, overallocation 30, node pressure **0**, idle capacity 2. No single value of a fixed default catches all of this at once, because the default is a single number and the real need of 22 pods is not a single number. The cost of mandating declaration, though, is paid not on the cluster's side but on the side of whoever writes the declaration. This topic's devolution balance sheet is this: the acceptance decision rests on three items per request — which tenant, how much it wants, how much it actually uses; **180 items** for sixty requests. The first two items come from declaration and are cheap; the third comes only from measurement, and the quota never sees it at all. Because the system does not know actual usage, it either overallocates and wastes capacity, or underallocates and puts nodes under pressure. #### Summary - A namespace and a quota do separate jobs: a namespace divides names and authorization, not resources; the object that divides resources is the quota. - In a quota-free cluster, share is distributed by order, not by need: the measurement network takes 80% of the cluster with a 76% accept rate, while reporting stays at 6% with a 31% accept rate, and 25 requests are rejected. - A quota ends starvation but makes you pay two costs: under the equal split, the measurement network gets 21 quota rejections while 11 units of reporting's quota and 13 units of the cluster sit idle. - The weighted split brings waste down from 13 to 2 units, shifting rejection onto reporting in exchange; its correctness depends on demand being known in advance. - Unused quota passes to nobody: a quota is not a capacity allocation, it is an upper bound. - At a default request of 0, the cluster thinks it has 17 units idle while actual usage is 92 and 22 pods create node pressure; at a default of 4, pressure drops to zero but 44 units are overallocated. Mandating declaration brings pressure down to 0 while keeping waste at 30. #### Next Step Throughout this topic, objects were declared one by one: the smallest deployment unit, the replica set and version object, the service object and the ingress rule, the configuration and secret object, the persistent volume claim, the stateful set, the job and the cron job, the namespace and the quota. All of them share a common gap, and this lesson exposed it with the last table: declarations have a **request**, quotas have an **upper bound**, but nowhere does it say "this pod will open on which node." The quota accepting a request does not finish the job; the accepted pod has to land on one of six nodes, and that node genuinely has to have room. The storage measurement already showed half of this problem: a volume in single-writer mode eliminated twenty-six of thirty-six options. So what makes the choice among the remaining options, what information does it look at when deciding, what happens if the information it needs to look at has gone stale, and where does a pod wait when no option fits? The next topic writes this decision as a constraint solver and measures the placement decision, the count of staying unsolvable, and the scaling decision's own delay. ## Scheduling and Scaling Tying the placement and growth decision to numbers: the resource request and limit's effect on the scheduler's decision, placement constraints through label selection, affinity, and topology spread, taints and tolerations for node separation, eviction order and priorities under resource pressure, the delay-oscillation balance of metric-based horizontal scaling, vertical resizing, and node addition. ### Resource Requests and Limits Source: https://academia.sh/en/courses/orchestration/resource-requests-and-limits Tying the placement decision to two numbers: the resource request sets the scheduler's ledger, the limit sets runtime behavior; the same workload set is placed under three request settings and compared by placed pod count against spare capacity; the request's drift from actual usage is scanned, and the waste of overstatement is weighed against the contention of understatement. The objects running on the cluster have been declared: a pod was defined, its copies were turned into a replica set, it was placed inside a namespace, and that namespace's total resources were shared out with a quota. Through all of this, one question was never asked: which pod lands on which node. A quota sets a ceiling; it does not place. The replica count says how many there will be, not where. Placement is the first mechanical decision handed from a human to the system, and it rests on two numbers. These two numbers, sitting side by side in the pod declaration, do their work in separate places: the resource request in the scheduler's ledger, the limit in the node's runtime. How many nodes are needed is not asked here — that calculation, as the larger of the load constraint and the failure constraint, was done in capacity planning, and it is an input to this lesson. The question here is what placement looks like given a fixed number of nodes. What is measured is not a real cluster but a model built with `node`. The workload is the services of the regional measurement network (the running fiction): the reading collector, the verifier, billing, the work order service, and the nightly batch job. #### The Request Keeps the Ledger, the Limit Does the Cutting The model consists of three nodes and eighteen pods. The scheduler places on the first node that fits and sees only the declared request; it never sees actual usage. Actual usage is derived from the seed of a generator we wrote ourselves and is spread across a 35 percent band around the baseline. ```js // scheduling/request.mjs — the effect of the request and the limit on placement. What gets built is a MODEL; // no real cluster runs. The workload is the regional measurement network's (the running fiction) services. export const NODE = { cpu: 3000, memory: 4096 }; // SS1: capacity assignable per node export const NODE_COUNT = 3; // SS2: service, pod count, base cpu unit, base memory (MiB) const SERVICE = [ ["reading-collector", 6, 420, 384], ["verifier", 4, 650, 704], ["billing", 3, 300, 512], ["work-order", 3, 180, 256], ["nightly-batch", 2, 900, 1408], ]; const SEED = 20260801; // the generator is our own; the seed is visible export function generator(seed = SEED) { let s = seed % 2147483647; return () => (s = (s * 48271) % 2147483647) / 2147483647; } // SS3: actual usage falls in a +/-35% band around the base; it is measured, not declared export function workload() { const r = generator(), k = []; for (const [name, count, c, m] of SERVICE) for (let i = 0; i < count; i++) { const deviation = 0.65 + 0.7 * r(); k.push({ name: `${name}-${i + 1}`, baseC: c, baseM: m, actualC: Math.round(c * deviation), actualM: Math.round(m * deviation) }); } return k; } // The scheduler sees only the DECLARED request; it never sees actual usage. export function place(pods, n = NODE_COUNT) { const nodes = Array.from({ length: n }, () => ({ reqC: 0, reqM: 0, pods: [] })); const unplaced = []; for (const k of pods) { const d = nodes.find((d) => d.reqC + k.reqC <= NODE.cpu && d.reqM + k.reqM <= NODE.memory); if (d === undefined) { unplaced.push(k); continue; } d.reqC += k.reqC; d.reqM += k.reqM; d.pods.push(k); } return { nodes, unplaced }; } // SS4: limit = request x 1.25. Exceeding the cpu limit throttles, exceeding the memory limit kills. export const LIMIT_FACTOR = 1.25; export function measure(pods, n = NODE_COUNT) { const { nodes, unplaced } = place(pods, n); let waste = 0, contention = 0, throttled = 0, killed = 0, placed = 0; for (const d of nodes) { const aC = d.pods.reduce((t, k) => t + k.actualC, 0); d.actualC = aC; contention += Math.max(0, aC - NODE.cpu); placed += d.pods.length; for (const k of d.pods) { waste += Math.max(0, k.reqC - k.actualC); if (k.limC > 0 && k.actualC > k.limC) throttled++; if (k.limM > 0 && k.actualM > k.limM) killed++; } } return { nodes, placed, unplaced: unplaced.length, waste, contention, throttled, killed, spareRequest: nodes.reduce((t, d) => t + (NODE.cpu - d.reqC), 0) }; } export function configure(pods, mode) { return pods.map((k) => { const c = mode === "none" ? 0 : mode === "realistic" ? k.baseC : k.baseC * 2; const m = mode === "none" ? 0 : mode === "realistic" ? k.baseM : k.baseM * 2; return { ...k, reqC: c, reqM: m, limC: mode === "none" ? 0 : Math.round(c * LIMIT_FACTOR), limM: mode === "none" ? 0 : Math.round(m * LIMIT_FACTOR) }; }); } if (process.argv[1].endsWith("request.mjs")) { const k = workload(); const totC = k.reduce((t, x) => t + x.actualC, 0), totM = k.reduce((t, x) => t + x.actualM, 0); console.log(`model: ${k.length} pods, ${NODE_COUNT} nodes x ${NODE.cpu} cpu ` + `units / ${NODE.memory} MiB = ${NODE_COUNT * NODE.cpu} / ` + `${NODE_COUNT * NODE.memory}`); console.log(`actual total usage ${totC} cpu units, ${totM} MiB (seed ${SEED})\n`); const bas = (s) => s.padStart(16); console.log(`${"setting".padEnd(18)}${bas("placed")}${bas("unplaced")}${bas("spare request")}` + `${bas("waste")}${bas("contention")}${bas("throttled")}${bas("killed")}`); for (const mode of ["none", "realistic", "inflated"]) { const o = measure(configure(k, mode)); console.log(`${mode.padEnd(18)}${bas(String(o.placed))}${bas(String(o.unplaced))}` + `${bas(String(o.spareRequest))}${bas(String(o.waste))}${bas(String(o.contention))}` + `${bas(String(o.throttled))}${bas(String(o.killed))}`); } console.log(`\nactual occupancy per node (cpu units / ${NODE.cpu}):`); for (const mode of ["none", "realistic", "inflated"]) { const o = measure(configure(k, mode)); console.log(` ${mode.padEnd(12)}` + o.nodes.map((d, i) => `d${i}: ${d.pods.length} pods ${((d.actualC / NODE.cpu) * 100).toFixed(0)}%`) .join(" ")); } console.log(`\ninformation needed for the decision: ${k.length} pods x 2 declarations = ${k.length * 2}, ` + `${NODE_COUNT} nodes x 2 measurements = ${NODE_COUNT * 2}, total ${k.length * 2 + NODE_COUNT * 2}`); console.log(`item the scheduler does not see: actual usage (${k.length} pods x 2 = ${k.length * 2})`); } ``` ``` model: 18 pods, 3 nodes x 3000 cpu units / 4096 MiB = 9000 / 12288 actual total usage 8503 cpu units, 10193 MiB (seed 20260801) setting placed unplaced spare request waste contention throttled killed none 18 0 9000 0 5503 0 0 realistic 18 0 640 675 557 1 1 inflated 11 7 280 3934 0 0 0 actual occupancy per node (cpu units / 3000): none d0: 18 pods 283% d1: 0 pods 0% d2: 0 pods 0% realistic d0: 8 pods 108% d1: 5 pods 110% d2: 5 pods 65% inflated d0: 4 pods 49% d1: 4 pods 56% d2: 3 pods 55% information needed for the decision: 18 pods x 2 declarations = 36, 3 nodes x 2 measurements = 6, total 42 item the scheduler does not see: actual usage (18 pods x 2 = 36) ``` #### Three Settings, Three Separate Failures Three rows build three separate worlds. In the **no request** setting, every node in the ledger is empty, so all eighteen pods pile onto the first node: the scheduler thinks it made no mistake while the node's actual occupancy reaches 283 percent and 5,503 units of contention are born. Resource sharing is left to the operating system here, and who is being slowed down cannot be read from any declaration. In the **realistic request** setting, eighteen pods spread across three nodes and spare request capacity drops to 640 units. The setting's imperfection shows up in two columns: because actual usage falls short of the declaration, 675 units are reserved and unused, while where it exceeds the declaration, 557 units of contention remain. One pod is throttled for exceeding its cpu limit, and one pod is killed for exceeding its memory limit. **The limit's enforcement is independent of placement**: both pods are placed, both are within the ledger's rules. In the **inflated request** setting there is neither contention nor a single kill; the price is that seven pods never get placed at all. The nodes actually run half-empty (49, 56, 55 percent) and 3,934 units sit reserved and unused. This setting looks like a safety setting; it is a capacity loss. #### Scanning the Drift Separates Two Costs Once the declaration is written as a multiple of actual usage and scanned, the cost of each direction can be counted separately. The same scan also shows the effect of a declaration that is made once and left in place, and of the choice of limit factor. The factor reads like a measure of information quality: the further it drifts from one, the more badly the declaration misrepresents the truth. ```js // scheduling/deviation.mjs — the request's drift from actual usage is scanned: the waste of overstatement, // the contention of understatement. The model and workload come from the previous file. import { NODE, NODE_COUNT, LIMIT_FACTOR, workload, measure } from "./request.mjs"; const pods = workload(); const CAPACITY = NODE_COUNT * NODE.cpu; // Request = actual usage x factor. A factor of 1 means the declaration is exact. const declare = (k, factor, growth = 1) => k.map((x) => { const c = Math.round(x.actualC * factor), m = Math.round(x.actualM * factor); return { ...x, reqC: c, reqM: m, actualC: Math.round(x.actualC * growth), actualM: Math.round(x.actualM * growth), limC: Math.round(c * LIMIT_FACTOR), limM: Math.round(m * LIMIT_FACTOR) }; }); const s = (x) => String(x).padStart(12); console.log(`the request's drift from actual usage (${pods.length} pods, capacity ${CAPACITY}):`); console.log(`${"factor".padEnd(10)}${s("placed")}${s("unplaced")}${s("waste")}` + `${s("contention")}${s("throttled")}${s("killed")}`); for (const factor of [0.5, 0.75, 0.9, 1.0, 1.25, 1.5, 2.0]) { const o = measure(declare(pods, factor)); console.log(`${("x" + factor.toFixed(2)).padEnd(10)}${s(o.placed)}${s(o.unplaced)}` + `${s(o.waste)}${s(o.contention)}${s(o.throttled)}${s(o.killed)}`); } // The declaration is made once while usage grows: the cost of a stale declaration. console.log(`\nwith the declaration held fixed at x1.00 while actual usage grows (SS5):`); console.log(`${"growth".padEnd(10)}${s("placed")}${s("contention")}${s("throttled")}${s("killed")}`); for (const g of [1.0, 1.1, 1.2, 1.3, 1.5]) { const o = measure(declare(pods, 1.0, g)); console.log(`${(((g - 1) * 100).toFixed(0) + "%").padEnd(10)}${s(o.placed)}${s(o.contention)}` + `${s(o.throttled)}${s(o.killed)}`); } // SS6: usage rises 30% during a short-lived spike. The limit factor is scanned. const SPIKE = 1.3; console.log(`\nlimit factor (request declared at x1.00, usage 30% higher at the spike):`); console.log(`${"factor".padEnd(10)}${s("throttled")}${s("killed")}${s("contention")}${s("limit total")}`); for (const c of [1.0, 1.1, 1.25, 1.5, 2.0]) { const k2 = pods.map((x) => ({ ...x, reqC: x.actualC, reqM: x.actualM, actualC: Math.round(x.actualC * SPIKE), actualM: Math.round(x.actualM * SPIKE), limC: Math.round(x.actualC * c), limM: Math.round(x.actualM * c) })); const o = measure(k2); console.log(`${("x" + c.toFixed(2)).padEnd(10)}${s(o.throttled)}${s(o.killed)}${s(o.contention)}` + `${s(k2.reduce((t, x) => t + x.limC, 0))}`); } console.log(`\nat the x2.00 limit factor the limit total reaches ${(pods.reduce((t, x) => t + x.actualC, 0) * 2 / CAPACITY).toFixed(2)}x capacity; the limit is not a reservation, it is a ceiling`); ``` ``` the request's drift from actual usage (18 pods, capacity 9000): factor placed unplaced waste contention throttled killed x0.50 18 0 0 2965 18 18 x0.75 18 0 0 1916 18 18 x0.90 18 0 0 557 0 0 x1.00 17 1 0 0 0 0 x1.25 14 4 1720 0 0 0 x1.50 13 5 2876 0 0 0 x2.00 9 9 4322 0 0 0 with the declaration held fixed at x1.00 while actual usage grows (SS5): growth placed contention throttled killed 0% 17 0 0 0 10% 17 470 0 0 20% 17 1058 0 0 30% 17 1646 17 17 50% 17 2879 17 17 limit factor (request declared at x1.00, usage 30% higher at the spike): factor throttled killed contention limit total x1.00 17 17 1646 8503 x1.10 17 17 1646 9355 x1.25 17 17 1646 10630 x1.50 0 0 1646 12760 x2.00 0 0 1646 17006 at the x2.00 limit factor the limit total reaches 1.89x capacity; the limit is not a reservation, it is a ceiling ``` #### The Waste of Overstatement, the Contention of Understatement The scan is not symmetric. As the factor drops below 1, every pod still gets placed and the bill piles up as contention: 2,965 units at half declaration, 1,916 units at three-quarter declaration, and all eighteen of the eighteen pods are both throttled and killed. Once the factor climbs above 1, contention drops to zero and the bill shifts instead to unplaced pods and waste: at the 1.5x factor five pods fail to place and 2,876 units sit idle, and at the 2x factor nine pods fail to place. The exact-declaration row teaches something too. At a factor of exactly 1, contention is zero and waste is zero, and yet **one pod still fails to place**. This is not a shortage of information; it is that the first-fit placement rule never looks back — the remaining gaps are, one by one, not quite enough, so fragmentation remains. Even with perfect information, the placement rule produces a loss. The second table measures how a declaration **goes stale**. While the declaration stays put, a 10 percent growth in actual usage produces 470 units of contention, a 20 percent growth produces 1,058 units, and at 30 percent seventeen pods are simultaneously throttled and killed. A declaration is not a measurement but a statement, and unless it is refreshed like a measurement, it is the oldest piece of information the system holds. The third table shows what the limit is not. When the factor is raised from 1 to 2, the number of throttled pods drops from seventeen to zero, while **contention never moves from 1,646 units**. The limit only cuts off the top of a single pod; it does not protect the node's total. At the 2x factor the sum of limits can reach 1.89 times the cluster's capacity, because the limit is not a reservation but a ceiling. The only number that protects the node's total is the request. #### The Cost of Information and the Unseen Item For the placement decision, the system holds forty-two information items: two declarations each for eighteen pods (thirty-six items) and two measurements each for three nodes (six items). These items do not cost the same. The six measurements arrive regularly from the node agent and are fresh. The thirty-six declarations are written once, nobody verifies them, and they quietly go wrong as load changes. Measuring this item is not impossible, only costly: it would require collecting, storing, and comparing a continuous usage series against the declaration for every pod. This lesson does not hand that series to the scheduler; what happens when it is handed over is the subject of this topic's last two lessons. The forty-third item does not exist at all: **actual usage**. The scheduler never sees actual usage in any table in this lesson. All three wrong decisions come from the same gap — piling eighteen pods onto a single node in the no-request setting, missing 557 units of contention in the realistic setting, and leaving seven pods waiting while half-empty nodes sit idle in the inflated setting. In all three, the scheduler's ledger is consistent; what is inconsistent is the ledger and the node. #### Summary - The resource request sets the scheduler's ledger, the limit sets the node's runtime: a pod can be placed and still be throttled or killed. - The three settings' costs are distinct: no request means eighteen pods on one node at 283 percent occupancy, realistic means 640 units of spare request with 557 units of contention, inflated means seven unplaced pods with 3,934 units of waste. - Understatement's cost is contention and cutoffs (2,965 units at half declaration, all eighteen pods throttled); overstatement's cost is unplaced pods and waste (nine pods fail to place at the 2x factor, 4,322 units sit idle). - Even with an exact declaration, one pod still fails to place: even with complete information, the first-fit placement rule produces fragmentation. - Raising the limit factor fourfold drops throttled pods from seventeen to zero and never touches the node's 1,646 units of contention; the sum of limits can reach 1.89 times capacity. - The decision rests on forty-two information items, thirty-six of them declarations that are never refreshed; the forty-third item, actual usage, never appears, and it is the source of all three wrong decisions. #### Next Step This lesson's scheduler asks a single question: does the remaining request fit. Placement has other sides, and none of them concern capacity. Having the verifier run on nodes with fast disks, keeping copies of the same service from piling onto one node, guaranteeing that a surviving copy remains after a zone is lost — each of these adds a new constraint to the scheduler and narrows the set of eligible nodes. The next lesson writes the scheduler as a constraint solver and counts how many pods become unplaceable as constraints are added. ### Scheduling Constraints Source: https://academia.sh/en/courses/orchestration/scheduling-constraints Writing the scheduler as a constraint solver: as label selection, affinity, anti-affinity, and topology spread are added, the eligible node set narrows; unplaceable pods are counted, two constraints render each other unsolvable, and topology spread changes the number of replicas that survive a zone loss. The previous lesson's scheduler asked a single question: does the remaining request suffice. Placement has other sides, and none of them concern capacity. Having the verifier run on nodes with a fast disk, keeping two copies of the same service from piling onto one node, keeping billing on the same node as the verifier, guaranteeing a surviving copy after a zone is lost — each of these turns a node with sufficient capacity into an ineligible one. These constraints turn the scheduler into a constraint solver. The decision stays the same (which pod goes to which node), but it is now made within a search space: for each pod, the eligible node set is computed first, and then one node is chosen from that set. If the set empties out, no decision can be made and the pod does not get placed. This is what this lesson measures: how much the set narrows as constraints are added, and at what point it empties out. The model is nine nodes spread across three zones; the workload is three services of the regional measurement network (the running fiction). No real cluster runs. #### The Constraint Solver and the Eligible Node Set Four kinds of constraint are added cumulatively. Label selection requires one of the node's labels, affinity requires the presence of another service, anti-affinity requires the absence of the same service, and topology spread bounds the replica-count difference across zones. The selection rule is the same as in the previous lesson: the first node in the eligible set. The constraints are written with our own small schema. A node declaration carries two labels: zone and disk type. A pod declaration places a constraint with three fields — the `disk` field requires the node's disk label, the `antiAffinity` field asks for a node with no copy of the same service, the `affinity` field asks for a node where the named service is present. Topology spread belongs to the solver, not the pod, and is tuned with a spread count. The solver reads nothing beyond these fields; whatever it does not read, however real it is on the node, lies outside the decision. ```js // scheduling/constraint.mjs — the scheduler as a CONSTRAINT SOLVER. Not a real cluster, a model. // Constraints are added cumulatively; the shrinking of the eligible node set is measured at each stage. export const ZONE = ["b1", "b2", "b3"]; // SS7: three zones x three nodes, 1500 cpu units per node // SS8: a fast disk only on the first node of each zone export const NODES = ZONE.flatMap((b, i) => [0, 1, 2].map((j) => ({ name: `${b}-d${j}`, zone: b, disk: j === 0 ? "fast" : "standard", capacity: 1500 }))); // SS9: workload — the verifier wants a fast disk, billing wants the verifier's node export const WORKLOAD = [ { service: "verifier", replicas: 6, unit: 650, disk: "fast", antiAffinity: true }, { service: "billing", replicas: 3, unit: 300, affinity: "verifier", antiAffinity: true }, { service: "reading-collector", replicas: 9, unit: 420 }, ]; export const expandPods = (load = WORKLOAD) => load.flatMap((s) => Array.from({ length: s.replicas }, (_, i) => ({ ...s, name: `${s.service}-${i + 1}` }))); // SS10: topology spread — a service's replica-count difference across zones is at most 1 export const SPREAD = 1; export function solve(constraint, load = WORKLOAD, nodes = NODES) { const state = nodes.map((d) => ({ ...d, used: 0, pods: [] })); const zoneCounts = new Map(); // service -> zone -> count const unplaced = []; const eligibleCounts = []; const count = (s, b) => (zoneCounts.get(s) ?? {})[b] ?? 0; for (const k of expandPods(load)) { const eligible = state.filter((d) => { if (d.used + k.unit > d.capacity) return false; if (constraint.label && k.disk && d.disk !== k.disk) return false; if (constraint.antiAffinity && k.antiAffinity && d.pods.some((x) => x.service === k.service)) return false; if (constraint.affinity && k.affinity && !d.pods.some((x) => x.service === k.affinity)) return false; if (constraint.topology) { const min = Math.min(...ZONE.map((b) => count(k.service, b))); if (count(k.service, d.zone) + 1 - min > SPREAD) return false; } return true; }); eligibleCounts.push(eligible.length); if (eligible.length === 0) { unplaced.push(k); continue; } const d = eligible[0]; // first-fit node: the rule never looks back d.used += k.unit; d.pods.push(k); const m = zoneCounts.get(k.service) ?? {}; m[d.zone] = (m[d.zone] ?? 0) + 1; zoneCounts.set(k.service, m); } return { state, unplaced, eligibleCounts, zoneCounts }; } export const distribution = (o, service) => ZONE.map((b) => (o.zoneCounts.get(service) ?? {})[b] ?? 0); const STAGE = [ ["capacity only", {}], ["+ label selection", { label: 1 }], ["+ affinity", { label: 1, affinity: 1 }], ["+ anti-affinity", { label: 1, affinity: 1, antiAffinity: 1 }], ["+ topology spread", { label: 1, affinity: 1, antiAffinity: 1, topology: 1 }], ]; if (process.argv[1].endsWith("constraint.mjs")) { const total = WORKLOAD.reduce((t, s) => t + s.replicas, 0); console.log(`model: ${NODES.length} nodes / ${ZONE.length} zones, ${total} pods, ` + `fast-disk nodes ${NODES.filter((d) => d.disk === "fast").length}`); const s = (x, n = 15) => String(x).padStart(n); console.log(`${"stage".padEnd(22)}${s("avg eligible")}${s("min eligible")}${s("placed")}` + `${s("unplaced")}${s("busy nodes")}`); for (const [ad, constraint] of STAGE) { const o = solve(constraint); const avg = o.eligibleCounts.reduce((t, x) => t + x, 0) / o.eligibleCounts.length; console.log(`${ad.padEnd(22)}${s(avg.toFixed(2))}${s(Math.min(...o.eligibleCounts))}` + `${s(total - o.unplaced.length)}${s(o.unplaced.length)}` + `${s(o.state.filter((d) => d.pods.length > 0).length)}`); } console.log(`\nper-service zone distribution (${ZONE.join("/")}):`); console.log(`${"stage".padEnd(22)}${s("verifier")}${s("billing")}${s("collector", 16)}`); for (const [ad, constraint] of STAGE) { const o = solve(constraint); console.log(`${ad.padEnd(22)}${s(distribution(o, "verifier").join("/"))}` + `${s(distribution(o, "billing").join("/"))}${s(distribution(o, "reading-collector").join("/"), 16)}`); } } ``` ``` model: 9 nodes / 3 zones, 18 pods, fast-disk nodes 3 stage avg eligible min eligible placed unplaced busy nodes capacity only 5.83 3 18 0 7 + label selection 3.83 1 18 0 7 + affinity 3.17 0 15 3 6 + anti-affinity 4.28 0 15 3 6 + topology spread 3.00 0 15 3 6 per-service zone distribution (b1/b2/b3): stage verifier billing collector capacity only 6/0/0 0/3/0 0/7/2 + label selection 2/2/2 3/0/0 4/5/0 + affinity 2/2/2 0/0/0 6/3/0 + anti-affinity 1/1/1 1/1/1 7/2/0 + topology spread 1/1/1 1/1/1 3/3/3 ``` #### The Narrowing Is Not One-Directional The first table shows the expected narrowing: the average number of eligible nodes per pod drops from 5.83 to 3.83 with label selection, then to 3.17 with affinity. The minimum-eligible-nodes column is sharper — label selection shrinks the set to a single node for one pod, and once affinity is added the set **empties out** and three billing pods fail to place. The reason is in the second table: once the verifier's copies fill the three fast-disk nodes two apiece, no room is left on those nodes for billing. The affinity constraint does not ask for room, but it requires room to be left. The next row is the lesson's inverted result. Once the anti-affinity constraint is added, the average eligible-node count does not fall; it rises from 3.17 to **4.28**. The reason the search space widens even as the constraint count grows is this: anti-affinity limits the verifier to one copy per node, so three copies become unplaceable, and the capacity they no longer hold enlarges the eligible set for the pods that follow. The number of placed pods stays at fifteen, but its composition changes: billing's three pods now get placed, and three of the verifier's copies do not. **One constraint rescues one service while dropping another**, and the total count does not show it. In the topology-spread row the placed count is again fifteen, and the average eligible set is 3.00. The only thing that changes is the last column: the reading collector's zone distribution goes from 7/2/0 to 3/3/3. This constraint drops no pod at all; it only reshapes placement — and its payoff only becomes visible once a zone is lost. #### Zone Loss, Unsolvability, and a Wrong Label ```js // scheduling/zone.mjs — topology spread's payoff on a zone loss, constraints that make each // other unsolvable, and an incorrect label declaration. The model comes from the previous file. import { ZONE, NODES, WORKLOAD, solve, distribution } from "./constraint.mjs"; const FULL = { label: 1, affinity: 1, antiAffinity: 1, topology: 1 }; const NO_TOPOLOGY = { label: 1, affinity: 1, antiAffinity: 1 }; const s = (x, n = 12) => String(x).padStart(n); const SERVICES = ["verifier", "billing", "reading-collector"]; // SS12: a zone loss means all three nodes in that zone become unreachable at once. console.log("replicas surviving a zone loss (placed / after the loss):"); console.log(`${"setting".padEnd(16)}${"loss".padEnd(8)}` + ["verifier", "billing", "collector"].map((x) => s(x, 14)).join("") + s("total remaining", 18)); for (const [ad, constraint] of [["no topology", NO_TOPOLOGY], ["with topology", FULL]]) { const o = solve(constraint); for (const loss of ZONE) { const remaining = SERVICES.map((sv) => { const d = distribution(o, sv), i = ZONE.indexOf(loss); return [d.reduce((t, x) => t + x, 0), d.reduce((t, x, j) => t + (j === i ? 0 : x), 0)]; }); console.log(`${ad.padEnd(16)}${loss.padEnd(8)}` + remaining.map(([t, k]) => s(`${t} / ${k}`, 14)).join("") + s(remaining.reduce((t, [, k]) => t + k, 0), 18)); } } // Constraints making each other unsolvable: verifier has 6 replicas, 3 fast-disk nodes, each node // takes at most one replica. The third row is unsolvable; the last row widens the node label. const FAST6 = NODES.map((d, i) => ({ ...d, disk: i % 3 < 2 ? "fast" : "standard" })); const VERIFIER_ONLY = WORKLOAD.filter((x) => x.service === "verifier"); console.log("\nverifier at 6 replicas, constraint combinations:"); console.log(`${"combination".padEnd(38)}${s("fast nodes")}${s("placed")}${s("unsolvable")}`); for (const [ad, constraint, dgm] of [ ["unconstrained", {}, NODES], ["label selection only", { label: 1 }, NODES], ["anti-affinity only", { antiAffinity: 1 }, NODES], ["label + anti-affinity", { label: 1, antiAffinity: 1 }, NODES], ["label + anti-affinity, 6 fast nodes", { label: 1, antiAffinity: 1 }, FAST6], ]) { const o = solve(constraint, VERIFIER_ONLY, dgm); console.log(`${ad.padEnd(38)}${s(dgm.filter((d) => d.disk === "fast").length)}` + `${s(6 - o.unplaced.length)}${s(o.unplaced.length)}`); } // SS11: a label is a declaration, not a measurement. What happens if a node's label is wrong. console.log("\nwhen a label declaration is corrupted (full constraint set):"); console.log(`${"declaration".padEnd(34)}${s("placed", 14)}${s("unplaced", 14)}${s("busy nodes", 14)}`); for (const [ad, dgm] of [ ["correct", NODES], ["b2-d0 missing its fast label", NODES.map((d) => d.name === "b2-d0" ? { ...d, disk: "standard" } : d)], ["b3-d1 mistakenly labeled fast", NODES.map((d) => d.name === "b3-d1" ? { ...d, disk: "fast" } : d)], ]) { const o = solve(FULL, WORKLOAD, dgm); console.log(`${ad.padEnd(34)}${s(18 - o.unplaced.length, 14)}${s(o.unplaced.length, 14)}` + `${s(o.state.filter((d) => d.pods.length > 0).length, 14)}`); } const labelItems = NODES.length * 2; const podItems = WORKLOAD.reduce((t, x) => t + x.replicas * ((x.disk ? 1 : 0) + (x.antiAffinity ? 1 : 0) + (x.affinity ? 1 : 0)) + 1, 0); console.log(`\ndeclarations read by the constraint solver: ${labelItems} node labels + ` + `${podItems} pod constraints = ${labelItems + podItems}; none of them are measurements`); ``` ``` replicas surviving a zone loss (placed / after the loss): setting loss verifier billing collector total remaining no topology b1 3 / 2 3 / 2 9 / 2 6 no topology b2 3 / 2 3 / 2 9 / 7 11 no topology b3 3 / 2 3 / 2 9 / 9 13 with topology b1 3 / 2 3 / 2 9 / 6 10 with topology b2 3 / 2 3 / 2 9 / 6 10 with topology b3 3 / 2 3 / 2 9 / 6 10 verifier at 6 replicas, constraint combinations: combination fast nodes placed unsolvable unconstrained 3 6 0 label selection only 3 6 0 anti-affinity only 3 6 0 label + anti-affinity 3 3 3 label + anti-affinity, 6 fast nodes 6 6 0 when a label declaration is corrupted (full constraint set): declaration placed unplaced busy nodes correct 15 3 6 b2-d0 missing its fast label 13 5 5 b3-d1 mistakenly labeled fast 16 2 6 declarations read by the constraint solver: 18 node labels + 21 pod constraints = 39; none of them are measurements ``` #### Three Measurements **Zone loss.** Without topology spread, the outcome depends on which zone is lost: if b1 goes down, only two of the nine reading collectors remain; if b3 goes down, all nine remain. The total surviving replica count swings between six and thirteen, and nobody chose which number would come out. With topology spread, all three rows are identical: whichever zone goes down, six collectors and ten replicas in total survive. **What the constraint buys is not the average but the worst case** — the worst case rises from six to ten, the best case falls from thirteen to ten. **Unsolvability.** The second table shows in numbers that two constraints, harmless alone, are unsolvable together. Label selection alone places all six replicas, anti-affinity alone places all six replicas, and the two together place only three, leaving three replicas unsolvable. What collides is not the constraints themselves but a number: anti-affinity wants as many separate nodes as there are replicas, while label selection narrows the eligible node count to three. The last row gives the fix in numbers too — once the fast-disk node count is raised to six, all six of the six replicas get placed. **The unsolvability is not in the declaration; it is in the cluster's hardware.** **A wrong label.** The third table exposes the kind of information the constraints rest on. None of the thirty-nine declarations is a measurement: eighteen node labels were written by a person, and so were the twenty-one pod constraints. When a node's fast-disk label goes missing, the placed count drops from fifteen to thirteen; the error is visible, because something fails to place. The real problem is in the next row. When a node without a fast disk is mistakenly labeled fast, the placed count **rises** to sixteen. In the solver's ledger, the situation has improved; in reality, a verifier copy is running on a slow-disk node, and no counter reports it. The scheduler does not measure the node's disk; it reads its label. The source of the wrong decision is, once again, not the information item itself but the fact that **the item is a declaration**. #### Summary - As constraints are added, the average number of eligible nodes per pod drops from 5.83 to 3.83, then to 3.17; under the affinity constraint the set empties out and three billing pods fail to place. - Adding a constraint does not always narrow the search space: the anti-affinity constraint raises the average from 3.17 to 4.28, because the capacity the three unplaceable verifier copies no longer hold is left over for the pods that follow. - The same placed count can hide different compositions: while fifteen pods get placed, one constraint rescues three of billing's pods while dropping three of the verifier's. - Topology spread fixes the worst case without dropping a single pod: surviving replicas on a zone loss move from the 6–13 range to a fixed ten, and the reading collector's distribution becomes 3/3/3 instead of 7/2/0. - Two constraints are harmless alone and unsolvable together: six replicas, three fast-disk nodes, and a one-copy-per-node rule yield three unsolvable pods; the unsolvability ends once the fast-node count rises to six. - None of the thirty-nine items the solver reads is a measurement; a mis-written label raises the placed count from fifteen to sixteen and makes the error invisible. #### Next Step The constraints so far write what the pod wants from the node. The reverse direction was never written: which pod the node will accept. In the regional measurement network, the nightly batch job consumes the whole disk and all the memory, and a service copy that lands on the same node slows down for the whole night. The way to set these nodes aside for the batch job is not to write one more constraint into every pod; it is to place a taint on the node itself and let only the pods that carry a matching toleration arrive. The next lesson measures this separation and the idle ratio of the set-aside capacity. ### Taints and Tolerations Source: https://academia.sh/en/courses/orchestration/taints-and-tolerations Setting up node separation with taints and tolerations: how many nodes the general workload fits into once tainted nodes are set aside, the idle ratio of the set-aside capacity, measuring the taint's three effects (placement, continuing to run, eviction) separately, and how many pods a wrongly given toleration drops onto the wrong node. All of the previous lesson's constraints looked in the same direction: the pod wanted something from the node. The reverse direction was never written. In the regional measurement network (the running fiction), the nightly batch job saturates the disk and the memory; a service copy that lands on the same node slows down for the whole night. The way to set these nodes aside for the batch job is not to write a "do not go to these nodes" constraint into each of the eighteen pods — that solution has to be rewritten for every new service, and one forgotten declaration breaks the separation. A **taint** is placed on the node and flips the default: the node rejects every pod that does not say otherwise. A **toleration** is written on the pod and lifts only that rejection. For a pod to arrive, both declarations must exist, and yet the two together are not enough — a toleration permits, it does not attract. What still carries a pod to the set-aside node is the previous lesson's label selection. What is measured is a model; no real cluster runs. #### The Condition the Node Sets The model is nine nodes, eighteen general pods, and six batch pods. The tainted-node count is scanned from zero to five, and at each step three things are counted: how many nodes the general workload fits into, how many pods fail to place, and how much of the set-aside capacity sits idle. ```js // scheduling/taint.mjs — node separation: taints and tolerations. Not a real cluster, a model. // A taint is placed on the node, a toleration is written on the pod; a toleration permits, it does not attract. export const NODE_COUNT = 9, CAPACITY = 1500; // SS13: nine nodes, 1500 units per node export const TAINT_ORDER = [0, 3, 6, 1, 4]; // tainting order: one from each zone first export const makeNodes = (tainted, effect = "block") => Array.from({ length: NODE_COUNT }, (_, i) => ({ name: `d${i}`, capacity: CAPACITY, used: 0, pods: [], taint: TAINT_ORDER.slice(0, tainted).includes(i) ? { key: "batch", effect } : null, })); // SS14: general workload and the nightly batch job (regional measurement network, the running fiction) export const GENERAL = [["reading-collector", 9, 420], ["verifier", 6, 650], ["billing", 3, 300]]; export const BATCH = [["nightly-batch", 6, 700]]; export const makePods = (list, extra = {}) => list.flatMap(([name, n, unit]) => Array.from({ length: n }, (_, i) => ({ name: `${name}-${i + 1}`, service: name, unit, ...extra }))); // The taint's first effect: a pod with no toleration cannot be placed on a tainted node. // The third effect (deprioritize) pushes a tainted node to last resort; it does not forbid it. export function place(podList, nodes) { const unplaced = [], misplaced = []; for (const k of podList) { const eligible = nodes.filter((d) => { if (d.used + k.unit > d.capacity) return false; if (k.target && d.taint?.key !== k.target) return false; return d.taint === null || k.toleration === true || d.taint.effect === "deprioritize"; }); const preferred = eligible.filter((d) => d.taint === null || d.taint.effect !== "deprioritize" || k.toleration === true); const d = (preferred.length > 0 ? preferred : eligible)[0]; if (d === undefined) { unplaced.push(k); continue; } d.used += k.unit; d.pods.push(k); if (d.taint !== null && k.service !== "nightly-batch") misplaced.push(k); } return { nodes, unplaced, misplaced }; } export const occupied = (nodes) => nodes.filter((d) => d.pods.length > 0).length; export const TOTAL_GENERAL = GENERAL.reduce((t, [, n]) => t + n, 0); if (process.argv[1].endsWith("taint.mjs")) { const gb = GENERAL.reduce((t, [, n, b]) => t + n * b, 0); const tb = BATCH.reduce((t, [, n, b]) => t + n * b, 0); console.log(`model: ${NODE_COUNT} nodes x ${CAPACITY} units = ${NODE_COUNT * CAPACITY}; ` + `general workload ${gb} units / ${TOTAL_GENERAL} pods, nightly batch job ${tb} units / 6 pods`); console.log("SS15: the batch job runs 4 hours a day; a set-aside node waits the day's full 24 hours\n"); const s = (x, n = 16) => String(x).padStart(n); console.log(`${"tainted".padEnd(10)}${s("general placed")}${s("unplaced")}` + `${s("general nodes")}${s("batch placed")}${s("reserved h/day")}${s("idle ratio")}`); for (let k = 0; k <= 5; k++) { const d = makeNodes(k); const g = place(makePods(GENERAL), d); const generalNodes = occupied(d); const t = place(makePods(BATCH, { toleration: true, target: "batch" }), d); const batchNodes = d.filter((x) => x.pods.some((y) => y.service === "nightly-batch")).length; const reserved = k * 24, used = batchNodes * 4; console.log(`${String(k).padEnd(10)}${s(TOTAL_GENERAL - g.unplaced.length)}` + `${s(g.unplaced.length)}${s(generalNodes)}${s(6 - t.unplaced.length)}${s(reserved)}` + `${s(k === 0 ? "-" : (100 * (1 - used / reserved)).toFixed(1) + "%")}`); } } ``` ``` model: 9 nodes x 1500 units = 13500; general workload 8580 units / 18 pods, nightly batch job 4200 units / 6 pods SS15: the batch job runs 4 hours a day; a set-aside node waits the day's full 24 hours tainted general placed unplaced general nodes batch placed reserved h/day idle ratio 0 18 0 7 0 0 - 1 18 0 7 2 24 83.3% 2 18 0 7 4 48 83.3% 3 15 3 6 6 72 83.3% 4 13 5 5 6 96 87.5% 5 11 7 4 6 120 90.0% ``` #### Two Bills for the Separation The first row gives, in numbers, the difference between a toleration and label selection. With no tainted node, none of the batch job's six pods get placed, because the batch pods carry a target that seeks a tainted node, and no such node exists. Their tolerations sit unused; a toleration alone carries no pod anywhere. The first bill belongs to the general workload. With one or two nodes set aside, the general workload still fits into seven nodes and all eighteen pods get placed; once three nodes are set aside, it is squeezed into six and three pods fail to place. Once five nodes are set aside, the unplaced count rises to seven. Setting nodes aside is an isolation decision, and its price is paid directly from the general workload's capacity. The second bill is in the set-aside capacity itself. The batch job runs four hours a day, while a set-aside node waits the full twenty-four hours: the idle ratio is 83.3 percent in the best case. This ratio is not the inefficiency of separation but its definition — separation sets aside a node, not time. Once more than three nodes are set aside, the ratio only worsens (87.5 percent, then 90 percent), because the batch job never uses more than three nodes and every extra set-aside node spends its entire day idle. With three nodes set aside, 72 node-hours a day are reserved while 12 node-hours are used. These two bills must be read together with the declaration cost of the separation. Building the same separation on the pod side requires writing a constraint into each of the eighteen general pods; building it with a taint takes three node taints and six tolerations — nine items in all. The difference is not only in the count but in **who can forget**: in the pod-side setup, a new service that forgets to write the constraint silently breaks the separation; in the node-side setup, a service that forgets it only fails to enter the set-aside nodes. A separation built in the wrong direction can be broken again by every new service. The idle ratio also comes down to a single number. Because a set-aside node waits the entire day, the ratio follows directly from the width of the window: 83.3 percent for a four-hour window, 66.7 percent for an eight-hour window, 95.8 percent for a one-hour window. The decision to set a node aside is therefore not a resource decision but a **window decision**; the shorter the batch job's duration, the larger the set-aside node's cost grows in proportion. #### The Taint's Three Effects and a Wrong Toleration A taint does not do just one thing. The same taint can be placed with three separate effects: blocking a new pod from being placed, evicting pods that are already running too, or only lowering the preference order. The three are measured separately, and then what happens when a toleration is given incorrectly is scanned. The distinction among the three effects is not merely a setting: the first binds future placements, the second binds pods already running, the third binds only ordering. The same separation intent yields three separate outcomes, and the difference between them is only visible once it is counted. ```js // scheduling/toleration.mjs — the taint's three effects, one at a time, and the cost of a wrongly given toleration. import { GENERAL, BATCH, TAINT_ORDER, TOTAL_GENERAL, makeNodes, makePods, place } from "./taint.mjs"; const TAINTED = 3; // SS16: three nodes are set aside for the nightly batch job const generalPods = (tolerating = 0) => makePods(GENERAL).map((k, i) => (i < tolerating ? { ...k, toleration: true } : k)); const batchPods = () => makePods(BATCH, { toleration: true, target: "batch" }); const s = (x, n = 16) => String(x).padStart(n); // Second effect: if the taint is placed AFTER pods are already running, those without a toleration are evicted. function taintAfterward(tolerating = 0) { const d = makeNodes(0); place(generalPods(tolerating), d); const set = new Set(TAINT_ORDER.slice(0, TAINTED)); const evicted = []; d.forEach((x, i) => { if (!set.has(i)) return; x.taint = { key: "batch", effect: "evict" }; evicted.push(...x.pods.filter((k) => k.toleration !== true)); x.pods = x.pods.filter((k) => k.toleration === true); x.used = x.pods.reduce((t, k) => t + k.unit, 0); }); const requeued = place(evicted, d); const retained = d.filter((x, i) => set.has(i)) .reduce((t, x) => t + x.pods.filter((k) => k.toleration === true).length, 0); return { evicted: evicted.length, unplaced: requeued.unplaced.length, retained }; } console.log(`the taint's three effects, one at a time (${TAINTED} nodes set aside, toleration given correctly):`); console.log(`${"effect".padEnd(24)}${s("general placed")}${s("misplaced")}${s("evicted")}` + `${s("unplaced")}`); for (const effect of ["block", "deprioritize"]) { const d = makeNodes(TAINTED, effect); const g = place(generalPods(), d); console.log(`${(effect === "block" ? "placement block" : "deprioritization").padEnd(24)}` + `${s(TOTAL_GENERAL - g.unplaced.length)}${s(g.misplaced.length)}${s(0)}${s(0)}`); } const t2 = taintAfterward(); console.log(`${"eviction block".padEnd(24)}` + `${s(TOTAL_GENERAL - t2.unplaced)}${s(0)}${s(t2.evicted)}${s(t2.unplaced)}`); // SS17: a toleration spreads by copy-paste. The count of wrongly given tolerations is scanned. console.log(`\nwrongly given toleration (${TAINTED} nodes set aside, placement block):`); console.log(`${"general tolerating".padEnd(20)}${s("misplaced")}${s("general placed")}` + `${s("batch placed")}${s("retained")}`); for (const n of [0, 2, 4, 6, 9]) { const d = makeNodes(TAINTED); const g = place(generalPods(n), d); const t = place(batchPods(), d); const retained = taintAfterward(n); console.log(`${String(n).padEnd(20)}${s(g.misplaced.length)}` + `${s(TOTAL_GENERAL - g.unplaced.length)}${s(6 - t.unplaced.length)}` + `${s(retained.retained)}`); } // SS18: declarations read for the decision. The taint sits on the node, the toleration on the pod; both are declarations. const items = TAINTED + TOTAL_GENERAL + 6; console.log(`\ndeclarations read: ${TAINTED} node taints + ${TOTAL_GENERAL + 6} pod toleration ` + `fields = ${items}; no item measures their correctness`); ``` ``` the taint's three effects, one at a time (3 nodes set aside, toleration given correctly): effect general placed misplaced evicted unplaced placement block 15 0 0 0 deprioritization 18 3 0 0 eviction block 15 0 8 3 wrongly given toleration (3 nodes set aside, placement block): general tolerating misplaced general placed batch placed retained 0 0 15 6 0 2 2 18 4 2 4 3 18 4 3 6 3 18 4 3 9 3 18 4 3 declarations read: 3 node taints + 24 pod toleration fields = 27; no item measures their correctness ``` #### The Three Effects, One at a Time The **placement block** binds only the future: fifteen general pods get placed, not a single general pod sits on the set-aside nodes, and nothing is evicted. The separation is clean, and its price is three unplaced pods. The **eviction block** binds today. When the taint is placed after the pods are already running, eight pods are evicted from their nodes; when they try to place back into the remaining capacity, three of them find no room. The same final state (fifteen pods standing) is reached by both paths, but the second path costs eight pods a restart. **When a taint is placed matters as much as which effect it is placed with.** **Deprioritization** blocks nothing. All eighteen of the eighteen pods get placed, but three general pods land on the set-aside nodes. A soft taint works only until capacity gets tight, and it dissolves on its own exactly when capacity gets tight — exactly when the separation is needed. **A separation needs to be an enforcement, not a note.** Read together, the three effects' numbers also clarify the scope of the delegated decision. The authority handed to the system covers not only "where to place" but also "whom to evict" and "whom to deprioritize." All three rest on a single field — the effect of the taint on the node — and when this field is chosen wrongly, the result is either a separation that stays on paper (deprioritization) or a sudden wave of eight pod restarts (eviction block). What produces the difference between the two is not the state of the cluster but a single-word declaration. #### A Wrong Toleration Makes the Error Look Like an Improvement The final table counts the price of a toleration that spreads by copy-paste. Once an unnecessary toleration is written onto two general pods, those pods land on the set-aside nodes and two of the nightly batch job's six pods become unplaceable. Once four or more pods carry the toleration, the number of foreign pods on the set-aside nodes saturates at three. This table's dangerous column is the middle one. Once the wrong toleration is given, the general placed count **rises** from fifteen to eighteen. No indicator that looks at the placement decision reports a problem: more pods have been placed, no pod is left unsolvable, the cluster's occupancy has gone up. The only thing broken is the separation itself, and it only becomes visible at night, when the batch job runs with pods missing. The last column adds a second loss: once the eviction block is placed, these pods are **not** evicted, because their tolerations protect them. A wrong declaration also stands in the way of the correction mechanism. The decision rests on twenty-seven items of information: three node taints and the toleration field of twenty-four pods. None of these items is a measurement, and there is no single counter that tests whether a toleration is genuinely necessary. The system places the wrong pod on the set-aside node not because it fails to know something is wrong, but because **no item exists that it could know it from**. #### Summary - A taint is written on the node, a toleration on the pod; a toleration permits, it does not attract. With no tainted node, none of the batch job's six pods get placed, because what carries them is label selection. - The separation's first bill is on the general workload: once three nodes are set aside, the general workload squeezes into six nodes and three pods fail to place; once five nodes are set aside, seven pods fail to place. - The second bill is in the set-aside capacity: reserving twenty-four hours for a four-hour window puts the idle ratio at 83.3 percent in the best case, rising to 90 percent as the set-aside node count grows. - The three effects measure three separate things: the placement block leaves zero foreign pods on the set-aside node, the eviction block evicts eight pods and three of them fail to re-place, deprioritization does not block the three foreign pods at all. - An unnecessary toleration lets three foreign pods onto the set-aside nodes and makes two of the nightly batch job's pods unplaceable; because the placed count rises from fifteen to eighteen, the error looks like an improvement. - The decision rests on twenty-seven declaration items, and none measures whether a toleration is necessary; a wrong toleration also blocks the correction, because it protects the pod under the eviction effect. #### Next Step Every decision so far has been made at the moment of placement: a pod lands on a node and stays there. Resource pressure, by contrast, is born after placement. Once a node's memory fills up, something has to be evicted, and the decision-maker here is not the scheduler but the node itself. Which pod gets evicted depends on how far below or above its request it is running, and on a priority class written onto the pod. The next lesson models the eviction order, and counts how many pods a priority addition affects when making room, and how many times the chain grows longer. ### Eviction and Priorities Source: https://academia.sh/en/courses/orchestration/eviction-and-priorities Modeling behavior under resource pressure: computing the eviction order under memory pressure for pods below their request, above their request, and with no request at all; three sort rules that evict different pods; how many pods a priority-class addition affects when making room and how many times the chain grows longer; and the system evicting the wrong pod because of the item it cannot see. Every decision so far has been made at the moment of placement: a pod lands on a node and stays there. Resource pressure, by contrast, is born after placement, and the decision-maker is no longer the scheduler. Once a node's memory fills up, something has to be evicted, the node makes the eviction decision, and its input is the same two numbers from the first lesson: the declared request and the measured usage. Here, for the first time, the delegated decision is **irreversible**. A wrongly placed pod fails to place and keeps waiting; a wrongly evicted pod is stopped while it is running. This lesson measures two things: which pod gets evicted under pressure, and, once a priority class is added, how many pods are affected by making room for a high-priority pod. What is measured is a model; the workload is the services of the regional measurement network (the running fiction). #### Pressure Does Not Show in the Scheduler's Ledger The model has two nodes and seventeen pods. The nightly batch job's two pods declare no request at all and use 1,400 MiB; billing and the reading collector run a little above their request, the verifier a little below. Eviction is tried with three separate sort rules. ```js // scheduling/eviction.mjs — eviction order under memory pressure. What gets built is a MODEL. // When deciding who to evict, a node sees only the declared request and the measured usage. export const MEMORY = 4096, NODE_COUNT = 2; export const THRESHOLD = 0.9; // SS19: memory pressure starts once occupancy exceeds 90% const SEED = 20260803; // name, count, request, actual/base factor, declared priority, actual criticality (unseen by the system) export const SERVICE = [ ["billing", 3, 512, 1.15, 900, 1], ["verifier", 4, 704, 0.90, 700, 2], ["reading-collector", 6, 384, 1.10, 500, 2], ["nightly-batch", 2, 0, 0, 100, 3], ["archive-scan", 2, 256, 1.00, 200, 3], // SS21: a low-priority service that declares a request ]; export const BATCH_ACTUAL = 1400; // SS20: a batch pod that declares no request uses 1400 MiB export function workload(billingRequest = 512) { let s = SEED % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; return SERVICE.flatMap(([name, n, req, factor, pri, crit]) => Array.from({ length: n }, (_, i) => { const rq = name === "billing" ? billingRequest : req; const a = factor === 0 ? BATCH_ACTUAL : Math.round(req * factor * (0.95 + 0.1 * r())); return { name: `${name}-${i + 1}`, service: name, request: rq, actual: a, priority: pri, critical: crit }; })); } export const place = (pods, n = NODE_COUNT) => { const d = Array.from({ length: n }, (_, i) => ({ name: `d${i}`, request: 0, pods: [] })); for (const k of pods) { const x = d.find((y) => y.request + k.request <= MEMORY); if (x !== undefined) { x.request += k.request; x.pods.push(k); } } return d; }; export const usage = (d) => d.pods.reduce((t, k) => t + k.actual, 0); // Three sort rules. Whoever sorts first gets evicted first. export const RULE = { "highest usage": (a, b) => b.actual - a.actual, "request gap": (a, b) => (b.actual - b.request) - (a.actual - a.request), "priority + request gap": (a, b) => a.priority - b.priority || (b.actual - b.request) - (a.actual - a.request), }; export function evict(node, sortFn) { const target = MEMORY * THRESHOLD, evicted = []; const order = [...node.pods].sort(sortFn); while (usage(node) > target && order.length > 0) { const k = order.shift(); node.pods = node.pods.filter((x) => x !== k); evicted.push(k); } return evicted; } if (process.argv[1].endsWith("eviction.mjs")) { const d = place(workload()); console.log(`model: ${NODE_COUNT} nodes x ${MEMORY} MiB, ${workload().length} pods ` + `(seed ${SEED}); pressure threshold ${MEMORY * THRESHOLD} MiB`); d.forEach((x) => console.log(` ${x.name}: ${x.pods.length} pods, request ${x.request} MiB, ` + `actual ${usage(x)} MiB, occupancy ${((usage(x) / MEMORY) * 100).toFixed(1)}%`)); const s = (x, n = 17) => String(x).padStart(n); console.log(`\neviction order on d0 (until it drops to ${(MEMORY * THRESHOLD).toFixed(0)} MiB):`); console.log(`${"rule".padEnd(34)}${s("evicted")}${s("freed MiB")}${s("critical loss")}` + `${s("occupancy left")}`); for (const [ad, sortFn] of Object.entries(RULE)) { const x = place(workload())[0]; const c = evict(x, sortFn); console.log(`${ad.padEnd(34)}${s(c.length)}${s(c.reduce((t, k) => t + k.actual, 0))}` + `${s(c.filter((k) => k.critical === 1).length)}` + `${s(((usage(x) / MEMORY) * 100).toFixed(1) + "%")}`); } const x = place(workload(640))[0]; const c = evict(x, RULE["request gap"]); console.log(`${"request gap, billing request 640".padEnd(34)}${s(c.length)}` + `${s(c.reduce((t, k) => t + k.actual, 0))}${s(c.filter((k) => k.critical === 1).length)}` + `${s(((usage(x) / MEMORY) * 100).toFixed(1) + "%")}`); console.log(`\nevicted pods (rule: request gap): ` + place(workload()).slice(0, 1).flatMap((y) => evict(y, RULE["request gap"])) .map((k) => `${k.name}(${k.actual}/${k.request})`).join(", ")); } ``` ``` model: 2 nodes x 4096 MiB, 17 pods (seed 20260803); pressure threshold 3686.4 MiB d0: 9 pods, request 4032 MiB, actual 6898 MiB, occupancy 168.4% d1: 8 pods, request 3136 MiB, actual 3266 MiB, occupancy 79.7% eviction order on d0 (until it drops to 3686 MiB): rule evicted freed MiB critical loss occupancy left highest usage 3 3452 0 84.1% request gap 3 3401 1 85.4% priority + request gap 3 3224 0 89.7% request gap, billing request 640 2 2800 0 89.7% evicted pods (rule: request gap): nightly-batch-1(1400/0), nightly-batch-2(1400/0), billing-2(601/512) ``` #### Three Sort Rules, Three Separate Victims The first two lines shown above explain why the decision falls to the node. In the scheduler's ledger, d0 is full and consistent: 4,032 of the 4,096 MiB has been allocated, and no further pod is accepted for the remaining 64 MiB. Actual usage, however, is 6,898 MiB — 168.4 percent. The entire gap comes from the two pods that declare no request. The ledger is correct, the node is under pressure, and the component keeping the ledger has no idea. The three rules evict three separate pods. The **highest-usage** rule takes the two batch pods and one verifier; it frees 3,452 MiB and loses no critical pod. The **request-gap** rule — which looks at how far usage exceeds the request — evicts billing after the two batch pods. The listing at the bottom shows why: billing declared 512 MiB and used 601, an 89 MiB overshoot; the reading collector declared 384 and used 422, a 38 MiB overshoot. The rule protects not the pod that declared more honestly, but the one whose declaration sat closer to its own request. Once the priority class is added, the rule is fixed: one more declaration item per pod is added (seventeen items in all), and the critical loss drops from one to zero. The memory freed is 3,224 MiB instead of 3,401 — less — but in exchange, what gets evicted are the lowest-priority pods. The fourth line gives the price of this information being a **declaration**. If billing's request had been written as 640 instead of 512 — with nothing else changing, only the declaration inflated — the pod would appear to be under its request and would drop to the back of the eviction order: two evictions, zero critical loss. Inflating the request was capacity waste in the first lesson; here it is a **protection device**, and the system cannot tell the two apart. #### Making Room and the Chain Pressure is not one-directional. When a high-priority pod arrives and no node has room, the system makes room by its own hand: it evicts lower-priority pods. An evicted pod does not vanish; it goes back into the queue and can, in turn, evict one lower than itself. That is how the chain grows. ```js // scheduling/make-room.mjs — evicting a low-priority pod to make room for a high-priority one, and the resulting chain. import { MEMORY, workload, place } from "./eviction.mjs"; // SS22: urgent billing pods arriving at midnight: priority 1000, request 700 MiB const URGENT = (n) => Array.from({ length: n }, (_, i) => ({ name: `urgent-billing-${i + 1}`, service: "urgent-billing", request: 700, actual: 700, priority: 1000, critical: 1, depth: 0 })); const TURN_LIMIT = 60; // SS23: the chain is cut off at this turn count // SS24: two node-selection rules — the node that evicts the fewest pods, or the first node that fits. function makeRoom(incoming, rule) { const d = place(workload()); const queue = [...incoming]; let evictions = 0, chained = 0, deepest = 0, turn = 0; const unplaced = []; while (queue.length > 0 && turn++ < TURN_LIMIT) { const k = queue.shift(); const free = (x) => MEMORY - x.request; const direct = d.find((x) => free(x) >= k.request); if (direct !== undefined) { direct.request += k.request; direct.pods.push(k); continue; } // For each node: how many pods must be evicted to make room. Only LOWER priority can be evicted. const plans = d.map((x) => { const candidates = x.pods.filter((p) => p.priority < k.priority) .sort((a, b) => a.priority - b.priority || a.request - b.request); let freed = free(x); const evicted = []; for (const p of candidates) { if (freed >= k.request) break; freed += p.request; evicted.push(p); } return { x, evicted, ok: freed >= k.request }; }).filter((p) => p.ok); if (plans.length === 0) { unplaced.push(k); continue; } const p = rule === "fewest evicted" ? plans.reduce((a, b) => (b.evicted.length < a.evicted.length ? b : a)) : plans[0]; for (const c of p.evicted) { p.x.pods = p.x.pods.filter((y) => y !== c); p.x.request -= c.request; evictions++; if (k.depth > 0) chained++; c.depth = (k.depth ?? 0) + 1; deepest = Math.max(deepest, c.depth); queue.push(c); } p.x.request += k.request; p.x.pods.push(k); } const placed = incoming.filter((g) => d.some((x) => x.pods.includes(g))).length; return { evictions, chained, deepest, unplaced: unplaced.length, placed, turn: turn - 1 }; } const s = (x, n = 16) => String(x).padStart(n); console.log("making room: the urgent pod count is scanned (free capacity d0 64 MiB, d1 960 MiB)"); console.log(`${"rule".padEnd(16)}${"urgent".padEnd(8)}${s("urgent placed")}${s("evicted")}` + `${s("chained")}${s("deepest")}${s("unplaced")}`); for (const rule of ["fewest evicted", "first fit"]) for (const n of [1, 2, 3, 4, 5]) { const o = makeRoom(URGENT(n), rule); console.log(`${rule.padEnd(16)}${String(n).padEnd(8)}${s(o.placed)}${s(o.evictions)}` + `${s(o.chained)}${s(o.deepest)}${s(o.unplaced)}`); } // The item the system does not see: actual criticality. The criticality mix of the evicted pods. const o = makeRoom(URGENT(3), "first fit"); console.log(`\nfor three urgent pods, the 'first fit' rule: ${o.evictions} evictions, ${o.chained} chained, ` + `${o.unplaced} pods stranded; under the 'fewest evicted' rule, ` + `${makeRoom(URGENT(3), "fewest evicted").evictions} evictions`); console.log(`declared priority items: 17 pods + the incoming urgent pods; measured criticality items: 0`); ``` ``` making room: the urgent pod count is scanned (free capacity d0 64 MiB, d1 960 MiB) rule urgent urgent placed evicted chained deepest unplaced fewest evicted 1 1 0 0 0 0 fewest evicted 2 2 2 0 1 2 fewest evicted 3 3 4 0 1 4 fewest evicted 4 4 6 0 1 6 fewest evicted 5 5 10 2 2 7 first fit 1 1 0 0 0 0 first fit 2 2 6 2 2 2 first fit 3 3 9 4 2 4 first fit 4 4 12 6 2 6 first fit 5 5 15 8 2 7 for three urgent pods, the 'first fit' rule: 9 evictions, 4 chained, 4 pods stranded; under the 'fewest evicted' rule, 4 evictions declared priority items: 17 pods + the incoming urgent pods; measured criticality items: 0 ``` #### Same Outcome, Three Times the Disruption A single urgent pod evicts nothing; the free request headroom is enough. From the second pod on, making room begins, and the two node-selection rules produce the same outcome at very different cost. The **fewest-evicted** rule makes two evictions for two urgent pods; the **first-fit** rule makes six evictions for the same two pods, two of which are chained. At five urgent pods the gap is ten evictions against fifteen, and the stranded-pod count is seven under both rules. **The outcome is the same, the disruption is three times as large.** Node selection looks like a placement detail, and it directly determines how many pods get restarted. The chained column shows the delegation's second cost: when an evicted pod returns to the queue, it triggers an eviction of its own, and the chain reaches a depth of at most two. The depth staying at two is not a design success but an accident. The ladder has six distinct priority values, and each rung can only evict rungs below it; the chain is cut off at the second step because the pods on the ladder's bottom rung declare no request, and evicting them frees no room in the ledger at all. If the same model had a realistic request written onto the bottom pods, the chain would run one step longer — meaning the chain's length is determined as much by the declaration itself as by the priority design. The third and quietest result is the stranded pods. While five urgent pods get placed, seven pods are left without a node, and they never come back, because no evictable pod with lower priority remains. The nightly batch job's pods never make room in this table at all: because they declare no request, evicting them frees zero MiB in the ledger. In reality 1,400 MiB is freed, but the room-making decision looks at the request, not at usage. The decision rests on the seventeen pods' priority declarations and the priorities of the incoming pods. The measured criticality item is zero. The system evicts billing not because it thinks billing is unimportant, but because **it holds no item at all about its importance**; a priority class is a declaration that fills this gap, and no measurement tests whether that declaration is true. When every pod is written with the highest priority, the ordering disappears entirely and the system falls back to the request gap. #### Summary - Memory pressure does not show in the scheduler's ledger: d0's ledger has allocated 4,032 of 4,096 MiB and is consistent, actual usage is 168.4 percent, and the gap comes from the two pods that declare no request. - Three sort rules evict three separate pods: the highest-usage rule frees 3,452 MiB and loses no critical pod, the request-gap rule frees 3,401 MiB and evicts one critical pod, and once priority is added the critical loss drops to zero. - The request-gap rule rewards whoever inflates their request: had billing's declaration been 640 instead of 512, evictions would drop from three to two and critical loss from one to zero, with nothing actually changing. - In making room, the node-selection rule determines not the outcome but the cost: for two urgent pods, the fewest-evicted rule makes two evictions and the first-fit rule makes six; at five pods it is ten against fifteen, with the stranded count at seven under both. - Chaining tops out at a depth of two, bounded by the number of rungs in the priority ladder; pods that declare no request free zero MiB in the ledger and are therefore never used to make room. - The decision rests on seventeen priority declarations, and the measured criticality item is zero; the source of a wrong eviction is not a bad rule but the absence of a single measurement that carries business value. #### Next Step These two lessons have shared out the cluster's fixed capacity: who gets in where, and who gets evicted once it is tight. A third option was never asked — changing the replica count. The regional measurement network's reading collector sees a load that fluctuates over the day, and as long as the replica count stays fixed, the price of that fluctuation is either spare capacity or a queue. This decision, too, can be delegated to a controller: a metric is watched, compared against a target, and the replica count is adjusted. The next lesson writes that controller and counts how the oscillation grows as the metric's delay increases. ### Horizontal Scaling Source: https://academia.sh/en/courses/orchestration/horizontal-scaling Writing metric-based autoscaling as a controller and running it under fluctuating load: how oscillation, excess replica-turns, and shortfall turns change as the metric's delay grows, and how much the wait window and the step limit lower the oscillation and by how many turns they slow the reaction. The previous two lessons shared out the cluster's fixed capacity: who gets in where, and who gets evicted once it is tight. A third option was never asked — changing the replica count. The regional measurement network's (the running fiction) reading collector sees a load that fluctuates over the day; as long as the replica count stays fixed, the price of that fluctuation is either spare capacity or a queue. This decision can be delegated to a controller: a metric is watched, compared against a target, and the replica count is adjusted. The arithmetic of how many replicas are needed is not repeated here — per-replica saturation throughput and the occupancy ceiling are inputs coming from capacity planning. What this lesson measures is the decision itself: what the controller does once it sees the metric delayed. The controller's rule is one line: the desired replica count is the current replica count multiplied by the measured occupancy and divided by the target occupancy. The trap in this line is multiplying the current replica count by a **stale** metric. What is measured is a model; no real cluster runs. #### The Controller and Metric Delay The load series is a 120-turn sequence: a 40-turn wave, a sudden jump at turn 60, and noise in an 8 percent band from our own generator. The required replica count is computed separately for each turn, and the controller's replica count is scored against it. ```js // scheduling/autoscale.mjs — metric-based horizontal autoscaling controller. A MODEL; no real // cluster runs. Per-replica saturation throughput and the occupancy ceiling are inputs from M19. export const SATURATION = 1176.47; // requests/s per replica (M19, computed; an input here) export const TARGET = 0.70; // target occupancy (M19's capacity share; an input here) export const TURNS = 120; // SS25: one turn is one measurement interval export const MAX_REPLICAS = 12; // SS27: replica ceiling const SEED = 20260805; // SS26: the load series is fiction -- a 40-turn wave, a sudden jump at turn 60, +/-8% noise. export function generateSeries() { let s = SEED % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; return Array.from({ length: TURNS }, (_, t) => { const wave = 1200 + 900 * Math.sin((2 * Math.PI * t) / 40); const step = t >= 60 && t < 90 ? 1800 : 0; return Math.max(200, Math.round((wave + step) * (0.92 + 0.16 * r()))); }); } export const SERIES = generateSeries(); export const required = (load) => Math.max(1, Math.ceil(load / (SATURATION * TARGET))); // The controller multiplies the delayed metric by the CURRENT replica count. As delay grows, it overcorrects. export function run({ delay = 0, window = 1, step = 0, series = SERIES } = {}) { let replicas = 2, lastDirection = 0, lowStreak = 0; const occupancies = []; let oscillation = 0, excess = 0, shortfall = 0, total = 0, jumpShortfall = 0; let changes = 0, peakReplicas = 0, reactionTurns = -1; for (let t = 0; t < series.length; t++) { occupancies.push(series[t] / (replicas * SATURATION)); const metric = occupancies[Math.max(0, t - delay)]; let desired = Math.min(MAX_REPLICAS, Math.max(1, Math.ceil((replicas * metric) / TARGET))); if (step > 0) desired = Math.min(replicas + step, Math.max(replicas - step, desired)); if (desired < replicas) { lowStreak++; if (lowStreak < window) desired = replicas; } else lowStreak = 0; const direction = Math.sign(desired - replicas); if (direction !== 0) { changes++; if (lastDirection !== 0 && direction !== lastDirection) oscillation++; lastDirection = direction; } replicas = desired; const req = required(series[t]); total += replicas; excess += Math.max(0, replicas - req); if (replicas < req) { shortfall++; if (t >= 60 && t < 90) jumpShortfall++; } if (t >= 60 && reactionTurns < 0 && replicas >= req) reactionTurns = t - 60; peakReplicas = Math.max(peakReplicas, replicas); } return { oscillation, changes, excess, shortfall, total, jumpShortfall, peakReplicas, reactionTurns }; } if (process.argv[1].endsWith("autoscale.mjs")) { const g = SERIES.map(required); console.log(`model: ${TURNS} turns, load ${Math.min(...SERIES)}-${Math.max(...SERIES)} requests/s ` + `(seed ${SEED}); ${SATURATION} requests/s per replica, target occupancy ${TARGET * 100}%`); console.log(`required replicas ${Math.min(...g)}-${Math.max(...g)}, ideal total ` + `${g.reduce((a, b) => a + b, 0)} replica-turns\n`); const s = (x, n = 17) => String(x).padStart(n); console.log("metric delay is scanned (no wait window and no step limit):"); console.log(`${"delay".padEnd(10)}${s("changes")}${s("oscillation")}${s("excess replica-turns", 23)}` + `${s("shortfall turns")}${s("total replica-turns", 22)}${s("peak replicas", 16)}`); for (const delay of [0, 1, 2, 3, 4, 5, 6]) { const o = run({ delay }); console.log(`${String(delay).padEnd(10)}${s(o.changes)}${s(o.oscillation)}${s(o.excess, 23)}` + `${s(o.shortfall)}${s(o.total, 22)}${s(o.peakReplicas, 16)}`); } } ``` ``` model: 120 turns, load 278-3915 requests/s (seed 20260805); 1176.47 requests/s per replica, target occupancy 70% required replicas 1-5, ideal total 301 replica-turns metric delay is scanned (no wait window and no step limit): delay changes oscillation excess replica-turns shortfall turns total replica-turns peak replicas 0 14 8 0 0 301 5 1 63 30 52 29 319 8 2 67 23 186 40 422 12 3 59 17 225 40 457 12 4 56 15 261 44 489 12 5 53 13 286 42 515 12 6 43 11 275 41 510 12 ``` #### Delay Grows the Oscillation, Then Hides It The zero-delay row gives a lower bound: 14 changes, zero excess replica-turns, zero shortfall turns, and exactly 301 replica-turns — the ideal itself. When the metric arrives instantly, the delegated decision works flawlessly. Even the eight oscillations come from the load's own wave, not from a controller error. A single turn of delay wrecks this table. The change count jumps from 14 to 63, oscillation from 8 to **30**; 29 turns pass with insufficient capacity and the peak replica count climbs from five to eight. The reason lies in the controller's rule: the replica count was raised a turn ago, the metric still shows the occupancy from before the raise, and the controller corrects the same load a second time. **The system does not know whether its own last decision has been reflected in the metric yet**, and that is the one item it does not know. As delay grows, the column worth reading changes. Oscillation **drops** from 30 to 11, and this is not an improvement. Excess replica-turns climb from 52 to 275, total resources climb from 319 to 510 replica-turns, and the peak replica count sticks to the ceiling. The controller no longer oscillates, because it climbs to the ceiling and stays there: the shortfall-turn count settles at 41, meaning resources sit at 1.7 times the ideal while a third of the turns are still short. **A drop in oscillation is not, on its own, a sign of success**; misread without excess replica-turns and shortfall turns alongside it, it is misinterpreted. Delay itself is not a setting but a **sum**. The metric is first produced in the pod, accumulated over an interval, transmitted to the collector, folded into the average the controller reads, and only then turned into a decision. Each of these steps adds a turn. The delay column in the table is therefore not a choice but the result of the monitoring pipeline's design: shortening the collection interval lowers the delay, and in exchange raises the metric count, the storage load, and the noise. The delegated decision's cost is here split across two separate budgets — one in replica-turns, the other in metrics. #### Two Dampers: the Wait Window and the Step Limit Two settings work against oscillation. The **wait window** delays the scale-down decision: it is not applied until the desired replica count has come in low for a set number of turns in a row. The **step limit** caps the size of the change made in a single turn. Both are measured at the same delay — a three-turn metric delay — and the reaction is additionally counted with a noise-free jump test. ```js // scheduling/balance.mjs — wait window and step limit: the drop in oscillation and the slowdown // in reaction are counted together. The controller and load series come from the previous file. import { run } from "./autoscale.mjs"; const DELAY = 3; // SS28: the metric arrives three turns delayed const s = (x, n = 23) => String(x).padStart(n); const row = (ad, o) => console.log(`${ad.padEnd(22)}${s(o.changes)}${s(o.oscillation)}${s(o.excess)}` + `${s(o.shortfall)}${s(o.jumpShortfall)}${s(o.total)}`); console.log(`metric delay held fixed at ${DELAY} turns; results from a 120-turn run:`); console.log(`${"setting".padEnd(22)}${s("changes")}${s("oscillation")}${s("excess replica-turns")}` + `${s("shortfall turns")}${s("shortfall in jump")}${s("total replica-turns")}`); row("unadjusted", run({ delay: DELAY })); // SS29: wait window -- a scale-down decision is applied only if it comes in low for w turns straight. for (const window of [2, 3, 5, 8]) row(`wait window ${window}`, run({ delay: DELAY, window })); // SS30: step limit -- the replica count changes by at most a in a single turn. for (const step of [1, 2]) row(`step limit ${step}`, run({ delay: DELAY, step })); row("window 3 + step 1", run({ delay: DELAY, window: 3, step: 1 })); row("window 5 + step 1", run({ delay: DELAY, window: 5, step: 1 })); // Noise-free jump test: 60 turns at 800 requests/s, then 60 turns at 3200 requests/s (required 1 -> 4). const JUMP_LOAD = Array.from({ length: 120 }, (_, t) => (t < 60 ? 800 : 3200)); console.log("\njump test -- reaction turn when load jumps from 800 to 3200 requests/s at turn 60:"); console.log(`${"setting".padEnd(22)}${s("delay 0")}${s("delay 3")}${s("delay 6")}`); for (const [ad, ek] of [["unadjusted", {}], ["step limit 1", { step: 1 }], ["step limit 2", { step: 2 }], ["window 3 + step 1", { window: 3, step: 1 }]]) console.log(`${ad.padEnd(22)}` + [0, 3, 6] .map((g) => s(run({ ...ek, delay: g, series: JUMP_LOAD }).reactionTurns)).join("")); console.log("\nthe same two settings with no delay (delay 0):"); row("unadjusted", run({ delay: 0 })); row("window 3 + step 1", run({ delay: 0, window: 3, step: 1 })); const a = run({ delay: DELAY }), b = run({ delay: DELAY, window: 3, step: 1 }); console.log(`\noscillation ${a.oscillation} -> ${b.oscillation} (${(100 * (1 - b.oscillation / a.oscillation)).toFixed(0)}% decrease), ` + `shortfall in jump window ${a.jumpShortfall} -> ${b.jumpShortfall}`); console.log(`resources ${a.total} -> ${b.total} replica-turns (ideal 301); ` + `the controller reads 120 metrics, each ${DELAY} turns delayed`); ``` ``` metric delay held fixed at 3 turns; results from a 120-turn run: setting changes oscillation excess replica-turns shortfall turns shortfall in jump total replica-turns unadjusted 59 17 225 40 15 457 wait window 2 46 17 257 37 12 493 wait window 3 45 15 315 33 13 552 wait window 5 40 13 343 31 9 596 wait window 8 31 11 371 23 7 632 step limit 1 71 15 98 35 14 347 step limit 2 71 15 173 35 15 410 window 3 + step 1 59 13 122 32 11 385 window 5 + step 1 53 11 152 27 10 417 jump test -- reaction turn when load jumps from 800 to 3200 requests/s at turn 60: setting delay 0 delay 3 delay 6 unadjusted 0 3 6 step limit 1 2 5 8 step limit 2 1 4 7 window 3 + step 1 2 5 8 the same two settings with no delay (delay 0): unadjusted 14 8 0 0 0 301 window 3 + step 1 14 6 17 0 0 318 oscillation 17 -> 13 (24% decrease), shortfall in jump window 15 -> 11 resources 457 -> 385 replica-turns (ideal 301); the controller reads 120 metrics, each 3 turns delayed ``` #### The Drop in Oscillation and the Slowdown in Reaction The two settings take hold of the same problem from two different places. The wait window drops oscillation from 17 to 11 and pays for it in resources: total replica-turns climb from 457 to 632, excess replica-turns from 225 to 371. Because delaying the scale-down keeps capacity higher, the shortfall-turn count also drops from 40 to 23. **This setting does not reduce oscillation, it redirects it.** The step limit does the opposite. Allowing a single-replica change per turn drops excess replica-turns from 225 to 98 and total resources from 457 to 347 replica-turns; oscillation also drops to 15. The change count, however, climbs from 59 to 71, because the same distance is now covered in more steps. The step limit cuts the size of the overcorrection, and so it targets the problem born from delay directly. The price shows up in the jump test. When the load jumps from 800 to 3,200 requests/s, the unadjusted controller catches up exactly as many turns later as the delay: three turns at a three-turn delay, six turns at a six-turn delay. Once a step limit of 1 is added, those same numbers become 5 and 8 — **the reaction slows by two turns**, because climbing from one replica to four now takes three turns. With a step limit of 2 the gap narrows to one turn. The drop in oscillation and the slowdown in reaction are two faces of the same setting, and unless both are counted separately, the setting is a guess, not a choice. The row where both settings are used together gives the sum of the two: oscillation drops from 17 to 13, resources hold at 385 replica-turns, and the shortfall-in-jump count drops from 15 to 11. This row beats the unadjusted row in every column, yet it still sits 28 percent above the ideal 301 replica-turns. As long as the delay persists, this gap does not close. The last table states what these settings are not. With no delay, the wait window and the step limit fix nothing: oscillation drops from 8 to 6, and in exchange 17 extra replica-turns are spent, paying 318 replica-turns instead of the ideal 301. These two settings are a **repair for a measurement problem, not a load problem**. If the metric can be freshened instead, no settings are needed. The choice of metric is also part of the same calculation. In this model, what is watched is occupancy, and occupancy is a **derived** number: it comes from dividing usage by the replica count and the per-replica saturation throughput. Every derived number is as stale as the oldest of its inputs. A directly countable metric — the number of pending requests, say — is collected in fewer steps and shifts the delay column to the left; in exchange, what the target value should be can no longer be read straight from capacity planning. The decision rests on 120 metrics, each three turns stale, plus four constants — per-replica saturation throughput, target occupancy, and the lower and upper replica bounds. The freshness of these items translates directly into resources: with the same load, the same controller, and the same target, 510 replica-turns can be spent instead of 301, purely because of the metric's delay. #### Summary - When the metric arrives instantly, the delegated decision is flawless: the 301-replica-turn ideal matches resources exactly, with zero excess and zero shortfall turns. - A single turn of delay raises oscillation from 8 to 30 and leaves 29 turns short of capacity; because the controller does not know whether its own last decision has reached the metric yet, it corrects the same load twice. - As delay grows, oscillation drops to 11, but this is not an improvement: the controller sticks to the ceiling, resources climb to 510 replica-turns (1.7 times the ideal), and a third of the turns remain short. - The wait window drops oscillation from 17 to 11 while raising resources from 457 to 632 replica-turns; because it delays the scale-down, it drops shortfall turns from 40 to 23. - Because the step limit cuts the overcorrection, it lowers both oscillation and resources (457 down to 347 replica-turns), while in the jump test the reaction slows from 3 turns to 5 and the change count rises from 59 to 71. - With no delay, the same two settings spend 17 extra replica-turns to buy two fewer oscillations; damping settings are a repair for a measurement problem, not a load problem. #### Next Step This lesson's controller turns a single knob: the replica count. Two more knobs exist, and both meet the same load from a different place. The replica itself can be resized — a pod's request can be rewritten to match observed usage, but this has a price, and the price is paid in restarts. The cluster itself can be grown — a node can be added, but a new node is not ready instantly. The next lesson measures these two knobs and shows the case where all three break each other. ### Vertical and Cluster Scaling Source: https://academia.sh/en/courses/orchestration/vertical-and-cluster-scaling Measuring resource resizing and node addition: comparing vertical resizing's cost — the restart count — against its gain — the drop in spare capacity; scanning a new node's readiness delay to count how many turns are spent short of capacity; and showing, with an example, the case where all three scaling types break each other. The previous lesson's controller turned a single knob: the replica count. Two more knobs exist. The replica itself can be grown or shrunk — a pod's request can be rewritten to match observed usage. The cluster can be grown or shrunk — nodes can be added or removed. All three meet the same load, and all three have a different information cost. What these two knobs have in common is that the payoff cannot be collected instantly. Changing the replica count is a counter operation; changing a pod's request requires **restarting** it, and adding a node makes the system wait for that node to become **ready**. This lesson counts these two delays separately and finally shows what happens when all three knobs are turned at once. What is measured is a model; the workload is the services of the regional measurement network (the running fiction), and the arithmetic of how many replicas are needed is an input coming from capacity planning. #### Vertical Resizing: the Cost Is Restarts The model tracks six pods over 120 turns. The initial request is 1,024 units — declared well above actual usage. A suggestion generator takes the peak of the last ten turns and, once it deviates from the declared request by more than a set threshold, rewrites the request; every rewrite restarts the pod and takes it out of service for two turns. ```js // scheduling/vertical.mjs — vertical resizing: the cost is restarts, the gain is dropping spare // capacity. What gets built is a MODEL; no real cluster runs. export const TURNS = 120, PODS = 6; export const WINDOW = 10; // SS31: the suggestion is the peak of the last 10 turns export const OUTAGE = 2; // SS32: every resize takes a pod out of service for 2 turns const SEED = 20260806; export function generator(seed = SEED) { let s = seed % 2147483647; return () => (s = (s * 48271) % 2147483647) / 2147483647; } // SS33: usage fluctuates around 480 units, two pods drift upward slowly, +/-12% noise. export function usageSeries() { const r = generator(); return Array.from({ length: PODS }, (_, k) => Array.from({ length: TURNS }, (_, t) => { const base = 480 + 100 * Math.sin((2 * Math.PI * t) / 30) + (k % 3 === 0 ? 1.6 * t : 0); return Math.max(80, Math.round(base * (0.88 + 0.24 * r()))); })); } export const SERIES = usageSeries(); // Suggestion: the peak of the last WINDOW turns. Once the deviation threshold is crossed, the request changes and the pod restarts. export function resize(threshold) { let restarts = 0, waste = 0, shortfall = 0, outageTurns = 0, requestTurns = 0; for (const series of SERIES) { let request = 1024, remaining = 0; // SS34: initial request 1024 units (inflated) for (let t = 0; t < TURNS; t++) { const suggestion = Math.max(...series.slice(Math.max(0, t - WINDOW + 1), t + 1)); if (threshold >= 0 && Math.abs(suggestion - request) / request > threshold && remaining === 0) { request = suggestion; restarts++; remaining = OUTAGE; } if (remaining > 0) { remaining--; outageTurns++; } requestTurns += request; waste += Math.max(0, request - series[t]); if (series[t] > request) shortfall++; } } return { restarts, waste, shortfall, outageTurns, requestTurns }; } if (process.argv[1].endsWith("vertical.mjs")) { const usageTurns = SERIES.flat().reduce((a, b) => a + b, 0); console.log(`model: ${PODS} pods x ${TURNS} turns (seed ${SEED}); actual total usage ` + `${usageTurns} unit-turns, initial request 1024 units`); const s = (x, n = 16) => String(x).padStart(n); console.log(`\n${"threshold".padEnd(14)}${s("restarts")}${s("outage turns")}` + `${s("waste")}${s("waste ratio")}${s("shortfall turns", 18)}`); for (const [ad, threshold] of [["fixed request", -1], ["5%", 0.05], ["10%", 0.10], ["20%", 0.20], ["35%", 0.35], ["50%", 0.50]]) { const o = resize(threshold); console.log(`${ad.padEnd(14)}${s(o.restarts)}${s(o.outageTurns)}${s(o.waste)}` + `${s((o.waste / o.requestTurns * 100).toFixed(1) + "%")}${s(o.shortfall, 18)}`); } const a = resize(-1), b = resize(0.20); console.log(`\nfrom a fixed request to a 20% threshold: waste ${a.waste} -> ${b.waste} unit-turns ` + `(${(100 * (1 - b.waste / a.waste)).toFixed(1)}% decrease), at a cost of ${b.restarts} restarts ` + `and ${b.outageTurns} outage turns`); } ``` ``` model: 6 pods x 120 turns (seed 20260806); actual total usage 367748 unit-turns, initial request 1024 units threshold restarts outage turns waste waste ratio shortfall turns fixed request 0 0 369532 50.1% 0 5% 141 280 57939 13.7% 75 10% 87 174 56176 13.3% 113 20% 38 76 54704 13.2% 190 35% 10 20 64938 15.4% 180 50% 8 16 51564 13.2% 336 from a fixed request to a 20% threshold: waste 369532 -> 54704 unit-turns (85.2% decrease), at a cost of 38 restarts and 76 outage turns ``` #### Three Columns Do Not Improve at the Same Time The fixed-request row is the starting point: no restarts, no shortfall turns, and 50.1 percent of the request sits idle. This is the price paid when the declaration is inflated twofold — the first lesson's waste, showing up here as a time series. Once the threshold is set to 20 percent, the waste ratio drops to 13.2 percent: 369,532 unit-turns of waste falls to 54,704 — an 85.2 percent gain. Its price is 38 restarts and 76 outage turns. Once the threshold is lowered to 5 percent, the waste ratio only stays at 13.7 percent while the restart count climbs to 141 and outage turns to 280. **Tighter tracking does not buy more gain, only more disruption.** The third column is not a free lunch. Where the fixed-request row has zero shortfall turns, the 20 percent threshold has 190 and the 50 percent threshold has 336: as the request nears actual usage, the limit gets exceeded at peak moments. Vertical resizing turns spare capacity into shortfall, and the only thing that sets the balance between the two is the threshold. The threshold itself is also a guess — the 35 percent row produces more waste than the 20 percent row, because the suggestion gets stuck on a peak value and stays there. **The ranking is not one-directional, which shows that the setting cannot be chosen without measuring it.** #### Cluster Scaling: the Cost Is Readiness Delay The second knob is the node count. The load climbs from 800 requests/s to 12,000 requests/s; the scaling controller adds replicas, and once the replicas no longer fit on the nodes, the cluster autoscaler orders a node, which becomes ready a set number of turns later. The metric delay is zero in this model, so the only delay being measured is the node's readiness time. ```js // scheduling/cluster.mjs — cluster scaling: whether a new node's readiness delay keeps up with // the load, and the case where three scaling knobs break each other. A model; no real cluster runs. import { TURNS, generator } from "./vertical.mjs"; const SATURATION = 1176.47, TARGET = 0.70; // M19's inputs; the arithmetic is not repeated here const PODS_PER_NODE = 4; // SS35: a node fits four replicas const SHRINK_WINDOW = 5; // Load climbs from 800 to 12000 requests/s, plus a wave and +/-6% noise (fiction). const r = generator(20260807); const LOAD = Array.from({ length: TURNS }, (_, t) => Math.round((800 + 95 * t + 600 * Math.sin((2 * Math.PI * t) / 25)) * (0.94 + 0.12 * r()))); const required = (y) => Math.max(1, Math.ceil(y / (SATURATION * TARGET))); // SS36: vertical resizing takes a replica out of service for 2 turns every 10th turn. function run({ ready, nodeLimit = 99, vertical = false }) { let replicas = 1, readyNodes = 1, pending = [], idleStreak = 0; let shortfallTurns = 0, longestStreak = 0, streak = 0, nodeTurns = 0, replicaTurns = 0; let events = 0, orders = 0, restarts = 0; for (let t = 0; t < TURNS; t++) { pending = pending.filter((v) => { if (v <= t) { readyNodes++; return false; } return true; }); const capped = Math.min(readyNodes, nodeLimit) * PODS_PER_NODE; const restarting = vertical && t % 10 < 2 ? 1 : 0; if (vertical && t % 10 === 0) restarts++; const serving = Math.max(1, Math.min(replicas, capped) - restarting); const occupancy = LOAD[t] / (serving * SATURATION); const desired = Math.max(1, Math.ceil((replicas * occupancy) / TARGET)); if (desired !== replicas) events++; replicas = desired; if (replicas > capped) { const extra = Math.ceil((replicas - capped) / PODS_PER_NODE); for (let i = 0; i < extra; i++) pending.push(t + ready); orders += extra; } else if (capped - replicas >= PODS_PER_NODE && readyNodes > 1) { if (++idleStreak >= SHRINK_WINDOW) { readyNodes--; idleStreak = 0; } } else idleStreak = 0; const running = Math.min(replicas, Math.min(readyNodes, nodeLimit) * PODS_PER_NODE); replicaTurns += running; nodeTurns += readyNodes + pending.length; if (running < required(LOAD[t])) { shortfallTurns++; longestStreak = Math.max(longestStreak, ++streak); } else streak = 0; } return { shortfallTurns, longestStreak, nodeTurns, replicaTurns, events, orders, restarts }; } const s = (x, n = 18) => String(x).padStart(n); console.log(`load ${Math.min(...LOAD)}-${Math.max(...LOAD)} requests/s, ${TURNS} turns; required ` + `replicas ${required(LOAD[0])}-${Math.max(...LOAD.map(required))}, ${PODS_PER_NODE} replicas per node`); console.log(`\nnew node readiness delay is scanned:`); console.log(`${"delay".padEnd(10)}${s("shortfall turns")}${s("longest streak")}${s("node orders")}` + `${s("node-turns")}${s("replica-turns")}`); for (const ready of [0, 1, 2, 4, 6, 8]) { const o = run({ ready }); console.log(`${String(ready).padEnd(10)}${s(o.shortfallTurns)}${s(o.longestStreak)}${s(o.orders)}` + `${s(o.nodeTurns)}${s(o.replicaTurns)}`); } console.log(`\nthree scaling knobs together (readiness delay 4):`); console.log(`${"setting".padEnd(34)}${s("scaling events")}${s("shortfall turns")}` + `${s("restarts")}${s("node-turns")}`); for (const [ad, ek] of [["horizontal only (ample nodes)", { ready: 0, nodeLimit: 99 }], ["horizontal + cluster", { ready: 4 }], ["horizontal + cluster + vertical", { ready: 4, vertical: true }]]) { const o = run(ek); console.log(`${ad.padEnd(34)}${s(o.events)}${s(o.shortfallTurns)}` + `${s(o.restarts)}${s(o.nodeTurns)}`); } const b = run({ ready: 4 }), c = run({ ready: 4, vertical: true }); console.log(`\nonce vertical is added: scaling events ${b.events} -> ${c.events}, node orders ` + `${b.orders} -> ${c.orders}, node-turns ${b.nodeTurns} -> ${c.nodeTurns}`); ``` ``` load 792-12189 requests/s, 120 turns; required replicas 1-15, 4 replicas per node new node readiness delay is scanned: delay shortfall turns longest streak node orders node-turns replica-turns 0 3 1 3 302 1003 1 3 1 3 302 1003 2 5 2 6 319 1001 4 10 4 13 431 997 6 10 6 21 837 998 8 8 8 33 2353 1002 three scaling knobs together (readiness delay 4): setting scaling events shortfall turns restarts node-turns horizontal only (ample nodes) 36 3 0 302 horizontal + cluster 42 10 0 431 horizontal + cluster + vertical 54 8 12 445 once vertical is added: scaling events 42 -> 54, node orders 13 -> 13, node-turns 431 -> 445 ``` #### Delay Turns Shortfall into Resources The second column behaves as expected: the longest shortfall streak is exactly the readiness delay itself. At a four-turn delay, the longest uninterrupted shortfall is four turns; at a six-turn delay, it is six turns. Load waits until a new node is on its way, and that wait is the result of node-provisioning time, not a design choice. The first column behaves unexpectedly. At a six-turn delay, 10 turns run short; at an eight-turn delay, that number **drops** to 8. The conclusion that the system runs better here is wrong; the fourth column tells the real story. Node orders climb from 21 to 33, node-turns from 837 to **2,353**. As delay grows, the controller keeps seeing the same gap over and over and keeps ordering nodes over and over; the cluster eventually accumulates far more nodes than it needs, and the shortfall closes by burning resources. Because the shrink rule removes at most one node every five turns, this surplus never melts away over the course of the run. **Unless shortfall turns and node-turns are read together, delay looks like an improvement.** The replica-turns column staying around 1,000 across all six rows is a separate result: the number of running replicas barely changes at all; what changes is how late those replicas manage to be placed, and how many nodes wait behind them. #### When All Three Knobs Are Turned at Once The final table stacks the three scaling types on top of each other. With plenty of nodes available, horizontal scaling alone produces 36 scaling events and 3 shortfall turns. Once cluster scaling is added, the event count climbs to 42, shortfall turns to 10, and node-turns from 302 to 431: because the horizontal controller's decision can no longer be applied instantly, it repeats itself. Once vertical resizing is added, the breakdown can be counted exactly. Twelve restarts raise the scaling-event count from 42 to 54 — **twelve extra events, exactly matching the restart count**. The chain runs like this: vertical resizing takes a replica out of service for two turns, the serving replica count drops, the remaining replicas' occupancy rises, the horizontal controller thinks load has grown and adds a replica, and if the added replica does not fit on a node, the cluster autoscaler orders one. The actual load never changed throughout, and node-turns climbed from 431 to 445. None of the three controllers makes a mistake. Each reads its own metric correctly and applies its own rule correctly. The problem is that **one controller's action becomes another's measurement**: vertical resizing's restart enters the horizontal controller's occupancy metric as a load increase. Here too, the item the system cannot see is a single thing — the reason a replica is out of service. Had this item been added to the metric, that is, had out-of-service replicas been excluded from the occupancy calculation, none of the twelve extra events would have been born. The three knobs' information costs are not equal, either. Horizontal scaling works with a single occupancy metric per replica, and its decision is undone with a single counter change. Vertical resizing wants a usage window per pod — in this model, a ten-turn window for six pods, meaning sixty samples every turn — and its decision costs a restart. Cluster scaling adds node-provisioning time on top of both of these; that time is not measured but **declared**, and when it is declared wrongly, the shortfall-turn count and node-turns drift together. #### Summary - Vertical resizing's gain is measured in spare capacity: while the waste ratio at a fixed request is 50.1 percent, it drops to 13.2 percent at a 20 percent threshold, and waste falls from 369,532 to 54,704 unit-turns. - Its cost is restarts, and the cost only grows as the threshold narrows: 38 restarts and 76 outage turns at 20 percent, 141 restarts and 280 outage turns at 5 percent, while the gain stays at the same 13.7 percent. - Tight resizing turns spare capacity into shortfall: shortfall turns climb from zero to 190, then to 336 at the 50 percent threshold. - In cluster scaling, the longest shortfall streak equals the readiness delay; at a delay of 8, shortfall turns drop to 8 while node orders climb from 21 to 33 and node-turns from 837 to 2,353 — the shortfall is closed with resources. - When all three knobs are turned together, twelve restarts produce exactly twelve extra scaling events and node-turns climb from 431 to 445, even though the actual load never changed. - All three controllers apply their own rule correctly; the source of the breakdown is that one controller's action enters another's metric as load, and the reason for being out of service does not appear in that metric at all. #### Next Step This topic tied placement and growth decisions to numbers: request and limit, constraints, taints, eviction, and three kinds of scaling. Their shared assumption is that the desired state is written somewhere and the cluster tries to make it real. How that desired state gets rolled out, when a pod counts as healthy, which subject holds authority over which object, who the network is open from and to, and how all of this gets rolled back during a failure — none of that has been asked yet. The next topic takes on the cluster's operation: deployment patterns, health checks, authorization and network policy, observability, troubleshooting, and extending the cluster with its own object types. ## Operations and Extension Operating and extending the cluster: comparing the rolling update, blue-green, and canary deployment patterns in terms of escaped defects and rollback window, the distinction among liveness, readiness, and startup probes, bounding privilege with role-based access control, restricting traffic with network policies, security context, reusable charts and templating, extending the resource type, coding operating knowledge into a controller, cluster observability, and multi-cluster operation. ### Deployment Patterns Source: https://academia.sh/en/courses/orchestration/deployment-patterns The rolling update, blue-green, and canary patterns are run against the same faulty-release sequence: the requests touched by the escaped defect, the rounds that pass undetected, the peak capacity kept active, and rollback's per-round cost are all counted. For canary, sample size is swept. Placement and scaling decisions have already been delegated to the system: the scheduler decides which node gets which pod, and the scaling controller decides how many replicas stay up. Both rest on a shared assumption — a **working** version already exists somewhere. Two decisions remain undelegated. **How** does a new version enter the cluster, and who checks whether a replica that has come up **is actually serving**. This lesson measures the first one. Release mechanics were measured earlier: the request dropped when the pool changes, in-flight requests recovered through connection draining, the difference between hard and graceful shutdown. Those measurements are not repeated here. This lesson's question is one step further: if **not a single request drops** during the rollout, do the patterns still diverge from each other? They do — but not at the point of interruption. They diverge at the point where **the new version is broken**. That every rollout produces some escaped defects is a measured result; this lesson ties that result to pattern choice and runs all three patterns against the same faulty-release sequence. The delegated decision is this: a controller, not a human, decides when to stop the old version and how much traffic to open to the new one. The controller needs four information items for this: which replica is on which version, whether the replica is ready, what the error rate is, and what to do once the threshold is crossed. If the last item is undefined, the controller keeps advancing — not because it does not know, but because **it was never told to stop**. - **OO1.** The cluster runs the fictional regional measurement network's verification service with twelve replicas; six hundred requests arrive per round. Both the network and the figures are fictional. - **OO2.** The faulty version errors out on six percent of the requests it receives. The defect is present from the first request and stays constant through the version's lifetime; defects that fade with warm-up are outside this model. - **OO3.** The observer checks at the end of the round. Delays shorter than a round, metric-collection lag, and the alert channel's own latency are not modeled. - **OO4.** The alarm threshold is one and a half percent, chosen above the cluster's ordinary error share; where the threshold comes from is not this lesson's subject. - **OO5.** Over the rollback rounds, the new version's share declines to zero in equal steps; the rollback itself produces no errors. - **OO6.** The horizon is twelve rounds. If the alarm has not fired within that time, the defect is considered to have escaped. #### Measurement Setup What is measured is not a real cluster but a model built with `node`. The model carries three things: what share a pattern produces round by round, how requests are distributed according to that share, and what the observer sees at the end of the round. The random generator is hand-written, and its seed is visible. ```js // pattern.mjs — round-based model of three deployment patterns; not a real cluster export const SEED = 20260801; export const SERVICE = 12, ROUND_REQUESTS = 600, ERROR_RATE = 0.06, THRESHOLD = 0.015, HORIZON = 12; export function generator(seed) { // linear congruential generator, seed visible let x = seed >>> 0; return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; }; } // Each step: replicas serving the new version, replicas serving the old version, // total replicas kept active at that moment, rollback cost in rounds if the alarm fires this round. export const rolling = { name: 'rolling update', plan: [ { newer: 3, older: 9, active: 15, rollback: 1 }, { newer: 6, older: 6, active: 15, rollback: 2 }, { newer: 9, older: 3, active: 15, rollback: 3 }, { newer: 12, older: 0, active: 12, rollback: 4 } ] }; export const blueGreen = { name: 'blue-green', plan: [ { newer: 0, older: 12, active: 24, rollback: 0 }, { newer: 12, older: 0, active: 24, rollback: 1 }, { newer: 12, older: 0, active: 12, rollback: 3 } ] }; export const canary = (k, observe = 3) => ({ name: `canary (sample ${k})`, plan: [ ...Array.from({ length: observe }, () => ({ newer: k, older: 12, active: 12 + k, rollback: 1 })), { newer: 12, older: 0, active: 24, rollback: 1 }, { newer: 12, older: 0, active: 12, rollback: 3 } ] }); export function run({ name, plan }, labeled = false) { const rnd = generator(SEED); let i = 0, alarm = 0, remaining = 0, share = 0, cost = 0, peak = SERVICE; let errors = 0, touched = 0, step = { newer: 0, older: SERVICE, active: SERVICE, rollback: 0 }; for (let round = 1; round <= HORIZON; round += 1) { if (remaining > 0) { step = { ...step, newer: Math.max(0, step.newer - share), older: SERVICE }; remaining -= 1; } else if (alarm) step = { newer: 0, older: SERVICE, active: SERVICE, rollback: 0 }; else { step = plan[Math.min(i, plan.length - 1)]; i += 1; } peak = Math.max(peak, step.active); let roundErrors = 0, roundTouched = 0; const ratio = step.newer / (step.newer + step.older); for (let r = 0; r < ROUND_REQUESTS; r += 1) { if (rnd() >= ratio) continue; // request went to the old version roundTouched += 1; if (rnd() < ERROR_RATE) roundErrors += 1; } errors += roundErrors; touched += roundTouched; if (!alarm) { // the observer checks at the end of the round const observed = labeled ? (roundTouched ? roundErrors / roundTouched : 0) : roundErrors / ROUND_REQUESTS; const sufficient = !labeled || roundTouched >= 100; if (sufficient && observed > THRESHOLD) { alarm = round; cost = step.rollback; remaining = cost; share = Math.ceil(step.newer / Math.max(1, cost)); } } } return { name, alarm, cost, window: alarm ? alarm - 1 : HORIZON, peak, errors, touched }; } ``` The entire difference between the three patterns' plans collects into three columns. The rolling update opens the new version in shares of three and keeps three extra replicas active through the transition. Blue-green brings the new version fully up before opening any traffic to it — its share is zero in the first round, with twenty-four replicas active — then switches over in a single step. Canary opens a small sample to traffic, waits a few rounds, and promotes if the alarm does not fire. #### Three Patterns, One Faulty Release ```js // run1.mjs — the same faulty-release sequence is run against all three patterns import { run, rolling, blueGreen, canary, SEED, SERVICE, ROUND_REQUESTS, ERROR_RATE, THRESHOLD, HORIZON } from './pattern.mjs'; console.log(`seed ${SEED} | ${SERVICE} replicas, ${ROUND_REQUESTS} requests per round, ${HORIZON}-round horizon`); console.log(`faulty version: ${ERROR_RATE * 100}% of requests to the new version error out`); console.log(`observer: alarms once the cluster-wide error rate exceeds ${THRESHOLD * 100}%\n`); console.log('pattern alarm round undetected faulty req touched peak repl rollback rounds'); for (const k of [rolling, blueGreen, canary(2)]) { const s = run(k); console.log(`${s.name.padEnd(23)} ${String(s.alarm || '-').padEnd(11)} ${String(s.window).padEnd(13)} ` + `${String(s.errors).padEnd(13)} ${String(s.touched).padEnd(10)} ${String(s.peak).padEnd(11)} ${s.cost || '-'}`); } ``` ``` seed 20260801 | 12 replicas, 600 requests per round, 12-round horizon faulty version: 6% of requests to the new version error out observer: alarms once the cluster-wide error rate exceeds 1.5% pattern alarm round undetected faulty req touched peak repl rollback rounds rolling update 2 1 33 555 15 2 blue-green 2 1 30 600 24 1 canary (sample 2) 4 3 53 851 24 1 ``` All three patterns carried the same defect, and all three detected it, but their costs piled up in different places. The rolling update gave the new version a one-quarter share in the first round; at that share, the cluster's total error rate did not clear the threshold, so the alarm fired in the second round. Blue-green gave no share at all in the first round and the full share in the second — the alarm fired in the same round, but in that single round **all six hundred requests** went to the faulty version. Over the same span, the rolling update touched five hundred fifty-five requests; the difference looks small only because the rolling update was also splitting the load roughly in half by the second round. The real divergence is in two columns. Peak capacity: the rolling update got through with fifteen replicas, blue-green with twenty-four — meaning the cluster had to find **a full extra set of room** for the new version at cutover. Rollback cost runs the other way: for blue-green, rollback is a single round, because the old version is still standing; for the rolling update, a defect caught in the second round costs two rounds of rollback, and would have cost four had it been caught in the fourth round. As the rolling update advances, rollback grows more expensive; for blue-green, the expense jumps **all at once** — in the round the old environment is torn down, rollback cost leaps from one round to three. The canary row is this table's surprise. A two-replica sample kept the cluster's total error rate under the threshold for all three observation rounds: with two of fourteen serving replicas faulty, the total error rate sits around one percent, and the threshold is one and a half. The observer saw nothing, the canary was promoted, and the fourth round opened it fully. The escaped defect touched eight hundred fifty-one requests — the worst of the three patterns. When the sample is chosen wrong, the canary stops being the pattern that catches the defect and becomes the one that **delays catching it and then opens it up fully**. #### Sample Size Which is why the canary's one setting has to be swept. The same model is run with two observers: the first watches the cluster's total error rate, the second splits requests by version and watches only the new version's error rate. The second one demands an extra information item — which version each request went to — and needs to see enough requests in the round before it can decide. ```js // run2.mjs — sample size is swept for the canary; the second observer splits requests by version import { run, canary, ROUND_REQUESTS } from './pattern.mjs'; console.log('sample cluster-wide observer per-version observer'); console.log('repl alarm undetected faulty touched alarm undetected faulty touched'); for (const k of [1, 2, 3, 4, 6, 8]) { const t = run(canary(k)), l = run(canary(k), true); const fmt = (s) => `${String(s.alarm || '-').padEnd(6)} ${String(s.window).padEnd(13)} ` + `${String(s.errors).padEnd(7)} ${String(s.touched).padEnd(10)}`; console.log(`${String(k).padEnd(6)} ${fmt(t)} ${fmt(l)}`); } console.log(`\nrequests reaching the canary during a round of observation: for sample k, ${ROUND_REQUESTS} x k/(12+k)`); console.log('the per-version observer does not decide without seeing at least 100 requests in the round'); for (const k of [1, 2, 3, 4, 6, 8]) { console.log(` sample ${k}: ${Math.round(ROUND_REQUESTS * k / (12 + k))} req/round` + ` -> ${ROUND_REQUESTS * k / (12 + k) >= 100 ? 'can decide' : 'sample too small'}`); } ``` ``` sample cluster-wide observer per-version observer repl alarm undetected faulty touched alarm undetected faulty touched 1 4 3 42 730 4 3 42 730 2 4 3 53 851 4 3 53 851 3 4 3 55 948 1 0 7 107 4 3 2 24 461 1 0 8 147 6 1 0 11 191 1 0 11 191 8 1 0 14 243 1 0 14 243 requests reaching the canary during a round of observation: for sample k, 600 x k/(12+k) the per-version observer does not decide without seeing at least 100 requests in the round sample 1: 46 req/round -> sample too small sample 2: 86 req/round -> sample too small sample 3: 120 req/round -> can decide sample 4: 150 req/round -> can decide sample 6: 200 req/round -> can decide sample 8: 240 req/round -> can decide ``` Read on its own, the left half leads to the conclusion that the canary is a bad pattern. For samples of one, two, and three replicas, the alarm fired in the fourth round — that is, **after** promotion; the number of touched requests grew as the sample grew, because the same missed span carried a larger observed share. At four replicas the alarm moved to the third round, at six and eight replicas to the first. The lowest total damage is not at eight but at six: the eight-replica sample caught the defect in the first round too, but that single round still touched two hundred forty-three requests, against one hundred ninety-one for the six-replica sample. The right half gives the sweep's real result. The per-version observer fired the alarm in the first round for samples of three and four replicas; the faulty-request count fell to seven and eight. At one and two replicas, the right half stayed identical to the left — because at those sample sizes only forty-six and eighty-six requests per round reach the new version, below the observer's decision threshold. The small sample's problem is not only the metric being diluted; it is **the smallness of the sample itself**. A labeled metric solves the dilution problem; it does not solve the sample problem. The setting worth reading off this is three replicas: it catches the defect in the first round, touches one hundred seven requests, and the extra capacity it keeps active is thirteen replicas, not fifteen. Its cost is that every request must carry a version label and the observer must keep a separate series for that label — one more information item. #### What the Pattern Hides None of the three patterns is "safer"; all three pile the same three quantities into different places. Blue-green makes **rollback** cheap and pays for it in capacity: twenty-four replicas, a full extra set of room, sitting idle for the whole cutover window. The rolling update makes **capacity** cheap and pays for it in rollback: the further it advances, the longer the way back. Canary shrinks the **blast radius** and pays for it in observation time — nothing advances for three rounds. What stays silent is that all three assume a threshold has been set correctly. In the model, every round the alarm did not fire, the controller moved on to the plan's next step; no pattern ever said "I am not sure, let me wait." The two-replica canary's promotion was not a malfunction — it was **behavior consistent with the plan**: the observer saw nothing crossing the threshold, so the controller advanced. This is exactly where the delegated decision's cost sits — the controller does not know the error rate, it knows **the number the observer reported**. If that number is diluted, the wrong decision comes not from wrong information but from **a missing distinction**. #### Summary - All three patterns were run against the same faulty release: the rolling update caught it in the second round with thirty-three faulty requests, blue-green in the second round with thirty, and the two-replica canary in the fourth round with fifty-three. - Peak capacity and rollback cost run in opposite directions: blue-green needs twenty-four replicas and rolls back in a single round; the rolling update gets through with fifteen replicas, but rollback grows to two, three, or four rounds the further it has advanced. - For blue-green, the cheap rollback window ends in the round the old environment is torn down; after that round, rollback jumps from one round to three. - For canary, a small sample misses the defect and lets it get promoted: the two-replica sample touched eight hundred fifty-one requests, the worst of the three patterns. - The observer's distinction matters before sample size does: the per-version metric moved the alarm from the fourth round to the first for the three-replica sample, cutting faulty requests from fifty-five to seven; at one and two replicas it changed nothing, because the sample sat below the decision threshold. - The delegated decision is "advance or stop"; the controller knows the number the observer reported, not the error rate itself, and if that number is diluted, the wrong decision comes from a missing distinction. #### Next Step Throughout this lesson's model, a replica could serve the moment it entered traffic. In reality, a process spends an interval between coming up and being ready: it builds its connection pool, warms its cache, polls its dependencies. If the controller does not know about this interval, it opens the replica to traffic too early; or, the reverse, it mistakes a slow-starting replica for a failed one and kills it. The next lesson measures this interval with three separate checks: it sweeps the thresholds of the checks that distinguish whether a replica is alive, can take requests, and is still starting up, and counts how many unnecessary restarts and how many faulty requests the wrong setting produces. ### Health Checks Source: https://academia.sh/en/courses/orchestration/health-checks The distinction between liveness, readiness, and startup probes is established with numbers: the unnecessary restarts and restart loop produced by a wrongly scoped liveness probe, the failed requests sent to a replica opened to traffic too early without a readiness probe, and the killing of a slow-starting replica without a startup probe are counted separately. Throughout the previous lesson's model, a replica could serve the moment it entered the pool. The deployment controller advanced on that same assumption: once a replica came up, it counted; once it counted, it took a share. In reality there is an interval between a process starting and being able to answer requests — the connection pool gets built, the cache gets warmed, dependencies get polled. If nobody reports this interval, the controller opens the replica to traffic too early. The reverse also happens: it treats a slow-starting replica as failed, kills it, and repeats this at every startup. There are two delegated decisions here, and they are often mistaken for one. The first is **should this replica be restarted**, the second **should requests be sent to this replica**. If the answer to the first is wrong, a working process gets killed; if the answer to the second is wrong, a request goes to a process that is not working. What the system needs to know to make these two decisions are separate information items: is the process itself responding, does it hold its dependencies, and is it still starting up. Three questions, three probes. This lesson measures what it costs to merge the questions. The term readiness probe was used earlier in the context of single-machine composition; here the same concept appears at cluster scale, as the probe that determines pool membership. - **OO7.** The cluster runs the fictional regional measurement network's verification service with eight replicas, with a warm start; fifty requests arrive per round. Both the network and the figures are fictional. - **OO8.** Six replicas become able to serve in six rounds, two in twenty-five rounds. The two slow ones warm a cache at startup; this difference comes from the application, not from the cluster. - **OO9.** The events are hand-written; there is no randomness. Transient dependency loss resolves on its own once its duration elapses; permanent stuck state is resolved only by a restart. - **OO10.** A probe's answer is binary: a replica either answers or it does not. Network-caused false negatives, the probe endpoint's own latency, and timeout settings are not modeled. - **OO11.** A restart happens instantly. Rescheduling the pod, pulling the image, and node selection are outside this measurement; all of them would enlarge the delay without changing its direction. - **OO12.** The "actually healthy" column is the average number of healthy replicas per round. The target is eight, and this number looks at the replicas' real state, not at what the probes believe. #### Measurement Setup What is measured is not a real cluster but a model built with `node`. In every round, a replica is in one of four states: starting, stuck, missing a dependency, healthy. The probes do not see this state directly; each only gets a yes or no answer to the one question within its scope. ```js // health.mjs — round-based model of liveness, readiness, and startup probes export const ROUND = 180, ROUND_REQUESTS = 50, STARTUP = [6, 6, 6, 6, 6, 6, 25, 25]; export const SCHEDULED = [{ replica: 6, round: 20 }, { replica: 7, round: 20 }]; // two replicas move to the new version export const INCIDENT = [ // events hand-written, no randomness { replica: 3, round: 50, type: 'noDependency', duration: 18 }, // transient: resolves on its own { replica: 1, round: 80, type: 'stuck' }, // permanent: only a restart fixes it { replica: 4, round: 120, type: 'noDependency', duration: 10 }, { replica: 5, round: 140, type: 'stuck' } ]; export function run({ name, liveness, readiness, startup }) { const reps = STARTUP.map(() => ({ readyRound: 0, stuck: false, cMiss: 0, hMiss: 0, bMiss: 0, startupDone: true, inPool: true, restarts: 0 })); const s = { name, restarts: 0, unnecessary: 0, duringStartup: 0, early: 0, failed: 0, poolRounds: 0, detect: [] }; const stuckRound = {}; const status = (i, t) => t < reps[i].readyRound ? 'starting' : reps[i].stuck ? 'stuck' : INCIDENT.some((a) => a.replica === i && a.duration && t >= a.round && t < a.round + a.duration) ? 'noDependency' : 'healthy'; const start = (i, t) => Object.assign(reps[i], { readyRound: t + STARTUP[i], stuck: false, cMiss: 0, hMiss: 0, bMiss: 0, startupDone: !startup, inPool: !readiness }); for (let t = 0; t < ROUND; t += 1) { for (const p of SCHEDULED) if (p.round === t) start(p.replica, t); // scheduled startup, not counted for (const a of INCIDENT) if (a.round === t && a.type === 'stuck') { reps[a.replica].stuck = true; stuckRound[a.replica] = t; } for (let i = 0; i < reps.length; i += 1) { const d = status(i, t); if (!reps[i].startupDone) { // other probes do not look until the startup probe finishes if (t % startup.p) continue; if (d !== 'starting') reps[i].startupDone = true; else if ((reps[i].bMiss += 1) >= startup.e) { s.restarts += 1; s.duringStartup += 1; s.unnecessary += 1; reps[i].restarts += 1; start(i, t); } continue; } if (liveness && t % liveness.p === 0) { const bad = liveness.scope === 'dependency' ? d !== 'healthy' : d === 'starting' || d === 'stuck'; if (!bad) reps[i].cMiss = 0; else if ((reps[i].cMiss += 1) >= liveness.e) { s.restarts += 1; reps[i].restarts += 1; if (d === 'stuck') s.detect.push(t - stuckRound[i]); else s.unnecessary += 1; if (d === 'starting') s.duringStartup += 1; start(i, t); } } if (readiness && t % readiness.p === 0) { if (d === 'healthy') { reps[i].hMiss = 0; reps[i].inPool = true; } else if ((reps[i].hMiss += 1) >= readiness.e) reps[i].inPool = false; } } const pool = reps.map((_, i) => i).filter((i) => reps[i].inPool); // the pool the probe believes in s.poolRounds += reps.filter((_, i) => status(i, t) === 'healthy').length; // actually able to serve for (let r = 0; r < ROUND_REQUESTS; r += 1) { if (!pool.length) { s.failed += 1; continue; } const d = status(pool[r % pool.length], t); if (d === 'healthy') continue; s.failed += 1; if (d === 'starting') s.early += 1; } } s.loop = reps.filter((c) => c.restarts >= 3).length; return s; } ``` Two lines carry this model's entire distinction. If the liveness probe's `scope` field is `process`, the probe only fails on starting and stuck states — it does not see dependency loss. If the same field is `dependency`, the probe asks **the same question** as the readiness probe; a single probe endpoint then drives both restarts and pool membership. The second line is the difference between `pool` and `poolRounds`: the former is the pool the probe believes in, the latter is the number of replicas actually able to serve. Wherever the two can diverge, there is a wrong decision. #### Four Settings, One Event Sequence ```js // run3.mjs — four settings are run against the same event sequence import { run, ROUND, ROUND_REQUESTS } from './health.mjs'; const SETTING = [ { name: 'no probes', liveness: null, readiness: null, startup: null }, { name: 'single combined probe', liveness: { p: 5, e: 3, scope: 'dependency' }, readiness: { p: 5, e: 3 }, startup: null }, { name: 'liveness + readiness', liveness: { p: 5, e: 3, scope: 'process' }, readiness: { p: 2, e: 1 }, startup: null }, { name: 'plus startup', liveness: { p: 5, e: 3, scope: 'process' }, readiness: { p: 2, e: 1 }, startup: { p: 5, e: 6 } }, ]; console.log(`8 replicas, ${ROUND} rounds, ${ROUND_REQUESTS} requests per round; two replicas start up over 25 rounds`); console.log('round 20: two slow replicas restart on schedule | rounds 80 and 140: permanent stuck state'); console.log('rounds 50 and 120: transient dependency loss, lasts 18 and 10 rounds, resolves on its own\n'); console.log('setting restarts unnecessary during restart early total actually detect'); console.log(' restarts startup loop failed failed healthy stuck'); for (const a of SETTING) { const s = run(a); console.log(`${s.name.padEnd(21)} ${String(s.restarts).padEnd(8)} ${String(s.unnecessary).padEnd(9)} ` + `${String(s.duringStartup).padEnd(9)} ${String(s.loop).padEnd(8)} ${String(s.early).padEnd(7)} ` + `${String(s.failed).padEnd(7)} ${(s.poolRounds / ROUND).toFixed(2).padEnd(10)} ${s.detect.join(', ') || '-'}`); } ``` ``` 8 replicas, 180 rounds, 50 requests per round; two replicas start up over 25 rounds round 20: two slow replicas restart on schedule | rounds 80 and 140: permanent stuck state rounds 50 and 120: transient dependency loss, lasts 18 and 10 rounds, resolves on its own setting restarts unnecessary during restart early total actually detect restarts startup loop failed failed healthy stuck no probes 0 0 0 0 300 1408 6.79 - single combined probe 23 21 20 2 0 330 5.89 10, 10 liveness + readiness 22 20 20 2 0 0 5.89 10, 10 plus startup 2 0 0 0 0 0 7.39 10, 10 ``` The no-probes row gives the baseline: no replica was restarted, the two permanent stuck states went undetected for a hundred rounds and forty rounds, and one thousand four hundred eight requests failed. Three hundred of those went to replicas still starting up — meaning the sole cost of having no readiness probe is three hundred requests. The remaining one thousand one hundred eight are requests sent to replicas that were stuck or had lost a dependency. The single combined probe cut failed requests from one thousand four hundred to three hundred thirty and detected the stuck state in ten rounds. Its cost is in the second and third columns: twenty-one of twenty-three restarts were unnecessary. This probe also counts dependency loss as a failure, so an outside outage ends up killing a healthy process. This is the price of being unable to distinguish permanent stuck state from transient dependency loss: a condition that would resolve on its own gets fixed with a restart instead. The third row limits the liveness probe's scope to the process and separates out the readiness probe. Failed requests dropped to zero — the pool now pulls a replica that has lost its dependency out within two rounds, without killing it. Unnecessary restarts are still twenty. Their source is not transient failure: all twenty are **killed during startup**. The liveness probe checks every five rounds and kills on three consecutive failures; that fifteen-round allowance is not enough for the two replicas that take twenty-five rounds to start. Those two replicas get killed in the middle of every startup and start over — two replicas caught in a restart loop. This row's real measurement is the second-to-last column. Failed requests are zero, but the actually-healthy replica count is 5.89 — in a cluster targeting eight, **more than two replicas are permanently missing**. Because the readiness probe keeps the looping replicas out of the pool, this loss produces not a single failure record. Anyone watching the cluster's error rate sees everything looking fine; the loss stops at capacity and, when load rises, falls on the remaining six replicas. The fourth row adds the startup probe: restarts drop from twenty-two to two — that is, down to **only the two real stuck states** — unnecessary restarts go to zero, and the healthy-replica count rises to 7.39. Detection time for the stuck state has not changed. What the startup probe does is not catch a new kind of failure; it is **silencing the liveness probe for the duration of the startup window**. #### Threshold and Period The third row's killing during startup could also be removed by loosening the liveness probe's settings. The sweep shows what that costs. ```js // run4.mjs — the liveness probe's period and threshold are swept, first without a startup probe import { run, ROUND } from './health.mjs'; const READINESS = { p: 2, e: 1 }; const write = (label, startup) => { console.log(`\n${label}`); console.log('period thresh tolerance detect killed restart actually'); console.log(' (rounds) stuck at startup loop healthy'); for (const p of [2, 5, 10]) for (const e of [2, 3, 6]) { const s = run({ name: '', liveness: { p, e, scope: 'process' }, readiness: READINESS, startup }); console.log(`${String(p).padEnd(8)} ${String(e).padEnd(5)} ${String(p * e).padEnd(8)} ` + `${(s.detect.join(', ') || '-').padEnd(11)} ${String(s.duringStartup).padEnd(10)} ` + `${String(s.loop).padEnd(8)} ${(s.poolRounds / ROUND).toFixed(2)}`); } }; console.log('two slow replicas become ready over 25 rounds; permanent stuck state at rounds 80 and 140'); write('no startup probe:', null); write('startup probe present (period 5, threshold 6 — a 30-round startup allowance):', { p: 5, e: 6 }); ``` ``` two slow replicas become ready over 25 rounds; permanent stuck state at rounds 80 and 140 no startup probe: period thresh tolerance detect killed restart actually (rounds) stuck at startup loop healthy 2 2 4 2, 2 113 4 5.29 2 3 6 4, 4 52 2 5.96 2 6 12 10, 10 26 2 5.89 5 2 10 5, 5 32 2 5.94 5 3 15 10, 10 20 2 5.89 5 6 30 25, 25 0 0 7.22 10 2 20 10, 10 16 2 5.89 10 3 30 20, 20 2 0 7.06 10 6 60 50 0 0 7.03 startup probe present (period 5, threshold 6 — a 30-round startup allowance): period thresh tolerance detect killed restart actually (rounds) stuck at startup loop healthy 2 2 4 2, 2 0 0 7.48 2 3 6 4, 4 0 0 7.46 2 6 12 10, 10 0 0 7.39 5 2 10 5, 5 0 0 7.44 5 3 15 10, 10 0 0 7.39 5 6 30 25, 25 0 0 7.22 10 2 20 10, 10 0 0 7.39 10 3 30 20, 20 0 0 7.28 10 6 60 50 0 0 7.03 ``` In the top table, a single setting does two jobs at once, and the two jobs pull in opposite directions. As the tolerance window shortens, the stuck state gets detected faster — in two rounds, with a four-round window — but killing during startup jumps to one hundred thirteen and four replicas fall into the loop; the healthy-replica count drops to 5.29. As the window lengthens, killing falls off and detection latency grows. Two rows never kill during startup, with tolerances of thirty and sixty rounds: they detect the stuck state in twenty-five and fifty rounds. The last row shows a single number in the stuck-detection column — with a sixty-round window, it never detected the second stuck state, at round one hundred forty, before the measurement horizon ran out. The bottom table repeats the same sweep with a startup probe that grants a thirty-round startup allowance. The killed-during-startup column is **zero in every row**, and so is the loop column. The tension is gone, because the two jobs were split into two settings: the startup probe covers the startup window, and the liveness probe now only looks at replicas that have finished starting up. Period and threshold can now be chosen by a single criterion — detection latency for the stuck state. The fastest row is the four-round window: it detects in two rounds, and its healthy-replica count of 7.48 is the sweep's highest. The distinction among the three probes is exactly the difference between these two tables. The same liveness setting produces 113 unnecessary kills in the top table and zero in the bottom. The only thing that changed is the system's ability to distinguish **starting up from failed**. That distinction is an information item, and its cost is running one more probe; its payoff is that the liveness setting becomes independent of the application's startup time. #### Summary - In the cluster without probes, two permanent stuck states went entirely unnoticed and 1408 requests failed; 300 of those went to replicas still starting up — the share attributable to having no readiness probe. - The single combined probe cut failed requests to 330, but 21 of 23 restarts were unnecessary: a probe that cannot tell dependency loss apart from stuck state fixes a self-resolving condition with a restart. - Limiting the liveness probe's scope to the process and separating out the readiness probe cut failed requests to zero, but the two slow-starting replicas kept getting killed mid-startup and fell into a loop. - The restart loop is invisible in the error rate: with failed requests at zero, the actually-healthy replica count was 5.89 against a target of 8; the loss shows up in capacity, not in the failure log. - When one setting does two jobs, shortening the tolerance window speeds up detection and pushes killing during startup to 113; lengthening it ends the killing, but a 60-round window never catches the second stuck state at all. - Once the startup probe is added, killing during startup goes to zero in all nine rows of the sweep; the liveness setting can now be chosen by detection latency alone, and the 4-round window is both the fastest and the highest-capacity row. #### Next Step The controllers in these two lessons act on the cluster without restriction: they change the deployment object, kill replicas, write pool membership. People and other automated processes use the same interface, and none of them has yet run into the question "are you authorized to do this." The next lesson turns that question into a number: it builds a subject × verb × object privilege matrix, measures how many operations drop out going from broad privilege to least privilege and how many lines of role definition that costs — and counts the paths by which privilege granted to one subject can reach another subject. ### Role-Based Access Control Source: https://academia.sh/en/courses/orchestration/role-based-access-control A subject × verb × object × namespace privilege matrix is built, and the number of operations that drop going from broad privilege down to least privilege is counted: permitted operations, the line cost of role definitions, and the escalation paths by which one subject can reach another subject's privilege are all measured. The controllers in the previous two lessons did their work in the cluster without restriction. The deployment controller changed the deployment object, deleted pods, wrote pool membership; the health probes killed processes. People and other automated processes use the same interface, and none of them has run into the question "are you authorized to do this" yet. Identity and least privilege were measured earlier; the difference here is that privilege is defined **over cluster objects**, and every object carries the possibility of reaching other objects. The delegated decision is this: the cluster interface decides whether to execute a request by looking at who made it. It needs three items to do this — who, what they want to do, on what — and a fourth is added to those three: in which namespace. The product of these items is the table the system has to fill in. This lesson measures that table's size, the cost of filling it in, and the price of **leaving it empty**. - **OO13.** The model carries the fictional regional measurement network's eight subjects, seven verbs, eight object types, and three namespaces. No real cluster is run; what is measured is a model built with `node`. - **OO14.** The operation trace is treated as if extracted from a thirty-day record, and is hand-written. An operation absent from the trace is taken as "not needed." Maintenance jobs that run rarely do not show up in a trace like this; least privilege drops them too, and they get blocked the first time they run. - **OO15.** The namespace here is the **cluster's** namespace: the partition into which objects are grouped and privilege is bounded. It is the same word as the namespace in the sense of the single-machine isolation's visibility mapping, but a different object. - **OO16.** Privilege escalation is reduced to two rules: a subject that can write a role definition can grant itself every privilege in that namespace; a subject that can read a secret object can take over a subject's identity in that namespace. In reality there are more paths, so the counted value is a lower bound. - **OO17.** Role-definition lines are counted with our own small schema, defined within the lesson. Line count depends on formatting choices, not on the order of magnitude. - **OO18.** Subjects with the same privilege set collapse into a single role. Role names are generated; readable names would not change the line count. #### Privilege Matrix ```js // privilege.mjs — subject x verb x object x namespace privilege matrix; a model, not a real cluster export const SUBJECT = ['reading-collector', 'verifier', 'billing', 'work-order', 'nightly-batch-job', 'release-controller', 'on-call', 'dashboard-reader']; export const VERB = ['read', 'list', 'watch', 'create', 'update', 'delete', 'authorize']; export const OBJECT = ['pod', 'deployment', 'service-object', 'config-object', 'secret-object', 'quota', 'role-definition', 'node']; export const NAMESPACE = ['development', 'staging', 'production']; const P = ['production'], S = ['staging', 'production'], T = NAMESPACE; // Thirty-day operation trace; hand-written, no randomness. export const TRACE = [ ['reading-collector', 'read', 'config-object', P], ['reading-collector', 'list', 'pod', P], ['verifier', 'read', 'config-object', S], ['verifier', 'read', 'secret-object', S], ['verifier', 'list', 'service-object', S], ['billing', 'read', 'config-object', P], ['billing', 'read', 'secret-object', P], ['work-order', 'read', 'config-object', P], ['work-order', 'list', 'service-object', P], ['nightly-batch-job', 'read', 'secret-object', P], ['nightly-batch-job', 'create', 'pod', P], ['nightly-batch-job', 'list', 'pod', P], ['release-controller', 'update', 'deployment', T], ['release-controller', 'list', 'pod', T], ['release-controller', 'delete', 'pod', T], ['release-controller', 'watch', 'pod', T], ['on-call', 'read', 'pod', T], ['on-call', 'list', 'pod', T], ['on-call', 'watch', 'pod', T], ['on-call', 'read', 'secret-object', P], ['dashboard-reader', 'list', 'pod', T], ['dashboard-reader', 'list', 'service-object', T], ]; export const key = (s, v, o, n) => `${s}|${v}|${o}|${n}`; export function leastPrivilege() { const set = new Set(); for (const [s, v, o, namespaces] of TRACE) for (const n of namespaces) set.add(key(s, v, o, n)); return set; } export function broadPrivilege() { // every subject gets every verb, object, namespace const set = new Set(); for (const s of SUBJECT) for (const v of VERB) for (const o of OBJECT) for (const n of NAMESPACE) set.add(key(s, v, o, n)); return set; } // Escalation: a subject that can write a role definition can grant itself any privilege in that // namespace; a subject that can read a secret object can take over a subject's identity in that namespace. export function escalation(granted) { const has = (s, v, o, n) => granted.has(key(s, v, o, n)); const namespaceSubjects = (n) => SUBJECT.filter((s) => [...granted].some((k) => k.startsWith(`${s}|`) && k.endsWith(`|${n}`))); const edge = new Set(); for (const s of SUBJECT) for (const n of NAMESPACE) { const writesRole = ['create', 'update', 'authorize'].some((v) => has(s, v, 'role-definition', n)); const readsSecret = has(s, 'read', 'secret-object', n); if (!writesRole && !readsSecret) continue; for (const t of namespaceSubjects(n)) if (t !== s) edge.add(`${s}->${t}`); } let n = edge.size, prev; // transitive closure do { prev = n; for (const k1 of [...edge]) for (const k2 of [...edge]) { const [a1, b1] = k1.split('->'), [a2, b2] = k2.split('->'); if (b1 === a2 && a1 !== b2) edge.add(`${a1}->${b2}`); } n = edge.size; } while (n !== prev); return edge; } ``` The difference between a grant list and a matrix is this: the matrix carries every cell, the trace only the filled ones. The `broadPrivilege` function fills every cell — it hands every subject everything, without asking a single question. `leastPrivilege` fills in only what appears in the trace. The entire difference between the two is the cost of looking at the thirty-day record. #### From Broad Privilege to Least Privilege ```js // run5.mjs — broad privilege is compared with least privilege import { SUBJECT, VERB, OBJECT, NAMESPACE, TRACE, leastPrivilege, broadPrivilege } from './privilege.mjs'; const g = broadPrivilege(), e = leastPrivilege(); console.log(`subject ${SUBJECT.length} x verb ${VERB.length} x object ${OBJECT.length} x namespace ${NAMESPACE.length}` + ` = ${SUBJECT.length * VERB.length * OBJECT.length * NAMESPACE.length} possible operations`); console.log(`observed trace: ${TRACE.length} lines, ${e.size} distinct operations once opened across namespaces\n`); console.log('metric broad privilege least privilege diff'); const row = (name, x, y) => console.log(`${name.padEnd(35)} ${String(x).padEnd(17)} ${String(y).padEnd(17)} ${x - y}`); row('permitted operations', g.size, e.size); row('privileged subjects', SUBJECT.length, new Set([...e].map((k) => k.split('|')[0])).size); const sv = (s) => new Set([...s].map((k) => k.split('|').slice(0, 2).join('|'))).size; row('subject-verb pairs', sv(g), sv(e)); const oo = (s) => new Set([...s].map((k) => k.split('|')[2])).size; row('object types touched', oo(g), oo(e)); const sec = (s) => [...s].filter((k) => k.split('|')[1] === 'read' && k.split('|')[2] === 'secret-object').length; row('secret-object read grants', sec(g), sec(e)); const role = (s) => [...s].filter((k) => k.split('|')[2] === 'role-definition').length; row('grants touching role-definition', role(g), role(e)); ``` ``` subject 8 x verb 7 x object 8 x namespace 3 = 1344 possible operations observed trace: 22 lines, 43 distinct operations once opened across namespaces metric broad privilege least privilege diff permitted operations 1344 43 1301 privileged subjects 8 8 0 subject-verb pairs 56 18 38 object types touched 8 5 3 secret-object read grants 24 5 19 grants touching role-definition 168 0 168 ``` One thousand three hundred one operations dropped out — ninety-seven percent of the matrix. The remaining forty-three operations are everything the eight subjects actually did over thirty days. The second row shows **which axis this drop did not happen on**: the number of privileged subjects went from eight to eight — unchanged. Least privilege does not remove anyone from the cluster; it narrows what everyone **can do**. The narrowing is sharpest on the object axis. Only five of eight object types are touched; three types — quota, role definition, and node — were used by no subject at all over thirty days. Two of these belong to the cluster's own operation, and there is no reason for workload subjects to reach them. The last row gives the count: broad privilege had one hundred sixty-eight grants touching role definitions, least privilege has zero. All one hundred sixty-eight of those grants sat open unused, and every one of them could have been used to widen itself. The secret-object read row, on the other hand, shows where the drop stops: not from twenty-four to zero, but to **five**. Four subjects genuinely access secrets — the verifier across two namespaces, billing, the nightly batch job, and on-call in production. These five grants are in the trace, so least privilege cannot drop them. #### Writing Cost and Escalation Paths Least privilege's second cost does not show up in the first table: the matrix's unfilled cells are free, but every filled one has to be written. To count this, the privilege set is written out to text in our own schema. ```js // run6.mjs — role definitions for both models are written in our own schema, and lines are counted import { SUBJECT, leastPrivilege, broadPrivilege, escalation } from './privilege.mjs'; // Schema: "role " / " : ," / "bind -> @ " function write(grants) { const grouped = new Map(); // (subject, namespace) -> object:verbs for (const k of grants) { const [s, v, o, n] = k.split('|'); const m = grouped.get(`${s}|${n}`) ?? new Map(); m.set(o, (m.get(o) ?? new Set()).add(v)); grouped.set(`${s}|${n}`, m); } const role = new Map(), bind = []; // the same privilege set collapses to one role for (const [sn, m] of grouped) { const body = [...m].sort().map(([o, vs]) => ` ${o}: ${[...vs].sort().join(',')}`).join('\n'); if (!role.has(body)) role.set(body, `role-${role.size + 1}`); const [s, n] = sn.split('|'); bind.push(`bind ${role.get(body)} -> ${s} @ ${n}`); } const text = [...role].map(([b, name]) => `role ${name}\n${b}`).concat(bind).join('\n'); return { text, roles: role.size, binds: bind.length, lines: text.split('\n').length }; } const g = write(broadPrivilege()), e = write(leastPrivilege()); console.log('model roles binds total lines grants grants/line'); for (const [name, y, grants] of [['broad privilege', g, broadPrivilege()], ['least privilege', e, leastPrivilege()]]) { console.log(`${name.padEnd(17)} ${String(y.roles).padEnd(6)} ${String(y.binds).padEnd(6)} ` + `${String(y.lines).padEnd(12)} ${String(grants.size).padEnd(7)} ${(grants.size / y.lines).toFixed(1)}`); } console.log('\ntwo roles and two bindings from the least-privilege definition:'); console.log(e.text.split('\n').slice(0, 6).join('\n')); console.log(e.text.split('\n').filter((l) => l.startsWith('bind')).slice(0, 2).join('\n')); // what happens to escalation paths if secret-object read is removed const trimmed = new Set([...leastPrivilege()].filter((k) => !(k.split('|')[1] === 'read' && k.split('|')[2] === 'secret-object'))); console.log('\nescalation path count and subjects reaching every subject:'); for (const [name, s] of [['broad privilege', broadPrivilege()], ['least privilege', leastPrivilege()], ['secret read removed', trimmed]]) { const k = escalation(s); const full = SUBJECT.filter((o) => [...k].filter((x) => x.startsWith(`${o}->`)).length === SUBJECT.length - 1).length; console.log(` ${name.padEnd(20)} paths ${String(k.size).padEnd(4)} reach every subject ${full} grants ${s.size}`); } ``` ``` model roles binds total lines grants grants/line broad privilege 1 24 33 1344 40.7 least privilege 9 15 42 43 1.0 two roles and two bindings from the least-privilege definition: role role-1 config-object: read pod: list role role-2 config-object: read secret-object: read bind role-1 -> reading-collector @ production bind role-2 -> verifier @ staging escalation path count and subjects reaching every subject: broad privilege paths 56 reach every subject 8 grants 1344 least privilege paths 28 reach every subject 4 grants 43 secret read removed paths 0 reach every subject 0 grants 38 ``` Writing cost went from thirty-three lines to forty-two — a twenty-seven percent increase, nine lines. This is the entire cost of dropping one thousand three hundred one operations. The last column reduces the relationship between the two to a single number: every line written under broad privilege opens forty operations, under least privilege exactly one. Broad privilege's cheapness comes from the lines it **does not write**; under least privilege, every permitted operation has to be written out individually, because no rule generalizes. This is also where the maintenance burden of forty-two lines comes from: when the trace changes, the definition has to change. The bottom table is this lesson's real result. Under broad privilege there are fifty-six escalation paths, and all eight subjects reach every other subject — the expected result. Under least privilege, permitted operations dropped to forty-three, but escalation paths dropped only by half, to twenty-eight, and **four subjects still reach every other subject**. Dropping ninety-seven percent of permitted operations dropped only fifty percent of escalation paths. The two numbers do not measure the same thing. The third row gives the reason. When secret-object read is removed, permitted operations drop from forty-three to thirty-eight — five operations — and escalation paths drop from twenty-eight to **zero**. Those five grants alone were carrying all twenty-eight paths. None of the remaining thirty-eight grants lets one subject stand in for another. These five grants were not unnecessary; they were in the trace, meaning those processes genuinely accessed secrets. What the number says is not that the grant should be removed, but that **the secret should reach the process by some route other than being read from the cluster**. The decision now becomes tied to a measure: removing one grant costs five operations and buys back twenty-eight escalation paths. The same calculation done for quota, role definition, and node would have returned zero gain, because those grants were already closed. #### What the System Does Not Know When the cluster interface evaluates a request, it looks for the answer to a single question: is this in the four-column table. It does not know intent, context, time, or sequence. On-call reading a secret object in production falls into the same cell whether it happens during an outage or on an ordinary Tuesday morning; the table cannot tell the two apart, because it carries no item that would let it. For the same reason, the question "when was this grant last used" cannot be answered from the table either — which of the forty-three grants are still live is known only by re-gathering the trace. This means the delegated decision is an information item that has to be fed continuously: the table does not stay correct once it is built correctly; it ages as the trace changes. Aging has two directions, and their costs are not symmetric. When the trace expands — a new service, a new maintenance job — the table stays narrow and the system **rejects a legitimate operation**; this shows up immediately, because someone reports being blocked. When the trace narrows — a process retires, a verb falls out of use — the table stays broad and the system **keeps accepting an unnecessary operation**; this never shows up, because no one complains. The one hundred sixty-eight role-definition grants under broad privilege sat exactly like that: unused, bothering no one, and carrying most of the fifty-six escalation paths. The value of least privilege derived from a thirty-day trace is that it closes this second direction **once**; keeping it closed requires the trace measurement to continue. #### Summary - Eight subjects, seven verbs, eight object types, and three namespaces build a 1344-cell matrix; the thirty-day trace fills 43 of those, and dropping to least privilege removes 1301 operations. - The drop is not on the subject axis: the number of privileged subjects stayed fixed at eight; what narrows is what each subject can do. - 168 grants touching role definitions drop to zero under least privilege; none of them had been used in thirty days. - Least privilege's writing cost goes from 33 lines to 42, a twenty-seven percent increase; under broad privilege each written line opens 40.7 operations, under least privilege exactly 1. - The count of permitted operations and the count of escalation paths do not measure the same thing: when ninety-seven percent of operations dropped, paths fell only by half, and four subjects still reached every other subject. - The remaining 28 paths were carried entirely by five secret-read grants; once those five close, the path count drops to zero. The cost and the gain of removing a grant can be measured separately. #### Next Step This lesson's matrix bounds what a subject can do **through the cluster interface**. Traffic between two pods does not pass through this matrix: the verifier connects directly to the billing service's endpoint and never touches the cluster interface at all. Two subjects that cannot reach each other in the privilege matrix can reach each other freely over the network. The next lesson builds the reachability matrix with policy, and counts the difference between default-open and default-closed: how many paths close, how many legitimate paths close by mistake, how many rules the policy needs, and how many rules get updated when a new service is added. ### Network Policies Source: https://academia.sh/en/courses/orchestration/network-policies This time the reachability matrix is built with policy, and the difference between default-open and default-closed is counted: how many paths close, how many legitimate paths close by mistake, how many rules the policy needs, and how many rules get updated when a new service is added. The previous lesson's matrix bounded what a subject could do through the cluster interface. Traffic between two pods does not pass through that matrix: the verifier connects directly to the billing service's endpoint and never touches the cluster interface at all. Two subjects that cannot reach each other in the privilege matrix can reach each other freely over the network. A reachability matrix was already built and measured earlier, in the context of single-machine network isolation — which pod can reach which pod. That measurement is not repeated here; this lesson's question is building the same matrix **with policy**. The delegated decision is this: the network proxy decides whether to admit a connection request by looking at who the source and destination are. It needs the following items for this: which group a pod belongs to, which flow between which groups is legitimate, and through which channel. The first item is already in the cluster — pod labels. So is the third. What is missing is the second, and nobody measures it; **someone has to write it**. This lesson measures the cost of writing it, and of not writing it. - **OO19.** There are ten service groups and three channel types, all belonging to the fictional regional measurement network. No real cluster is run; what is measured is a model built with `node`. - **OO20.** The set of legitimate flows is twenty-two, and four of them do not appear in the thirty-day trace. This distinction was hand-written; in reality, which flow is sparse can only be known by extending the observation period. - **OO21.** Channel type is reduced to three values. In reality, port number and protocol are separate axes; they would enlarge the matrix without changing the result's direction. - **OO22.** Policy is written only by destination, meaning it bounds incoming traffic. There are no rules in this model that separately bound outgoing traffic; if there were, the rule count would grow without the closed-path count growing. - **OO23.** Rule count is counted with our own small schema, defined within the lesson. A label assignment does not count as a rule; it is shown in a separate column. - **OO24.** A path is either open or closed. Rate limiting, mutual authentication, and per-connection authorization are not modeled. #### Policy Model Policy is written in our own small schema: a rule specifies the target label, the label the source has to carry, and the channel. Rules only grant; under a default-closed setup, every path that is not written is closed. ```js // network.mjs — reachability matrix between pod groups and policy; a model, not a real cluster export const CHANNEL = ['api', 'data', 'metric']; export const SERVICE = { // service -> labels ingress: ['edge'], 'reading-collector': ['app'], verifier: ['app'], billing: ['app'], 'work-order': ['app'], 'nightly-batch-job': ['batch'], 'data-store': ['store'], queue: ['store'], 'metric-collector': ['monitoring'], dashboard: ['monitoring'], }; // Genuinely legitimate flows; "sparse" ones did not appear in the thirty-day trace. export const LEGITIMATE = [ ['ingress', 'reading-collector', 'api'], ['ingress', 'billing', 'api'], ['ingress', 'work-order', 'api'], ['reading-collector', 'queue', 'data'], ['verifier', 'queue', 'data'], ['verifier', 'data-store', 'data'], ['reading-collector', 'data-store', 'data'], ['billing', 'data-store', 'data'], ['work-order', 'data-store', 'data'], ['nightly-batch-job', 'data-store', 'data'], ['nightly-batch-job', 'queue', 'data'], ['metric-collector', 'reading-collector', 'metric'], ['metric-collector', 'verifier', 'metric'], ['metric-collector', 'billing', 'metric'], ['metric-collector', 'work-order', 'metric'], ['dashboard', 'metric-collector', 'api'], ['billing', 'work-order', 'api'], ['work-order', 'queue', 'data'], ['nightly-batch-job', 'billing', 'api', 'sparse'], ['metric-collector', 'nightly-batch-job', 'metric', 'sparse'], ['verifier', 'billing', 'api', 'sparse'], ['ingress', 'dashboard', 'api', 'sparse'], ]; // Label policy, in our own schema: "target