--- title: 'Operator Pattern' source: 'https://academia.sh/en/courses/orchestration/controller-pattern' course: 'Container Orchestration' language: en updated: '2026-09-13T05:08:27+00:00' license: 'CC BY-SA 4.0' objectives: - "Counts the operating knowledge coded into the controller item by item and shows each item's payoff" - 'Compares manual operation with the controller by a single measure: how many steps, how many rounds, to close one event' - 'Shows with numbers why a condition the controller does not know produces oscillation, not a fix' --- # Operator Pattern For the previous lesson's resource type, a controller that genuinely runs is written: it reads the desired state, reads the real state, closes the gap, and does zero work on the second round. A manual operation's steps and minutes to close one event are compared against the controller's rounds; the coded operating knowledge is counted item by item, and the oscillation the one uncoded item produces is measured across eighty rounds. The previous lesson defined the reading-pipeline type and measured that a schema says a declaration is valid, not that it is correct. An accepted declaration does nothing on its own in the cluster: something needs to bring the real state to match it, and today that something is an on-call human. The **operator pattern** is what moves this watch itself into code. The reconciliation loop's mechanics — desired state, real state, gap, fix, and idleness — were measured in M22/K01's **Infrastructure Provisioning Models** lesson; they are not repeated here. This lesson's subject is not the loop itself but **what gets written into the loop**: how many items of the on-call knowledge in the operator's head made it into code, how many did not, and what the ones that did not cost. The delegated decision is plain: **a controller, not a human, decides what to do about an event.** The measurement is again over the **regional measurement network** and is **fictional**; no real cluster is run, and what is measured is a **model** built with `node`. - **OO43.** The controller is written for the previous lesson's reading-pipeline type; the desired state comes from that declaration — four workers and a 960-second latency limit. - **OO44.** One round is one minute of model time; 80 readings arrive per round, and each running worker processes 40 readings. - **OO45.** Eighty rounds are run, and the target service is under maintenance between rounds 10 and 59; the window is **hand-written**, there is no randomness. - **OO46.** The controller's coded knowledge is five items, and they are listed item by item. - **OO47.** The manual-operation recipe is nine steps; step durations are hand-written, and it is a **model**. - **OO48.** A restart finishes in one round and puts forty readings back on the queue. ## What the Controller Knows ```js // world.mjs — the reading pipeline's fictional world, and the controller's coded knowledge. export const DESIRED = { workers: 4, status: "running", limit: 960 }; // from the declaration in lesson 07 export const newWorld = () => ({ status: "stopped", workers: 0, queue: 400, latency: 300, target: "open", stubborn: 0, wait: 3 }); export function advance(d) { // one round = one minute of model time const capacity = d.status === "running" && d.target === "open" ? d.workers * 40 : 0; const previous = d.latency; d.queue = Math.max(0, d.queue + 80 - capacity); // 80 readings arrive per round d.latency = Math.round((d.queue / 80) * 60); d.stubborn = d.latency >= previous ? d.stubborn + 1 : 0; } export const RULE = [ // operating knowledge coded into the controller { name: "a stopped pipeline gets started", f: (d, i) => d.status === "stopped" && ((d.status = "running"), true) }, { name: "worker count is completed to the declared value", f: (d, i) => d.workers < i.workers && ((d.workers = i.workers), true) }, { name: "a worker is added when latency is above the limit", f: (d, i) => d.latency > i.limit && d.workers < 8 && (d.workers++, true) }, { name: "excess workers are reclaimed when latency is below the limit", f: (d, i) => d.latency <= i.limit && d.workers > i.workers && (d.workers--, true) }, { name: "the pipeline restarts when latency stays stubbornly high", f: (d, i) => d.latency > i.limit && d.stubborn >= d.wait && ((d.stubborn = 0), (d.workers = i.workers), (d.queue += 40), true) }, ]; export const UNCODED = { name: "do not touch while the target is under maintenance", f: (d) => d.target === "maintenance" }; export function reconcile(d, i, ruleset, backoff = false) { if (ruleset.wait && ruleset.wait(d)) { d.stubborn = 0; return []; } // once the sixth item is added const actions = []; for (const k of ruleset.list) if (k.f(d, i)) actions.push(k.name); if (backoff && actions.includes("the pipeline restarts when latency stays stubbornly high")) d.wait = Math.min(d.wait * 2, 48); // growing retry interval return actions; } ``` The five rules are five sentences from the on-call operator's head. The first two can be read from the declaration: the pipeline should be running, the worker count should be four. The last three cannot be read from the declaration — they are knowledge that belongs not to the declaration but to **the person running the pipeline**: what to do once the latency limit is exceeded, what to do once latency returns to normal, and what to do if latency stays stubbornly high. This is exactly the delegation the operator pattern describes — these three sentences. The items the world model chooses to hold are countable too. The controller can read four items — status, worker count, queue length, and the latency derived from it. A fifth item sits in the model too, and directly determines the pipeline's behavior: whether the target service is open or under maintenance. That item exists in the world, but **it appears in no rule**; the controller does not read it. Every deviation measured throughout this lesson traces back to this one line. ## Idleness and Comparison with Manual Operation ```js // run14.mjs — idleness, comparison with manual operation, cost of uncoded knowledge, backoff. import { DESIRED, newWorld, advance, RULE, UNCODED, reconcile } from "./world.mjs"; const ROUND = 80, MAINTENANCE = [10, 59]; const runModel = (ruleset, backoff = false) => { const d = newWorld(); const s = { actions: 0, restarts: 0, workerChanges: 0, peakQueue: 0, exceededRounds: 0, firstRound: 0, secondRound: 0, reads: 0, trace: [] }; for (let t = 0; t < ROUND; t++) { d.target = t >= MAINTENANCE[0] && t <= MAINTENANCE[1] ? "maintenance" : "open"; const i = reconcile(d, DESIRED, ruleset, backoff); s.reads += 3; // three items are read every round s.actions += i.length; if (t === 0) s.firstRound = i.length; if (t === 1) s.secondRound = i.length; s.restarts += i.filter((x) => x.startsWith("the pipeline restarts")).length; s.workerChanges += i.filter((x) => x.includes("worker")).length; advance(d); if (t >= 30 && t < 50) s.trace.push(d.workers); s.peakQueue = Math.max(s.peakQueue, d.queue); if (d.latency > DESIRED.limit) s.exceededRounds++; } return s; }; const missing = { list: RULE }, full = { list: RULE, wait: UNCODED.f }; const a = runModel(missing), b = runModel(full), c = runModel(missing, true); console.log(`coded knowledge ${RULE.length} items; uncoded 1 item: ${UNCODED.name}`); console.log(`idleness: ${a.firstRound} actions in round 1, ${a.secondRound} actions in round 2`); console.log(`\n${ROUND} rounds, target under maintenance between rounds ${MAINTENANCE[0]}-${MAINTENANCE[1]}`); console.log("run actions restarts worker changes peak queue exceeded rounds reads"); const show = (name, s) => console.log(`${name.padEnd(29)} ${String(s.actions).padStart(4)}` + `${String(s.restarts).padStart(11)}${String(s.workerChanges).padStart(17)}` + `${String(s.peakQueue).padStart(12)}${String(s.exceededRounds).padStart(18)}${String(s.reads).padStart(7)}`); show("5 items (missing info)", a); show("5 items + backoff", c); show("6 items (maintenance known)", b); console.log(`\nworker-count trace mid-maintenance (rounds 30-49)`); console.log(` 5 items : ${a.trace.join(" ")}`); console.log(` 6 items : ${b.trace.join(" ")}`); const MANUAL = [["alert box opens", 2], ["pipeline status is read", 3], ["queue length is read", 3], ["worker count is read", 2], ["target service's status is queried", 8], ["maintenance window is confirmed", 6], ["a decision is made", 4], ["the action is applied", 3], ["a record is logged", 3]]; const min = MANUAL.reduce((t, [, m]) => t + m, 0); console.log(`\nmanual operation for one event: ${MANUAL.length} steps, ${min} minutes` + ` with the controller: 1 round to detect + 1 round to act = 2 rounds, 15 sec/round -> 0.5 minutes`); console.log(`the same event a second time: manual is still ${MANUAL.length} steps/${min} min, controller still 2 rounds`); ``` ``` coded knowledge 5 items; uncoded 1 item: do not touch while the target is under maintenance idleness: 2 actions in round 1, 0 actions in round 2 80 rounds, target under maintenance between rounds 10-59 run actions restarts worker changes peak queue exceeded rounds reads 5 items (missing info) 56 12 43 4440 48 240 5 items + backoff 22 3 18 4120 45 240 6 items (maintenance known) 10 0 9 4000 46 240 worker-count trace mid-maintenance (rounds 30-49) 5 items : 4 5 6 4 5 6 4 5 6 4 5 6 4 5 6 4 5 6 4 5 6 items : 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 manual operation for one event: 9 steps, 34 minutes with the controller: 1 round to detect + 1 round to act = 2 rounds, 15 sec/round -> 0.5 minutes the same event a second time: manual is still 9 steps/34 min, controller still 2 rounds ``` Idleness shows up in two lines: the first round does two things — starts the stopped pipeline and raises the worker count from zero to four — the second round **does zero work**. The controller does not remember what it did; it re-reads the real state every round and, finding no gap, does not touch anything. The comparison with manual operation reduces to one line too. When the same event is closed manually, it takes nine steps and thirty-four minutes; **five of the nine steps are pure reading** — the pipeline's status, queue length, worker count, target service's status, the maintenance window — and these five steps eat twenty-two of the thirty-four minutes. Deciding and acting together take seven minutes. The expensive part of manual operation is not the work done, it is the information gathered to do the work. With the controller, an event closes in two rounds: one round to see the gap, one round to apply the action; at fifteen seconds per round, that is half a minute. The real difference is in the second line: **when the same event happens a second time, the manual path is still nine steps and thirty-four minutes**, and the controller is still two rounds. In manual operation, cost multiplies by the number of events; with the controller, it is paid once. Information's cost is countable too: the controller reads three items in every one of the eighty rounds — status, worker count, latency — two hundred forty reads in total. These reads happen even when there is no event; on quiet rounds, the only thing the controller produces is a read. ## Uncoded Knowledge and Oscillation All three runs see the same world and live through the same fifty-round maintenance window. The only difference between them is what the controller knows. The five-item controller performs **fifty-six actions** through maintenance: forty-three worker changes and twelve restarts. The worker trace shows the oscillation bare — `4 5 6 4 5 6 4 5 6` — a three-round cycle. Because latency is above the limit, the third rule adds a worker; because latency does not fall, the fifth rule restarts the pipeline three rounds later; the restart pulls the worker count back to four and adds forty readings to the queue. Then the same three rounds start over. The real point here is not the action count. **The controller cannot fix a condition it does not know about, and it also cannot give up trying to fix it.** The cause of the latency is not in the pipeline itself; the target service is under maintenance, and how many workers the pipeline runs with changes nothing. All the controller sees is "latency above the limit," so it tries the two tools it has, one after the other. Each of the twelve restarts puts forty readings back on the queue; the peak queue climbs to four thousand four hundred forty — four thousand for the controller that knows. The four hundred forty readings in between are **the delay the fixing attempt itself produces**. There are two separate remedies, and they measure two separate things. The first is **widening the retry interval**: once the wait time doubles after every restart (three, six, twelve, twenty-four rounds), actions drop from fifty-six to twenty-two and restarts from twelve to three. The controller is still doing the wrong thing, only less often. The second is **adding the sixth item of knowledge**: do not touch the pipeline while the target is under maintenance. Then actions drop to ten, restarts to zero, and the worker trace becomes a flat line. The distinction between the two remedies matters, because they belong to different classes of thing. Backoff **does not code the uncoded knowledge**; it never changes what the controller knows, it only lowers how often the wrong decision fires. The sixth item, on the other hand, adds an information item and creates a read obligation in return: the target service's maintenance status must now be read every round. Put into numbers, the gap between the two paths is twelve actions (twenty-two against ten), but there is a side the number does not say — backoff has to be retuned for every new event type, while the sixth item gets written once. The sixth item does not **solve** the event. The exceeded-round count sits between forty-five and forty-eight across all three runs; the latency limit still gets exceeded, because the target is genuinely closed. What gets won is not the event's duration, it is **the damage added on top of the event**: forty-six actions, twelve restarts, and the four-hundred-forty-reading extra queue all disappear. ## Delegation's Balance Sheet | Measure | Manual operation | 5 items | 5 items + backoff | 6 items | |---|---|---|---|---| | closing one event | 9 steps, 34 min | 2 rounds | 2 rounds | 2 rounds | | the same event a second time | 9 steps, 34 min | 2 rounds | 2 rounds | 2 rounds | | actions in 80 rounds | — | 56 | 22 | 10 | | restarts | — | 12 | 3 | 0 | | worker changes | — | 43 | 18 | 9 | | peak queue | — | 4440 | 4120 | 4000 | | exceeded rounds | — | 48 | 45 | 46 | | reads | 5 steps / event | 240 | 240 | 240 | Delegation's information items are countable: five coded items, one uncoded item. Two of the coded items are read from the declaration, three are not — all three come from the on-call operator's experience and sit nowhere until they are written into the controller. Information's cost is countable too: three reads per round, two hundred forty reads across eighty rounds, and the same two hundred forty reads even when there is no event. The table's last row carries delegation's silent side. Manual operation reads only when there is an event — five steps per event — while the controller reads on every single round. The price paid for forty-eight exceeded rounds is two hundred forty reads, and most of that accumulates on rounds where nothing changed at all. Delegating the decision makes it mandatory to produce the information the decision rests on **even when there is no event**. The wrong decision's source sits in exactly one place: **the controller does not read the target service's maintenance status.** Because it does not read it, it cannot tell "latency is high" apart from "the cause of the latency is in me," so it tries every tool it has, one after another. Delegating a decision requires producing every information item that decision rests on, continuously and on time; a missing item does not stop the fixing, it just **points it in the wrong direction**. ## Summary - The controller did 2 things in the first round, 0 in the second; it re-reads the real state every round and does not remember what it did. - A manual event is 9 steps and 34 minutes, a controller event is 2 rounds; when the event happens a second time, the manual cost repeats exactly, the controller's does not. - Coded operating knowledge is 5 items; 2 are read from the declaration, 3 belong only to the person running the pipeline. - The one uncoded item produced 56 actions, 12 restarts, and a three-round oscillation shaped like `4 5 6 4 5 6` across 80 rounds; once the retry interval doubled, actions dropped to 22, restarts to 3. - Adding the sixth item dropped actions to 10, restarts to 0, but exceeded rounds only moved from 48 to 46: what gets won is not the event's duration, it is the 440-reading damage added on top of it. ## Next Step Every number in this lesson came out of a single pipeline's eighty-round trace, and what kept that trace was measurement: three items per round, two hundred forty reads in total. A cluster holds not one but hundreds of resources, and every one of their controllers does the same reading. The next lesson measures what it costs to collect all these reads: how many resources, how many series per resource, how many records and how many bytes once multiplied by a retention period. It counts why series count grows multiplicatively rather than additively as label cardinality rises, and how much detail gets lost against how many bytes get won once sampling and aggregation are applied.