Local and Managed Clusters

This topic measured what a cluster is, which components it is built from, and how it reconciles. One operational decision remains: whether you will run this plane yourself, or leave its operation to a party that takes it over. The two options are referred to here as types — the cluster you operate yourself and the cluster with delegated operations — and are compared on the same items.

This decision is one layer above this course’s measure. The previous lessons counted the cost of delegating the placement and replica-count decision to a system; this lesson counts the cost of delegating the operation of that system itself. The questions are the same: which items get delegated, what information and control get lost in exchange, and is the decision reversible.

The measurement runs, once again, on the regional measurement network, and it is fictional; what runs are models built with node.

CA26. Operation is represented by twenty-one items; each item’s monthly hours, owner, and whether it is rebuilt on exit are written down. CA27. For shared items, half the hours go to the provider. CA28. For items rebuilt on exit, the initial setup load is four times the monthly hours. CA29. A release ships every four months, its support lasts twelve months; the run is thirty-six months. CA30. Under self-run operation, upgrades happen according to the team’s capacity; under delegated operation, they happen the month support ends, and the calendar belongs to the provider. CA31. The load per upgrade is 8 control-plane hours plus 7 node hours under self-run operation, and only 7 node hours under delegated operation. CA32. Report delay and startup time are taken from the previous lessons: average 13.53 and worst 26.36 seconds, startup time 5 seconds.

The Same Items, Two Owners

// items.mjs — the cluster operation's items. [name, monthly hours, who, rebuilt on exit?]
// "who" points to the same name in both options: the party taking over operations, shared, or you.
export const ITEMS = [
  ["control plane setup", 6, "provider", true],
  ["control plane version upgrades", 8, "provider", true],
  ["state store backup", 5, "provider", true],
  ["state store recovery", 4, "provider", true],
  ["API server certificates", 2, "provider", true],
  ["control plane redundancy", 5, "provider", true],
  ["control plane metric collection", 3, "provider", true],
  ["node provisioning", 6, "shared", false],
  ["node version upgrades", 7, "shared", false],
  ["node OS patching", 5, "shared", false],
  ["cluster autoscaling rules", 3, "shared", false],
  ["network configuration", 4, "shared", false],
  ["storage attachment", 3, "shared", false],
  ["incident response", 6, "shared", false],
  ["workload declarations", 9, "you", false],
  ["resource requests and limits", 5, "you", false],
  ["authorization rules", 4, "you", false],
  ["network policies", 3, "you", false],
  ["observability and alerting", 8, "you", false],
  ["release rollout", 7, "you", false],
  ["cost tracking", 3, "you", false],
];
const SHARE = 0.5;                       // for a shared item, half the hours go to the provider

if (import.meta.url === `file://${process.argv[1]}`) {
  const byOwner = (k) => ITEMS.filter(([, , x]) => x === k);
  const hours = (l) => l.reduce((t, [, s]) => t + s, 0);
  const total = hours(ITEMS);
  const delegated = hours(byOwner("provider")) + hours(byOwner("shared")) * SHARE;
  console.log("item  total  provider  shared  you");
  console.log(`count ${String(ITEMS.length).padStart(5)}${String(byOwner("provider").length).padStart(10)}` +
    `${String(byOwner("shared").length).padStart(9)}${String(byOwner("you").length).padStart(6)}`);
  console.log(`hours ${String(total).padStart(5)}${String(hours(byOwner("provider"))).padStart(10)}` +
    `${String(hours(byOwner("shared"))).padStart(9)}${String(hours(byOwner("you"))).padStart(6)}`);
  console.log(`\ncluster you operate  : items ${ITEMS.length}   monthly hours ${total}`);
  console.log(`operations delegated : items ${byOwner("shared").length + byOwner("you").length}` +
    `   monthly hours ${total - delegated}   delegated ${delegated}   share ${(100 * delegated / total).toFixed(1)}%`);
  const exitItems = ITEMS.filter(([, , , c]) => c);
  console.log(`exit cost            : items to rebuild ${exitItems.length}` +
    `   monthly hours ${hours(exitItems)}   initial setup ${hours(exitItems) * 4} hours (four months' equivalent)`);
}

item  total  provider  shared  you
count    21         7        7     7
hours   106        33       34    39

cluster you operate  : items 21   monthly hours 106
operations delegated : items 14   monthly hours 56   delegated 50   share 47.2%
exit cost            : items to rebuild 7   monthly hours 33   initial setup 132 hours (four months' equivalent)

Seven of the twenty-one items get delegated, seven get shared, seven stay with you no matter what. On the hours side, the delegated share is 50 of the monthly 106 hours — 47.2 percent. Item count falls from 21 to 14, monthly load from 106 hours to 56.

The most commonly misread part of this table is the remaining column. Delegating provides a reduction of close to half, but more than half stands unchanged: workload declarations, resource requests, authorization rules, network policies, observability, release rollout, and cost tracking do not shrink under any delegation, because all of these are items of your application, not the cluster. The delegation decision touches none of these items, and assuming it does is the most common wrong guess about post-delegation team load. The declarations that need to be written for the measurement network’s four services and nightly batch job are independent of who operates the cluster.

The middle seven items — the shared column — belong to exactly no one in either option, and this is where the surprises come from. Node provisioning, node version upgrades, network configuration, and incident response stay half yours even after delegating; if the delegation contract does not write down the boundary of these items, which half of the monthly 34 hours falls to whom becomes clear only at the first incident. On the measurement network, the counterpart of this is: if the collector’s node cannot take a copy because its capacity is full, both sides go looking separately for whether the problem is in the cluster or in the workload declaration.

The last line gives the decision’s reversibility. The seven delegated items must be rebuilt on exit; against a 33-hour monthly maintenance load, their initial setup is 132 hours. Exit cost is measured in item count: seven items, and none of these seven show up in day-to-day work, because as long as they are delegated, no one looks at them. The decision is reversible, but the cost of reversing it is a cost not paid on the day the delegation decision was made.

The Version-Upgrade Window

// window.mjs — a 36-month version-upgrade window. A release ships every 4 months, support lasts 12.
const MONTHS = 36, INTERVAL = 4, SUPPORT = 12, HOURS_PLANE = 8, HOURS_NODE = 7;
const lastRelease = (m) => Math.floor(m / INTERVAL) * INTERVAL;

function run({ name, team, mandatory }) {          // team: your own capacity (months); mandatory: the provider's
  let s = 0, upgrades = 0, unsupported = 0, oldest = 0, hours = 0;
  for (let m = 1; m <= MONTHS; m++) {
    if (mandatory ? m === s + SUPPORT : m % team === 0 && lastRelease(m) > s) {
      s = lastRelease(m); upgrades++; hours += mandatory ? HOURS_NODE : HOURS_PLANE + HOURS_NODE;
    }
    const age = m - s;
    oldest = Math.max(oldest, age);
    if (age > SUPPORT) unsupported++;
  }
  return [name, upgrades, unsupported, oldest, hours, mandatory ? "provider" : "you"];
}

const row = (r) => console.log(`${String(r[0]).padEnd(27)}${String(r[1]).padStart(10)}` +
  `${String(r[2]).padStart(16)}${String(r[3]).padStart(14)}${String(r[4]).padStart(7)}  ${r[5]}`);
row(["option", "upgrades", "unsupported mo", "oldest (mo)", "hours", "calendar"]);
for (const team of [4, 6, 12, 18, 24]) row(run({ name: `self-run (team ${team} mo)`, team }));
row(run({ name: "operations delegated", mandatory: true }));

option                       upgrades  unsupported mo   oldest (mo)  hours  calendar
self-run (team 4 mo)                9               0             3    135  you
self-run (team 6 mo)                6               0             7     90  you
self-run (team 12 mo)               3               0            11     45  you
self-run (team 18 mo)               2              12            19     30  you
self-run (team 24 mo)               1              11            23     15  you
operations delegated                3               0            11     21  provider

Six rows describe the same thirty-six months. Under self-run operation, upgrade frequency depends on the team’s capacity, and the results are linear: a team upgrading every four months spends 9 upgrades and 135 hours, and the release is never more than 3 months old. A team upgrading every twelve months spends 3 upgrades, 45 hours, and the release ages at most 11 months. Neither of these two rows is a wrong answer; both are a choice, and you are the one making it.

The lower bound shows up in the fifth and sixth rows. A team that can only upgrade every eighteen months spends twelve of the thirty-six months unsupported and ages the release up to nineteen months. Self-run operation gives you the calendar, but keeping to it is also your job; a calendar not kept turns into unsupported months, and there is no alarm reported to anyone for this.

The delegated row sits in an interesting spot: 3 upgrades, 0 unsupported months, an oldest release of 11 months — the same freshness as a team upgrading every twelve months, but 21 hours instead of 45. The difference is not in the hours; it is in the last column. The party that decides when an upgrade happens changes. The measurement network’s billing period falls in the first week of the month, and the nightly batch job starts every night at 02:30; if a mandatory upgrade window falls on either of these two calendars, the only thing you can do to avoid the collision is talk to the provider. Under self-run operation, this same collision is a scheduling problem; under delegated operation, it is a conversation, and the two are resolved on two different timescales.

The Loss of Access to the Control Plane

// access.mjs — access to setting and observability items, and its effect on the previous lessons' relation.
const SETTINGS = [                       // [item, when you run it yourself, when operations are delegated]
  ["reporting period (sets G)", "full", "none"], ["liveness threshold", "full", "none"],
  ["loop period (U)", "full", "none"], ["proxy read period", "full", "partial"],
  ["scheduler priorities", "full", "partial"], ["API server audit log", "full", "partial"],
  ["direct state-store reads", "full", "none"], ["control plane logs", "full", "partial"],
  ["controller manager metrics", "full", "none"], ["startup time (B)", "full", "full"],
  ["desired copy count", "full", "full"], ["node type and count", "full", "full"],
];
const count = (col, level) => SETTINGS.filter((a) => a[col] === level).length;
console.log("access               full  partial  none   items");
for (const [col, name] of [[1, "cluster you operate"], [2, "operations delegated"]])
  console.log(`${name.padEnd(21)}${String(count(col, "full")).padStart(6)}${String(count(col, "partial")).padStart(9)}` +
    `${String(count(col, "none")).padStart(7)}${String(SETTINGS.length).padStart(8)}`);

// The relation tested in the previous lesson: overshoot = desired x (ceil((G+B)/U) - 1)
const DESIRED = 6, B = 5;
const overshoot = (G, U) => DESIRED * (Math.ceil((G + B) / U) - 1);
console.log("\novershoot with the measured report delay (desired 6 copies, B=5 sec)");
console.log("G(sec)   U selectable (35)   U fixed (15)   B that zeroes overshoot   feasible?");
for (const G of [13.53, 26.36]) {
  const neededB = 15 - G;
  console.log(`${G.toFixed(2).padStart(6)}${String(overshoot(G, 35)).padStart(20)}` +
    `${String(overshoot(G, 15)).padStart(15)}${neededB.toFixed(2).padStart(24)} sec` +
    `${(neededB > 0 ? "very hard" : "impossible").padStart(14)}`);
}

access               full  partial  none   items
cluster you operate      12        0      0      12
operations delegated      3        4      5      12

overshoot with the measured report delay (desired 6 copies, B=5 sec)
G(sec)   U selectable (35)   U fixed (15)   B that zeroes overshoot   feasible?
 13.53                   0              6                    1.47 sec     very hard
 26.36                   0             12                  -11.36 sec    impossible

The top table gives the size of the loss: of twelve setting and observability items, all of them are fully accessible on the cluster you operate yourself, while on the cluster with delegated operations only three are fully accessible; four are partial, five are none at all. The five inaccessible items were not picked at random — they are exactly the items this topic’s three lessons measured: the reporting period, the liveness threshold, the loop period, direct state-store reads, and the controller manager’s metrics.

The bottom table turns this into a number. The previous lesson tested, across fifty-six cells, that overshoot equals desired × (⌈(G+B)/U⌉ − 1). With the measured report delay — average 13.53, worst 26.36 seconds — when the loop period is selectable it can be pulled to 35 seconds and overshoot is zero in both cases. Once the loop period is pinned to the provider’s default, overshoot comes out to 6 at the average delay, 12 at the worst delay.

The one lever left is the startup time, and the last two columns show that it is not enough either: zeroing out overshoot requires the startup time to fall to 1.47 seconds, and at the worst delay, the required value is negative 11.36 seconds — impossible. Delegating here did not delegate an hourly item; it took away the setting the previous lesson found.

The observability side of the loss is quieter than the settings side. This topic’s second lesson traced a single request, counting that it passed through four components and produced 50 item reads and 7 writes. That measurement requires direct state-store reads. Once access is gone, so is the measurement: the question of why a request was not applied now gets answered from the outside, by looking at results, not from inside the plane. Looking at results does not give a wrong answer, it gives a late one; the previous lesson’s oscillation is exactly this kind of thing — an error that converges to the correct number in the end and is invisible along the way.

The distinction is this: most delegated items are work, and load falls when they are delegated. But some are information and control; delegating them does not reduce load, it only changes whose hands they are in. The second class’s cost shows up not in monthly hours but in the number of options you have left when a problem appears.

The decision is therefore not one-directional. The 50 hours gained is 6.25 hours per person per month on an eight-person team, and on a setup that needs none of the five lost setting items, this trade is a clear win. The same trade, on a delay-sensitive workload — a cluster that needs to tune its loop period — buys, for 50 hours, an oscillation that cannot be closed. What needs to be measured is not the hours but which items need to stay with you; this list changes from setup to setup, and if it is not written before the delegation contract is signed, it cannot be written afterward.

Summary

• 7 of 21 operational items get delegated, 7 get shared, 7 stay with you no matter what; 50 of the monthly 106 hours (47.2 percent) get delegated, and item count falls from 21 to 14.
• The remaining items are items of the application, not the cluster — declarations, resource requests, authorization, network policy, observability, release rollout — and none of them shrink under any delegation.
• Exit cost is 7 items: a monthly 33-hour maintenance load, a 132-hour rebuild; for as long as the delegation lasts, these are items no one looks at.
• Over thirty-six months, delegated operation gave 3 upgrades, 0 unsupported months, and an oldest release of 11 months; keeping the same freshness with self-run operation took 45 hours, against 21 hours under delegation, but the calendar passed to the provider.
• Under self-run operation the calendar stays with you, but a calendar not kept writes a cost: with eighteen-month capacity, 12 of thirty-six months went unsupported, and the release aged up to 19 months.
• Of twelve setting and observability items, only 3 are fully accessible on the delegated cluster; the 5 inaccessible items are exactly what this topic measured, and once the loop period is pinned, overshoot goes from 0 to 6 (12 at the worst delay); the one lever left demands an impossible startup time.

Next Step

Across this topic, what a cluster is, which parts it is built from, what those parts report, how the loop reconciles, and who operates it were all measured. One thing was never defined: the thing being reconciled itself. Across four lessons, items like “desired copy count,” “placement constraint,” and “service” appeared, and all of them stood as a field in the model’s own small schema. What actually gets declared to the cluster — what the smallest deployable unit of a workload is, which object copies get counted by, which copies a name resolves to, where configuration and secrets get written — was never defined. The next topic builds these objects and tests each one by the same measure: which decision does this object carry, and what information does it need in order to carry it.