Vertical and Cluster Scaling

The previous lesson’s controller turned a single knob: the replica count. Two more knobs exist. The replica itself can be grown or shrunk — a pod’s request can be rewritten to match observed usage. The cluster can be grown or shrunk — nodes can be added or removed. All three meet the same load, and all three have a different information cost.

What these two knobs have in common is that the payoff cannot be collected instantly. Changing the replica count is a counter operation; changing a pod’s request requires restarting it, and adding a node makes the system wait for that node to become ready. This lesson counts these two delays separately and finally shows what happens when all three knobs are turned at once. What is measured is a model; the workload is the services of the regional measurement network (the running fiction), and the arithmetic of how many replicas are needed is an input coming from capacity planning.

Vertical Resizing: the Cost Is Restarts

The model tracks six pods over 120 turns. The initial request is 1,024 units — declared well above actual usage. A suggestion generator takes the peak of the last ten turns and, once it deviates from the declared request by more than a set threshold, rewrites the request; every rewrite restarts the pod and takes it out of service for two turns.

// scheduling/vertical.mjs — vertical resizing: the cost is restarts, the gain is dropping spare
// capacity. What gets built is a MODEL; no real cluster runs.
export const TURNS = 120, PODS = 6;
export const WINDOW = 10;         // SS31: the suggestion is the peak of the last 10 turns
export const OUTAGE = 2;          // SS32: every resize takes a pod out of service for 2 turns
const SEED = 20260806;

export function generator(seed = SEED) {
  let s = seed % 2147483647;
  return () => (s = (s * 48271) % 2147483647) / 2147483647;
}

// SS33: usage fluctuates around 480 units, two pods drift upward slowly, +/-12% noise.
export function usageSeries() {
  const r = generator();
  return Array.from({ length: PODS }, (_, k) =>
    Array.from({ length: TURNS }, (_, t) => {
      const base = 480 + 100 * Math.sin((2 * Math.PI * t) / 30) + (k % 3 === 0 ? 1.6 * t : 0);
      return Math.max(80, Math.round(base * (0.88 + 0.24 * r())));
    }));
}
export const SERIES = usageSeries();

// Suggestion: the peak of the last WINDOW turns. Once the deviation threshold is crossed, the request changes and the pod restarts.
export function resize(threshold) {
  let restarts = 0, waste = 0, shortfall = 0, outageTurns = 0, requestTurns = 0;
  for (const series of SERIES) {
    let request = 1024, remaining = 0;                // SS34: initial request 1024 units (inflated)
    for (let t = 0; t < TURNS; t++) {
      const suggestion = Math.max(...series.slice(Math.max(0, t - WINDOW + 1), t + 1));
      if (threshold >= 0 && Math.abs(suggestion - request) / request > threshold && remaining === 0) {
        request = suggestion; restarts++; remaining = OUTAGE;
      }
      if (remaining > 0) { remaining--; outageTurns++; }
      requestTurns += request;
      waste += Math.max(0, request - series[t]);
      if (series[t] > request) shortfall++;
    }
  }
  return { restarts, waste, shortfall, outageTurns, requestTurns };
}

if (process.argv[1].endsWith("vertical.mjs")) {
  const usageTurns = SERIES.flat().reduce((a, b) => a + b, 0);
  console.log(`model: ${PODS} pods x ${TURNS} turns (seed ${SEED}); actual total usage ` +
    `${usageTurns} unit-turns, initial request 1024 units`);
  const s = (x, n = 16) => String(x).padStart(n);
  console.log(`\n${"threshold".padEnd(14)}${s("restarts")}${s("outage turns")}` +
    `${s("waste")}${s("waste ratio")}${s("shortfall turns", 18)}`);
  for (const [ad, threshold] of [["fixed request", -1], ["5%", 0.05], ["10%", 0.10],
                            ["20%", 0.20], ["35%", 0.35], ["50%", 0.50]]) {
    const o = resize(threshold);
    console.log(`${ad.padEnd(14)}${s(o.restarts)}${s(o.outageTurns)}${s(o.waste)}` +
      `${s((o.waste / o.requestTurns * 100).toFixed(1) + "%")}${s(o.shortfall, 18)}`);
  }
  const a = resize(-1), b = resize(0.20);
  console.log(`\nfrom a fixed request to a 20% threshold: waste ${a.waste} -> ${b.waste} unit-turns ` +
    `(${(100 * (1 - b.waste / a.waste)).toFixed(1)}% decrease), at a cost of ${b.restarts} restarts ` +
    `and ${b.outageTurns} outage turns`);
}

model: 6 pods x 120 turns (seed 20260806); actual total usage 367748 unit-turns, initial request 1024 units

threshold             restarts    outage turns           waste     waste ratio   shortfall turns
fixed request                0               0          369532           50.1%                 0
5%                         141             280           57939           13.7%                75
10%                         87             174           56176           13.3%               113
20%                         38              76           54704           13.2%               190
35%                         10              20           64938           15.4%               180
50%                          8              16           51564           13.2%               336

from a fixed request to a 20% threshold: waste 369532 -> 54704 unit-turns (85.2% decrease), at a cost of 38 restarts and 76 outage turns

Three Columns Do Not Improve at the Same Time

The fixed-request row is the starting point: no restarts, no shortfall turns, and 50.1 percent of the request sits idle. This is the price paid when the declaration is inflated twofold — the first lesson’s waste, showing up here as a time series.

Once the threshold is set to 20 percent, the waste ratio drops to 13.2 percent: 369,532 unit-turns of waste falls to 54,704 — an 85.2 percent gain. Its price is 38 restarts and 76 outage turns. Once the threshold is lowered to 5 percent, the waste ratio only stays at 13.7 percent while the restart count climbs to 141 and outage turns to 280. Tighter tracking does not buy more gain, only more disruption.

The third column is not a free lunch. Where the fixed-request row has zero shortfall turns, the 20 percent threshold has 190 and the 50 percent threshold has 336: as the request nears actual usage, the limit gets exceeded at peak moments. Vertical resizing turns spare capacity into shortfall, and the only thing that sets the balance between the two is the threshold. The threshold itself is also a guess — the 35 percent row produces more waste than the 20 percent row, because the suggestion gets stuck on a peak value and stays there. The ranking is not one-directional, which shows that the setting cannot be chosen without measuring it.

Cluster Scaling: the Cost Is Readiness Delay

The second knob is the node count. The load climbs from 800 requests/s to 12,000 requests/s; the scaling controller adds replicas, and once the replicas no longer fit on the nodes, the cluster autoscaler orders a node, which becomes ready a set number of turns later. The metric delay is zero in this model, so the only delay being measured is the node’s readiness time.

// scheduling/cluster.mjs — cluster scaling: whether a new node's readiness delay keeps up with
// the load, and the case where three scaling knobs break each other. A model; no real cluster runs.
import { TURNS, generator } from "./vertical.mjs";

const SATURATION = 1176.47, TARGET = 0.70;      // M19's inputs; the arithmetic is not repeated here
const PODS_PER_NODE = 4;                        // SS35: a node fits four replicas
const SHRINK_WINDOW = 5;

// Load climbs from 800 to 12000 requests/s, plus a wave and +/-6% noise (fiction).
const r = generator(20260807);
const LOAD = Array.from({ length: TURNS }, (_, t) =>
  Math.round((800 + 95 * t + 600 * Math.sin((2 * Math.PI * t) / 25)) * (0.94 + 0.12 * r())));
const required = (y) => Math.max(1, Math.ceil(y / (SATURATION * TARGET)));

// SS36: vertical resizing takes a replica out of service for 2 turns every 10th turn.
function run({ ready, nodeLimit = 99, vertical = false }) {
  let replicas = 1, readyNodes = 1, pending = [], idleStreak = 0;
  let shortfallTurns = 0, longestStreak = 0, streak = 0, nodeTurns = 0, replicaTurns = 0;
  let events = 0, orders = 0, restarts = 0;
  for (let t = 0; t < TURNS; t++) {
    pending = pending.filter((v) => { if (v <= t) { readyNodes++; return false; } return true; });
    const capped = Math.min(readyNodes, nodeLimit) * PODS_PER_NODE;
    const restarting = vertical && t % 10 < 2 ? 1 : 0;
    if (vertical && t % 10 === 0) restarts++;
    const serving = Math.max(1, Math.min(replicas, capped) - restarting);
    const occupancy = LOAD[t] / (serving * SATURATION);
    const desired = Math.max(1, Math.ceil((replicas * occupancy) / TARGET));
    if (desired !== replicas) events++;
    replicas = desired;
    if (replicas > capped) {
      const extra = Math.ceil((replicas - capped) / PODS_PER_NODE);
      for (let i = 0; i < extra; i++) pending.push(t + ready);
      orders += extra;
    } else if (capped - replicas >= PODS_PER_NODE && readyNodes > 1) {
      if (++idleStreak >= SHRINK_WINDOW) { readyNodes--; idleStreak = 0; }
    } else idleStreak = 0;
    const running = Math.min(replicas, Math.min(readyNodes, nodeLimit) * PODS_PER_NODE);
    replicaTurns += running;
    nodeTurns += readyNodes + pending.length;
    if (running < required(LOAD[t])) { shortfallTurns++; longestStreak = Math.max(longestStreak, ++streak); }
    else streak = 0;
  }
  return { shortfallTurns, longestStreak, nodeTurns, replicaTurns, events, orders, restarts };
}

const s = (x, n = 18) => String(x).padStart(n);
console.log(`load ${Math.min(...LOAD)}-${Math.max(...LOAD)} requests/s, ${TURNS} turns; required ` +
  `replicas ${required(LOAD[0])}-${Math.max(...LOAD.map(required))}, ${PODS_PER_NODE} replicas per node`);
console.log(`\nnew node readiness delay is scanned:`);
console.log(`${"delay".padEnd(10)}${s("shortfall turns")}${s("longest streak")}${s("node orders")}` +
  `${s("node-turns")}${s("replica-turns")}`);
for (const ready of [0, 1, 2, 4, 6, 8]) {
  const o = run({ ready });
  console.log(`${String(ready).padEnd(10)}${s(o.shortfallTurns)}${s(o.longestStreak)}${s(o.orders)}` +
    `${s(o.nodeTurns)}${s(o.replicaTurns)}`);
}

console.log(`\nthree scaling knobs together (readiness delay 4):`);
console.log(`${"setting".padEnd(34)}${s("scaling events")}${s("shortfall turns")}` +
  `${s("restarts")}${s("node-turns")}`);
for (const [ad, ek] of [["horizontal only (ample nodes)", { ready: 0, nodeLimit: 99 }],
                        ["horizontal + cluster", { ready: 4 }],
                        ["horizontal + cluster + vertical", { ready: 4, vertical: true }]]) {
  const o = run(ek);
  console.log(`${ad.padEnd(34)}${s(o.events)}${s(o.shortfallTurns)}` +
    `${s(o.restarts)}${s(o.nodeTurns)}`);
}
const b = run({ ready: 4 }), c = run({ ready: 4, vertical: true });
console.log(`\nonce vertical is added: scaling events ${b.events} -> ${c.events}, node orders ` +
  `${b.orders} -> ${c.orders}, node-turns ${b.nodeTurns} -> ${c.nodeTurns}`);

load 792-12189 requests/s, 120 turns; required replicas 1-15, 4 replicas per node

new node readiness delay is scanned:
delay        shortfall turns    longest streak       node orders        node-turns     replica-turns
0                          3                 1                 3               302              1003
1                          3                 1                 3               302              1003
2                          5                 2                 6               319              1001
4                         10                 4                13               431               997
6                         10                 6                21               837               998
8                          8                 8                33              2353              1002

three scaling knobs together (readiness delay 4):
setting                               scaling events   shortfall turns          restarts        node-turns
horizontal only (ample nodes)                     36                 3                 0               302
horizontal + cluster                              42                10                 0               431
horizontal + cluster + vertical                   54                 8                12               445

once vertical is added: scaling events 42 -> 54, node orders 13 -> 13, node-turns 431 -> 445

Delay Turns Shortfall into Resources

The second column behaves as expected: the longest shortfall streak is exactly the readiness delay itself. At a four-turn delay, the longest uninterrupted shortfall is four turns; at a six-turn delay, it is six turns. Load waits until a new node is on its way, and that wait is the result of node-provisioning time, not a design choice.

The first column behaves unexpectedly. At a six-turn delay, 10 turns run short; at an eight-turn delay, that number drops to 8. The conclusion that the system runs better here is wrong; the fourth column tells the real story. Node orders climb from 21 to 33, node-turns from 837 to 2,353. As delay grows, the controller keeps seeing the same gap over and over and keeps ordering nodes over and over; the cluster eventually accumulates far more nodes than it needs, and the shortfall closes by burning resources. Because the shrink rule removes at most one node every five turns, this surplus never melts away over the course of the run. Unless shortfall turns and node-turns are read together, delay looks like an improvement.

The replica-turns column staying around 1,000 across all six rows is a separate result: the number of running replicas barely changes at all; what changes is how late those replicas manage to be placed, and how many nodes wait behind them.

When All Three Knobs Are Turned at Once

The final table stacks the three scaling types on top of each other. With plenty of nodes available, horizontal scaling alone produces 36 scaling events and 3 shortfall turns. Once cluster scaling is added, the event count climbs to 42, shortfall turns to 10, and node-turns from 302 to 431: because the horizontal controller’s decision can no longer be applied instantly, it repeats itself.

Once vertical resizing is added, the breakdown can be counted exactly. Twelve restarts raise the scaling-event count from 42 to 54 — twelve extra events, exactly matching the restart count. The chain runs like this: vertical resizing takes a replica out of service for two turns, the serving replica count drops, the remaining replicas’ occupancy rises, the horizontal controller thinks load has grown and adds a replica, and if the added replica does not fit on a node, the cluster autoscaler orders one. The actual load never changed throughout, and node-turns climbed from 431 to 445.

None of the three controllers makes a mistake. Each reads its own metric correctly and applies its own rule correctly. The problem is that one controller’s action becomes another’s measurement: vertical resizing’s restart enters the horizontal controller’s occupancy metric as a load increase. Here too, the item the system cannot see is a single thing — the reason a replica is out of service. Had this item been added to the metric, that is, had out-of-service replicas been excluded from the occupancy calculation, none of the twelve extra events would have been born.

The three knobs’ information costs are not equal, either. Horizontal scaling works with a single occupancy metric per replica, and its decision is undone with a single counter change. Vertical resizing wants a usage window per pod — in this model, a ten-turn window for six pods, meaning sixty samples every turn — and its decision costs a restart. Cluster scaling adds node-provisioning time on top of both of these; that time is not measured but declared, and when it is declared wrongly, the shortfall-turn count and node-turns drift together.

Summary

• Vertical resizing’s gain is measured in spare capacity: while the waste ratio at a fixed request is 50.1 percent, it drops to 13.2 percent at a 20 percent threshold, and waste falls from 369,532 to 54,704 unit-turns.
• Its cost is restarts, and the cost only grows as the threshold narrows: 38 restarts and 76 outage turns at 20 percent, 141 restarts and 280 outage turns at 5 percent, while the gain stays at the same 13.7 percent.
• Tight resizing turns spare capacity into shortfall: shortfall turns climb from zero to 190, then to 336 at the 50 percent threshold.
• In cluster scaling, the longest shortfall streak equals the readiness delay; at a delay of 8, shortfall turns drop to 8 while node orders climb from 21 to 33 and node-turns from 837 to 2,353 — the shortfall is closed with resources.
• When all three knobs are turned together, twelve restarts produce exactly twelve extra scaling events and node-turns climb from 431 to 445, even though the actual load never changed.
• All three controllers apply their own rule correctly; the source of the breakdown is that one controller’s action enters another’s metric as load, and the reason for being out of service does not appear in that metric at all.

Next Step

This topic tied placement and growth decisions to numbers: request and limit, constraints, taints, eviction, and three kinds of scaling. Their shared assumption is that the desired state is written somewhere and the cluster tries to make it real. How that desired state gets rolled out, when a pod counts as healthy, which subject holds authority over which object, who the network is open from and to, and how all of this gets rolled back during a failure — none of that has been asked yet. The next topic takes on the cluster’s operation: deployment patterns, health checks, authorization and network policy, observability, troubleshooting, and extending the cluster with its own object types.