Horizontal Scaling

The previous two lessons shared out the cluster’s fixed capacity: who gets in where, and who gets evicted once it is tight. A third option was never asked — changing the replica count. The regional measurement network’s (the running fiction) reading collector sees a load that fluctuates over the day; as long as the replica count stays fixed, the price of that fluctuation is either spare capacity or a queue.

This decision can be delegated to a controller: a metric is watched, compared against a target, and the replica count is adjusted. The arithmetic of how many replicas are needed is not repeated here — per-replica saturation throughput and the occupancy ceiling are inputs coming from capacity planning. What this lesson measures is the decision itself: what the controller does once it sees the metric delayed.

The controller’s rule is one line: the desired replica count is the current replica count multiplied by the measured occupancy and divided by the target occupancy. The trap in this line is multiplying the current replica count by a stale metric. What is measured is a model; no real cluster runs.

The Controller and Metric Delay

The load series is a 120-turn sequence: a 40-turn wave, a sudden jump at turn 60, and noise in an 8 percent band from our own generator. The required replica count is computed separately for each turn, and the controller’s replica count is scored against it.

// scheduling/autoscale.mjs — metric-based horizontal autoscaling controller. A MODEL; no real
// cluster runs. Per-replica saturation throughput and the occupancy ceiling are inputs from M19.
export const SATURATION = 1176.47;      // requests/s per replica (M19, computed; an input here)
export const TARGET = 0.70;             // target occupancy (M19's capacity share; an input here)
export const TURNS = 120;               // SS25: one turn is one measurement interval
export const MAX_REPLICAS = 12;         // SS27: replica ceiling
const SEED = 20260805;

// SS26: the load series is fiction -- a 40-turn wave, a sudden jump at turn 60, +/-8% noise.
export function generateSeries() {
  let s = SEED % 2147483647;
  const r = () => (s = (s * 48271) % 2147483647) / 2147483647;
  return Array.from({ length: TURNS }, (_, t) => {
    const wave = 1200 + 900 * Math.sin((2 * Math.PI * t) / 40);
    const step = t >= 60 && t < 90 ? 1800 : 0;
    return Math.max(200, Math.round((wave + step) * (0.92 + 0.16 * r())));
  });
}
export const SERIES = generateSeries();
export const required = (load) => Math.max(1, Math.ceil(load / (SATURATION * TARGET)));

// The controller multiplies the delayed metric by the CURRENT replica count. As delay grows, it overcorrects.
export function run({ delay = 0, window = 1, step = 0, series = SERIES } = {}) {
  let replicas = 2, lastDirection = 0, lowStreak = 0;
  const occupancies = [];
  let oscillation = 0, excess = 0, shortfall = 0, total = 0, jumpShortfall = 0;
  let changes = 0, peakReplicas = 0, reactionTurns = -1;
  for (let t = 0; t < series.length; t++) {
    occupancies.push(series[t] / (replicas * SATURATION));
    const metric = occupancies[Math.max(0, t - delay)];
    let desired = Math.min(MAX_REPLICAS, Math.max(1, Math.ceil((replicas * metric) / TARGET)));
    if (step > 0) desired = Math.min(replicas + step, Math.max(replicas - step, desired));
    if (desired < replicas) { lowStreak++; if (lowStreak < window) desired = replicas; } else lowStreak = 0;
    const direction = Math.sign(desired - replicas);
    if (direction !== 0) {
      changes++;
      if (lastDirection !== 0 && direction !== lastDirection) oscillation++;
      lastDirection = direction;
    }
    replicas = desired;
    const req = required(series[t]);
    total += replicas;
    excess += Math.max(0, replicas - req);
    if (replicas < req) { shortfall++; if (t >= 60 && t < 90) jumpShortfall++; }
    if (t >= 60 && reactionTurns < 0 && replicas >= req) reactionTurns = t - 60;
    peakReplicas = Math.max(peakReplicas, replicas);
  }
  return { oscillation, changes, excess, shortfall, total, jumpShortfall, peakReplicas, reactionTurns };
}

if (process.argv[1].endsWith("autoscale.mjs")) {
  const g = SERIES.map(required);
  console.log(`model: ${TURNS} turns, load ${Math.min(...SERIES)}-${Math.max(...SERIES)} requests/s ` +
    `(seed ${SEED}); ${SATURATION} requests/s per replica, target occupancy ${TARGET * 100}%`);
  console.log(`required replicas ${Math.min(...g)}-${Math.max(...g)}, ideal total ` +
    `${g.reduce((a, b) => a + b, 0)} replica-turns\n`);
  const s = (x, n = 17) => String(x).padStart(n);
  console.log("metric delay is scanned (no wait window and no step limit):");
  console.log(`${"delay".padEnd(10)}${s("changes")}${s("oscillation")}${s("excess replica-turns", 23)}` +
    `${s("shortfall turns")}${s("total replica-turns", 22)}${s("peak replicas", 16)}`);
  for (const delay of [0, 1, 2, 3, 4, 5, 6]) {
    const o = run({ delay });
    console.log(`${String(delay).padEnd(10)}${s(o.changes)}${s(o.oscillation)}${s(o.excess, 23)}` +
      `${s(o.shortfall)}${s(o.total, 22)}${s(o.peakReplicas, 16)}`);
  }
}

model: 120 turns, load 278-3915 requests/s (seed 20260805); 1176.47 requests/s per replica, target occupancy 70%
required replicas 1-5, ideal total 301 replica-turns

metric delay is scanned (no wait window and no step limit):
delay               changes      oscillation   excess replica-turns  shortfall turns   total replica-turns   peak replicas
0                        14                8                      0                0                   301               5
1                        63               30                     52               29                   319               8
2                        67               23                    186               40                   422              12
3                        59               17                    225               40                   457              12
4                        56               15                    261               44                   489              12
5                        53               13                    286               42                   515              12
6                        43               11                    275               41                   510              12

Delay Grows the Oscillation, Then Hides It

The zero-delay row gives a lower bound: 14 changes, zero excess replica-turns, zero shortfall turns, and exactly 301 replica-turns — the ideal itself. When the metric arrives instantly, the delegated decision works flawlessly. Even the eight oscillations come from the load’s own wave, not from a controller error.

A single turn of delay wrecks this table. The change count jumps from 14 to 63, oscillation from 8 to 30; 29 turns pass with insufficient capacity and the peak replica count climbs from five to eight. The reason lies in the controller’s rule: the replica count was raised a turn ago, the metric still shows the occupancy from before the raise, and the controller corrects the same load a second time. The system does not know whether its own last decision has been reflected in the metric yet, and that is the one item it does not know.

As delay grows, the column worth reading changes. Oscillation drops from 30 to 11, and this is not an improvement. Excess replica-turns climb from 52 to 275, total resources climb from 319 to 510 replica-turns, and the peak replica count sticks to the ceiling. The controller no longer oscillates, because it climbs to the ceiling and stays there: the shortfall-turn count settles at 41, meaning resources sit at 1.7 times the ideal while a third of the turns are still short. A drop in oscillation is not, on its own, a sign of success; misread without excess replica-turns and shortfall turns alongside it, it is misinterpreted.

Delay itself is not a setting but a sum. The metric is first produced in the pod, accumulated over an interval, transmitted to the collector, folded into the average the controller reads, and only then turned into a decision. Each of these steps adds a turn. The delay column in the table is therefore not a choice but the result of the monitoring pipeline’s design: shortening the collection interval lowers the delay, and in exchange raises the metric count, the storage load, and the noise. The delegated decision’s cost is here split across two separate budgets — one in replica-turns, the other in metrics.

Two Dampers: the Wait Window and the Step Limit

Two settings work against oscillation. The wait window delays the scale-down decision: it is not applied until the desired replica count has come in low for a set number of turns in a row. The step limit caps the size of the change made in a single turn. Both are measured at the same delay — a three-turn metric delay — and the reaction is additionally counted with a noise-free jump test.

// scheduling/balance.mjs — wait window and step limit: the drop in oscillation and the slowdown
// in reaction are counted together. The controller and load series come from the previous file.
import { run } from "./autoscale.mjs";

const DELAY = 3;                 // SS28: the metric arrives three turns delayed
const s = (x, n = 23) => String(x).padStart(n);
const row = (ad, o) => console.log(`${ad.padEnd(22)}${s(o.changes)}${s(o.oscillation)}${s(o.excess)}` +
  `${s(o.shortfall)}${s(o.jumpShortfall)}${s(o.total)}`);

console.log(`metric delay held fixed at ${DELAY} turns; results from a 120-turn run:`);
console.log(`${"setting".padEnd(22)}${s("changes")}${s("oscillation")}${s("excess replica-turns")}` +
  `${s("shortfall turns")}${s("shortfall in jump")}${s("total replica-turns")}`);
row("unadjusted", run({ delay: DELAY }));

// SS29: wait window -- a scale-down decision is applied only if it comes in low for w turns straight.
for (const window of [2, 3, 5, 8])
  row(`wait window ${window}`, run({ delay: DELAY, window }));

// SS30: step limit -- the replica count changes by at most a in a single turn.
for (const step of [1, 2])
  row(`step limit ${step}`, run({ delay: DELAY, step }));

row("window 3 + step 1", run({ delay: DELAY, window: 3, step: 1 }));
row("window 5 + step 1", run({ delay: DELAY, window: 5, step: 1 }));

// Noise-free jump test: 60 turns at 800 requests/s, then 60 turns at 3200 requests/s (required 1 -> 4).
const JUMP_LOAD = Array.from({ length: 120 }, (_, t) => (t < 60 ? 800 : 3200));
console.log("\njump test -- reaction turn when load jumps from 800 to 3200 requests/s at turn 60:");
console.log(`${"setting".padEnd(22)}${s("delay 0")}${s("delay 3")}${s("delay 6")}`);
for (const [ad, ek] of [["unadjusted", {}], ["step limit 1", { step: 1 }],
                        ["step limit 2", { step: 2 }], ["window 3 + step 1", { window: 3, step: 1 }]])
  console.log(`${ad.padEnd(22)}` + [0, 3, 6]
    .map((g) => s(run({ ...ek, delay: g, series: JUMP_LOAD }).reactionTurns)).join(""));

console.log("\nthe same two settings with no delay (delay 0):");
row("unadjusted", run({ delay: 0 }));
row("window 3 + step 1", run({ delay: 0, window: 3, step: 1 }));

const a = run({ delay: DELAY }), b = run({ delay: DELAY, window: 3, step: 1 });
console.log(`\noscillation ${a.oscillation} -> ${b.oscillation} (${(100 * (1 - b.oscillation / a.oscillation)).toFixed(0)}% decrease), ` +
  `shortfall in jump window ${a.jumpShortfall} -> ${b.jumpShortfall}`);
console.log(`resources ${a.total} -> ${b.total} replica-turns (ideal 301); ` +
  `the controller reads 120 metrics, each ${DELAY} turns delayed`);

metric delay held fixed at 3 turns; results from a 120-turn run:
setting                               changes            oscillation   excess replica-turns        shortfall turns      shortfall in jump    total replica-turns
unadjusted                                 59                     17                    225                     40                     15                    457
wait window 2                              46                     17                    257                     37                     12                    493
wait window 3                              45                     15                    315                     33                     13                    552
wait window 5                              40                     13                    343                     31                      9                    596
wait window 8                              31                     11                    371                     23                      7                    632
step limit 1                               71                     15                     98                     35                     14                    347
step limit 2                               71                     15                    173                     35                     15                    410
window 3 + step 1                          59                     13                    122                     32                     11                    385
window 5 + step 1                          53                     11                    152                     27                     10                    417

jump test -- reaction turn when load jumps from 800 to 3200 requests/s at turn 60:
setting                               delay 0                delay 3                delay 6
unadjusted                                  0                      3                      6
step limit 1                                2                      5                      8
step limit 2                                1                      4                      7
window 3 + step 1                           2                      5                      8

the same two settings with no delay (delay 0):
unadjusted                                 14                      8                      0                      0                      0                    301
window 3 + step 1                          14                      6                     17                      0                      0                    318

oscillation 17 -> 13 (24% decrease), shortfall in jump window 15 -> 11
resources 457 -> 385 replica-turns (ideal 301); the controller reads 120 metrics, each 3 turns delayed

The Drop in Oscillation and the Slowdown in Reaction

The two settings take hold of the same problem from two different places. The wait window drops oscillation from 17 to 11 and pays for it in resources: total replica-turns climb from 457 to 632, excess replica-turns from 225 to 371. Because delaying the scale-down keeps capacity higher, the shortfall-turn count also drops from 40 to 23. This setting does not reduce oscillation, it redirects it.

The step limit does the opposite. Allowing a single-replica change per turn drops excess replica-turns from 225 to 98 and total resources from 457 to 347 replica-turns; oscillation also drops to 15. The change count, however, climbs from 59 to 71, because the same distance is now covered in more steps. The step limit cuts the size of the overcorrection, and so it targets the problem born from delay directly.

The price shows up in the jump test. When the load jumps from 800 to 3,200 requests/s, the unadjusted controller catches up exactly as many turns later as the delay: three turns at a three-turn delay, six turns at a six-turn delay. Once a step limit of 1 is added, those same numbers become 5 and 8 — the reaction slows by two turns, because climbing from one replica to four now takes three turns. With a step limit of 2 the gap narrows to one turn. The drop in oscillation and the slowdown in reaction are two faces of the same setting, and unless both are counted separately, the setting is a guess, not a choice.

The row where both settings are used together gives the sum of the two: oscillation drops from 17 to 13, resources hold at 385 replica-turns, and the shortfall-in-jump count drops from 15 to 11. This row beats the unadjusted row in every column, yet it still sits 28 percent above the ideal 301 replica-turns. As long as the delay persists, this gap does not close.

The last table states what these settings are not. With no delay, the wait window and the step limit fix nothing: oscillation drops from 8 to 6, and in exchange 17 extra replica-turns are spent, paying 318 replica-turns instead of the ideal 301. These two settings are a repair for a measurement problem, not a load problem. If the metric can be freshened instead, no settings are needed.

The choice of metric is also part of the same calculation. In this model, what is watched is occupancy, and occupancy is a derived number: it comes from dividing usage by the replica count and the per-replica saturation throughput. Every derived number is as stale as the oldest of its inputs. A directly countable metric — the number of pending requests, say — is collected in fewer steps and shifts the delay column to the left; in exchange, what the target value should be can no longer be read straight from capacity planning.

The decision rests on 120 metrics, each three turns stale, plus four constants — per-replica saturation throughput, target occupancy, and the lower and upper replica bounds. The freshness of these items translates directly into resources: with the same load, the same controller, and the same target, 510 replica-turns can be spent instead of 301, purely because of the metric’s delay.

Summary

• When the metric arrives instantly, the delegated decision is flawless: the 301-replica-turn ideal matches resources exactly, with zero excess and zero shortfall turns.
• A single turn of delay raises oscillation from 8 to 30 and leaves 29 turns short of capacity; because the controller does not know whether its own last decision has reached the metric yet, it corrects the same load twice.
• As delay grows, oscillation drops to 11, but this is not an improvement: the controller sticks to the ceiling, resources climb to 510 replica-turns (1.7 times the ideal), and a third of the turns remain short.
• The wait window drops oscillation from 17 to 11 while raising resources from 457 to 632 replica-turns; because it delays the scale-down, it drops shortfall turns from 40 to 23.
• Because the step limit cuts the overcorrection, it lowers both oscillation and resources (457 down to 347 replica-turns), while in the jump test the reaction slows from 3 turns to 5 and the change count rises from 59 to 71.
• With no delay, the same two settings spend 17 extra replica-turns to buy two fewer oscillations; damping settings are a repair for a measurement problem, not a load problem.

Next Step

This lesson’s controller turns a single knob: the replica count. Two more knobs exist, and both meet the same load from a different place. The replica itself can be resized — a pod’s request can be rewritten to match observed usage, but this has a price, and the price is paid in restarts. The cluster itself can be grown — a node can be added, but a new node is not ready instantly. The next lesson measures these two knobs and shows the case where all three break each other.