Replica Sets and Version Objects

The previous lesson drew the pod boundary and took needing six replicas as a given. That assumption alone does nothing: if a pod goes down, nothing replaces it; if a node is lost, the pods on it are lost too. For the number six to have any effect, it must be declared somewhere, and a controller must continuously compare that declaration against reality. The object that carries this declaration is the replica set: a single pod template and a single number.

The number is not enough. The template itself also changes — a new version ships, and the content of the six pods has to change. The object that manages this is the version object; it holds two replica sets and shifts the desired count between them step by step. This lesson writes both objects with node and counts the rollout round by round. What is measured is a model; no real cluster runs.

CO7 — the desired replica count for the reading collector is six. CO8 — the only thing the controller knows is the desired count and the actual count; the inside of the pods is invisible to it. CO9 — a newly created pod becomes ready after two rounds; removal finishes in the same round. CO10 — the version object holds two replica sets and splits the desired count between them. CO11 — the rollout setting is two numbers: the maximum extra pods allowed, the maximum missing ready replicas allowed. CO12 — the new version turns out broken at the end of round four; rollback runs the same settings in reverse.

The Desired Count and the Actual Count

The controller’s job is one line: take the gap, produce an action in whatever direction its sign points. The run below carries this one line through four events — cold start, two pods crashing, the desired count changing, and two pods being added by hand from outside.

// replica.mjs — replica set controller: reconciles desired vs. actual replica count (MODEL).
export const READY_ROUNDS = 2;                            // CO9: a new pod is ready in 2 rounds

export const set = (name, desired) => ({ name, desired, ready: 0, pending: [] });
export const total = (k) => k.ready + k.pending.length;

export function advance(k) {                              // pending pods become ready
  k.pending = k.pending.map((x) => x - 1);
  const done = k.pending.filter((x) => x <= 0).length;
  k.ready += done;
  k.pending = k.pending.filter((x) => x > 0);
  return done;
}
export const create = (k, n) => { for (let i = 0; i < n; i++) k.pending.push(READY_ROUNDS); };
export function remove(k, n) {                             // pending pods go first, then ready ones
  let left = n;
  while (left > 0 && k.pending.length) { k.pending.pop(); left--; }
  const d = Math.min(left, k.ready); k.ready -= d;
  return n - (left - d);
}

// Controller: turns the gap into an action. All it knows is the desired and actual count. CO8.
export function reconcile(k) {
  const gap = k.desired - total(k);
  if (gap > 0) { create(k, gap); return `+${gap} create`; }
  if (gap < 0) { remove(k, -gap); return `${gap} remove`; }
  return 'no gap';
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const K = set('reading-collector', 6);                  // CO7
  const EVENT = { 4: ['two pods crashed', (k) => (k.ready -= 2)],
    8: ['desired became 8', (k) => (k.desired = 8)],
    11: ['two pods added by hand', (k) => create(k, 2)] };
  const s = (x, n) => String(x).padStart(n);
  let created = 0, removed = 0, shortRounds = 0;
  console.log(`${'round'.padEnd(6)}${'event'.padEnd(25)}${s('desired', 8)}${s('ready', 6)}`
    + `${s('pending', 9)}   action`);
  for (let t = 1; t <= 14; t++) {
    const o = EVENT[t]; if (o) o[1](K);
    advance(K);
    const before = total(K);
    const action = reconcile(K);
    const d = total(K) - before;
    if (d > 0) created += d; else removed += -d;
    if (K.ready < K.desired) shortRounds++;
    console.log(`${String(t).padEnd(6)}${(o ? o[0] : '').padEnd(25)}${s(K.desired, 8)}`
      + `${s(K.ready, 6)}${s(K.pending.length, 9)}   ${action}`);
  }
  console.log(`\nover 14 rounds: ${created} created, ${removed} removed, ${shortRounds} rounds`
    + ` with ready count below desired`);
}

round event                     desired ready  pending   action
1                                     6     0        6   +6 create
2                                     6     0        6   no gap
3                                     6     6        0   no gap
4     two pods crashed                6     4        2   +2 create
5                                     6     4        2   no gap
6                                     6     6        0   no gap
7                                     6     6        0   no gap
8     desired became 8                8     6        2   +2 create
9                                     8     6        2   no gap
10                                    8     8        0   no gap
11    two pods added by hand          8     8        0   -2 remove
12                                    8     8        0   no gap
13                                    8     8        0   no gap
14                                    8     8        0   no gap

over 14 rounds: 10 created, 2 removed, 6 rounds with ready count below desired

The four events come from four separate causes, but the controller’s response has one shape. Six pods on cold start, two on crash, two when the desired count rises; in every case the rule applied is the same, and in none of them is the reason for the drop ever asked. The controller does not distinguish a crash from a manual deletion, because under CO8 both reduce to the same number. This is devolving’s most concrete payoff: four separate procedures are not written for four different scenarios.

Round eleven is the other side of this. The two pods added from outside are deleted in the same round; a change made outside the declaration does not outlive the next reconciliation round. Manual work done in the cluster is not permanent, and the only evidence of this is the -2 remove line in the table.

The cost of the numbers shows up in rounds two and five: in the two rounds marked no gap, genuinely no work is done, but the ready replica count is still below the desired one. The total measure is six rounds — in six of fourteen rounds the cluster ran with short capacity. All of this comes from CO9, that is, from readiness taking two rounds; the controller cannot close the gap in the round it sees it, it can only issue the order to close it. The capacity shortfall is the delay between a decision being made and its effect becoming visible.

The no gap on ten rounds of the table is also a result. The same rule runs again every round and does nothing when the gap is closed; this is a property of the reconciliation loop that does not change at cluster scale either. The controller is not a procedure that fires once and finishes, it is a comparison that runs continuously and produces no work in most rounds. Work per round is nonzero in four of fourteen rounds.

The Rollout’s Settings

When the replica count stays fixed but the template changes, the work stops being reconciliation and turns into a rollout. The version object manages two replica sets at once and obeys two numbers: at most how many extra pods can exist at once, at most how many replicas can be missing at once.

// rollout.mjs — version object: staged rollout and rollback between two replica sets (MODEL).
import { set, total, advance, create, remove } from './replica.mjs';

const N = 6;                                              // CO7: desired replica count

// surplus: max extra pods allowed at once; shortfall: max missing ready replicas allowed. CO11.
function rollout(surplus, shortfall, badRound = null) {
  const old = set('version-1', N), fresh = set('version-2', 0);
  old.ready = N;
  const created = { 'version-1': 0, 'version-2': 0 };
  let t = 0, belowRounds = 0, peak = N, removedCount = 0, backRound = null, stall = 0, dir = 'forward';
  while (t < 80) {
    t++;
    advance(old); advance(fresh);
    if (badRound && t === badRound && dir === 'forward') { dir = 'back'; backRound = t - 1; }
    const target = dir === 'forward' ? fresh : old, source = dir === 'forward' ? old : fresh;
    const room = Math.min(N + surplus - (total(old) + total(fresh)), N - total(target));
    if (room > 0) { create(target, room); created[target.name] += room; }
    const drop = Math.min(old.ready + fresh.ready - (N - shortfall), total(source));
    if (drop > 0) { remove(source, drop); removedCount += drop; }
    peak = Math.max(peak, total(old) + total(fresh));
    if (old.ready + fresh.ready < N) belowRounds++;
    stall = room <= 0 && drop <= 0 && !old.pending.length && !fresh.pending.length
      ? stall + 1 : 0;
    if (stall >= 2) return { round: null, belowRounds, peak, created, removedCount, backRound };
    if (total(source) === 0 && target.ready >= N) break;
  }
  return { round: t, belowRounds, peak, created, removedCount, backRound };
}

const SETTING = [[0, 0], [0, 1], [1, 0], [1, 1], [2, 2], [6, 0]];
const s = (x, n) => String(x).padStart(n);
console.log(`desired replicas ${N}, ready in 2 rounds`);
console.log(`${s('surplus', 8)}${s('shortfall', 10)}${s('round', 11)}${s('below capacity', 16)}`
  + `${s('peak pods', 13)}${s('created', 9)}${s('removed', 9)}`);
for (const [f, e] of SETTING) {
  const r = rollout(f, e), o = r.created['version-1'] + r.created['version-2'];
  console.log(`${s(f, 8)}${s(e, 10)}${s(r.round ?? 'stalls', 11)}${s(`${r.belowRounds} rounds`, 16)}`
    + `${s(r.peak, 13)}${s(o, 9)}${s(r.removedCount, 9)}`);
}

console.log(`\nif the new version turns out broken at the end of round 4 (CO12):`);
console.log(`${s('surplus', 8)}${s('shortfall', 10)}${s('forward rounds', 15)}${s('back rounds', 12)}`
  + `${s('total', 8)}${s('below capacity', 16)}${s('wasted pods', 13)}`);
for (const [f, e] of SETTING.slice(1)) {
  const r = rollout(f, e, 5);
  if (r.backRound === null) {
    console.log(`${s(f, 8)}${s(e, 10)}   rollout finished before round 4: no old replica to roll back to`);
    continue;
  }
  console.log(`${s(f, 8)}${s(e, 10)}${s(r.backRound, 15)}${s(r.round - r.backRound, 12)}`
    + `${s(r.round, 8)}${s(`${r.belowRounds} rounds`, 16)}${s(r.created['version-2'], 13)}`);
}

desired replicas 6, ready in 2 rounds
 surplus shortfall      round  below capacity    peak pods  created  removed
       0         0     stalls        0 rounds            6        0        0
       0         1         19       18 rounds            6        6        6
       1         0         18        0 rounds            7        6        6
       1         1         10        9 rounds            7        6        6
       2         2          6        5 rounds            8        6        6
       6         0          3        0 rounds           12        6        6

if the new version turns out broken at the end of round 4 (CO12):
 surplus shortfall forward rounds back rounds   total  below capacity  wasted pods
       0         1              4           6      10        9 rounds            1
       1         0              4           5       9        0 rounds            2
       1         1              4           6      10        9 rounds            3
       2         2              4           6      10        9 rounds            6
       6         0   rollout finished before round 4: no old replica to roll back to

The first row gives the sharpest sense of what the two numbers mean: if both are zero, the rollout does not start. There is no room for an extra pod and no permission to run short a replica; the version object cannot take a single step without violating the rules, and the model sees the stall and exits. That a declarative object needs slack in at least one direction to make progress is this table’s first result.

The next two rows create and delete the same number of pods yet produce diametrically opposite costs. Allowing one shortfall and no surplus stretches the rollout to nineteen rounds, and in eighteen of them the cluster runs with short capacity; peak pod count stays at six, so no extra resource is spent at all. Allowing one surplus and no shortfall takes eighteen rounds, never drops below capacity, and in exchange asks for only one pod’s worth of extra resource. The round gap between the two settings is one; the capacity gap is eighteen rounds. Being willing to give extra resource is the price of getting through the same span without interruption.

The bottom two rows show where speed comes from. When one surplus and one shortfall are allowed together, the round count drops from eighteen to ten; at two and two, to six. Speed is fed by both settings, but below-capacity rounds come along with it. The last row is the tip of this curve: when six surplus pods are allowed, the rollout finishes in three rounds and capacity never drops — peak pod count is twelve, meaning resources temporarily double. This setting is the numerical equivalent of a blue-green rollout.

The second table makes that three-round speed pay its hidden cost. When the new version turns out broken at the end of round four, rollback with one surplus and zero shortfall takes five rounds, capacity never drops, and two pods go to waste. With the two-two setting rollback takes six rounds and six pods go to waste — the new version was fully installed within four rounds. With the six-surplus setting there is no such thing as rollback: the rollout finished before round four, and the old replica set has already been zeroed out. The rollback window is the same thing as the rollout’s duration. Every setting that speeds up the rollout shortens this window, and past a certain point closes it.

The Devolved Decision and What It Does Not Know

There are two devolved decisions in this lesson: how many replicas run and how fast to move to the new template. For the first, the only information item the system needs to know is the desired count — this is why it is cheap and answers four different events with one rule. The second needs two items: surplus and shortfall. These two numbers are not a preference, they are the mutual price of two resources; one is temporary capacity, the other temporary disruption.

The cost of the two items is not equal either. The desired count is written once and rarely changes; it needs no measurement, does not age, and when wrong its result is directly visible. Rollout settings, though, rest on an estimate that must be reassessed at every release: how long the new version will take to come up, and if it is broken, how long that will take to become clear. Because the second estimate cannot be measured, the settings are usually written once and stay the same for every release — yet as the table shows, the same setting can entirely close the rollback window on a version that comes up fast.

What the system does not know stands in the same table. The version object sees that new replicas are ready, not that they are working correctly; under CO8 the inside of the pod is closed to it. The breakage arrives from somewhere at the end of round four, and in the model this is an assumption. Because of this information item the system cannot measure, six pods are built for nothing under the two-two setting, and under the six-surplus setting there is nowhere to roll back to because no old version is left at all. The wrong decision here is not an arithmetic error: speed is proceeding without waiting for a piece of information the system cannot see.

Summary

• The replica set controller applies one rule: create if the gap is positive, remove if it is negative. Cold start, crash, the desired count changing, and manual addition are all handled by the same procedure.
• The two pods added by hand outside the declaration were deleted in the same round; manual work done in the cluster does not outlive the next reconciliation round.
• In six of fourteen rounds the ready replica count stayed below desired; the source of this is readiness taking two rounds, that is, the delay between a decision and its effect.
• If both rollout numbers are zero, the rollout never starts; allowing one shortfall spends eighteen of nineteen rounds below capacity, and allowing one surplus does the same work in eighteen rounds without ever dropping capacity.
• Two surplus and two shortfall bring the rollout down to six rounds, six surplus to three; in the latter, peak pod count rises to twelve, that is, it doubles.
• The rollback window is the same as the rollout’s duration: breakage noticed in round four wastes two pods under the one-zero setting and six pods under the two-two setting; a rollout that finishes in three rounds leaves no old replica to roll back to.

Next Step

This lesson measured keeping six pods up and changing what they contain. In both runs pods were continually created and deleted: ten creations and two deletions in fourteen rounds, six creations and six deletions in the rollout. Every new pod is a new pod, and the previous one’s address went with it. In that case, which one will another component sending it requests connect to? A replica set guarantees six pods but not which six. The next lesson measures the object that fills this gap: a name that stays fixed on top while the replica set changes underneath it, the round that resolving that name adds to a request, and the number of requests that go to a replica that is not ready.