Immutable Infrastructure

The previous two lessons changed infrastructure in place: the queue’s partition count grew, the machine count grew, existing resources were untouched. A machine’s package version advances the same way — the machine stays the same machine, the software on it changes. This is an in-place patch, and every application of it adds a layer to the machine’s history.

The second way never touches the machine at all: a new image is prepared, the old machine is shut down, and a new machine is born from the image in its place. This is immutable infrastructure — a running resource is not changed, a new one replaces it. This lesson runs the same update two ways over eight periods and counts four things: the number of diverged servers, the update duration, the rollback round count, and what a rebuild cannot carry across. At the end, the case where a rebuild gains nothing is shown with the same measure too.

The fleet is the collector machines of the regional measurement network: fictional software that takes meter readings from the field and writes them to the queue. Machines are not really provisioned; the fleet, the events, and the durations are modeled with node, and the generator is self-written.

IF13. The fleet is twelve machines, updated over eight periods. A machine’s fingerprint consists of its version and the traces it carries; if two machines carry the same fingerprint, they have not diverged. IF14. Events are generated once (seed 8140723, the generator is self-written) and replayed on both paths in the same order; the difference comes from the path, not from the events. IF15. In a given period, the probability of a patch failing per machine is 0.10; the probability of a hotfix being applied is 0.10. IF16. On the rebuild path, a machine is born from the current image every period. A hotfix applied within a period is baked into the image on even periods, and lost on odd periods. IF17. Durations: an in-place patch is 40 seconds, a rebuild is 360 seconds plus 90 seconds of draining; a wave is four machines. A failed patch is retried twice more, a failed health check takes one more round. IF18. Rollback happens in periods 3 and 6. A reverse patch fails on a machine carrying traces, and half of what remains is fixed on each round.

The Fleet and Eight Periods

// fleet.mjs — eight periods of a twelve-machine collector fleet. Model; the generator is self-written.
export const SEED = 8140723, MACHINES = 12, PERIODS = 8;
const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;

// Events are generated once and replayed in the same order on both paths; the difference comes from the path.
const r = generator(SEED);
export const events = [];
for (let d = 1; d <= PERIODS; d++)
  for (let m = 0; m < MACHINES; m++)
    events.push({ d, m, patchFails: r() < 0.10, hotfix: r() < 0.10 });
const periodOf = (d) => events.filter((o) => o.d === d);
const fingerprint = (x) => `${x.version}|${[...x.traces].sort().join(",")}`;
export const diverged = (s) => new Set(s.map(fingerprint)).size;
const emptyFleet = () => Array.from({ length: MACHINES }, () => ({ version: "1.0.0", traces: new Set() }));

export const pathInPlace = () => {                        // in-place patch: traces accumulate, never wiped
  const s = emptyFleet(), series = [];
  let failed = 0, hotfix = 0;
  for (let d = 1; d <= PERIODS; d++) {
    for (const o of periodOf(d)) {
      if (o.patchFails) { s[o.m].traces.add(`half-patch-${o.d}`); failed++; }
      else s[o.m].version = `1.${d}.0`;
      if (o.hotfix) { s[o.m].traces.add(`hotfix-${d}`); hotfix++; }
    }
    series.push(diverged(s));
  }
  return { s, series, failed, hotfix, lost: 0 };
};

export const pathRebuild = () => {                        // rebuild: from the image, every period
  const s = emptyFleet(), series = [], imageTraces = new Set();
  let failed = 0, hotfix = 0, lost = 0;
  for (let d = 1; d <= PERIODS; d++) {
    for (let m = 0; m < MACHINES; m++) s[m] = { version: `1.${d}.0`, traces: new Set(imageTraces) };
    const toBake = [];
    for (const o of periodOf(d)) {
      if (o.patchFails) failed++;                          // health check fails: one more round
      if (o.hotfix) { hotfix++; s[o.m].traces.add(`hotfix-${d}`);  // applied to a single machine within the period
        if (d % 2 === 0) toBake.push(`hotfix-${d}`); else lost++; }
    }
    series.push(diverged(s));
    for (const x of toBake) imageTraces.add(x);            // the fix is baked into the image on even periods
  }
  return { s, series, failed, hotfix, lost };
};

Measurement

// measure.mjs — divergence, update duration, rollback rounds, and what a rebuild cannot carry across.
import { SEED, MACHINES, PERIODS, events, diverged, pathInPlace, pathRebuild } from "./fleet.mjs";

const A = pathInPlace(), B = pathRebuild();
console.log(`seed ${SEED}   machines ${MACHINES}   periods ${PERIODS}   events ${events.length}`);
console.log(`diverged servers, period by period (distinct fingerprint)`);
console.log(`  in-place patch     ${A.series.join("  ")}   -> final ${diverged(A.s)}/${MACHINES}`);
console.log(`  rebuild            ${B.series.join("  ")}   -> final ${diverged(B.s)}/${MACHINES}`);
console.log(`  failed patch/health ${A.failed}   hotfixes applied ${A.hotfix}` +
  `   lost, never baked into image ${B.lost}\n`);

// Durations are in seconds; a wave is 4 machines. In-place patch 40 s, rebuild 360 s plus 90 s drain.
const WAVE = Math.ceil(MACHINES / 4), PATCH = 40, BUILD = 360, DRAIN = 90;
const periodFailed = (d) => events.filter((o) => o.d === d && o.patchFails).length;
let sA = 0, sB = 0;
for (let d = 1; d <= PERIODS; d++) {
  sA += WAVE * PATCH + periodFailed(d) * 2 * PATCH;          // a failed patch is retried twice more
  sB += WAVE * (DRAIN + BUILD) + periodFailed(d) * BUILD;    // a failed health check adds one more round
}
console.log(`update duration (${PERIODS}-period total, seconds)`);
console.log(`  in-place patch     ${sA}   period average ${(sA / PERIODS).toFixed(0)}`);
console.log(`  rebuild            ${sB}   period average ${(sB / PERIODS).toFixed(0)}` +
  `   ratio ${(sB / sA).toFixed(1)}x\n`);

// Rollback: the 3rd and 6th period's update is rolled back. A reverse patch fails on a machine carrying traces.
const rollback = (s) => {
  const traced = s.filter((x) => x.traces.size).length;
  let round = 1, remaining = traced;
  while (remaining > 0 && round < 3) { remaining = Math.floor(remaining / 2); round++; }
  return { round: remaining > 0 ? round + 1 : round, traced, manual: remaining };
};
const g = rollback(A.s);
console.log(`rollback (period 3 and period 6, twice)`);
console.log(`  in-place patch     round ${g.round} x 2 = ${g.round * 2}   machines the reverse patch fails on` +
  ` ${g.traced}   left for manual repair ${g.manual}`);
console.log(`  rebuild            round 1 x 2 = 2   the prior image tag is selected, fleet returns in` +
  ` ${WAVE * (DRAIN + BUILD)} seconds\n`);

// What a rebuild cannot carry across: local state, open connections, a warm cache.
const BUFFER = 1800, PROCESS_RATE = 20, CONNECTIONS = 40, WARMUP = 720, READS = 15, FACTOR = 3.4;
console.log(`what a rebuild cannot carry across (per period, ${MACHINES} machines)`);
console.log(`  queue buffer       reads dropped if not drained ${MACHINES * BUFFER}` +
  `   extra time if drained ${WAVE * DRAIN} s (buffer empties in ${BUFFER / PROCESS_RATE} s)`);
console.log(`  open connections   broken ${MACHINES * CONNECTIONS}   0 on in-place patch`);
console.log(`  warm cache         warmup ${WARMUP} s, data-store reads ${FACTOR}x` +
  `   extra reads ${MACHINES * WARMUP * READS * (FACTOR - 1)}\n`);

// What a rebuild does not gain: a single machine carrying state.
const DATA_GB = 400, SPEED_MBS = 180;
const transfer = Math.round(DATA_GB * 1024 / SPEED_MBS);
console.log(`a single state-carrying machine (measurement store, ${DATA_GB} GB local data)`);
console.log(`  in-place patch ${PATCH} s   rebuild ${transfer} s` +
  ` (${(transfer / 60).toFixed(1)} min, data copy at ${SPEED_MBS} MB/s)`);
console.log(`  diverged servers 1/1 -> 1/1   gain 0   cost ${(transfer / PATCH).toFixed(0)}x duration`);

seed 8140723   machines 12   periods 8   events 96
diverged servers, period by period (distinct fingerprint)
  in-place patch     3  6  6  6  7  7  8  10   -> final 10/12
  rebuild            2  2  1  1  1  2  2  2   -> final 2/12
  failed patch/health 7   hotfixes applied 9   lost, never baked into image 3

update duration (8-period total, seconds)
  in-place patch     1520   period average 190
  rebuild            13320   period average 1665   ratio 8.8x

rollback (period 3 and period 6, twice)
  in-place patch     round 4 x 2 = 8   machines the reverse patch fails on 10   left for manual repair 2
  rebuild            round 1 x 2 = 2   the prior image tag is selected, fleet returns in 1350 seconds

what a rebuild cannot carry across (per period, 12 machines)
  queue buffer       reads dropped if not drained 21600   extra time if drained 270 s (buffer empties in 90 s)
  open connections   broken 480   0 on in-place patch
  warm cache         warmup 720 s, data-store reads 3.4x   extra reads 311040

a single state-carrying machine (measurement store, 400 GB local data)
  in-place patch 40 s   rebuild 2276 s (37.9 min, data copy at 180 MB/s)
  diverged servers 1/1 -> 1/1   gain 0   cost 57x duration

Diverged Servers and Rollback

The same ninety-six events were replayed on both paths, and by the end of eight periods, ten of the twelve machines on the in-place patch path had become unique: ten distinct fingerprints, meaning groups of 1.2 machines on average. The series shows accumulation — three to ten, with no reversal. On the rebuild path, the series oscillates between one and two and never accumulates. The difference is not one of care: on both paths seven patches failed and nine hotfixes were applied. On the in-place patch path, their trace stays on the machine; on the rebuild path, the trace is wiped because the machine is reborn from scratch the next period.

Wiping has a cost, and it too is counted: three of the nine hotfixes were never written back into the image and were lost. A rebuild makes writing the fix back into the image mandatory; if it is not written back, the fix disappears the next period and the problem it fixed comes back. On the in-place patch path there is no such requirement — the fix stays. This is exactly what produces divergence: persistence.

The cost of divergence does not show up in the series itself; it shows up in a later operation. Ten unique fingerprints mean the next change gets tried against ten separate starting states. A patch working on one machine and not on another is no longer a possibility — it is the accumulated result of eight periods. On the rebuild path, the same change gets tried against at most two starting states, and one of the two is a single hotfix applied within the period — it too disappears once the period closes.

The duration table runs the other way. Rebuild takes 13,320 seconds over eight periods, in-place patching 1,520 seconds — 8.8 times as long. The gap between 190 seconds and 1,665 seconds per period is large enough to determine the size of a maintenance window. The sign flips for rollback: in-place patching spent a total of eight rounds across two rollbacks, because the reverse patch failed on the ten machines carrying traces and two machines ended up needing manual repair. Rebuild finished in two rounds — the prior image tag is selected, and the fleet returns to that tag in 1,350 seconds. Rollback’s round count and duration come apart here: a rebuild’s round count is fixed and predictable, an in-place patch’s depends on the trace the environment has accumulated and cannot be known in advance.

What a Rebuild Cannot Carry Across

The third table shows that a rebuild’s cost is not just duration. When a machine is shut down, it carries three things, and none of them are part of the image. Reads waiting in the queue buffer: without draining, 21,600 reads are dropped per period; with draining, none are dropped, but ninety seconds are added to every wave. Open connections: four hundred eighty long-lived connections from the field break and are re-established every period; on in-place patching this number is zero. Warm cache: a newly born machine’s cache is empty, data-store reads run at 3.4 times their rate through a twelve-minute warmup, and 311,040 extra reads are born per period.

All three have a way to reduce them, and all three write their cost somewhere else. Draining saves the buffer and lengthens the duration. Broken connections become lossless if the field unit waits to reconnect and holds its reads locally — but that means adding a buffer to the field unit. A warming cache is hidden by warming the machine up before it takes traffic; that too is extra time per wave. The measure does not change, it only changes column.

What these three items have in common is that none of them is written in the record. The image version, the package list, and the settings go into the record; buffer occupancy, connection count, and cache temperature do not. A rebuild reproduces exactly what is written in the record, and zeroes out, every time, what is not written in it. The mechanism that prevents divergence and the mechanism that destroys local state are the same mechanism; one cannot be separated from the other.

Where It Gains Nothing

The last table shows where a rebuild does not help. The measurement store is a single machine and carries four hundred gigabytes of local data. Being a single machine, there is nothing to diverge: the fingerprint count is one both before and after the rebuild. The gain is zero. The cost is the copy duration — 2,276 seconds, fifty-seven times the in-place patch — and the whole of that duration goes to moving data, not to updating software.

The same calculation gives a different result on the measurement network’s other components. Verifier machines carry no state and number four; there, a rebuild produces a gain close to what it does on the collectors. The queue and the object store are already resources the provider operates, and they carry no operating environment to patch; the question itself is never asked for these two components. The decision comes not from seeing the fleet as one piece, but from separating it role by role.

The criterion that follows fits in one sentence: a rebuild’s gain is proportional to the number of machines sharing a role and to the trace accumulated on those machines. On twelve collectors, the gain falls from ten to one; on a single state-carrying machine, there is no gain. Immutable infrastructure is not decided as a principle, it is decided role by role, and the decision’s input is these two numbers.

Summary

• When the same 96 events were replayed on both paths, in-place patching made 10 of 12 machines unique by the end of 8 periods (series 3 to 10); on rebuild, the series stayed between 1 and 2.
• Both paths saw 7 failed patches and 9 applied hotfixes; what produced the difference was the persistence of the trace.
• Rebuild lost 3 of the 9 hotfixes; writing a fix back into the image becomes mandatory.
• Update duration is 8.8 times longer on rebuild (1,665 versus 190 seconds per period), but rollback finished in 2 rounds instead of 8, and the round count is known in advance.
• The three items a rebuild cannot carry across are 21,600 dropped reads, 480 broken connections, and 311,040 extra data-store reads per period; on a single state-carrying machine, gain is 0, cost is 57 times the duration.

Next Step

In this lesson, two jobs got mixed together. Which packages the image carries, what the settings on the machine are — that is one job; how many machines there are, which network they attach to, which disk they carry — that is another. In-place patching touches the first, rebuild touches the second — and the hotfix getting lost happened exactly on the boundary between these two jobs.

The next lesson draws that boundary. An environment’s items are split into two layers — the layer that creates the resource and the layer that configures the inside of the resource — and where the boundary is drawn gets counted: how many items are defined in both layers, how many in neither, how many items get applied twice when the boundary shifts, and which layer wins when both write to the same item.