---
title: 'Immutable Infrastructure'
source: 'https://academia.sh/en/courses/infrastructure-as-code/immutable-infrastructure'
course: 'Infrastructure as Code'
language: en
updated: '2026-09-13T05:08:24+00:00'
license: 'CC BY-SA 4.0'
objectives:
    - 'Runs the same update two ways and separates the outcome with a single criterion: how many machines became unique by the end of the period'
    - 'Counts the update duration, the rollback round count, and the number of machines a rollback falls back to'
    - 'Measures the local state a rebuild cannot carry across, and the case where it gains nothing'
---

# Immutable Infrastructure

A twelve-machine fleet is updated two ways over eight periods: an in-place patch and a rebuild from an image. The same event sequence is replayed on both paths; the number of servers that diverge from one another, the update duration, the rollback round count, the local state and connections a rebuild cannot carry across, and the case where a rebuild gains nothing are counted.

The previous two lessons changed infrastructure in place: the queue's partition count grew, the
machine count grew, existing resources were untouched. A machine's package version advances the
same way — the machine stays the same machine, the software on it changes. This is an **in-place
patch**, and every application of it adds a layer to the machine's history.

The second way never touches the machine at all: a new image is prepared, the old machine is shut
down, and a new machine is born from the image in its place. This is **immutable infrastructure** —
a running resource is not changed, a new one replaces it. This lesson runs the same update two ways
over eight periods and counts four things: the number of **diverged servers**, the update duration,
the rollback round count, and what a rebuild **cannot carry across**. At the end, the case where a
rebuild gains nothing is shown with the same measure too.

The fleet is the collector machines of the **regional measurement network**: **fictional** software
that takes meter readings from the field and writes them to the queue. Machines are not really
provisioned; the fleet, the events, and the durations are modeled with `node`, and the generator is
self-written.

**IF13.** The fleet is twelve machines, updated over eight periods. A machine's **fingerprint**
consists of its version and the traces it carries; if two machines carry the same fingerprint, they
have not diverged.
**IF14.** Events are generated once (seed 8140723, the generator is self-written) and replayed on
both paths **in the same order**; the difference comes from the path, not from the events.
**IF15.** In a given period, the probability of a patch failing per machine is 0.10; the probability
of a hotfix being applied is 0.10.
**IF16.** On the rebuild path, a machine is born from the current image every period. A hotfix
applied within a period is baked into the image on even periods, and **lost** on odd periods.
**IF17.** Durations: an in-place patch is 40 seconds, a rebuild is 360 seconds plus 90 seconds of
draining; a wave is four machines. A failed patch is retried twice more, a failed health check takes
one more round.
**IF18.** Rollback happens in periods 3 and 6. A reverse patch fails on a machine carrying traces,
and half of what remains is fixed on each round.

## The Fleet and Eight Periods

```js
// fleet.mjs — eight periods of a twelve-machine collector fleet. Model; the generator is self-written.
export const SEED = 8140723, MACHINES = 12, PERIODS = 8;
const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;

// Events are generated once and replayed in the same order on both paths; the difference comes from the path.
const r = generator(SEED);
export const events = [];
for (let d = 1; d <= PERIODS; d++)
  for (let m = 0; m < MACHINES; m++)
    events.push({ d, m, patchFails: r() < 0.10, hotfix: r() < 0.10 });
const periodOf = (d) => events.filter((o) => o.d === d);
const fingerprint = (x) => `${x.version}|${[...x.traces].sort().join(",")}`;
export const diverged = (s) => new Set(s.map(fingerprint)).size;
const emptyFleet = () => Array.from({ length: MACHINES }, () => ({ version: "1.0.0", traces: new Set() }));

export const pathInPlace = () => {                        // in-place patch: traces accumulate, never wiped
  const s = emptyFleet(), series = [];
  let failed = 0, hotfix = 0;
  for (let d = 1; d <= PERIODS; d++) {
    for (const o of periodOf(d)) {
      if (o.patchFails) { s[o.m].traces.add(`half-patch-${o.d}`); failed++; }
      else s[o.m].version = `1.${d}.0`;
      if (o.hotfix) { s[o.m].traces.add(`hotfix-${d}`); hotfix++; }
    }
    series.push(diverged(s));
  }
  return { s, series, failed, hotfix, lost: 0 };
};

export const pathRebuild = () => {                        // rebuild: from the image, every period
  const s = emptyFleet(), series = [], imageTraces = new Set();
  let failed = 0, hotfix = 0, lost = 0;
  for (let d = 1; d <= PERIODS; d++) {
    for (let m = 0; m < MACHINES; m++) s[m] = { version: `1.${d}.0`, traces: new Set(imageTraces) };
    const toBake = [];
    for (const o of periodOf(d)) {
      if (o.patchFails) failed++;                          // health check fails: one more round
      if (o.hotfix) { hotfix++; s[o.m].traces.add(`hotfix-${d}`);  // applied to a single machine within the period
        if (d % 2 === 0) toBake.push(`hotfix-${d}`); else lost++; }
    }
    series.push(diverged(s));
    for (const x of toBake) imageTraces.add(x);            // the fix is baked into the image on even periods
  }
  return { s, series, failed, hotfix, lost };
};
```

## Measurement

```js
// measure.mjs — divergence, update duration, rollback rounds, and what a rebuild cannot carry across.
import { SEED, MACHINES, PERIODS, events, diverged, pathInPlace, pathRebuild } from "./fleet.mjs";

const A = pathInPlace(), B = pathRebuild();
console.log(`seed ${SEED}   machines ${MACHINES}   periods ${PERIODS}   events ${events.length}`);
console.log(`diverged servers, period by period (distinct fingerprint)`);
console.log(`  in-place patch     ${A.series.join("  ")}   -> final ${diverged(A.s)}/${MACHINES}`);
console.log(`  rebuild            ${B.series.join("  ")}   -> final ${diverged(B.s)}/${MACHINES}`);
console.log(`  failed patch/health ${A.failed}   hotfixes applied ${A.hotfix}` +
  `   lost, never baked into image ${B.lost}\n`);

// Durations are in seconds; a wave is 4 machines. In-place patch 40 s, rebuild 360 s plus 90 s drain.
const WAVE = Math.ceil(MACHINES / 4), PATCH = 40, BUILD = 360, DRAIN = 90;
const periodFailed = (d) => events.filter((o) => o.d === d && o.patchFails).length;
let sA = 0, sB = 0;
for (let d = 1; d <= PERIODS; d++) {
  sA += WAVE * PATCH + periodFailed(d) * 2 * PATCH;          // a failed patch is retried twice more
  sB += WAVE * (DRAIN + BUILD) + periodFailed(d) * BUILD;    // a failed health check adds one more round
}
console.log(`update duration (${PERIODS}-period total, seconds)`);
console.log(`  in-place patch     ${sA}   period average ${(sA / PERIODS).toFixed(0)}`);
console.log(`  rebuild            ${sB}   period average ${(sB / PERIODS).toFixed(0)}` +
  `   ratio ${(sB / sA).toFixed(1)}x\n`);

// Rollback: the 3rd and 6th period's update is rolled back. A reverse patch fails on a machine carrying traces.
const rollback = (s) => {
  const traced = s.filter((x) => x.traces.size).length;
  let round = 1, remaining = traced;
  while (remaining > 0 && round < 3) { remaining = Math.floor(remaining / 2); round++; }
  return { round: remaining > 0 ? round + 1 : round, traced, manual: remaining };
};
const g = rollback(A.s);
console.log(`rollback (period 3 and period 6, twice)`);
console.log(`  in-place patch     round ${g.round} x 2 = ${g.round * 2}   machines the reverse patch fails on` +
  ` ${g.traced}   left for manual repair ${g.manual}`);
console.log(`  rebuild            round 1 x 2 = 2   the prior image tag is selected, fleet returns in` +
  ` ${WAVE * (DRAIN + BUILD)} seconds\n`);

// What a rebuild cannot carry across: local state, open connections, a warm cache.
const BUFFER = 1800, PROCESS_RATE = 20, CONNECTIONS = 40, WARMUP = 720, READS = 15, FACTOR = 3.4;
console.log(`what a rebuild cannot carry across (per period, ${MACHINES} machines)`);
console.log(`  queue buffer       reads dropped if not drained ${MACHINES * BUFFER}` +
  `   extra time if drained ${WAVE * DRAIN} s (buffer empties in ${BUFFER / PROCESS_RATE} s)`);
console.log(`  open connections   broken ${MACHINES * CONNECTIONS}   0 on in-place patch`);
console.log(`  warm cache         warmup ${WARMUP} s, data-store reads ${FACTOR}x` +
  `   extra reads ${MACHINES * WARMUP * READS * (FACTOR - 1)}\n`);

// What a rebuild does not gain: a single machine carrying state.
const DATA_GB = 400, SPEED_MBS = 180;
const transfer = Math.round(DATA_GB * 1024 / SPEED_MBS);
console.log(`a single state-carrying machine (measurement store, ${DATA_GB} GB local data)`);
console.log(`  in-place patch ${PATCH} s   rebuild ${transfer} s` +
  ` (${(transfer / 60).toFixed(1)} min, data copy at ${SPEED_MBS} MB/s)`);
console.log(`  diverged servers 1/1 -> 1/1   gain 0   cost ${(transfer / PATCH).toFixed(0)}x duration`);
```

```
seed 8140723   machines 12   periods 8   events 96
diverged servers, period by period (distinct fingerprint)
  in-place patch     3  6  6  6  7  7  8  10   -> final 10/12
  rebuild            2  2  1  1  1  2  2  2   -> final 2/12
  failed patch/health 7   hotfixes applied 9   lost, never baked into image 3

update duration (8-period total, seconds)
  in-place patch     1520   period average 190
  rebuild            13320   period average 1665   ratio 8.8x

rollback (period 3 and period 6, twice)
  in-place patch     round 4 x 2 = 8   machines the reverse patch fails on 10   left for manual repair 2
  rebuild            round 1 x 2 = 2   the prior image tag is selected, fleet returns in 1350 seconds

what a rebuild cannot carry across (per period, 12 machines)
  queue buffer       reads dropped if not drained 21600   extra time if drained 270 s (buffer empties in 90 s)
  open connections   broken 480   0 on in-place patch
  warm cache         warmup 720 s, data-store reads 3.4x   extra reads 311040

a single state-carrying machine (measurement store, 400 GB local data)
  in-place patch 40 s   rebuild 2276 s (37.9 min, data copy at 180 MB/s)
  diverged servers 1/1 -> 1/1   gain 0   cost 57x duration
```

## Diverged Servers and Rollback

The same ninety-six events were replayed on both paths, and by the end of eight periods, ten of the
twelve machines on the in-place patch path had become **unique**: ten distinct fingerprints, meaning
groups of 1.2 machines on average. The series shows accumulation — three to ten, with no reversal.
On the rebuild path, the series oscillates between one and two and never accumulates. The difference
is not one of care: on both paths seven patches failed and nine hotfixes were applied. On the
in-place patch path, their trace **stays** on the machine; on the rebuild path, the trace is
**wiped** because the machine is reborn from scratch the next period.

Wiping has a cost, and it too is counted: three of the nine hotfixes were never written back into
the image and were lost. A rebuild makes writing the fix back into the image **mandatory**; if it is
not written back, the fix disappears the next period and the problem it fixed comes back. On the
in-place patch path there is no such requirement — the fix stays. This is exactly what produces
divergence: persistence.

The cost of divergence does not show up in the series itself; it shows up in a later operation. Ten
unique fingerprints mean the next change gets tried against ten separate starting states. A patch
working on one machine and not on another is no longer a possibility — it is the accumulated result
of eight periods. On the rebuild path, the same change gets tried against at most two starting
states, and one of the two is a single hotfix applied within the period — it too disappears once the
period closes.

The duration table runs the other way. Rebuild takes 13,320 seconds over eight periods, in-place
patching 1,520 seconds — **8.8 times as long**. The gap between 190 seconds and 1,665 seconds per
period is large enough to determine the size of a maintenance window. The sign flips for rollback:
in-place patching spent a total of eight rounds across two rollbacks, because the reverse patch
failed on the ten machines carrying traces and two machines ended up needing manual repair. Rebuild
finished in two rounds — the prior image tag is selected, and the fleet returns to that tag in 1,350
seconds. Rollback's **round count** and **duration** come apart here: a rebuild's round count is
fixed and predictable, an in-place patch's depends on the trace the environment has accumulated and
cannot be known in advance.

## What a Rebuild Cannot Carry Across

The third table shows that a rebuild's cost is not just duration. When a machine is shut down, it
carries three things, and none of them are part of the image. Reads waiting in the queue buffer:
without draining, 21,600 reads are dropped per period; with draining, none are dropped, but ninety
seconds are added to every wave. Open connections: four hundred eighty long-lived connections from
the field break and are re-established every period; on in-place patching this number is zero. Warm
cache: a newly born machine's cache is empty, data-store reads run at 3.4 times their rate through a
twelve-minute warmup, and 311,040 extra reads are born per period.

All three have a way to reduce them, and all three write their cost somewhere else. Draining saves
the buffer and lengthens the duration. Broken connections become lossless if the field unit waits to
reconnect and holds its reads locally — but that means adding a buffer to the field unit. A warming
cache is hidden by warming the machine up before it takes traffic; that too is extra time per wave.
The measure does not change, it only changes column.

What these three items have in common is that none of them is **written in the record**. The image
version, the package list, and the settings go into the record; buffer occupancy, connection count,
and cache temperature do not. A rebuild reproduces exactly what is written in the record, and
**zeroes out, every time,** what is not written in it. The mechanism that prevents divergence and
the mechanism that destroys local state are the same mechanism; one cannot be separated from the
other.

## Where It Gains Nothing

The last table shows where a rebuild does not help. The measurement store is a single machine and
carries four hundred gigabytes of local data. Being a single machine, there is nothing to diverge:
the fingerprint count is one both before and after the rebuild. The gain is zero. The cost is the
copy duration — 2,276 seconds, **fifty-seven times** the in-place patch — and the whole of that
duration goes to moving data, not to updating software.

The same calculation gives a different result on the measurement network's other components.
Verifier machines carry no state and number four; there, a rebuild produces a gain close to what it
does on the collectors. The queue and the object store are already resources the provider operates,
and they carry no operating environment to patch; the question itself is never asked for these two
components. The decision comes not from seeing the fleet as one piece, but from separating it **role
by role**.

The criterion that follows fits in one sentence: a rebuild's gain is proportional to the number of
machines sharing a role and to the trace accumulated on those machines. On twelve collectors, the
gain falls from ten to one; on a single state-carrying machine, there is no gain. Immutable
infrastructure is not decided as a principle, it is decided **role by role**, and the decision's
input is these two numbers.

## Summary

- When the same 96 events were replayed on both paths, in-place patching made 10 of 12 machines
  unique by the end of 8 periods (series 3 to 10); on rebuild, the series stayed between 1 and 2.
- Both paths saw 7 failed patches and 9 applied hotfixes; what produced the difference was the
  persistence of the trace.
- Rebuild lost 3 of the 9 hotfixes; writing a fix back into the image becomes mandatory.
- Update duration is 8.8 times longer on rebuild (1,665 versus 190 seconds per period), but rollback
  finished in 2 rounds instead of 8, and the round count is known in advance.
- The three items a rebuild cannot carry across are 21,600 dropped reads, 480 broken connections,
  and 311,040 extra data-store reads per period; on a single state-carrying machine, gain is 0, cost
  is 57 times the duration.

## Next Step

In this lesson, two jobs got mixed together. Which packages the image carries, what the settings on
the machine are — that is one job; how many machines there are, which network they attach to, which
disk they carry — that is another. In-place patching touches the first, rebuild touches the second —
and the hotfix getting lost happened exactly on the boundary between these two jobs.

The next lesson draws that boundary. An environment's items are split into two layers — the layer
that creates the resource and the layer that configures the inside of the resource — and where the
boundary is drawn gets counted: how many items are defined in both layers, how many in neither, how
many items get applied twice when the boundary shifts, and which layer wins when both write to the
same item.