Server Image Building

The previous lesson measured two ways of carrying the record to a running machine. Both assumed the same ordering: the machine boots empty first, then gets configured. Both the rollout duration and the partial failure came from that ordering — the machine had already started taking work while it did not yet match the record.

The ordering can be reversed: instead of the setting being carried to the machine after boot, it can be placed in advance inside the image the machine boots from. This lesson sets three paths side by side as kinds — configure at every boot, golden image, and hybrid (a baked base plus a small setting applied at boot). M22/K02 measured the base image’s bytes and its update overhead; that measurement is not repeated. What is measured here is not the image’s size, it is the image build flow.

IV31. The fleet is 50 machines running the regional measurement network’s — the fictional software collecting water-meter readings’ — collectors, across 6 roles and 3 environments. IV32. The three paths’ boot steps and seconds are fictional; each step’s dependency and number of calls is written down. IV33. Dependencies’ probability of being up at boot time is between 0.995 and 0.999, and a sudden spike in demand boots 40 machines at once. IV34. The numbers come from our own generator; the seeds 20260811–20260814 are visible and are an order of magnitude. IV35. An image bakes in 7–19 minutes, plus 6 minutes of testing on top; the image builder bakes 3 images at a time, and the fleet replaces 12 machines a day. IV36. The image’s content record writes only direct components, not the indirect ones that come along; the full rebake cycle is 30 days.

What Happens at Boot

All three paths boot the same service. The configure-at-every-boot path downloads and installs packages at boot time; the golden-image path finds them already in the image; the hybrid path finds the image ready but pulls the environment-varying setting at boot.

// measurement-network/boot.mjs — the same service's three boot paths (model; a kind, not a product)
export const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

// External dependency kinds and each one's probability of being up at boot time (fictional).
export const UP = {
  "name resolver": 0.999, "package repository": 0.995, "configuration server": 0.995,
  "secret store": 0.998, "image store": 0.997,
};

// Step: [name, seconds, dependency or null, number of calls made in that step].
export const PATH = {
  "every boot": [
    ["kernel boot", 12, null, 0], ["name resolution", 2, "name resolver", 1],
    ["package listing", 9, "package repository", 1], ["14 package installs", 128, "package repository", 14],
    ["config pull", 11, "configuration server", 1], ["secret pull", 6, "secret store", 1],
    ["service start", 8, null, 0],
  ],
  "golden image": [
    ["image pull", 19, "image store", 1], ["kernel boot", 12, null, 0],
    ["secret pull", 6, "secret store", 1], ["service start", 8, null, 0],
  ],
  "hybrid": [
    ["image pull", 19, "image store", 1], ["kernel boot", 12, null, 0],
    ["environment setting pull", 7, "configuration server", 1], ["secret pull", 6, "secret store", 1],
    ["service start", 8, null, 0],
  ],
};

export const measure = (ad) => {
  const step = PATH[ad];
  const calls = step.reduce((t, [, , , c]) => t + c, 0);
  const success = step.reduce((t, [, , b, c]) => t * (b ? UP[b] ** c : 1), 1);
  const dependency = [...new Set(step.filter(([, , b]) => b).map(([, , b]) => b))];
  return { ad, duration: step.reduce((t, [, s]) => t + s, 0), calls, success, dependency };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const NEW_MACHINES = 40;
  console.log("path".padEnd(14) + "boot".padEnd(9) + "ext calls".padEnd(11) +
    "dependency".padEnd(12) + "boot success".padEnd(17) + `not booting at ${NEW_MACHINES} machines`);
  for (const ad of Object.keys(PATH)) {
    const o = measure(ad);
    console.log(o.ad.padEnd(14) + `${o.duration} s`.padEnd(9) + String(o.calls).padEnd(11) +
      `${o.dependency.length} kinds`.padEnd(12) + `${(100 * o.success).toFixed(1)}%`.padEnd(17) +
      (NEW_MACHINES * (1 - o.success)).toFixed(1));
  }
  console.log("\nmachines that fail to boot when one dependency drops (out of 40):");
  const columns = Object.keys(PATH);
  console.log("dropped".padEnd(24) + columns.map((a) => a.padEnd(14)).join("").trimEnd());
  for (const b of Object.keys(UP))
    console.log((b.padEnd(24) + columns.map((ad) =>
      (measure(ad).dependency.includes(b) ? "0/40" : "40/40").padEnd(14)).join("")).trimEnd());
}

path          boot     ext calls  dependency  boot success     not booting at 40 machines
every boot    176 s    18         4 kinds     92.0%            3.2
golden image  45 s     2          2 kinds     99.5%            0.2
hybrid        52 s     3          3 kinds     99.0%            0.4

machines that fail to boot when one dependency drops (out of 40):
dropped                 every boot    golden image  hybrid
name resolver           0/40          40/40         40/40
package repository      0/40          40/40         40/40
configuration server    0/40          40/40         0/40
secret store            0/40          0/40          0/40
image store             40/40         0/40          0/40

Boot duration drops from 176 seconds to 45 seconds — roughly a fourfold cut. But the real difference is in the second column: 18 outbound calls drop to 2. This is a quantity independent of duration; every call is a separate point of failure, and under a demand spike, all of them happen at once. At a forty-machine boot, the expected number of machines that fail to boot drops from 3.2 to 0.2; the golden-image path does less to speed up boot than to make boot independent.

The second table writes down the limit of this. On the secret-store row, all three paths are zero: a sensitive value cannot be baked into an image — if it were, anyone who can read the image can read the secret too. The golden path removes seventeen of eighteen calls; it cannot remove the eighteenth. The image-store row is the other side of the trade-off: as the golden path cuts four dependencies down to two, it picks up a new one. That is the configure-at-every-boot path’s one advantage; it keeps booting even if the image store drops.

The gap between the record and reality exists here too. In the configure-at-every-boot path, the record says “these packages will be installed”; what actually gets installed is whatever the package repository serves that day, and two machines booting on two different days do not get the same content. In the golden path, the record is turned into reality once, and freezes into the image.

How Many Images Get Rebaked When a Fix Lands

The cost of frozen content is that changing it now requires a build step. When a security fix lands, the configure-at-every-boot path bakes nothing; the golden path rebakes every affected image. How many images get baked depends on where the environment is written.

// measurement-network/bake.mjs — how many images get rebaked when a security fix lands (model)
import { generator } from "./boot.mjs";

export const ROLE = 6, ENVIRONMENT = 3;    // the measurement network's six roles, three environments (fictional)
const LANE = 3, TEST = 6;                  // concurrent baking lanes; per-image test time (min)
const MACHINE = 50, REPLACE = 12;          // fleet size and daily machine-replacement capacity

// On the pure golden-image path, each role is baked separately for each environment; on the hybrid
// path the environment setting is applied at boot, so one image per role is enough.
export const IMAGE_COUNT = { "every boot": 0, "golden image": ROLE * ENVIRONMENT, "hybrid": ROLE };

// A fix touches the roles that carry the component; a base component is in every role.
const fixes = (seed = 20260811) => {
  const r = generator(seed);
  return ["core library", "encryption library", "name resolver plugin",
    "shell", "compression library"].map((ad) =>
      ({ ad, role: r() < 0.4 ? ROLE : 1 + Math.floor(r() * (ROLE - 1)) }));
};

// Window = baking wall-clock time + fleet renewal (REPLACE machines a day).
export const window = (n, seed = 20260812) => {
  const r = generator(seed);
  const single = Array.from({ length: n }, () => 7 + Math.round(r() * 12) + TEST);
  let wall = 0;
  for (let i = 0; i < n; i += LANE) wall += Math.max(...single.slice(i, i + LANE));
  return { wall, total: single.reduce((t, v) => t + v, 0), round: Math.ceil(n / LANE),
    days: wall / 1440 + MACHINE / REPLACE };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`${ROLE} roles, ${ENVIRONMENT} environments, ${MACHINE} machines; ` +
    `image count: golden ${IMAGE_COUNT["golden image"]}, hybrid ${IMAGE_COUNT["hybrid"]}\n`);
  console.log("fix".padEnd(26) + "roles touched".padEnd(15) +
    "golden image".padEnd(14) + "hybrid".padEnd(10) + "every boot");
  let golden = 0, hybrid = 0;
  for (const f of fixes()) {
    golden += f.role * ENVIRONMENT; hybrid += f.role;
    console.log(f.ad.padEnd(26) + `${f.role}/${ROLE}`.padEnd(15) +
      `${f.role * ENVIRONMENT} images`.padEnd(14) + `${f.role} image${f.role === 1 ? "" : "s"}`.padEnd(10) + "0 images");
  }
  console.log("total".padEnd(41) + `${golden} images`.padEnd(14) + `${hybrid} images`.padEnd(10) + "0 images");

  console.log("\nrollout window for the widest fix (every role):");
  console.log("path".padEnd(14) + "images baked".padEnd(14) + "round".padEnd(7) +
    "wall hours".padEnd(13) + "builder minutes".padEnd(17) + "window");
  for (const [ad, n] of Object.entries(IMAGE_COUNT)) {
    const p = window(n);
    console.log(ad.padEnd(14) + String(n).padEnd(14) + String(p.round).padEnd(7) +
      `${(p.wall / 60).toFixed(1)} hr`.padEnd(13) + String(p.total).padEnd(17) +
      `${p.days.toFixed(2)} days`);
  }
}

6 roles, 3 environments, 50 machines; image count: golden 18, hybrid 6

fix                       roles touched  golden image  hybrid    every boot
core library              6/6            18 images     6 images  0 images
encryption library        3/6            9 images      3 images  0 images
name resolver plugin      4/6            12 images     4 images  0 images
shell                     6/6            18 images     6 images  0 images
compression library       6/6            18 images     6 images  0 images
total                                    75 images     25 images 0 images

rollout window for the widest fix (every role):
path          images baked  round  wall hours   builder minutes  window
every boot    0             0      0.0 hr       0                4.17 days
golden image  18            6      2.1 hr       334              4.26 days
hybrid        6             2      0.7 hr       117              4.20 days

The first table shows the cost of where the environment gets written. If the environment is baked into the image, each role gets baked three times: five fixes produce 75 image bakes. If the environment is applied at boot, the same five fixes need only 25 bakes. The threefold gap between the two is the price of the single extra call the hybrid path pays at boot.

The second table corrects a common expectation. A golden image’s freshness is assumed to lag; once counted, the lag is 0.09 days — 4.26 days against 4.17 days. Baking wall-clock time is 2.1 hours, and holds about two percent of the window. What decides the window is not baking, it is the fleet’s renewal: since only 12 of 50 machines can be replaced a day, a fix cannot reach every machine before four days pass, and that is the cost all three paths share. The discipline of immutable infrastructure requires replacing a machine instead of patching it in place; the window is the price of that rule.

The builder-minutes column is a repeated cost: the same fix spends 334 builder-minutes on the pure golden path, 117 on the hybrid path. The image builder is itself an infrastructure component that needs watching; if its capacity drops, the window stretches directly.

What is Inside the Image

A record decides which images get rebaked: the image’s content record writes down which image carries which component. If this record is incomplete, the rebake set is incomplete too, and no one sees the gap, because baking finishes successfully.

// measurement-network/freshness.mjs — the gap between an image's content record and its real content (model)
import { generator } from "./boot.mjs";
import { ROLE, ENVIRONMENT } from "./bake.mjs";

const COMPONENT = ["core library", "encryption library", "compression library",
  "name resolver plugin", "certificate store", "timezone data", "shell",
  "localization data", "math library", "version compatibility layer",
  "logging library", "text parser", "network client", "scheduler"];
const CYCLE = 30;                          // full rebake cycle (days)
const ROLE_MACHINE = [5, 6, 7, 9, 11, 12]; // machines per role; 50 total

// Each role image: direct components written in the record + indirect components the record misses.
const catalog = (seed = 20260813) => {
  const r = generator(seed);
  return Array.from({ length: ROLE }, (_, i) => {
    const recorded = COMPONENT.filter(() => r() < 0.4);
    const indirect = COMPONENT.filter((b) => !recorded.includes(b) && r() < 0.35);
    return { recorded, actual: [...recorded, ...indirect], machine: ROLE_MACHINE[i] };
  });
};

// Five separate security fixes: components are chosen without replacement, the day comes from the generator.
const fix = (seed = 20260814) => {
  const r = generator(seed), pool = [...COMPONENT];
  return Array.from({ length: 5 }, () => ({
    component: pool.splice(Math.floor(r() * pool.length), 1)[0],
    day: 1 + Math.floor(r() * CYCLE),
  }));
};

const roleImages = catalog(), fixList = fix();
const recordedCount = roleImages.reduce((t, i) => t + i.recorded.length, 0);
const actualCount = roleImages.reduce((t, i) => t + i.actual.length, 0);
console.log(`${ROLE} role images, ${COMPONENT.length} component kinds; the content record writes ` +
  `${recordedCount} entries, the images have ${actualCount} entries: ${actualCount - recordedCount} entries ` +
  `out of record (${(100 * (actualCount - recordedCount) / actualCount).toFixed(0)}%)\n`);
console.log("fix".padEnd(26) + "in record".padEnd(11) + "in reality".padEnd(12) +
  "missed".padEnd(11) + "unpatched machines".padEnd(20) + "machine-days");
let totalDays = 0, totalMissed = 0;
for (const f of fixList) {
  const inRecord = roleImages.filter((i) => i.recorded.includes(f.component));
  const inActual = roleImages.filter((i) => i.actual.includes(f.component));
  const missed = inActual.filter((i) => !inRecord.includes(i));
  const machines = missed.reduce((t, i) => t + i.machine, 0);
  const days = machines * (CYCLE - f.day);
  totalDays += days; totalMissed += missed.length;
  console.log(f.component.padEnd(26) + `${inRecord.length} image${inRecord.length === 1 ? "" : "s"}`.padEnd(11) +
    `${inActual.length} image${inActual.length === 1 ? "" : "s"}`.padEnd(12) + `${missed.length} image${missed.length === 1 ? "" : "s"}`.padEnd(11) +
    `${machines}/50`.padEnd(20) + String(days));
}
console.log(`\nimages missed across five fixes: ${totalMissed} (${totalMissed * ENVIRONMENT} on the ` +
  `pure golden-image path); unpatched machine-days: ${totalDays}`);
console.log(`missed images close only at the next ${CYCLE}-day full rebake`);

6 role images, 14 component kinds; the content record writes 32 entries, the images have 52 entries: 20 entries out of record (38%)

fix                       in record  in reality  missed     unpatched machines  machine-days
math library              6 images   6 images    0 images   0/50                0
shell                     3 images   5 images    2 images   23/50               322
certificate store         2 images   2 images    0 images   0/50                0
encryption library        1 image    3 images    2 images   14/50               126
timezone data             0 images   0 images    0 images   0/50                0

images missed across five fixes: 4 (12 on the pure golden-image path); unpatched machine-days: 448
missed images close only at the next 30-day full rebake

The header row is this lesson’s number. The content record writes 32 component entries for six images; the images have 52 entries inside them. 20 entries, 38 percent, are out of record — components that ride along with another component. The record is not wrong; it is incomplete.

The cost of that incompleteness shows up in the third and fourth columns. On the shell fix, the record says 3 images are affected, in reality 5 images are affected; 2 images do not get rebaked, and 23 machines booted from those images stay unpatched. On the encryption library fix, the record says 1 image, the actual is 3. Across five fixes, 4 images are missed in total, and 448 unpatched machine-days accumulate. These machines sit exposed until the next 30-day full rebake, because no one is looking for them.

The last line also has an environment dimension: the same gap misses not 4 but 12 images on the pure golden path. Baking the environment into the image does not just triple the baking load, it triples every gap in the record too.

This completes the three paths’ measure. In the configure-at-every-boot path, there is no such thing as a rebake set, so nothing can even be computed wrong; the moment a fix lands in the package repository, it arrives at every machine’s next boot. The golden image makes boot independent and freezes the content — in exchange it turns inventory into a record, and that record’s gaps are a silent security hole. The 20 entries the record does not cover are what this course has been counting from the start.

Summary

• Boot duration drops from 176 seconds to 45, outbound calls from 18 to 2; the expected number of machines that fail to boot in a forty-machine boot drops from 3.2 to 0.2.
• No path escapes the secret-store call — a sensitive value cannot be baked into an image; the golden path cuts the dependency count from four to two while picking up the image store as a new dependency.
• If the environment is baked into the image, five fixes need 75 image bakes; if applied at boot, 25 bakes; builder-minutes are 334 against 117.
• The rollout window is 4.26 days against 4.17: baking’s contribution is 0.09 days, and what decides the window is the fleet’s ability to replace only 12 machines a day.
• The content record writes 32 entries, the images have 52; the 38 percent gap leaves 4 images (12 on the pure golden path) out of the rebake set across five fixes and accumulates 448 unpatched machine-days.
• A golden image removes drift from boot but turns inventory into a record; that record’s gaps are silent, because baking finishes successfully.

Next Step

The flow this lesson measured is itself a build step now: an image is compiled, tested, versioned, and shipped to environments. The same is true of an infrastructure record — a plan is produced, it passes through static checking and policy scanning, gets priced, gets applied. This topic’s lessons counted these steps one by one, but every one of them was run by hand.

M22/K04 ran an application change through gates and measured what the gate caught and what it missed; that measure is not repeated, it is applied. The next lesson runs an infrastructure change through the same pipeline and asks: when this course’s checks are lined up in a pipeline, how many changes stop at which gate, how many changes pass every gate and still produce drift, and how many minutes does the pipeline itself add?