Machine Images

The previous lesson passed over one item in a single line: re-creation, 35 minutes. The real work inside that duration is an empty disk turning into a running machine, and that transformation starts from a machine image. An image is the ready-made file system copied onto the instance’s root disk; whatever is inside it is not done at startup, and whatever is not inside it is done at startup.

M22/K05 measured how this image is produced — the bake pipeline, its versioning, the rollout window; that measurement is not repeated here. This lesson’s measure is a single axis: startup duration. The same question is asked for every item — if this item is embedded in the image, how many seconds does startup shorten by, how many more days does the image stay stale in exchange, and how many times a month does it need to be re-baked. The answer varies from item to item, and the size of that variation takes the choice between “embed everything in the image” and “embed nothing” out of the realm of taste.

CC7. Ten items are done at the startup of the regional measurement network — the fictional software that collects readings from water meters; eight can be embedded in the image, two cannot. Durations, sizes, and change intervals are fictional. CC8. The base image is 900 MB, the pull speed is 60 MB/s, and kernel startup is 11 seconds on every path. CC9. Baking is done in a weekly batch; a change waits an average of 3.5 days to enter the image. CC10. A single bake, together with its test, takes 18 minutes. CC11. The embedding order is greedy: the item that gains the most seconds per bake is embedded first. CC12. An item being “critical” means the service does not start while that item is missing; the distinction is fictional.

The Two Sides of an Item

Every piece of work done at startup is an independent item, and embedding each item in the image is a separate decision. Once an item is embedded, its seconds at startup disappear, but the image’s size grows — and pulling a larger image adds seconds to startup. Net gain is the difference between these two. The cost, though, is in a different unit: the more often an item changes, the more often embedding it in the image demands a re-bake.

// measurement-network/image-items.mjs — the startup gain and staleness cost of embedding an item in the image (model)

export const SPEED = 60, BASE_MB = 900, KERNEL = 11;   // MB/s, base image, kernel startup (s)
export const BAKE_WAIT = 3.5, BAKE_MIN = 18;            // average wait in a weekly batch bake

// [name, seconds at startup, MB it adds to the image, days between changes, dependency, calls, critical?]
export const ITEM = [
  ["kernel packages", 46, 320, 45, "package repository", 6, true],
  ["runtime", 38, 210, 30, "package repository", 3, true],
  ["measurement agent", 14, 60, 14, "package repository", 2, false],
  ["log shipping", 8, 20, 21, "package repository", 1, false],
  ["certificate bundle", 9, 12, 7, "configuration server", 1, true],
  ["application binary", 22, 90, 2, "build repository", 1, true],
  ["network settings", 6, 2, 3, "configuration server", 1, true],
  ["environment variables", 3, 1, 1, "configuration server", 1, false],
];
// Items that cannot be embedded: always done at startup.
export const NOT_EMBEDDABLE = [["node secret", 7, "secret store", 1, true],
  ["clock sync", 4, "time server", 1, false]];

export const measure = ([name, sec, mb, days, , calls]) => ({
  name, gain: sec - mb / SPEED, bakes: 30 / days, calls,
  stale: (30 / days) * BAKE_WAIT,
});
// Greedy order: the item that gains the most seconds per bake is embedded first.
export const ORDER = [...ITEM].sort((a, b) =>
  measure(b).gain / measure(b).bakes - measure(a).gain / measure(a).bakes);

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log("item".padEnd(23) + "at startup".padEnd(12) + "in image".padEnd(10) + "pull".padEnd(9) +
    "net gain".padEnd(12) + "bakes/month".padEnd(14) + "stale item-days".padEnd(17) + "ratio");
  for (const k of ORDER) {
    const m = measure(k);
    console.log(m.name.padEnd(23) + `${k[1]} s`.padEnd(12) + `${k[2]} MB`.padEnd(10) +
      `${(k[2] / SPEED).toFixed(1)} s`.padEnd(9) + `${m.gain.toFixed(1)} s`.padEnd(12) +
      m.bakes.toFixed(2).padEnd(14) + m.stale.toFixed(1).padEnd(17) +
      (m.gain / m.bakes).toFixed(1));
  }
  const g = NOT_EMBEDDABLE.reduce((t, x) => t + x[1], 0);
  console.log(`\nitems that cannot be embedded: ${NOT_EMBEDDABLE.map((x) => x[0]).join(", ")} — ` +
    `${g} s and ${NOT_EMBEDDABLE.length} calls stay at every startup`);
}

item                   at startup  in image  pull     net gain    bakes/month   stale item-days  ratio
kernel packages        46 s        320 MB    5.3 s    40.7 s      0.67          2.3              61.0
runtime                38 s        210 MB    3.5 s    34.5 s      1.00          3.5              34.5
measurement agent      14 s        60 MB     1.0 s    13.0 s      2.14          7.5              6.1
log shipping           8 s         20 MB     0.3 s    7.7 s       1.43          5.0              5.4
certificate bundle     9 s         12 MB     0.2 s    8.8 s       4.29          15.0             2.1
application binary     22 s        90 MB     1.5 s    20.5 s      15.00         52.5             1.4
network settings       6 s         2 MB      0.0 s    6.0 s       10.00         35.0             0.6
environment variables  3 s         1 MB      0.0 s    3.0 s       30.00         105.0            0.1

items that cannot be embedded: node secret, clock sync — 11 s and 2 calls stay at every startup

The last column splits items into two groups. kernel packages gains 40.7 seconds and demands 0.67 bakes a month; its ratio is 61. environment variables gains 3 seconds and demands 30 bakes a month; its ratio is 0.1. There is a six-hundredfold difference between these two items, and the difference comes not from size but from change frequency. The item best suited for embedding in the image is not the one that takes the longest at startup; it is the one that takes long and changes rarely.

The application binary row is where this distinction shows most clearly. It takes 22 seconds at startup — the third-longest item on the list — but changes every two days. Embedded, it triggers 15 bakes a month and accumulates 52.5 stale item-days. The runtime sells the same 20.5 seconds for only 3.5 stale item-days. Intuition says “embed the biggest item”; the ratio column says “embed the most stable item”.

The two items that cannot be embedded draw the measure’s floor. The node secret cannot be written into the image, because anyone who can read the image can also read the secret; clock synchronization cannot come from the image, because the thing being measured is the moment of startup. These two items are 11 seconds and 2 calls, and no bake decision can remove them.

Embedding Level Scan

Individual items have been compared; the next question is how many of them to embed. The scan below takes the list sorted by the ratio column from the top and recomputes startup at every level from zero to eight.

// measurement-network/embedding-scan.mjs — what happens to startup as more items are embedded in the image (model)
import { SPEED, BASE_MB, KERNEL, BAKE_MIN, ITEM, NOT_EMBEDDABLE, ORDER, measure } from "./image-items.mjs";

export const level = (k) => {
  const embedded = ORDER.slice(0, k), open = ORDER.slice(k);
  const mb = BASE_MB + embedded.reduce((t, x) => t + x[2], 0);
  const duration = KERNEL + mb / SPEED + open.reduce((t, x) => t + x[1], 0) +
    NOT_EMBEDDABLE.reduce((t, x) => t + x[1], 0);
  const calls = open.reduce((t, x) => t + x[5], 0) + NOT_EMBEDDABLE.reduce((t, x) => t + x[3], 0);
  const types = new Set([...open.map((x) => x[4]), ...NOT_EMBEDDABLE.map((x) => x[2])]).size;
  return { k, mb, duration, calls, types,
    bakes: embedded.reduce((t, x) => t + measure(x).bakes, 0),
    stale: embedded.reduce((t, x) => t + measure(x).stale, 0),
    last: k ? ORDER[k - 1][0] : "—" };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log("embedded".padEnd(10) + "last embedded".padEnd(23) + "image".padEnd(10) +
    "startup".padEnd(10) + "external calls".padEnd(16) + "dependency".padEnd(12) +
    "bakes/month".padEnd(14) + "baker minutes".padEnd(15) + "stale item-days");
  for (let k = 0; k <= ITEM.length; k++) {
    const d = level(k);
    console.log(`${k}/${ITEM.length}`.padEnd(10) + d.last.padEnd(23) + `${d.mb} MB`.padEnd(10) +
      `${d.duration.toFixed(1)} s`.padEnd(10) + String(d.calls).padEnd(16) + `${d.types} types`.padEnd(12) +
      d.bakes.toFixed(2).padEnd(14) + Math.round(d.bakes * BAKE_MIN).toString().padEnd(15) +
      d.stale.toFixed(1));
  }
  const s = level(0), t = level(4), h = level(ITEM.length);
  console.log(`\nfirst four items: startup shortens by ${(s.duration - t.duration).toFixed(1)} s, ` +
    `costing ${t.stale.toFixed(1)} stale item-days`);
  console.log(`last four items: startup shortens by ${(t.duration - h.duration).toFixed(1)} s, ` +
    `costing ${(h.stale - t.stale).toFixed(1)} stale item-days`);
  console.log(`staleness per second: first four ${(t.stale / (s.duration - t.duration)).toFixed(2)}, ` +
    `last four ${((h.stale - t.stale) / (t.duration - h.duration)).toFixed(2)} item-days/s`);
}

embedded  last embedded          image     startup   external calls  dependency  bakes/month   baker minutes  stale item-days
0/8       —                      900 MB    183.0 s   18              5 types     0.00          0              0.0
1/8       kernel packages        1220 MB   142.3 s   12              5 types     0.67          12             2.3
2/8       runtime                1430 MB   107.8 s   9               5 types     1.67          30             5.8
3/8       measurement agent      1490 MB   94.8 s    7               5 types     3.81          69             13.3
4/8       log shipping           1510 MB   87.2 s    6               4 types     5.24          94             18.3
5/8       certificate bundle     1522 MB   78.4 s    5               4 types     9.52          171            33.3
6/8       application binary     1612 MB   57.9 s    4               3 types     24.52         441            85.8
7/8       network settings       1614 MB   51.9 s    3               3 types     34.52         621            120.8
8/8       environment variables  1615 MB   48.9 s    2               2 types     64.52         1161           225.8

first four items: startup shortens by 95.8 s, costing 18.3 stale item-days
last four items: startup shortens by 38.3 s, costing 207.5 stale item-days
staleness per second: first four 0.19, last four 5.42 item-days/s

The scan is not one curve but two different curves. Startup duration drops from 183 seconds to 48.9 seconds with diminishing returns: the first four items gain 95.8 seconds, the last four gain 38.3 seconds. Staleness runs the opposite way and accelerates: the first four items cost 18.3 stale item-days, the last four cost 207.5 stale item-days. The last line turns the two curves into a single number — the price of a gained second is 0.19 item-days in the first four, 5.42 item-days in the last four. Twenty-eight times.

The baker minutes column writes the same thing from the labor side. With four items embedded, the image builder runs for 94 minutes a month; with eight items embedded, 1,161 minutes — roughly nineteen hours. This is a cost nobody writes into the plan but that gets paid every month, and it is why the bake pipeline itself is an infrastructure piece that needs monitoring.

The dependency column, though, does not fall smoothly — it falls step by step: from 5 types to 4 at the fourth item, to 3 at the sixth, to 2 at the eighth. The reason is that all four of the first four items use the same dependency — the package repository. A dependency type is removed only once the last item bound to it is also embedded; embedding the first three items drops calls from 18 to 7 but does not change the type count at all.

The Surface of What Remains at Startup

Call count is a surface measure: every call is a separate thing that must be standing at the moment of startup. But not every call carries the same weight. When some items are missing, the service does not start at all; when others are missing, the machine boots and runs incomplete.

// measurement-network/startup-surface.mjs — the external surface of the work left at startup and the decision ledger (model)
import { ITEM, NOT_EMBEDDABLE, ORDER } from "./image-items.mjs";
import { level } from "./embedding-scan.mjs";

const LEVELS = [0, 4, 8];
const open = (k) => [...ORDER.slice(k).map(([name, sec, , , dep, calls, crit]) => ({ name, sec, dep, calls, crit })),
  ...NOT_EMBEDDABLE.map(([name, sec, dep, calls, crit]) => ({ name, sec, dep, calls, crit }))];

const impact = (k, dep) => {
  const l = open(k).filter((x) => x.dep === dep);
  if (!l.length) return "no effect";
  if (l.some((x) => x.crit)) return "startup fails";
  return `-${l.reduce((t, x) => t + x.sec, 0)} s, ${l.length} item${l.length === 1 ? "" : "s"} missing`;
};

// [item, holder, reversible?]
const DECISIONS = [["which item goes into the image", "yours", "yes — new image"],
  ["base operating system version", "yours", "yes — new image"],
  ["bake frequency", "yours", "yes"],
  ["published image content", "yours", "no — does not change under the same identity"],
  ["secret embedded in the image", "yours", "no — deleting is not enough once copied"],
  ["startup script given to the instance", "yours", "no — does not change after creation"],
  ["image repository durability", "provider", "—"],
  ["image copy speed", "provider", "—"],
  ["virtualization beneath the base image", "provider", "—"]];

if (import.meta.url === `file://${process.argv[1]}`) {
  const DEPS = [...new Set([...ITEM.map((x) => x[4]), ...NOT_EMBEDDABLE.map((x) => x[2])])];
  console.log("dropped dependency".padEnd(24) +
    LEVELS.map((k) => `${k}/8 embedded`.padEnd(22)).join("").trimEnd());
  for (const dep of DEPS)
    console.log((dep.padEnd(24) + LEVELS.map((k) => impact(k, dep).padEnd(22)).join("")).trimEnd());
  console.log("");
  console.log("level".padEnd(9) + "startup".padEnd(10) + "open items".padEnd(13) +
    "critical open items".padEnd(22) + "dependency blocking startup");
  for (const k of LEVELS) {
    const a = open(k);
    console.log(`${k}/8`.padEnd(9) + `${level(k).duration.toFixed(1)} s`.padEnd(10) +
      String(a.length).padEnd(13) + String(a.filter((x) => x.crit).length).padEnd(22) +
      String(new Set(a.filter((x) => x.crit).map((x) => x.dep)).size));
  }
  const count = (v) => DECISIONS.filter((x) => x[1] === v).length;
  console.log(`\ndecision ledger: ${count("yours")} items kept, ${count("provider")} items with the provider`);
  for (const [name, , rev] of DECISIONS.filter((x) => x[2].startsWith("no")))
    console.log(`  cannot be undone — ${name.padEnd(38)} ${rev.slice(5)}`);
}

dropped dependency      0/8 embedded          4/8 embedded          8/8 embedded
package repository      startup fails         no effect             no effect
configuration server    startup fails         startup fails         no effect
build repository        startup fails         startup fails         no effect
secret store            startup fails         startup fails         startup fails
time server             -4 s, 1 item missing  -4 s, 1 item missing  -4 s, 1 item missing

level    startup   open items   critical open items   dependency blocking startup
0/8      183.0 s   10           6                     4
4/8      87.2 s    6            4                     3
8/8      48.9 s    2            1                     1

decision ledger: 6 items kept, 3 items with the provider
  cannot be undone — published image content                does not change under the same identity
  cannot be undone — secret embedded in the image           deleting is not enough once copied
  cannot be undone — startup script given to the instance   does not change after creation

The number of dependencies that block startup drops from 4 to 1, and that is a gain independent of seconds. With no item embedded, four separate systems must be standing at the same time; with eight embedded, a single system — the secret store — remains. The secret store row is identical across all three columns and cannot be changed by an embedding decision. The clock server row is the one “soft” row: if it drops, startup still completes, and the machine begins taking work 4 seconds early and unsynchronized.

This row is where a silent trap sits. The measurement network’s readings are timestamped; the records an unsynchronized machine produces are harder to notice than a failed startup, because the machine looks healthy. A non-critical item starting up missing is not a bug, it is a decision.

The decision ledger leaves six of nine items with you: what goes into the image, what the base version is, how often it gets baked, and three items that cannot be undone. The three that stay with the provider — image repository durability, copy speed, the virtualization beneath it — determine the pull seconds but cannot be chosen.

The three items that cannot be undone are irreversible each in a different way. Published image content does not change under the same identity; fixing it means a new image, and that image does not delete the old one. A secret embedded in the image cannot be undone by deletion once the image has been copied; that is why the secret store call is a deliberate choice, not a gap. The startup script given to an instance, however, is set when the instance is created and does not change for as long as that instance lives — fixing the script does not fix the running machine, it only fixes machines created from this point on.

Summary

• An item’s suitability for embedding is determined by change frequency, not size: kernel packages gains 61 seconds per bake, environment variables gains 0.1 — a six-hundredfold difference.
• application binary takes 22 seconds at startup but changes every two days; embedded, it costs 15 bakes and 52.5 stale item-days a month, while runtime sells the same seconds for 3.5 item-days.
• In the embedding level scan, startup drops from 183 seconds to 48.9 seconds; the price of a gained second is 0.19 stale item-days in the first four items, 5.42 in the last four.
• External calls drop from 18 to 2, but dependency type count falls step by step: a type is removed only once the last item bound to it is also embedded; baker load climbs from 94 minutes to 1,161 minutes.
• Dependencies that block startup drop from 4 to 1, not to zero; if the clock server drops, the machine boots and takes work unsynchronized — a condition harder to notice than a failed startup.
• Six of nine items are kept, three are with the provider; the three that cannot be undone are published image content, the secret embedded in the image, and the startup script given to the instance.

Next Step

This lesson counted how long startup takes and what it depends on, but it never asked how long the machine will live. The entire calculation was built on the assumption that the instance stays up for as long as it is wanted: both 183 seconds and 48.9 seconds are paid once, and then the machine keeps doing work.

There is also an interruptible form of renting — capacity is given cheaply, and in exchange the provider can take it back with a short notice. The previous course measured this form’s price difference; here price becomes an input, and the question shifts to the type of job. The next lesson runs four job types on interruptible capacity and asks: as the notice period shortens, how much work does each job lose, what does restarting cost, and which job does not tolerate interruption at all?