Virtual Machine Management

The previous two lessons measured a running guest: how many layers stand beneath it, how many items it shares with the host. But the real feature that separates a virtual machine from a physical one shows up not while it runs, but while it is stopped and copied. A guest is nothing more than a file: it can be snapshotted and stored, its image can be cloned twenty-four times and distributed to twenty-four machines, and it can be returned to a week-old state in a few seconds. These three operations lighten management, and every one of them carries a boundary item.

What is measured is not how easy the operations are, but the items they carry. For cloning the question is this: how many items do machines produced from one image carry in common, and if one of those items falls, how many machines are reached. For the snapshot the question is chain length: how many links need to be sound to return to a given point. For rollback the question is loss: how many vulnerabilities, closed between the point returned to and today, reopen.

VR13 — In the fictional branch fleet, 24 guests are cloned from a single golden image; the item list is a model, the count is real. VR14 — A secret item is a hidden value that is identical across two machines when copied. VR15 — De-duplication is the step that regenerates one item on a clone’s first boot; its cost is written as steps and the number of service restarts. VR16 — The snapshot chain is incremental: each link is written on top of the one before it, and opening a point requires the base plus every link up to that point. VR17 — The validator guest’s schedule spans 42 days; it contains six snapshots and six patch events, and the patch events close 17 vulnerabilities in total. VR18 — The guest produces 240 audit records a day. VR19 — Four components run on the guest, and each is consistent only at a given readiness level; the cost of that readiness level is the interruption seconds given to the guest.

The Secret a Clone Carries

A golden image is installed once, hardened, and distributed to branches. Part of the files inside it are secret values specific to that installation, and cloning copies those too.

// clone.mjs — secret items carried by clones produced from a golden image (MODEL).
// Fictional branch fleet: 24 guests are cloned from a single golden image.
const CLONES = 24;
// [item, renewed on first boot?, extra steps the renewal needs, service restarts]
const SECRET = [
  ['host key', true, 1, 1], ['machine identity', true, 1, 0],
  ['management agent token', false, 1, 1], ['log shipping key', false, 1, 1],
  ['admin session token cache', false, 1, 0],
  ['local admin password hash', false, 1, 0], ['certificate private key', false, 2, 1],
];
const s = (x, n) => String(x).padStart(n);
console.log(`golden image carries ${SECRET.length} secret items; ${CLONES} clones are produced`);
console.log(`${'item'.padEnd(36)}${s('renewed', 12)}${s('machines staying identical', 28)}`);
for (const [ad, y] of SECRET)
  console.log(`${ad.padEnd(36)}${s(y ? 'yes' : 'no', 12)}${s(y ? 1 : CLONES, 28)}`);

const shared = SECRET.filter(([, y]) => !y);
console.log(`\nrenewed ${SECRET.length - shared.length}, shared ${shared.length}; `
  + `shared item-machine pairs ${shared.length * CLONES}`);
console.log(`required items for one shared secret item: 1, machines reached ${CLONES}`);
const [steps, restarts] = shared.reduce(([a, r], [, , x, y]) => [a + x, r + y], [0, 0]);
console.log(`cost of renewing all five shared items: ${steps} first-boot steps, `
  + `${restarts} service restarts, per clone`);
console.log(`fleet-wide ${steps * CLONES} steps; if a single item is left unrenewed, machines reached `
  + `is still ${CLONES}`);

golden image carries 7 secret items; 24 clones are produced
item                                     renewed  machines staying identical
host key                                     yes                           1
machine identity                             yes                           1
management agent token                        no                          24
log shipping key                              no                          24
admin session token cache                     no                          24
local admin password hash                     no                          24
certificate private key                       no                          24

renewed 2, shared 5; shared item-machine pairs 120
required items for one shared secret item: 1, machines reached 24
cost of renewing all five shared items: 6 first-boot steps, 3 service restarts, per clone
fleet-wide 144 steps; if a single item is left unrenewed, machines reached is still 24

Two of the seven items are renewed. These two are renewed because, unrenewed, the system breaks visibly: if two machines’ machine identity is the same, network management cannot tell them apart; if the host key is the same, connecting clients raise a warning. The remaining five are not renewed because, unrenewed, nothing breaks. Twenty-four machines carry the same management agent token, the same log shipping key, and the same certificate private key, and the fleet runs without issue.

The shared item-machine pair count is one hundred twenty. This number is not damage, it is surface. The real number lies beneath it: for a secret item, the item required to cross the boundary is one, and the machines reached when that one item falls is twenty-four. A single private key read from one machine’s disk is the same key at twenty-four branches. Cloning does not open a path; it raises the number of machines at the end of an existing path to twenty-four.

The cost column is small this time. De-duplicating all five items requires six first-boot steps and three service restarts per clone, one hundred forty-four steps fleet-wide — a one-time, scriptable job. The last line is the harshest spot in the measurement: if a single item is left unrenewed, the machines reached are still twenty-four. De-duplication cannot be done partially; if four are renewed and one is forgotten, the number gained is zero.

The Snapshot Chain

A snapshot is not a copy but a difference: each link holds the change written on top of the one before it. This makes taking a snapshot cheap and ties reading it to the length of the chain.

// chain.mjs — snapshot chain and the vulnerabilities a rollback brings back (MODEL).
// Fictional validator guest's 42-day schedule: six snapshots, six patch events.
const TODAY = 42, DAILY = 240;                  // 240 audit records a day
const POINTS = [['base', 0], ['snap-1', 3], ['snap-2', 10], ['snap-3', 17],
  ['snap-4', 24], ['snap-5', 31], ['snap-6', 38]];
const PATCH = [[5, 3], [12, 1], [19, 4], [26, 2], [33, 5], [40, 2]];  // [day, vulnerabilities closed]
const BASE_GIB = 40, LINK_GIB = 1.9;

const reopened = (day) => PATCH.filter(([g]) => g > day).reduce((t, [, n]) => t + n, 0);
const s = (x, n) => String(x).padStart(n);
console.log(`today is day ${TODAY}; ${POINTS.length - 1} snapshots, ${PATCH.length} patch events, `
  + `vulnerabilities closed ${reopened(0)}`);
console.log(`\n${'point restored'.padEnd(16)}${s('day', 5)}${s('links required', 16)}`
  + `${s('vulnerabilities reopened', 26)}${s('audit records lost', 23)}`);
POINTS.forEach(([ad, day], i) => {
  console.log(`${ad.padEnd(16)}${s(day, 5)}${s(i + 1, 16)}${s(reopened(day), 26)}`
    + `${s((TODAY - day) * DAILY, 23)}`);
});

const links = POINTS.length - 1;
console.log(`\nchain: base ${BASE_GIB} GiB + ${links} links x ${LINK_GIB} GiB = `
  + `${(BASE_GIB + links * LINK_GIB).toFixed(1)} GiB; opening the newest point reads ${POINTS.length} links`);
const CORRUPT = 3;                              // snap-3's delta is corrupt
const [ad, day] = POINTS[CORRUPT - 1];
console.log(`link ${CORRUPT} corrupt -> newest point restorable ${ad} (day ${day}), `
  + `vulnerabilities reopened ${reopened(day)}, records lost ${(TODAY - day) * DAILY}`);
console.log(`the ${POINTS.length - CORRUPT} points after the corrupt link cannot be opened; `
  + `the chain requires ${POINTS.length} sound links`);

today is day 42; 6 snapshots, 6 patch events, vulnerabilities closed 17

point restored    day  links required  vulnerabilities reopened     audit records lost
base                0               1                        17                  10080
snap-1              3               2                        17                   9360
snap-2             10               3                        14                   7680
snap-3             17               4                        13                   6000
snap-4             24               5                         9                   4320
snap-5             31               6                         7                   2640
snap-6             38               7                         2                    960

chain: base 40 GiB + 6 links x 1.9 GiB = 51.4 GiB; opening the newest point reads 7 links
link 3 corrupt -> newest point restorable snap-2 (day 10), vulnerabilities reopened 14, records lost 7680
the 4 points after the corrupt link cannot be opened; the chain requires 7 sound links

The third column writes the chain’s price: returning to the newest point requires all seven of the seven links to be sound. This number is good for isolation, bad for resilience — wanting the item required to cross a boundary to be high is desirable, wanting the links required to open a recovery point to be high is not. As the snapshot chain grows longer, rollback does not get easier — it becomes more tied to a single point.

The fourth column is rollback’s real cost. In forty-two days, six patch events closed seventeen vulnerabilities. Returning to the base reopens all seventeen; returning to the day-38 point reopens only two. The difference is not a choice but a scheduling problem: the patch was applied on day 40, the closest snapshot was taken on day 38. The cost of a rollback is not the age of the point restored, but how many patch events remain between that point and today.

The fifth column counts a quieter item. Rollback does not only bring back the vulnerability, it also deletes the audit records produced between the point restored and today. Returning to the base takes away 10,080 records, returning to day 38 takes away 960. Returning to a clean state after an incident means also deleting the records that carry that incident’s trace — which is why copying the records out of the guest before a rollback is a separate step.

The last two rows show the chain’s fragility. When the third link is corrupted, all four of the four points after it become impossible to open. The newest point restorable drops thirty-two days back, to day 10; the vulnerabilities reopened climb from two to fourteen, the records lost from 960 to 7,680. There were seven recovery points, and a single corrupt link took away four of them at once. The number of points in a chain is not the number of independent recovery points.

The Moment a Snapshot Is Consistent

The chain table counted every point as a single “state.” That hides a question that was never asked: at the moment the snapshot was taken, the guest was running, and the host captured the disk’s state at that instant. What is on disk and what the components hold in memory are not the same instant.

// consistency.mjs — the moment a snapshot is consistent vs. the moment the components are consistent (MODEL).
// A host-level snapshot captures the disk's state at that instant; a pending in-memory write is not on disk.
// [component, readiness level required to be consistent, manual recovery steps on restore]
const COMPONENTS = [['file system metadata', 1, 1], ['file queue', 1, 1],
  ['database pages', 2, 3], ['in-memory application state', 3, 2]];
// [readiness level, name, interruption given to the guest (seconds)]
const LEVEL = [[0, 'no preparation', 0], [1, 'flushing writes inside the guest', 0.4],
  [2, 'pausing the application', 3.2], [3, 'stopping the guest', 95]];

const s = (x, n) => String(x).padStart(n);
console.log(`${'readiness'.padEnd(33)}${s('consistent', 12)}${s('half-done', 11)}${s('interruption s', 16)}`
  + `${s('manual recovery steps', 24)}`);
for (const [d, ad, sn] of LEVEL) {
  const half = COMPONENTS.filter(([, g]) => g > d);
  console.log(`${ad.padEnd(33)}${s(`${COMPONENTS.length - half.length}/${COMPONENTS.length}`, 12)}`
    + `${s(half.length, 11)}${s(sn.toFixed(1), 16)}${s(half.reduce((t, [, , a]) => t + a, 0), 24)}`);
}
const full = LEVEL.find(([d]) => COMPONENTS.every(([, g]) => g <= d));
console.log(`\nall four of the four components are consistent only with "${full[1]}"; interruption ${full[2]} seconds`);
console.log(`an unprepared snapshot leaves ${COMPONENTS.filter(([, g]) => g > 0).length} items half-done; `
  + `the snapshot operation still returns successful`);

readiness                          consistent  half-done  interruption s   manual recovery steps
no preparation                            0/4          4             0.0                       7
flushing writes inside the guest          2/4          2             0.4                       5
pausing the application                   3/4          1             3.2                       2
stopping the guest                        4/4          0            95.0                       0

all four of the four components are consistent only with "stopping the guest"; interruption 95 seconds
an unprepared snapshot leaves 4 items half-done; the snapshot operation still returns successful

The first row is the lesson’s most misleading row: interruption is zero, the operation returns successful, and all four of the four components are half-done. The host does not see what happens inside the guest; it copies the disk’s state at that instant and counts the job as complete. Flushing writes inside the guest makes two components consistent, at a cost of 0.4 seconds. Database pages become consistent only by pausing the application, in-memory application state only by stopping the guest — in exchange for a ninety-five-second interruption.

The right column writes the difference on the restore side: restoring from an unprepared snapshot requires seven manual recovery steps, restoring from one taken with the application paused requires two. The moment a snapshot is consistent, for the host, is the moment the disk was frozen; for the data layer it is only the moment pending writes reach disk, and the two are not the same moment unless preparation is done.

The Boundary’s Three Columns

This lesson’s boundary is the one drawn by management operations, and it fills the three columns as follows. The boundary’s name is the clone and snapshot boundary. The number of items required to cross the boundary is one: any one of the five secret items shared across twenty-four machines. The number of items reached when the boundary falls is twenty-four machines — cloning is an operation that multiplies the number of machines behind a single item.

The snapshot adds a second row to this table, and in that row the number works in the reverse direction. The sound links required to open a point is seven; this is not an isolation gain but a resilience debt. Rollback, meanwhile, does not cross the boundary — it reopens boundaries that were already closed: it can go back as far as seventeen vulnerabilities, and this is done without using any escape path.

What the three measurements share is that all three are operations that make management easier. Cloning sets up twenty-four machines from one image, a snapshot fits a week’s work into a single file, rollback brings a broken system back to running in seconds. The boundary item of all three comes from the same place: a machine’s state becoming a copyable object. When that object is copied, the secret inside it is copied too; when it is stored, the patch level at that instant is stored too; when it is restored, that level is returned to.

Summary

• Two of the golden image’s seven secret items are renewed on first boot, five stay identical across twenty-four clones; the shared item-machine pair count is 120.
• For a shared secret item, the item required to cross the boundary is one and the machines reached is twenty-four; if a single item is left unrenewed, this number is still twenty-four.
• The cost of de-duplicating all five items is six first-boot steps and three service restarts per clone, 144 steps fleet-wide.
• In the seven-point incremental chain, opening the newest point requires seven links to be sound; when the third link is corrupted, four points become impossible to open at once.
• Rollback undoes the patches between the point restored and today: returning to the base reopens 17 vulnerabilities and deletes 10,080 audit records, returning to day 38 reopens 2 vulnerabilities and deletes 960 records.
• In an unprepared snapshot, all four of the four components are half-done and restoring requires seven manual steps; making all four consistent requires stopping the guest and a 95-second interruption.

Next Step

The three lessons so far asked the same question about a single technology: where is the guest system’s boundary, how many items cross it, and what is reached if it is crossed. There is a second way to do the same job, and in that way there is no guest system at all: processes separated from one another run on top of the same kernel. If two ways do the same job, they can be compared with the same measure. The next lesson does this: for the same eight targets, how many items are required to cross the boundary in a guest system, how many in a compartment sharing the same kernel; what a shared kernel means in shared items, and where the difference between the two numbers comes from.