Backup and Resilience

Across four lessons, the boundary was always measured from the same direction: what happens if someone crosses it. That is not the only way a boundary falls. A storage device fails, a guest is deleted by mistake, a host becomes unbootable — no one crosses the boundary, yet the guest is still lost. Virtualization has two answers to this question, and both are called “bringing it back”: the snapshot and the backup.

They do not bring back the same thing. The snapshot sits on the host, on the same storage device the guest’s image sits on, and its retention is short; the backup sits outside the host, its retention is long, and its scope is limited by a list. This lesson runs both through the same seven scenarios and counts the recovered and unrecovered items separately.

VR26 — The fictional validator guest carries eight items. VR27 — The snapshot sits on the host, on the guest image’s storage device; retention is 7 days, the capture interval is 1 hour. VR28 — The backup sits outside the host; retention is 90 days, the capture interval is 24 hours. VR29 — Two items sit outside the backup’s scope: local secrets and in-memory session state; this scope gap was measured in the Servers, Networking and Hardening course, and here it is an input. VR30 — A method, if it works in a given scenario, brings back every item in its scope. VR31 — The backup chain is a 28-day window; the sizes are fictional.

Two Methods, the Same Loss Scenario

// loss.mjs — snapshot vs. backup in the same loss scenarios: recovered and unrecovered items.
// Fictional validator guest carries 8 items. Scope gaps are an input from the previous course.
const ITEMS = ['operating system root files', 'application binaries', 'configuration',
  'database pages', 'work order queue', 'audit log', 'local secrets', 'in-memory session state'];
const OUT_OF_SCOPE = ['local secrets', 'in-memory session state'];   // outside the backup's scope
const SNAPSHOT = { scope: ITEMS.length, retention: 7, window: 1 };   // on the host, same storage device
const BACKUP = { scope: ITEMS.length - OUT_OF_SCOPE.length, retention: 90, window: 24 };
// [scenario, age of the loss (days), storage device lost, host lost, guest object deleted]
const SCENARIOS = [
  ['guest file system corruption', 0, false, false, false],
  ['misconfiguration (2 hours ago)', 0, false, false, false],
  ['files inside the guest being encrypted', 0, false, false, false],
  ['a record deleted 30 days ago', 30, false, false, false],
  ['loss of the storage device', 0, true, false, false],
  ['loss of the host machine', 0, false, true, false],
  ['guest deleted from the management interface', 0, false, false, true],
];
const snapshotOk = ([, age, b, k, s]) => !b && !k && !s && age <= SNAPSHOT.retention;
const backupOk = ([, age]) => age <= BACKUP.retention;

const s = (x, n) => String(x).padStart(n);
console.log(`${ITEMS.length} items; snapshot scope ${SNAPSHOT.scope}, retention ${SNAPSHOT.retention} days; `
  + `backup scope ${BACKUP.scope}, retention ${BACKUP.retention} days`);
console.log(`\n${'scenario'.padEnd(46)}${s('snapshot', 15)}${s('backup', 9)}${s('best', 10)}`);
let a = 0, y = 0, best = 0;
for (const sen of SCENARIOS) {
  const ka = snapshotOk(sen) ? SNAPSHOT.scope : 0, ky = backupOk(sen) ? BACKUP.scope : 0;
  a += ka; y += ky; best += Math.max(ka, ky);
  console.log(`${sen[0].padEnd(46)}${s(`${ka}/${ITEMS.length}`, 15)}${s(`${ky}/${ITEMS.length}`, 9)}`
    + `${s(`${Math.max(ka, ky)}/${ITEMS.length}`, 10)}`);
}
const T = SCENARIOS.length * ITEMS.length;
console.log(`${'total'.padEnd(46)}${s(`${a}/${T}`, 15)}${s(`${y}/${T}`, 9)}${s(`${best}/${T}`, 10)}`);
console.log(`\nsnapshot works in ${SCENARIOS.filter(snapshotOk).length}/${SCENARIOS.length} scenarios, `
  + `and when it works it brings back ${SNAPSHOT.scope}/${ITEMS.length} items`);
console.log(`backup works in ${SCENARIOS.filter(backupOk).length}/${SCENARIOS.length} scenarios, `
  + `every time with ${OUT_OF_SCOPE.length} items out of scope: ${OUT_OF_SCOPE.join(', ')}`);
console.log(`the two together: ${best}/${T}; unrecoverable ${T - best} items`);
console.log(`data-loss window: snapshot ${SNAPSHOT.window} hour, backup ${BACKUP.window} hours`);

8 items; snapshot scope 8, retention 7 days; backup scope 6, retention 90 days

scenario                                             snapshot   backup      best
guest file system corruption                              8/8      6/8       8/8
misconfiguration (2 hours ago)                            8/8      6/8       8/8
files inside the guest being encrypted                    8/8      6/8       8/8
a record deleted 30 days ago                              0/8      6/8       6/8
loss of the storage device                                0/8      6/8       6/8
loss of the host machine                                  0/8      6/8       6/8
guest deleted from the management interface               0/8      6/8       6/8
total                                                   24/56    42/56     48/56

snapshot works in 3/7 scenarios, and when it works it brings back 8/8 items
backup works in 7/7 scenarios, every time with 2 items out of scope: local secrets, in-memory session state
the two together: 48/56; unrecoverable 8 items
data-loss window: snapshot 1 hour, backup 24 hours

The two columns are incomplete in two different ways. The snapshot is complete where it works, and zero where it does not: it brings back eight of eight items in three scenarios, none in four. Three of the zeros come from the same cause — the snapshot sits right next to the thing that is lost: when the storage device, the host, or the guest object goes, the snapshots go with it. The fourth has a different cause; the thirty-day-old record is outside the seven-day retention.

The backup is incomplete everywhere: it works in all seven and brings back six items every time. The two items it misses are written by name — local secrets and in-memory session state — and they are out of scope regardless of scenario. The snapshot’s shortfall depends on the condition; the backup’s depends on scope.

The bottom rows give the total: snapshot 24/56, backup 42/56, 48/56 when the better of the two is picked in every scenario. The two methods do not replace each other, they fill each other’s gap — but the remaining eight items come in no arrangement, and all of them are the out-of-scope items in the four scenarios where the snapshot does not work. On top of this comes the data-loss window: one hour for the snapshot, twenty-four hours for the backup.

The Point You Can Return to When the Chain Breaks

The backup is not a single file either. Taking a full copy every day is the most expensive arrangement; incremental and differential arrangements lower the cost, and in exchange tie the points to one another.

// backupchain.mjs — the point restorable when full, incremental, and differential chains break (MODEL).
// 28-day window; full backup on day 0, one point every day after. Sizes are fictional.
const DAYS = 28, FULL_GIB = 40, DELTA_GIB = 1.9, CORRUPT = 10;

// files required to open point d (day numbers)
const REQUIRED = {
  full: (d) => [d],
  incremental: (d) => Array.from({ length: d + 1 }, (_, i) => i),
  differential: (d) => (d === 0 ? [0] : [0, d]),
};
const SIZE = {
  full: () => DAYS * FULL_GIB,
  incremental: () => FULL_GIB + (DAYS - 1) * DELTA_GIB,
  differential: () => FULL_GIB + DELTA_GIB * ((DAYS - 1) * DAYS) / 2,
};
const lost = (type, corrupt) => Array.from({ length: DAYS }, (_, d) => d)
  .filter((d) => REQUIRED[type](d).includes(corrupt));

const s = (x, n) => String(x).padStart(n);
console.log(`${DAYS} days, full backup on day 0; full ${FULL_GIB} GiB, daily delta ${DELTA_GIB} GiB`);
console.log(`\n${'type'.padEnd(13)}${s('files', 6)}${s('files per point', 20)}`
  + `${s(`day ${CORRUPT} corrupt: lost`, 22)}${s('newest restorable', 21)}${s('storage GiB', 14)}`);
for (const type of Object.keys(REQUIRED)) {
  const n = Array.from({ length: DAYS }, (_, d) => REQUIRED[type](d).length);
  const k = lost(type, CORRUPT);
  const sound = Array.from({ length: DAYS }, (_, d) => d).filter((d) => !k.includes(d));
  console.log(`${type.padEnd(13)}${s(DAYS, 6)}${s(n[0] === n[DAYS - 1] ? n[0] : `${Math.min(...n)}-${Math.max(...n)}`, 20)}`
    + `${s(`${k.length} points`, 22)}${s(`day ${Math.max(...sound)}`, 21)}${s(SIZE[type]().toFixed(1), 14)}`);
}
console.log(`\nif the full backup (day 0) is corrupted, points lost: `
  + Object.keys(REQUIRED).map((t) => `${t} ${lost(t, 0).length}`).join(', '));
console.log(`differential requires ${(SIZE.differential() / SIZE.incremental()).toFixed(1)}x the storage of incremental`
  + ` and, when day ${CORRUPT} corrupts, recovers ${lost('incremental', CORRUPT).length - lost('differential', CORRUPT).length}`
  + ' more points');

28 days, full backup on day 0; full 40 GiB, daily delta 1.9 GiB

type          files     files per point  day 10 corrupt: lost    newest restorable   storage GiB
full             28                   1              1 points               day 27        1120.0
incremental      28                1-28             18 points                day 9          91.3
differential     28                 1-2              1 points               day 27         758.2

if the full backup (day 0) is corrupted, points lost: full 1, incremental 28, differential 28
differential requires 8.3x the storage of incremental and, when day 10 corrupts, recovers 17 more points

All three types have the same number of points: twenty-eight. The difference is how tied the points are to one another. In the full arrangement, every point is one file, and one file’s corruption takes only that point. In the incremental arrangement, opening day 27 requires all twenty-eight of the twenty-eight files; when the single file on day 10 corrupts, eighteen points go at once and the newest point restorable drops to day 9. In the differential arrangement, every point is two files, and day 10 corrupting takes only day 10.

The storage column writes the trade-off: 1,120, 91.3, and 758.2 GiB. The incremental arrangement is the cheapest and the most fragile; the differential arrangement recovers seventeen more points in the same corruption, at 8.3 times the storage. The backup type looks like a storage decision; measured, it turns out to be a dependency decision. All three share a weakness in the last row: if the full backup itself is corrupted, all twenty-eight of the twenty-eight points go at once in both the incremental and differential arrangements.

Where the Copy Sits

Chain dependency has been measured; where the copy sits is a separate number. The same scenarios now run over a set of copies.

// copies.mjs — copy distribution: how many copies survive in each scenario (MODEL).
// [name, location, online?, deletable with the same management identity?, recovery minutes]
const COPIES = [
  ['host snapshot', 'host', true, true, 8],
  ['backup on the branch server', 'branch', true, true, 45],
  ['backup in the central object store', 'central', true, true, 210],
  ['offline copy at central', 'central', false, false, 720],
];
const SCENARIOS = [
  ['loss of the host machine', (k) => k[1] === 'host'],
  ['loss of the branch facility', (k) => k[1] === 'host' || k[1] === 'branch'],
  ['loss of the central object store', (k) => k[1] === 'central' && k[2]],
  ['compromise of the management identity', (k) => k[3]],
  ['files inside the guest being encrypted', () => false],
];
const remaining = (list, lost) => list.filter((k) => !lost(k));
const s = (x, n) => String(x).padStart(n);
console.log(`${'scenario'.padEnd(40)}${s('remaining', 11)}   fastest recovery`);
let worst = COPIES.length;
for (const [ad, lost] of SCENARIOS) {
  const k = remaining(COPIES, lost);
  const fastest = k.length ? k.reduce((a, b) => (b[4] < a[4] ? b : a)) : null;
  worst = Math.min(worst, k.length);
  console.log(`${ad.padEnd(40)}${s(`${k.length}/${COPIES.length}`, 11)}   `
    + (fastest ? `${fastest[0]} (${fastest[4]} min)` : 'none'));
}
const locations = new Set(COPIES.map((k) => k[1])).size;
console.log(`\n${COPIES.length} copies, ${locations} locations; remaining in the worst scenario ${worst}`);
const online = COPIES.filter((k) => k[2]);
const after = Math.min(...SCENARIOS.map(([, y]) => remaining(online, y).length));
console.log(`once the offline copy is removed, copies ${COPIES.length} -> ${online.length}, `
  + `remaining in the worst scenario ${worst} -> ${after}`);

scenario                                  remaining   fastest recovery
loss of the host machine                        3/4   backup on the branch server (45 min)
loss of the branch facility                     2/4   backup in the central object store (210 min)
loss of the central object store                3/4   host snapshot (8 min)
compromise of the management identity           1/4   offline copy at central (720 min)
files inside the guest being encrypted          4/4   host snapshot (8 min)

4 copies, 3 locations; remaining in the worst scenario 1
once the offline copy is removed, copies 4 -> 3, remaining in the worst scenario 1 -> 0

In three physical scenarios, at least two copies stay standing. The fourth row is different from the others: compromising the management identity destroys no hardware, yet it still takes three of the four copies at once — those three are the online copies the same identity can delete, and location diversity means nothing here.

The last row is the summary of the measure: when the offline copy is removed, the copy count drops from four to three, but the copies remaining in the worst scenario drop from one to zero. Resilience is not the number of copies, but the number of items that can destroy those copies at the same time.

Recovery Testing

All of these numbers rest on one assumption: the backup file is readable. Testing this assumption is called recovery testing; it is the same restore drill from the Servers, Networking and Hardening course, and it was measured there. With this course’s measure, there is only one thing to say: when recovery testing is not done, the item required to cross the recovery boundary is one — one corrupt file, one out-of-scope item, or one unreadable key — and when that item is in place, the accessible recovery points are zero.

Summary

• In seven loss scenarios, the snapshot works in 3 and brings back 8/8 items when it does; the backup works in 7 and brings back 6/8 every time.
• The snapshot’s shortfall depends on the condition — it sits right next to the thing that is lost; the backup’s depends on scope, and two items stay outside it regardless of scenario.
• The two methods together recover 48 of 56 items; the remaining 8 items come in no arrangement. The data-loss window is 1 hour for the snapshot, 24 for the backup.
• In the twenty-eight-point chain, one file corrupting takes 1 point in the full arrangement, 18 in incremental, 1 in differential; storage is 1,120, 91.3, and 758.2 GiB respectively. If the full backup itself is corrupted, all 28 of the 28 points go at once in both incremental and differential.
• In the four-copy arrangement, the worst scenario is compromise of the management identity, taking 3 copies at once; once the offline copy is removed, the copies remaining in the worst scenario drop from 1 to 0.
• When recovery testing is not done, the item required to cross the recovery boundary is one, and when that item is in place, the accessible recovery points are zero.

Course Wrap-Up

Lesson	Boundary Name	Items Required to Cross	Items Reached if It Falls
Computer Hardware Components	8 components, 43 items	remote 1–3, physical 0	3–43; firmware alone 43
Operating System Families	admin boundary, 14 requests	1, 2, 3 by family	13, 9, 5 requests
Installation and Configuration	default boundary, 20 items	1 risky item	17 items; 5 after hardening
File Permissions and Ownership	permission boundary, 70 requests	1 at ownership	all of the object’s permission decisions; 6 deviations
Graphical Interface and Command Line	audit trail boundary, 21 records	1 deletable item	10 records invisible; 3 jobs untraced
Troubleshooting Method	symptom–cause boundary, 16 possibilities	3.50 steps; 5.81 in the worst order	14 scenarios to a single cause; 2 inseparable
Hypervisor Types	hypervisor boundary, 19 and 26 items	1 to the neighbor; 2 and 3 to the hardware	15/19 and 22/26
Guest and Host System	sharing boundary, 11 items	1 in 4 paths, 2 in 10 paths, 3 in 2 paths	6 of 16 paths cannot be closed
Virtual Machine Management	clone and snapshot boundary, 7 secrets	1 for a shared secret	24 machines; 17 vulnerabilities on rollback
Comparison with Containers	isolation boundary, 52 items	guest 2, compartment 1, nested 3	39/52, 52/52, nested 9/52
Backup and Resilience	recovery boundary, 56 items	1 for a corrupt file	8 items unrecoverable; 18 points

The third column reads the course’s rule the same way across all eleven rows: isolation is not an adjective, it is the number of items that must be wrong at the same time to cross the boundary; if that number is one, there is no boundary. All eleven of the eleven rows have at least one “1.” A risky default item, an ownership record, a deletable audit record, a shared secret, a corrupt backup file — each is sufficient on its own, and none of them uses an escape path.

The places where the number is greater than one come from three sources: layer stacked on layer (two and three items to the hardware, three in the nested arrangement), a verification chain (signed boot raises the required item from one to four), and split privilege (the admin boundary is one item in one family, three in the other). All three have their cost written down — four extra patch items, three links re-signed on every update, nine requests unreachable even as an admin. A boundary whose cost is not written down has not been measured.

The fourth column shows where the boundary is not enough: six sharing paths that cannot be closed, eight items that cannot be recovered, two causes that cannot be separated, and one installation item that never closes. None of them close with a better configuration.

All of these numbers were measured on a single machine and the guest systems on it. But the branch server does not stand alone: meter endpoints and field teams’ laptops connect to it, and it in turn connects to the center. Each of these connections is a path that runs outside the boundary the operating system draws; neither permission bits, nor the hypervisor, nor the compartment sees it. M24/K03 Network Security begins with this question: how many paths does each connection between two machines open, and who counts those paths.