Restore Drill

The previous lesson tied backup coverage, frequency, and retention to numbers; those numbers were read from a backup set. The item the default leaves is the quietest one: the backup step returns successful. The archive is produced, its size is written, a record is kept — and that record says nothing about whether the backup can actually be restored. The measure is proof of reversibility: a digest comparing the byte in production with the byte that comes back.

The Drill Really Runs

• HB31 — The fictional regional measurement network’s production tree is really written to disk: five items, their contents from our own generator, seed 51703; the temp directory is deleted at the end of the run.
• HB32 — The archive container is our own format; the password and salt are fictional.
• HB33 — Three of the five defects are really injected (an out-of-scope item, a flipped byte, a different key); wrong order and missing permission are modeled.
• HB34 — The recovery time objective is input from the Resilience and Reliability course: 240 minutes. Download time derives from the previous lesson’s 649.8 GiB full backup and a fictional 80 MiB/s line; the other four components are modeled.
• HB35 — The production set gets three config changes a month and one item a quarter; one drill is six hours of one person’s time, and runs in a separate restore environment.
• HB36 — Time is measured but not printed; it is only passed through a band check.

// drill/container.mjs — the archive container is our own format: manifest + sequential contents.
// The password and salt are fictional, no real secret is written.
import { readFileSync, writeFileSync, readdirSync, mkdirSync } from "node:fs";
import { join } from "node:path";
import { gzipSync, gunzipSync } from "node:zlib";
import { createCipheriv, createDecipheriv, createHash, scryptSync } from "node:crypto";

export const SEED = 51703;              // we write the generator ourselves, the seed is visible
let x = SEED >>> 0;
export const generate = (n) => {
  const b = Buffer.allocUnsafe(n);
  for (let i = 0; i < n; i += 4) { x = (Math.imul(x, 1103515245) + 12345) >>> 0; b.writeUInt32LE(x, i); }
  return b;
};
export const digest = (b) => createHash("sha256").update(b).digest("hex");
const salt = Buffer.from("mock-salt-example-invalid");
export const KEY = scryptSync("mock-backup-password-example-invalid", salt, 32);
export const OTHER = scryptSync("mock-other-password-example-invalid", salt, 32);
const IV = Buffer.from("mock-iv-12b");  // 12 bytes, fixed and fictional

export function pack(dir, skip = []) {       // skip: item left out of the plan
  const names = readdirSync(dir).sort().filter((a) => !skip.includes(a));
  const body = names.map((a) => readFileSync(join(dir, a)));
  const manifest = Buffer.from(JSON.stringify(names.map((a, i) => [a, body[i].length])));
  const head = Buffer.allocUnsafe(4);
  head.writeUInt32LE(manifest.length, 0);
  return Buffer.concat([head, manifest, ...body]);
}
export function unpack(archive, dir) {
  const n = archive.readUInt32LE(0);
  let p = 4 + n;
  mkdirSync(dir, { recursive: true });
  for (const [a, size] of JSON.parse(archive.subarray(4, 4 + n).toString())) {
    writeFileSync(join(dir, a), archive.subarray(p, p + size));
    p += size;
  }
}
export function encrypt(archive) {
  const s = createCipheriv("aes-256-gcm", KEY, IV);
  return { body: Buffer.concat([s.update(gzipSync(archive)), s.final()]), tag: s.getAuthTag() };
}
export function decrypt({ body, tag }, key = KEY) {
  try {                                  // a corrupt archive or the wrong key falls through
    const d = createDecipheriv("aes-256-gcm", key, IV).setAuthTag(tag);
    return gunzipSync(Buffer.concat([d.update(body), d.final()]));
  } catch { return null; }
}

// drill/drill.mjs — REALLY runs: a fictional production tree is written to disk, archived,
// DELETED, and restored; being byte-for-byte identical is verified with a digest. Time is not printed.
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, readdirSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { SEED, generate, digest, pack, unpack, encrypt, decrypt, OTHER } from "./container.mjs";

const ITEM = [["metric.data", 262144], ["work-order.data", 131072], ["rate.data", 65536],
  ["service.config", 2048], ["zone.record", 1024]];   // fictional measurement network tree
const CONTENT = ITEM.map(([name, n]) => [name, generate(n)]);
const root = mkdtempSync(join(tmpdir(), "drill-"));
const production = join(root, "production");
const state = () => readdirSync(production).sort().map((a) => [a, digest(readFileSync(join(production, a)))]);

function run({ skip = [], corrupt = false, key = undefined }) {
  rmSync(production, { recursive: true, force: true });
  mkdirSync(production, { recursive: true });
  for (const [name, b] of CONTENT) writeFileSync(join(production, name), b);
  const before = state();
  let arc = encrypt(pack(production, skip));
  if (corrupt) arc = { ...arc, body: Buffer.concat([arc.body.subarray(0, 64),         // one byte is flipped
    Buffer.from([arc.body[64] ^ 1]), arc.body.subarray(65)]) };
  rmSync(production, { recursive: true });                  // production is deleted: return comes from the archive
  const container = decrypt(arc, key);
  if (container) unpack(container, production);
  const after = new Map(container ? state() : []);
  return { before, opened: !!container, archiveBytes: arc.body.length,
    same: before.filter(([name, o]) => after.get(name) === o).length };
}

const t0 = process.hrtime.bigint();
const healthy = run({});
const sec = Number(process.hrtime.bigint() - t0) / 1e9;
const bytes = CONTENT.reduce((a, [, b]) => a + b.length, 0);
const back = new Map(state());
console.log(`${ITEM.length} items, ${bytes} bytes (seed ${SEED}); archive ${healthy.archiveBytes} bytes`);
console.log("item                | prod digest (first 16)   | same on return");
console.log("--------------------|---------------------------|-----------------");
for (const [name, o] of healthy.before)
  console.log(`${name.padEnd(19)} | ${o.slice(0, 16).padStart(25)} | ` +
    `${(back.get(name) === o ? "yes" : "no").padStart(16)}`);

// MODEL: if order breaks, how many steps come up with empty data
const DEPENDS = { secret: [], data: ["secret"], config: [], service: ["data", "config"] };
const WRONG_ORDER = ["service", "data", "config", "secret"];
const empty = WRONG_ORDER.filter((a, i) => DEPENDS[a].some((b) => WRONG_ORDER.indexOf(b) > i)).length;
// MODEL: required/granted permissions for the restore identity
const REQUIRED = { "backup-store": "read", "object-store": "write", "secret-store": "read", "service-manager": "start" };
const GRANTED = { "backup-store": "read", "secret-store": "read" };
const missing = Object.keys(REQUIRED).filter((k) => GRANTED[k] !== REQUIRED[k]);

console.log("\ntrial                         |   opened |   match |   missing | backup step");
console.log("------------------------------|----------|---------|-----------|------------------");
let found = 2;                                        // two MODEL defects to start
for (const [name, option] of [["healthy drill", {}],
  ["missing item (config)", { skip: ["service.config"] }],
  ["corrupt archive (one byte)", { corrupt: true }], ["undecryptable cipher", { key: OTHER }]]) {
  const r = run(option), n = r.before.length;
  if (r.same < n) found += 1;
  console.log(`${name.padEnd(29)} | ${(r.opened ? "yes" : "no").padStart(8)} | ` +
    `${(r.same + "/" + n).padStart(7)} | ${String(n - r.same).padStart(9)} | ` +
    `${(r.same < n ? "looked successful" : "successful").padStart(16)}`);
}
console.log(`wrong order (MODEL)           | ${empty}/${WRONG_ORDER.length} steps open with empty data` +
  `${" ".repeat(2)}| looked successful`);
console.log(`missing permission (MODEL)    | ${missing.length} items: ${missing.join(", ")}  | looked successful`);
console.log(`\ndrill found ${found} defects; seen in the backup step 0`);
const mib = bytes / 1024 / 1024, [LOW, HIGH] = [0.5, 20000];   // MiB/s band; the raw measurement is not printed
console.log(`backup+delete+restore speed within ${LOW}-${HIGH} MiB/s band: ` +
  `${mib / sec >= LOW && mib / sec <= HIGH ? "yes" : "no"}`);
rmSync(root, { recursive: true, force: true });               // the temp tree is deleted

5 items, 461824 bytes (seed 51703); archive 462107 bytes
item                | prod digest (first 16)   | same on return
--------------------|---------------------------|-----------------
metric.data         |          3edf42c01d4f883f |              yes
rate.data           |          434443d05cecee54 |              yes
service.config      |          4ead0d5566a44d75 |              yes
work-order.data     |          5c4512d9a8016d51 |              yes
zone.record         |          b7c0e693cd032524 |              yes

trial                         |   opened |   match |   missing | backup step
------------------------------|----------|---------|-----------|------------------
healthy drill                 |      yes |     5/5 |         0 |       successful
missing item (config)         |      yes |     4/5 |         1 | looked successful
corrupt archive (one byte)    |       no |     0/5 |         5 | looked successful
undecryptable cipher          |       no |     0/5 |         5 | looked successful
wrong order (MODEL)           | 2/4 steps open with empty data  | looked successful
missing permission (MODEL)    | 2 items: object-store, service-manager  | looked successful

drill found 5 defects; seen in the backup step 0
backup+delete+restore speed within 0.5-20000 MiB/s band: yes

The first table is this lesson’s proof: the production tree was written, archived, deleted, and brought back from the archive alone; digests came out identical for all five of the five items. Not a sentence, five comparisons.

The last column of the second table is the finding. The config left out of scope does not come back, but the archive still opens; the archive with one flipped byte and the archive encrypted with a different key do not open at all, and all five items go missing at once. In the wrong order, two of the four steps open with empty data, and the identity is missing two permissions. All five of the five defects were invisible in the backup step.

Duration Splits Into Components

// drill/duration.mjs — restore duration splits into components, is compared against the target, and
// drill frequency is scanned. MODEL; download time derives from 05's full backup size.
const TARGET = 240;                       // minutes; input from the previous course
const FULL_GIB = 649.8, LINE_MIBS = 80;    // full backup size; line speed is fictional
const DOWNLOAD = (FULL_GIB * 1024) / LINE_MIBS / 60;
const COMPONENT = {                        // [undrilled, drilled] minutes
  find: [95, 12], download: [DOWNLOAD, DOWNLOAD], extract: [65, 38],
  verify: [40, 15], "start service": [85, 22],
};
const g = (x, n, d = 1) => x.toFixed(d).padStart(n);
const sum = (i) => Object.values(COMPONENT).reduce((a, b) => a + b[i], 0);
const SHARE = TARGET / Object.keys(COMPONENT).length;         // equal share

console.log(`target ${TARGET} min; full backup ${FULL_GIB} GiB, line ${LINE_MIBS} MiB/s, equal share ${SHARE} min.`);
console.log("\ncomponent       |   undrilled |    drilled |    percent of target");
console.log("----------------|-------------|------------|---------------------");
for (const [name, [y, t]] of Object.entries(COMPONENT))
  console.log(`${name.padEnd(15)} | ${g(y, 11)} | ${g(t, 10)} | ${(g((100 * t) / TARGET, 6) + "%").padStart(20)}`);
console.log(`${"total".padEnd(15)} | ${g(sum(0), 11)} | ${g(sum(1), 10)} | ` +
  `${(g((100 * sum(1)) / TARGET, 6) + "%").padStart(20)}`);
const over = Object.entries(COMPONENT).filter(([, [, t]]) => t > SHARE).map(([a]) => a);
console.log(`${over.length} components exceed the equal share in drilled time: ${over.join(", ")}`);

const CHANGE_RATE = 3 / 30 + 1 / 90;         // daily change: 3 configs a month, 1 item a quarter
const ITEM = 16, HOURS = 6;              // inventory; one drill is six hours
console.log("\nfrequency        |   yearly |   yearly hours |        unverified change |      restore |     target");
console.log("-----------------|----------|----------------|--------------------------|--------------|-----------");
for (const days of [90, 180, 365, null]) {
  const n = days ? Math.floor(365 / days) : 0;
  const uncertain = days ? days * CHANGE_RATE : 365 * CHANGE_RATE + ITEM;
  const dur = days ? sum(1) : sum(0);
  console.log(`${(days ? days + " days" : "never").padEnd(16)} | ${String(n).padStart(8)} | ` +
    `${String(n * HOURS).padStart(14)} | ${(g(uncertain, 4) + (days ? " items" : " items+set")).padStart(25)}` +
    ` | ${g(dur, 12)} | ${(dur > TARGET ? "exceeded" : "held").padStart(10)}`);
}

target 240 min; full backup 649.8 GiB, line 80 MiB/s, equal share 48 min.

component       |   undrilled |    drilled |    percent of target
----------------|-------------|------------|---------------------
find            |        95.0 |       12.0 |                 5.0%
download        |       138.6 |      138.6 |                57.8%
extract         |        65.0 |       38.0 |                15.8%
verify          |        40.0 |       15.0 |                 6.3%
start service   |        85.0 |       22.0 |                 9.2%
total           |       423.6 |      225.6 |                94.0%
1 components exceed the equal share in drilled time: download

frequency        |   yearly |   yearly hours |        unverified change |      restore |     target
-----------------|----------|----------------|--------------------------|--------------|-----------
90 days          |        4 |             24 |                10.0 items |        225.6 |       held
180 days         |        2 |             12 |                20.0 items |        225.6 |       held
365 days         |        1 |              6 |                40.6 items |        225.6 |       held
never            |        0 |              0 |            56.6 items+set |        423.6 |   exceeded

In the undrilled arrangement, restore takes 423.6 minutes, missing the target by 183.6 minutes; in the drilled arrangement it drops to 225.6 minutes and holds the target with a 14.4-minute margin. What the drill shortens is knowledge: finding the right point drops from 95 minutes to 12, starting services from 85 to 22 — rehearsed steps.

The download column does not change at all: 138.6 minutes, 57.8 percent of the target, and the only component that exceeds the equal share. This time does not come from a lack of knowledge, it comes from pushing 649.8 GiB through an 80 MiB/s line; the drill measures it but does not shorten it. The way to hit the target is not more drilling, it is moving the backup closer to where it will be restored. The frequency scan puts the cost and the alternative side by side: a drill every ninety days is twenty-four hours a year and 10 items of uncertainty, a drill once a year is six hours and 40.6 items. Never drilling costs zero, the uncertainty is all sixteen items, and duration misses the target.

Summary

• The backup step returned successful for all five of the five defects; all of them were seen only in the drill: an out-of-scope item, a corrupt archive, an undecryptable cipher, wrong order, missing permission.
• Proof is a digest comparison: five items were written, archived, deleted, restored, and digests came out identical.
• Restore duration is 423.6 minutes undrilled, 225.6 minutes drilled; the target holds at 94 percent full. The drill shortens four components and does not touch download: 138.6 minutes is 57.8 percent of the target and the only component exceeding the equal share.
• Drill frequency is a cost-uncertainty trade-off: a drill every ninety days is 24 hours and 10 items a year, never drilling is 0 hours and all sixteen items.

Course Wrap-Up

No row in the second column shows a trace of a decision: the default is not a decision, it is the decision nobody made. In the third column, wrong closing fills fourteen of the sixteen rows: no step is free. In the fourth column, three types of surface repeat: a new identity or write authorization in five lessons, a single point carrying the whole load in four lessons, a new layer that itself needs configuring in four lessons. Every defense brings its own surface — a closed item does not disappear, it changes shape.

Seven courses in, every layer has been built and hardened separately. Every row in the table is a sequence of decisions: which port, which threshold, which item. But does every team have to rebuild all of this from scratch? Platform Engineering starts with this question.