--- title: 'Restore Drill' source: 'https://academia.sh/en/courses/servers-and-hardening/restore-drill' course: 'Servers, Networking and Hardening' language: en updated: '2026-09-13T05:08:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Backs up and restores real files, and verifies they are byte-for-byte identical with a digest comparison' - 'Counts the defects seen only through the drill and compares them with what the backup step saw' - 'Splits restore duration into five components and compares it against the recovery time objective' - 'Scans drill frequency against its own cost and the uncertainty that accumulates between two drills' --- # Restore Drill Proving that the backup actually works: real files are backed up, deleted, and restored; being byte-for-byte identical is verified with a digest; five defects seen only through the drill are counted; restore duration is split into five components and compared against the recovery target; and drill frequency is scanned alongside the uncertainty that accumulates between drills. The previous lesson tied backup coverage, frequency, and retention to numbers; those numbers were read from a backup **set**. The item the default leaves is the quietest one: **the backup step returns successful.** The archive is produced, its size is written, a record is kept — and that record says nothing about whether the backup can actually be restored. The measure is **proof** of reversibility: a digest comparing the byte in production with the byte that comes back. ## The Drill Really Runs - **HB31** — The fictional regional measurement network's production tree is really written to disk: five items, their contents from our own generator, seed 51703; the temp directory is deleted at the end of the run. - **HB32** — The archive container is our own format; the password and salt are fictional. - **HB33** — Three of the five defects are really injected (an out-of-scope item, a flipped byte, a different key); wrong order and missing permission are modeled. - **HB34** — The recovery time objective is input from the Resilience and Reliability course: 240 minutes. Download time derives from the previous lesson's 649.8 GiB full backup and a fictional 80 MiB/s line; the other four components are modeled. - **HB35** — The production set gets three config changes a month and one item a quarter; one drill is six hours of one person's time, and runs in a separate restore environment. - **HB36** — Time is measured but not printed; it is only passed through a band check. ```js // drill/container.mjs — the archive container is our own format: manifest + sequential contents. // The password and salt are fictional, no real secret is written. import { readFileSync, writeFileSync, readdirSync, mkdirSync } from "node:fs"; import { join } from "node:path"; import { gzipSync, gunzipSync } from "node:zlib"; import { createCipheriv, createDecipheriv, createHash, scryptSync } from "node:crypto"; export const SEED = 51703; // we write the generator ourselves, the seed is visible let x = SEED >>> 0; export const generate = (n) => { const b = Buffer.allocUnsafe(n); for (let i = 0; i < n; i += 4) { x = (Math.imul(x, 1103515245) + 12345) >>> 0; b.writeUInt32LE(x, i); } return b; }; export const digest = (b) => createHash("sha256").update(b).digest("hex"); const salt = Buffer.from("mock-salt-example-invalid"); export const KEY = scryptSync("mock-backup-password-example-invalid", salt, 32); export const OTHER = scryptSync("mock-other-password-example-invalid", salt, 32); const IV = Buffer.from("mock-iv-12b"); // 12 bytes, fixed and fictional export function pack(dir, skip = []) { // skip: item left out of the plan const names = readdirSync(dir).sort().filter((a) => !skip.includes(a)); const body = names.map((a) => readFileSync(join(dir, a))); const manifest = Buffer.from(JSON.stringify(names.map((a, i) => [a, body[i].length]))); const head = Buffer.allocUnsafe(4); head.writeUInt32LE(manifest.length, 0); return Buffer.concat([head, manifest, ...body]); } export function unpack(archive, dir) { const n = archive.readUInt32LE(0); let p = 4 + n; mkdirSync(dir, { recursive: true }); for (const [a, size] of JSON.parse(archive.subarray(4, 4 + n).toString())) { writeFileSync(join(dir, a), archive.subarray(p, p + size)); p += size; } } export function encrypt(archive) { const s = createCipheriv("aes-256-gcm", KEY, IV); return { body: Buffer.concat([s.update(gzipSync(archive)), s.final()]), tag: s.getAuthTag() }; } export function decrypt({ body, tag }, key = KEY) { try { // a corrupt archive or the wrong key falls through const d = createDecipheriv("aes-256-gcm", key, IV).setAuthTag(tag); return gunzipSync(Buffer.concat([d.update(body), d.final()])); } catch { return null; } } ``` ```js // drill/drill.mjs — REALLY runs: a fictional production tree is written to disk, archived, // DELETED, and restored; being byte-for-byte identical is verified with a digest. Time is not printed. import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, readdirSync, rmSync } from "node:fs"; import { join } from "node:path"; import { tmpdir } from "node:os"; import { SEED, generate, digest, pack, unpack, encrypt, decrypt, OTHER } from "./container.mjs"; const ITEM = [["metric.data", 262144], ["work-order.data", 131072], ["rate.data", 65536], ["service.config", 2048], ["zone.record", 1024]]; // fictional measurement network tree const CONTENT = ITEM.map(([name, n]) => [name, generate(n)]); const root = mkdtempSync(join(tmpdir(), "drill-")); const production = join(root, "production"); const state = () => readdirSync(production).sort().map((a) => [a, digest(readFileSync(join(production, a)))]); function run({ skip = [], corrupt = false, key = undefined }) { rmSync(production, { recursive: true, force: true }); mkdirSync(production, { recursive: true }); for (const [name, b] of CONTENT) writeFileSync(join(production, name), b); const before = state(); let arc = encrypt(pack(production, skip)); if (corrupt) arc = { ...arc, body: Buffer.concat([arc.body.subarray(0, 64), // one byte is flipped Buffer.from([arc.body[64] ^ 1]), arc.body.subarray(65)]) }; rmSync(production, { recursive: true }); // production is deleted: return comes from the archive const container = decrypt(arc, key); if (container) unpack(container, production); const after = new Map(container ? state() : []); return { before, opened: !!container, archiveBytes: arc.body.length, same: before.filter(([name, o]) => after.get(name) === o).length }; } const t0 = process.hrtime.bigint(); const healthy = run({}); const sec = Number(process.hrtime.bigint() - t0) / 1e9; const bytes = CONTENT.reduce((a, [, b]) => a + b.length, 0); const back = new Map(state()); console.log(`${ITEM.length} items, ${bytes} bytes (seed ${SEED}); archive ${healthy.archiveBytes} bytes`); console.log("item | prod digest (first 16) | same on return"); console.log("--------------------|---------------------------|-----------------"); for (const [name, o] of healthy.before) console.log(`${name.padEnd(19)} | ${o.slice(0, 16).padStart(25)} | ` + `${(back.get(name) === o ? "yes" : "no").padStart(16)}`); // MODEL: if order breaks, how many steps come up with empty data const DEPENDS = { secret: [], data: ["secret"], config: [], service: ["data", "config"] }; const WRONG_ORDER = ["service", "data", "config", "secret"]; const empty = WRONG_ORDER.filter((a, i) => DEPENDS[a].some((b) => WRONG_ORDER.indexOf(b) > i)).length; // MODEL: required/granted permissions for the restore identity const REQUIRED = { "backup-store": "read", "object-store": "write", "secret-store": "read", "service-manager": "start" }; const GRANTED = { "backup-store": "read", "secret-store": "read" }; const missing = Object.keys(REQUIRED).filter((k) => GRANTED[k] !== REQUIRED[k]); console.log("\ntrial | opened | match | missing | backup step"); console.log("------------------------------|----------|---------|-----------|------------------"); let found = 2; // two MODEL defects to start for (const [name, option] of [["healthy drill", {}], ["missing item (config)", { skip: ["service.config"] }], ["corrupt archive (one byte)", { corrupt: true }], ["undecryptable cipher", { key: OTHER }]]) { const r = run(option), n = r.before.length; if (r.same < n) found += 1; console.log(`${name.padEnd(29)} | ${(r.opened ? "yes" : "no").padStart(8)} | ` + `${(r.same + "/" + n).padStart(7)} | ${String(n - r.same).padStart(9)} | ` + `${(r.same < n ? "looked successful" : "successful").padStart(16)}`); } console.log(`wrong order (MODEL) | ${empty}/${WRONG_ORDER.length} steps open with empty data` + `${" ".repeat(2)}| looked successful`); console.log(`missing permission (MODEL) | ${missing.length} items: ${missing.join(", ")} | looked successful`); console.log(`\ndrill found ${found} defects; seen in the backup step 0`); const mib = bytes / 1024 / 1024, [LOW, HIGH] = [0.5, 20000]; // MiB/s band; the raw measurement is not printed console.log(`backup+delete+restore speed within ${LOW}-${HIGH} MiB/s band: ` + `${mib / sec >= LOW && mib / sec <= HIGH ? "yes" : "no"}`); rmSync(root, { recursive: true, force: true }); // the temp tree is deleted ``` ``` 5 items, 461824 bytes (seed 51703); archive 462107 bytes item | prod digest (first 16) | same on return --------------------|---------------------------|----------------- metric.data | 3edf42c01d4f883f | yes rate.data | 434443d05cecee54 | yes service.config | 4ead0d5566a44d75 | yes work-order.data | 5c4512d9a8016d51 | yes zone.record | b7c0e693cd032524 | yes trial | opened | match | missing | backup step ------------------------------|----------|---------|-----------|------------------ healthy drill | yes | 5/5 | 0 | successful missing item (config) | yes | 4/5 | 1 | looked successful corrupt archive (one byte) | no | 0/5 | 5 | looked successful undecryptable cipher | no | 0/5 | 5 | looked successful wrong order (MODEL) | 2/4 steps open with empty data | looked successful missing permission (MODEL) | 2 items: object-store, service-manager | looked successful drill found 5 defects; seen in the backup step 0 backup+delete+restore speed within 0.5-20000 MiB/s band: yes ``` The first table is this lesson's proof: the production tree was written, archived, **deleted**, and brought back from the archive alone; digests came out identical for all five of the five items. Not a sentence, five comparisons. The last column of the second table is the finding. The config left out of scope does not come back, but the archive still opens; the archive with one flipped byte and the archive encrypted with a different key do not open at all, and all five items go missing at once. In the wrong order, two of the four steps open with empty data, and the identity is missing two permissions. **All five of the five defects were invisible in the backup step.** ## Duration Splits Into Components ```js // drill/duration.mjs — restore duration splits into components, is compared against the target, and // drill frequency is scanned. MODEL; download time derives from 05's full backup size. const TARGET = 240; // minutes; input from the previous course const FULL_GIB = 649.8, LINE_MIBS = 80; // full backup size; line speed is fictional const DOWNLOAD = (FULL_GIB * 1024) / LINE_MIBS / 60; const COMPONENT = { // [undrilled, drilled] minutes find: [95, 12], download: [DOWNLOAD, DOWNLOAD], extract: [65, 38], verify: [40, 15], "start service": [85, 22], }; const g = (x, n, d = 1) => x.toFixed(d).padStart(n); const sum = (i) => Object.values(COMPONENT).reduce((a, b) => a + b[i], 0); const SHARE = TARGET / Object.keys(COMPONENT).length; // equal share console.log(`target ${TARGET} min; full backup ${FULL_GIB} GiB, line ${LINE_MIBS} MiB/s, equal share ${SHARE} min.`); console.log("\ncomponent | undrilled | drilled | percent of target"); console.log("----------------|-------------|------------|---------------------"); for (const [name, [y, t]] of Object.entries(COMPONENT)) console.log(`${name.padEnd(15)} | ${g(y, 11)} | ${g(t, 10)} | ${(g((100 * t) / TARGET, 6) + "%").padStart(20)}`); console.log(`${"total".padEnd(15)} | ${g(sum(0), 11)} | ${g(sum(1), 10)} | ` + `${(g((100 * sum(1)) / TARGET, 6) + "%").padStart(20)}`); const over = Object.entries(COMPONENT).filter(([, [, t]]) => t > SHARE).map(([a]) => a); console.log(`${over.length} components exceed the equal share in drilled time: ${over.join(", ")}`); const CHANGE_RATE = 3 / 30 + 1 / 90; // daily change: 3 configs a month, 1 item a quarter const ITEM = 16, HOURS = 6; // inventory; one drill is six hours console.log("\nfrequency | yearly | yearly hours | unverified change | restore | target"); console.log("-----------------|----------|----------------|--------------------------|--------------|-----------"); for (const days of [90, 180, 365, null]) { const n = days ? Math.floor(365 / days) : 0; const uncertain = days ? days * CHANGE_RATE : 365 * CHANGE_RATE + ITEM; const dur = days ? sum(1) : sum(0); console.log(`${(days ? days + " days" : "never").padEnd(16)} | ${String(n).padStart(8)} | ` + `${String(n * HOURS).padStart(14)} | ${(g(uncertain, 4) + (days ? " items" : " items+set")).padStart(25)}` + ` | ${g(dur, 12)} | ${(dur > TARGET ? "exceeded" : "held").padStart(10)}`); } ``` ``` target 240 min; full backup 649.8 GiB, line 80 MiB/s, equal share 48 min. component | undrilled | drilled | percent of target ----------------|-------------|------------|--------------------- find | 95.0 | 12.0 | 5.0% download | 138.6 | 138.6 | 57.8% extract | 65.0 | 38.0 | 15.8% verify | 40.0 | 15.0 | 6.3% start service | 85.0 | 22.0 | 9.2% total | 423.6 | 225.6 | 94.0% 1 components exceed the equal share in drilled time: download frequency | yearly | yearly hours | unverified change | restore | target -----------------|----------|----------------|--------------------------|--------------|----------- 90 days | 4 | 24 | 10.0 items | 225.6 | held 180 days | 2 | 12 | 20.0 items | 225.6 | held 365 days | 1 | 6 | 40.6 items | 225.6 | held never | 0 | 0 | 56.6 items+set | 423.6 | exceeded ``` In the undrilled arrangement, restore takes 423.6 minutes, missing the target by 183.6 minutes; in the drilled arrangement it drops to 225.6 minutes and holds the target with a 14.4-minute margin. What the drill shortens is knowledge: finding the right point drops from 95 minutes to 12, starting services from 85 to 22 — rehearsed steps. The download column does not change at all: 138.6 minutes, 57.8 percent of the target, and the only component that exceeds the equal share. This time does not come from a lack of knowledge, it comes from pushing 649.8 GiB through an 80 MiB/s line; the drill measures it but does not shorten it. **The way to hit the target is not more drilling, it is moving the backup closer to where it will be restored.** The frequency scan puts the cost and the alternative side by side: a drill every ninety days is twenty-four hours a year and 10 items of uncertainty, a drill once a year is six hours and 40.6 items. Never drilling costs zero, the uncertainty is all sixteen items, and duration misses the target. ## Summary - The backup step returned successful for all five of the five defects; all of them were seen only in the drill: an out-of-scope item, a corrupt archive, an undecryptable cipher, wrong order, missing permission. - Proof is a digest comparison: five items were written, archived, deleted, restored, and digests came out identical. - Restore duration is 423.6 minutes undrilled, 225.6 minutes drilled; the target holds at 94 percent full. The drill shortens four components and does not touch download: 138.6 minutes is 57.8 percent of the target and the only component exceeding the equal share. - Drill frequency is a cost-uncertainty trade-off: a drill every ninety days is 24 hours and 10 items a year, never drilling is 0 hours and all sixteen items. ## Course Wrap-Up | Lesson | What the default leaves open | Closed / wrongly closed | New surface closing introduces | |---|---|---|---| | Web Server Roles | 18 items | 13 closed / 4 legitimate jobs | 3 new items | | Reverse Proxy Configuration | 4 pieces of missing info | 4 resolved / 2 items hidden | spoofed header, 4 requests | | TLS Termination | 19 weak items | 19 closed / 65 devices | certificate at 6 points | | Certificate Automation | 1,700 hours window | 0 hours / 12 unnecessary renewals | 4 new items | | Cache Server | 179 leaked requests | 0 leaks / 2x load | escaping broadcast, silent window | | Service Mesh | 40 touches for 40 services | 508 work items / expensive at 3 services | 162 sidecar items | | DNS Management | 4 dead, 2 takeover-able records | to 5 minutes / 288x queries | identity writing to the zone | | Sender Verification Records | unowned sending is free | 5/6 spoofed / 2 legitimate dropped | 254 excess authorized addresses | | Domain Policy Record | 61 spoofed reach | 49 blocked / 43 legitimate separated | 12 compliant impersonations | | Mail Delivery Issues | 12-point ceiling | ceiling lifted / 1,078 messages lost | recipient decision unmeasurable | | Firewall Configuration | 100 items, 107,265 attempts | 10 rows all / 2 flows | external row, 20,650 attempts | | Remote Access Hardening | 2,340 triples, 16,505 attempts | 16,505 closed / 0 legitimate | 34 key copies | | Secret Management Systems | 36 items, infinite window | 2.5 minutes / 9 items lifetime-less | single point, 18 instances | | Patch Management | 60 items, 28.94 days | to 1.22 days / 4 rollbacks, 82 outages | 15 items, patch agent | | Backup Strategy | 13 items out of scope | to 3 items / services still 2/4 | readable surface doubled | | Restore Drill | 5 defects invisible, 423.6 min | 5 found / 225.6 min | 24 hours, 2 missing permissions | No row in the second column shows a trace of a decision: **the default is not a decision, it is the decision nobody made.** In the third column, wrong closing fills fourteen of the sixteen rows: no step is free. In the fourth column, three types of surface repeat: **a new identity or write authorization** in five lessons, **a single point carrying the whole load** in four lessons, **a new layer that itself needs configuring** in four lessons. Every defense brings its own surface — a closed item does not disappear, it changes shape. Seven courses in, every layer has been built and hardened separately. Every row in the table is a sequence of decisions: which port, which threshold, which item. But does every team have to rebuild all of this from scratch? Platform Engineering starts with this question.