Backup Strategy

The previous lesson ended with an assumption: that a prior state exists to return to when a patch is rolled back. The same assumption sits under every recovery plan. Recovery time and data-loss objectives were measured in the Resilience and Reliability course; they are taken here as input. What is measured is the backup itself.

The item the default leaves is a gap here too: after install, the only backup that comes to mind is the nightly dump of the database. The rest — config, secrets, object store, data in an external service — stays out of the plan in a way nobody decided. The measure is the coverage gap: the number of items that exist in production but are not in the backup set.

Coverage Gap Item by Item

• HB25 — The production inventory is fictional: sixteen items; their type, size, and daily change are written in the model.

• HB26 — The item set each of the four services needs is written in the model; if one item is missing, that service cannot be restored.

• HB27 — Three items cannot be backed up; the reason is the item’s type.

• HB28 — The compression ratio is 0.42; an incremental backup is the interval’s share of the daily change, and every point has 0.02 GiB of overhead.

• HB29 — The data-loss window is the interval’s half on average, and the interval itself in the worst case.

• HB30 — The encryption section really runs; time is measured but not printed, it is passed through a band check.

// backup/model.mjs — production inventory is a MODEL: sixteen items, four services. No real backup is taken.
const K = (name, type, gib, change, capturable = true) => ({ name, type, gib, change, capturable });
export const INVENTORY = [
  K("read-database", "database", 420, 6.0), K("rate-database", "database", 4, 0.02),
  K("work-order-database", "database", 18, 0.6), K("invoice-document-store", "object store", 260, 1.4),
  K("field-photo-store", "object store", 610, 3.1), K("service-config", "config", 0.02, 0.001),
  K("firewall-rules", "config", 0.001, 0.0002),
  K("zone-file", "config", 0.0005, 0.0001),
  K("deployment-config", "config", 0.05, 0.01),
  K("secret-store-content", "secret", 0.002, 0.0005), K("certificate-private-keys", "secret", 0.0004, 0.00005),
  K("monitoring-metric-series", "metric", 95, 4.2), K("audit-log", "log", 140, 2.8),
  K("message-queue-state", "queue", 2.5, 2.5, false),
  K("external-billing-service-data", "external service", 12, 0.3, false),
  K("external-notification-lists", "external service", 0.8, 0.05, false),
];
// A service can only be restored if every item it needs is in the backup.
export const SERVICE = {
  collector: ["read-database", "service-config", "secret-store-content", "deployment-config"],
  validator: ["read-database", "service-config", "secret-store-content", "message-queue-state"],
  billing: ["read-database", "rate-database", "invoice-document-store", "service-config",
    "secret-store-content", "external-billing-service-data"],
  "work-order": ["work-order-database", "field-photo-store", "service-config",
    "certificate-private-keys", "zone-file"],
};
export const PLAN = {                             // which types go into the backup set
  default: ["database"],
  "data-and-object": ["database", "object store"],
  "config-added": ["database", "object store", "config"],
  "secret-added": ["database", "object store", "config", "secret"],
  full: ["database", "object store", "config", "secret", "metric", "log", "queue", "external service"],
};
export const covered = (p) =>
  INVENTORY.filter((k) => PLAN[p].includes(k.type) && k.capturable);

// backup/scope.mjs — coverage gap is counted item by item and restorable service is measured
import { INVENTORY, SERVICE, PLAN, covered } from "./model.mjs";

const TOTAL_GIB = INVENTORY.reduce((a, k) => a + k.gib, 0);
const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
console.log(`${INVENTORY.length} production items, ${g(TOTAL_GIB, 0, 1)} GiB, ${Object.keys(SERVICE).length} services; ` +
  `${INVENTORY.filter((k) => !k.capturable).length} items cannot be backed up.`);
console.log("\nplan                 | covered   | gap    | backed up GiB   | byte share    | restorable");
console.log("---------------------|-----------|--------|-----------------|---------------|------------------");
for (const p of Object.keys(PLAN)) {
  const inside = covered(p), names = new Set(inside.map((k) => k.name));
  const gib = inside.reduce((a, k) => a + k.gib, 0);
  const service = Object.entries(SERVICE).filter(([, needed]) => needed.every((x) => names.has(x)));
  console.log(`${p.padEnd(20)} | ${String(inside.length).padStart(9)} | ` +
    `${String(INVENTORY.length - inside.length).padStart(6)} | ${g(gib, 15, 1)} | ` +
    `${g(100 * gib / TOTAL_GIB, 12)}% | ${(service.length + "/" + Object.keys(SERVICE).length).padStart(18)}`);
}

console.log("\nCoverage gap item by item (left out of the config-added plan):");
const inside = new Set(covered("config-added").map((k) => k.name));
for (const k of INVENTORY.filter((x) => !inside.has(x.name)))
  console.log(`${k.name.padEnd(31)} | ${k.type.padEnd(18)} | ${g(k.gib, 7, 3)} GiB | ` +
    `${k.capturable ? "out of plan" : "UNCAPTURABLE"}`);

const blocking = INVENTORY.map((k) => ({ k, e: Object.keys(SERVICE).filter((s) => SERVICE[s].includes(k.name)) }))
  .filter((x) => x.e.length > 0).sort((a, b) => a.k.gib - b.k.gib).slice(0, 4);
const small = blocking.reduce((a, x) => a + x.k.gib, 0);
console.log(`\nfour smallest items blocking a service: ${blocking.map((x) => x.k.name).join(", ")}`);
console.log(`total ${g(small * 1024, 6)} MiB = ${g(100 * small / TOTAL_GIB, 6, 4)}% of production bytes; ` +
  `together blocking ${new Set(blocking.flatMap((x) => x.e)).size}/${Object.keys(SERVICE).length} services.`);

16 production items, 1562.4 GiB, 4 services; 3 items cannot be backed up.

plan                 | covered   | gap    | backed up GiB   | byte share    | restorable
---------------------|-----------|--------|-----------------|---------------|------------------
default              |         3 |     13 |           442.0 |        28.29% |                0/4
data-and-object      |         5 |     11 |          1312.0 |        83.97% |                0/4
config-added         |         9 |      7 |          1312.1 |        83.98% |                0/4
secret-added         |        11 |      5 |          1312.1 |        83.98% |                2/4
full                 |        13 |      3 |          1547.1 |        99.02% |                2/4

Coverage gap item by item (left out of the config-added plan):
secret-store-content            | secret             |   0.002 GiB | out of plan
certificate-private-keys        | secret             |   0.000 GiB | out of plan
monitoring-metric-series        | metric             |  95.000 GiB | out of plan
audit-log                       | log                | 140.000 GiB | out of plan
message-queue-state             | queue              |   2.500 GiB | UNCAPTURABLE
external-billing-service-data   | external service   |  12.000 GiB | UNCAPTURABLE
external-notification-lists     | external service   |   0.800 GiB | UNCAPTURABLE

four smallest items blocking a service: certificate-private-keys, zone-file, secret-store-content, service-config
total  23.45 MiB = 0.0015% of production bytes; together blocking 4/4 services.

The default plan covers three of the sixteen items, a coverage gap of 13. Once the object store is added, byte share jumps from 28.29 percent to 83.97 percent and the plan looks complete — but restorable services are still 0/4. Once config is added, byte share rises by 0.01 points, services are still 0/4. Once secrets are added, byte share does not change at all, and services go from 0 to 2. Byte share does not measure coverage. The last two rows give the reason: the four smallest blocking items add up to 23.45 MiB, 0.0015 percent of production bytes, but they block all four services.

Even in the full plan, services stay at 2/4: a consistent snapshot of the message queue state cannot be captured, and the data in the external billing service and the external notification lists have no export interface. The validator depends on the first, billing on the second; neither can be restored. There is no backup decision to make for these items — the decision is moving the item to a place that can be backed up.

Frequency, Retention, and Storage

// backup/frequency.mjs — frequency, retention, point count, and storage are scanned together (MODEL)
import { INVENTORY, covered } from "./model.mjs";

const COMPRESSION = 0.42, OVERHEAD = 0.02;         // overhead per point, GiB
const items = covered("full");
const FULL_GIB = items.reduce((a, k) => a + k.gib, 0) * COMPRESSION;
const DAILY_GIB = items.reduce((a, k) => a + k.change, 0) * COMPRESSION;
const PRODUCTION_GIB = INVENTORY.reduce((a, k) => a + k.gib, 0);
const SCHEDULE = {                                   // full: days, inc: hours (null = no incremental)
  "weekly-full": { full: 7, inc: null }, "nightly-full": { full: 1, inc: null },
  "nightly+6h": { full: 1, inc: 6 }, "weekly+1h": { full: 7, inc: 1 },
  "weekly+5min": { full: 7, inc: 1 / 12 },
};
const points = (d, R) => Math.floor(R / d.full) + 1 + (d.inc ? Math.floor(R * 24 / d.inc) : 0);
const stored = (d, R) => (Math.floor(R / d.full) + 1) * FULL_GIB + (d.inc ? R * DAILY_GIB : 0)
  + points(d, R) * OVERHEAD;
const g = (x, n, d = 2) => x.toFixed(d).padStart(n);

console.log(`backing up ${items.length} items: full backup ${g(FULL_GIB, 0, 1)} GiB (compression ${COMPRESSION}), ` +
  `daily change ${g(DAILY_GIB, 0, 2)} GiB, overhead per point ${OVERHEAD} GiB.`);
console.log("\nschedule     |      full |   incr. |   avg data loss (min) |   worst (min) | daily written GiB");
console.log("-------------|-----------|---------|-----------------------|---------------|--------------------");
for (const [name, d] of Object.entries(SCHEDULE)) {
  const min = (d.inc ?? d.full * 24) * 60;
  console.log(`${name.padEnd(12)} | ${(d.full + " days").padStart(9)} | ` +
    `${(d.inc ? (d.inc < 1 ? d.inc * 60 + " min" : d.inc + " h") : "none").padStart(7)} | ` +
    `${g(min / 2, 21, 1)} | ${g(min, 13, 1)} | ${g(FULL_GIB / d.full + (d.inc ? DAILY_GIB : 0), 19)}`);
}

const RETENTION = [7, 30, 90, 365];        // retention = oldest point you can go back to
console.log("\nstored GiB (by retention days) and points recoverable at 30 days:");
console.log(`schedule     |${RETENTION.map((R) => (R + " days").padStart(11)).join(" |")} | 30d point`);
console.log("-------------|------------|------------|------------|------------|----------");
for (const [name, d] of Object.entries(SCHEDULE))
  console.log(`${name.padEnd(12)} |${RETENTION.map((R) => g(stored(d, R), 11, 1)).join(" |")} | ` +
    `${String(points(d, 30)).padStart(9)}`);
console.log(`production data ${g(PRODUCTION_GIB, 0, 1)} GiB; stored in the weekly+1h schedule, at 7 days retention ` +
  `${g(stored(SCHEDULE["weekly+1h"], 7) / PRODUCTION_GIB, 0)}x, at 365 days ${g(stored(SCHEDULE["weekly+1h"], 365) / PRODUCTION_GIB, 0)}x.`);

backing up 13 items: full backup 649.8 GiB (compression 0.42), daily change 7.62 GiB, overhead per point 0.02 GiB.

schedule     |      full |   incr. |   avg data loss (min) |   worst (min) | daily written GiB
-------------|-----------|---------|-----------------------|---------------|--------------------
weekly-full  |    7 days |    none |                5040.0 |       10080.0 |               92.82
nightly-full |    1 days |    none |                 720.0 |        1440.0 |              649.77
nightly+6h   |    1 days |     6 h |                 180.0 |         360.0 |              657.39
weekly+1h    |    7 days |     1 h |                  30.0 |          60.0 |              100.44
weekly+5min  |    7 days |   5 min |                   2.5 |           5.0 |              100.44

stored GiB (by retention days) and points recoverable at 30 days:
schedule     |     7 days |    30 days |    90 days |   365 days | 30d point
-------------|------------|------------|------------|------------|----------
weekly-full  |     1299.6 |     3249.0 |     8447.3 |    34438.9 |         5
nightly-full |     5198.3 |    20143.5 |    59131.0 |   237823.5 |        31
nightly+6h   |     5252.2 |    20374.4 |    59823.6 |   240632.3 |       151
weekly+1h    |     1356.2 |     3491.8 |     9175.9 |    37393.7 |       725
weekly+5min  |     1393.2 |     3650.2 |     9651.1 |    39320.9 |      8645
production data 1562.4 GiB; stored in the weekly+1h schedule, at 7 days retention 0.87x, at 365 days 23.93x.

The nightly full backup is the most common default habit, and it loses on both ends: it writes 649.77 GiB a day, stores 20,143.5 GiB over thirty days, has a 720-minute window, and leaves 31 points. The weekly-full-plus-hourly-incremental schedule writes 100.44 GiB a day, stores 3,491.8 GiB over thirty days, drops the window to 30 minutes, and leaves 725 points — six times less writing, five times less storage, a window twenty-four times smaller.

Raising frequency has almost no effect on storage: once the incremental interval drops to five minutes, thirty-day storage rises from 3,491.8 GiB to 3,650.2 GiB — 4.5 percent — and the loss window drops to 2.5 minutes. What rises is not the data stored, it is the point count and the overhead. What is expensive is not frequency, it is retention: once retention goes from 7 days to 365 days, storage goes from 1,356.2 GiB to 37,393.7 GiB, from 0.87 times production data to 23.93 times.

Retention is also the oldest point you can go back to, and it is not a number until it is compared against how long silent corruption takes to notice: at seven-day retention, corruption noticed on the eighth day has no clean point to return to — all 725 points belong to after the corruption.

The Backup Itself Is a Surface

Copy placement is also a model. Copies lost in a single event are the copy-per-zone ratio rounded up; readable surface is the copy count times the number of reading identities.

// backup/key.mjs — the encryption section REALLY runs: recovery is tried with the correct and the wrong
// key. Time is measured but the RAW MEASUREMENT IS NOT PRINTED, only the band check is written.
import { createCipheriv, createDecipheriv, scryptSync } from "node:crypto";

const SEED = 90211, MIB = 24;           // data comes from our own generator, seed is visible: fictional
let x = SEED >>> 0;
const next = () => { x = (Math.imul(x, 1103515245) + 12345) >>> 0; return x; };
const plain = Buffer.allocUnsafe(MIB * 1024 * 1024);
for (let i = 0; i < plain.length; i += 4) plain.writeUInt32LE(next(), i);
const iv = Buffer.allocUnsafe(12);       // the init vector comes from the same generator: no randomness
for (let i = 0; i < 12; i += 4) iv.writeUInt32LE(next(), i);
const salt = Buffer.from("mock-salt-example-invalid");   // no real secret or key is written
const key = scryptSync("mock-password-example-invalid", salt, 32);
const other = scryptSync("mock-other-password-example-invalid", salt, 32);

const t0 = process.hrtime.bigint();
const s = createCipheriv("aes-256-gcm", key, iv);
const cipher = Buffer.concat([s.update(plain), s.final()]), tag = s.getAuthTag();
const decrypt = (a) => {                 // the integrity tag will not hold with the wrong key
  try {
    const d = createDecipheriv("aes-256-gcm", a, iv).setAuthTag(tag);
    return Buffer.concat([d.update(cipher), d.final()]);
  } catch { return null; }
};
const back = decrypt(key), wrong = decrypt(other);
const sec = Number(process.hrtime.bigint() - t0) / 1e9;

const [LOW, HIGH] = [5, 40000];           // MiB/s band; not the measured value, the check result is printed
const print = (e, d) => console.log(`${e.padEnd(44)}: ${d}`);
console.log(`\nmock backup set ${MIB} MiB (seed ${SEED}), encrypted size ${cipher.length} bytes`);
print("returned identical to input with the correct key", back && back.equals(plain) ? "yes" : "no");
print("could be recovered with a different key", wrong ? "yes" : "no");
print(`speed within ${LOW}-${HIGH} MiB/s band`, MIB / sec >= LOW && MIB / sec <= HIGH ? "yes" : "no");

mock backup set 24 MiB (seed 90211), encrypted size 25165824 bytes
returned identical to input with the correct key: yes
could be recovered with a different key     : no
speed within 5-40000 MiB/s band             : yes

The default location holds a full copy of production data in the same place as production, readable by the same nine identities, and unencrypted: readable surface is 9, and a single event takes out the whole copy. The second copy raises durability but pushes the surface to 18 — taking a backup is exposing the data a second time. Once access is narrowed, the surface drops to 6 with three copies: the copy count triples while the surface drops to a third. The fourth, immutable copy zeroes out who can delete.

The item encryption introduces shows up in the run: recovery with the correct key returns exactly the input, recovery with a different key does not. This is also the definition of the new single point — if the key is lost, the backup is lost too. The key is a secret, and it sits in the secret store; and the secret store’s content is itself one of the items that needs backing up: the backup’s key cannot be stored inside the backup.

Summary

• Coverage gap is not measured by byte share: once the object store is added, byte share rises to 83.97 percent while restorable services stay at 0/4; once secrets are added, byte share does not change at all but services rise to 2/4.
• Four small items — certificate private keys, the zone file, secret store content, service config — total 23.45 MiB, 0.0015 percent of production bytes, but block all four services.
• The three items that cannot be backed up are named: the instantaneous queue state, the data in the external billing service, the external notification lists. Even in the full plan, services stay at 2/4.
• Frequency is cheap, retention is expensive: dropping the incremental interval to five minutes raises thirty-day storage by 4.5 percent, and raising retention to 365 days carries it from 1,356.2 GiB to 37,393.7 GiB.
• The backup itself is a surface: readable surface is 9 by default, 18 with two copies, 6 with three access-narrowed copies. The encryption key is a new single point and cannot be stored inside the backup.

Next Step

Every number in this lesson came from a backup set; none of them came from actually performing a restore. The coverage table says an item is in the backup set, not that it is readable; the point count says you can return to a point, not how long the return takes. The next lesson tests this with a controlled drill: real files are backed up and restored, and what comes back is verified to be byte-for-byte identical to what went in. Three things get measured — the real time the drill takes, the item count the coverage table counted correctly, and which assumptions a first restore, in a setup without a drill, breaks all at once.