Recovery

The previous lesson counted that five of the 22 footholds could not be found by any discovery method, and showed that the only way to bring the missed count to zero is to rebuild the asset. A rebuilt asset does not come back empty; it is filled from a backup or an image. This lesson measures that choice.

Recovery is not measured by how quickly the service comes back up. What is measured is reinfection risk: which recovery point was taken before the first foothold, how many items returning to a clean one costs, how many items are manually rebuilt, and across how many stages the service is observed as it returns to production. Backup types, chain structure, and storage accounting were measured in the IT and Systems Fundamentals course; here a backup is treated only as a point in time.

Recovery Points

IR27: The first foothold was nine days ago; a recovery point is clean only if it was taken before that moment. IR28: The day each of the 22 footholds was established is a given; a backup contains every point that was already established at the moment it was taken. IR29: Daily transaction volume is 9,970 items — 8,400 meter readings, 1,250 invoice items, 320 work orders; lost items are the volume from the day returned to up to today. IR30: Configuration left outside the backup is manually rebuilt: six items baseline, four items per day returned to. IR31: When returning to a dirty backup, eradication reruns from scratch — the previous lesson’s full discovery regime, 642 minutes — and five points are missed again. IR32: Supervised restoration has four stages; each stage has an observation window and a probability of catching a reactivating point. IR33: In rebuilding from scratch, the system image is clean and data is taken from the latest backup; rebuilding nine assets from an image takes 430 minutes and 34 manually rebuilt items.

// recovery/model.mjs — supervised restoration model: no real backup or system involved.
// Recovery points, foothold establishment days and stages are data structures.

export const FIRST_FOOTHOLD = 9;               // how many days ago the first foothold occurred
export const POINT_COUNT = 22, MISSED = 5;     // from the previous lesson: total and missed points
// day each foothold was established (how many days ago) — 22 points (data)
export const ESTABLISHED = [9, 9, 9, 8, 8, 7, 7, 6, 6, 6, 5, 5, 4, 4, 4, 3, 3, 2, 2, 1, 1, 1];

// daily transaction volume: service -> items (data)
export const VOLUME = { "meter reading": 8400, "invoice item": 1250, "work order": 320 };
export const DAILY = Object.values(VOLUME).reduce((s, v) => s + v, 0);

// recovery point: how many days ago it was taken, restore minutes
export const POINTS = [[21, 260], [14, 245], [9.5, 240], [7, 215], [3, 190], [1, 175],
  [0.25, 165]];
// items to be manually rebuilt: configuration outside the backup, per day (data)
export const MANUAL_DAILY = 4, MANUAL_BASE = 6;

export const dirtyPoints = (daysAgo) =>
  ESTABLISHED.filter((k) => k >= daysAgo).length;   // points already established at that moment are in the backup
export const lost = (daysAgo) => Math.round(daysAgo * DAILY);
export const manual = (daysAgo) => MANUAL_BASE + Math.round(daysAgo * MANUAL_DAILY);

// supervised restoration stage: name, observation minutes, probability of catching a reactivating point
export const STAGES = [["opening on an isolated network", 120, 0.55],
  ["supervised limited traffic", 180, 0.70], ["partial traffic", 240, 0.85],
  ["full traffic", 0, 0]];

export function staged(count) {                  // the first `count` stages are applied
  const subset = STAGES.slice(0, count);
  const escaped = subset.reduce((p, [, , y]) => p * (1 - y), 1);
  return { min: subset.reduce((s, [, d]) => s + d, 0), capture: 1 - escaped, escaped };
}

// recovery regime: name, recovery point (days ago) or null (rebuild from scratch),
// data is taken from the latest backup on a from-scratch rebuild, and the system image is clean
export const REGIMES = [
  ["return to latest backup", 0.25, false], ["return to 1 day ago", 1, false],
  ["return to 3 days ago", 3, false], ["return to clean backup", 9.5, false],
  ["rebuild from scratch + data", 0.25, true],
];
export const SCRATCH_MIN = 430, SCRATCH_MANUAL = 34;   // rebuilding nine assets from an image
// eradication reruns from scratch when returning to a dirty backup (the previous lesson's full discovery regime)
export const CLEANUP_MIN = 642;

export function measure([name, day, scratch], stageCount) {
  const backupPoints = scratch ? 0 : dirtyPoints(day);
  const remaining = backupPoints > 0 ? MISSED : 0;   // remains even if eradication reruns
  const a = staged(stageCount);
  const restore = POINTS.find(([g]) => g === day)[1];
  return { name, day, backupPoints, remaining, lost: scratch ? lost(0.25) : lost(day),
    manual: scratch ? SCRATCH_MANUAL : manual(day),
    min: (scratch ? SCRATCH_MIN : restore) + (backupPoints ? CLEANUP_MIN : 0) + a.min,
    captured: +(remaining * a.capture).toFixed(1), escaped: +(remaining * a.escaped).toFixed(2),
    stage: stageCount, observe: a.min, rate: a.capture };
}

The measurement file first ranks the seven recovery points, then runs five recovery regimes against the same event and scans the stage count.

// recovery/measure.mjs — which backup is clean to return to, how many items are lost,
// how many items are manually rebuilt, and where the supervised restoration stages catch reinfection.
import { FIRST_FOOTHOLD, ESTABLISHED, VOLUME, DAILY, POINTS, STAGES, REGIMES, CLEANUP_MIN,
  MISSED, dirtyPoints, lost, manual, staged, measure } from "./model.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const t0 = process.hrtime.bigint();
const CLEAN_DAY = POINTS.find(([g]) => dirtyPoints(g) === 0 && g < FIRST_FOOTHOLD + 1)[0];

console.log(`first foothold ${FIRST_FOOTHOLD} days ago; ${ESTABLISHED.length} footholds, ` +
  `daily transaction volume ${DAILY} items (` +
  Object.entries(VOLUME).map(([a, v]) => `${a} ${v}`).join(", ") + ")");

console.log("\n1. recovery points");
const A = [-10, 8, 20, 14, 8, 15];
write(A, "days ago", "clean", "footholds in backup", "lost items", "manual", "restore");
for (const [g, dk] of POINTS)
  write(A, g, dirtyPoints(g) === 0 ? "yes" : "no", dirtyPoints(g), lost(g), manual(g), dk + " min");

console.log(`\n2. five recovery regimes (in a dirty backup, eradication reruns from scratch ` +
  `for ${CLEANUP_MIN} min, and ${MISSED} points are missed again)`);
const B = [-28, 10, 10, 14, 8, 12, 15];
write(B, "regime", "in backup", "remaining", "lost items", "manual", "total min", "escaped points");
const R = REGIMES.map((d) => measure(d, STAGES.length));
for (const r of R)
  write(B, r.name, r.backupPoints, r.remaining, r.lost, r.manual, r.min, r.escaped.toFixed(2));

console.log("\n3. supervised restoration stages");
const C = [-31, 12, 12, 18, 16];
write(C, "stage", "observe min", "capture", "cumul. capture", "cumul. minutes");
for (const [i, [name, dk, y]] of STAGES.entries()) {
  const a = staged(i + 1);
  write(C, name, dk, y.toFixed(2), a.capture.toFixed(4), a.min);
}

console.log("\n4. expected points reaching production by stage count");
const D = [-8, 12, 18, 16, 14, 14];
write(D, "stages", "observe min", "cumul. capture", "latest backup", "clean backup", "from scratch");
for (let n = 0; n <= STAGES.length; n++) {
  const a = staged(n);
  write(D, n, a.min, a.capture.toFixed(4), measure(REGIMES[0], n).escaped.toFixed(2),
    measure(REGIMES[3], n).escaped.toFixed(2), measure(REGIMES[4], n).escaped.toFixed(2));
}

console.log("\n5. difference between the two clean options");
const t = measure(REGIMES[3], STAGES.length), s = measure(REGIMES[4], STAGES.length);
const E = [-27, 14, 13, 12];
write(E, "measure", "clean backup", "from scratch", "difference");
write(E, "lost items", t.lost, s.lost, t.lost - s.lost);
write(E, "manual items", t.manual, s.manual, t.manual - s.manual);
write(E, "total minutes", t.min, s.min, t.min - s.min);
write(E, "points reaching production", t.escaped.toFixed(2), s.escaped.toFixed(2),
  (t.escaped - s.escaped).toFixed(2));

console.log("\n6. loss per service from returning to the clean backup");
const F = [-18, 12, 14, 10];
write(F, "service", "daily", `${CLEAN_DAY}-day`, "share");
for (const [name, v] of Object.entries(VOLUME))
  write(F, name, v, Math.round(v * CLEAN_DAY), (100 * v / DAILY).toFixed(1) + "%");

const ms = Number(process.hrtime.bigint() - t0) / 1e6;   // the raw measurement is not printed, only the band is checked
console.log(`\nfive regimes and the five stage counts within the 500 ms band: ${ms < 500 ? "yes" : "no"}`);

first foothold 9 days ago; 22 footholds, daily transaction volume 9970 items (meter reading 8400, invoice item 1250, work order 320)

1. recovery points
days ago     clean footholds in backup    lost items  manual        restore
21             yes                   0        209370      90        260 min
14             yes                   0        139580      62        245 min
9.5            yes                   0         94715      44        240 min
7               no                   7         69790      34        215 min
3               no                  17         29910      18        190 min
1               no                  22          9970      10        175 min
0.25            no                  22          2493       7        165 min

2. five recovery regimes (in a dirty backup, eradication reruns from scratch for 642 min, and 5 points are missed again)
regime                       in backup remaining    lost items  manual   total min escaped points
return to latest backup             22         5          2493       7        1347           0.10
return to 1 day ago                 22         5          9970      10        1357           0.10
return to 3 days ago                17         5         29910      18        1372           0.10
return to clean backup               0         0         94715      44         780           0.00
rebuild from scratch + data          0         0          2493      34         970           0.00

3. supervised restoration stages
stage                           observe min     capture    cumul. capture  cumul. minutes
opening on an isolated network          120        0.55            0.5500             120
supervised limited traffic              180        0.70            0.8650             300
partial traffic                         240        0.85            0.9798             540
full traffic                              0        0.00            0.9798             540

4. expected points reaching production by stage count
stages   observe min    cumul. capture   latest backup  clean backup  from scratch
0                  0            0.0000            5.00          0.00          0.00
1                120            0.5500            2.25          0.00          0.00
2                300            0.8650            0.68          0.00          0.00
3                540            0.9798            0.10          0.00          0.00
4                540            0.9798            0.10          0.00          0.00

5. difference between the two clean options
measure                      clean backup from scratch  difference
lost items                          94715         2493       92222
manual items                           44           34          10
total minutes                         780          970        -190
points reaching production           0.00         0.00        0.00

6. loss per service from returning to the clean backup
service                  daily       9.5-day     share
meter reading             8400         79800     84.3%
invoice item              1250         11875     12.5%
work order                 320          3040      3.2%

five regimes and the five stage counts within the 500 ms band: yes

The numbers belong to the measurement class; their inputs are the assumptions above.

Which Backup Is Clean

The first table splits the seven recovery points along the axis of cleanliness. Three are clean: 21, 14, and 9.5 days ago. The newest clean point is 9.5 days back, and returning to it costs 94,715 items of data and 44 manually rebuilt items. Four points are dirty, and the number of footholds they contain rises toward the present: 7 at seven days ago, 17 at three days ago, 22 at one day ago. Lost items and the foothold count in the backup move in opposite directions — there is no single row in this table that is a good option.

Restore minutes reflect none of this distinction: the oldest point takes 260 minutes, the newest 165. When recovery time is measured on its own, the latest backup looks like the best option. The decision depends not on restore minutes but on what the backup contains.

Five Regimes

The second table runs all five regimes together. In the three regimes that return to a dirty backup, eradication reruns from scratch and adds 642 minutes; total time falls between 1,347 and 1,372 minutes, and in all three, five points remain missed. Returning to the latest backup loses only 2,493 items of data, but it takes 1,347 minutes and makes eradication redo all of its work. Recovery can undo the work eradication has done.

Two regimes are clean. Returning to the clean backup is the fastest at 780 minutes and brings back no points at all, but it costs 94,715 items of data and 44 manually rebuilt items. Rebuilding from scratch gives the same cleanliness in 970 minutes; lost items drop to 2,493 and manually rebuilt items to 34. The fifth table writes out the difference item by item: for 190 more minutes, 92,222 items of data and 10 manually rebuilt items are gained. The 190 minutes paid in recovery is cheaper than nine and a half days of transaction data.

Staged Restoration

The third table breaks supervised restoration into stages. Opening on an isolated network catches 0.55 with 120 minutes of observation, supervised limited traffic 0.70 with 180 minutes, partial traffic 0.85 with 240 minutes; cumulative capture is 0.5500, 0.8650, and 0.9798 respectively. The fourth stage is full traffic, its observation window is zero, and it adds nothing — production itself is not an observation stage.

The fourth table converts stage count into the expected points reaching production. In the regime that returns to the latest backup, five points remain; if no stage is applied all five reach production, with one stage this drops to 2.25, with two to 0.68, with three to 0.10. The cost is 540 minutes of observation. In the clean regimes, the remaining point count is zero, and the stage count changes nothing.

Two separate decisions follow from this. If a dirty backup was returned to, staged restoration is mandatory, and 540 minutes of observation brings the expected points reaching production down from five to 0.10 — one in fifty. If the source was clean, the stages gain nothing on this measure — what they gain is on a different axis, verifying that cleanliness was actually achieved, and this model does not measure that.

The stages’ minutes are large compared with recovery itself. Of the 970 minutes for rebuilding from scratch, 540 are observation; the restore itself takes 430 minutes. If the observation window is shortened, the service opens earlier but the capture rate falls: 0.8650 at 300 minutes with two stages, 0.9798 at 540 minutes with three. The last stage’s 240 minutes carries cumulative capture from 0.8650 to 0.9798 — in a dirty regime, that means bringing the expected points reaching production down from 0.68 to 0.10.

Where the Loss Falls

The sixth table breaks the 94,715-item loss from returning to the clean backup down by service. Meter reading holds 84.3 percent of the share at a daily 8,400 items and comes to 79,800 items over 9.5 days; invoice item is 11,875, work order 3,040 items. The loss from the three services does not carry the same weight, and this model counts them as equal. A meter reading is a measurement that can be read again later; an invoice item and a work order are the record of a commitment made to someone outside. The lost-item count carries none of this distinction — whoever makes the decision also has to see which service the number came from.

Summary

• Three of the seven recovery points are clean; the newest clean point is 9.5 days back and costs 94,715 items of data and 44 manually rebuilt items.
• In the three regimes that return to a dirty backup, eradication reruns from scratch for 642 minutes, total time rises to 1,347–1,372 minutes, and five points are missed again.
• Rebuilding from scratch, for 190 more minutes than the clean backup, gains 92,222 items of data and 10 manually rebuilt items.
• Supervised restoration’s three observation stages catch a cumulative 0.9798 in 540 minutes; the fourth stage is full traffic and adds nothing.
• The expected points reaching production in a dirty regime is 5.00 with no stages, 2.25 with one stage, and 0.10 with three; in clean regimes it is zero regardless of stage count.

Next Step

The service is back and the expected reinfection is down to 0.10. What remains is the incident itself: what happened, when it began, which decision was made at which minute, and which item, done differently next time, would shorten the timeline. The next lesson takes up lessons learned along the axis of evidence and scope: which questions can be answered, which stay unanswered for lack of a record, and how many of the improvement items opened rest directly on a missing record.