Business Continuity and Disaster Recovery

The previous three lessons each measured something that flows continuously, and in all three the measurement was taken while normal operations continued. This lesson’s subject is the moment normal operations stop. There is no measurement on hand there, only a plan: which service comes back up in how many minutes, in what order, and who does which step is written down. Being written down is not the same as being verified.

Backup types, restore mechanics and the definition of recovery objectives were measured in the Information Technology and Systems Fundamentals course; here they are taken as input. This lesson’s measure is a single difference: the gap between plan and reality. It is written with three numbers — how many items’ assumptions fail to hold in the drill, how many minutes separate the target time from the measured time, and which item those minutes come from. The third question is independent of the first two, because one item’s delay may not be reflected in recovery time.

• SP19. The twelve recovery items, the dependency graph, the plan durations and the assumptions that fail to hold in the drill all belong to the fictional regional measurement network and are a model. No real recovery is being performed.
• SP20. An item starts only after all of its dependencies finish, and teams can work in parallel. The recovery team’s headcount was not treated as limited; if it were limited, the measured duration would come out longer.
• SP21. The minutes added by a failed assumption depend on the assumption type and do not vary by item. In reality the same assumption can cost one item ten minutes and another an hour.
• SP22. The drill actually runs every item. Tabletop exercises never test some assumptions at all; in that case the number of items that fail to hold looks lower than it really is.
• SP23. The fix cost is expressed in person-days and is fictional; fixes do not make each other cheaper.
• SP24. The data-loss objective was not measured in this lesson; the only thing measured is time.

// continuity/model.mjs — MODEL of the recovery plan: fictional regional measurement network's
// twelve recovery items, dependency graph, plan duration and the assumptions that fail to hold
// in the drill. No real recovery is being performed; backup types and recovery mechanics are
// taken as input.
export const TARGET_MIN = 300;                      // recovery target the plan declares (minutes)
// Assumption type: minutes added to an item's duration and the fix cost when it fails to hold in the drill
export const ASSUMPTION = {
  'authorized access': { min: 25, days: 2 },
  'current procedure': { min: 18, days: 3 },
  'readable backup': { min: 40, days: 5 },
  'contact list': { min: 12, days: 1 },
  'backup capacity': { min: 30, days: 8 },
};
// item: name, plan duration (min), dependencies, assumptions that FAIL TO HOLD in the drill
export const ITEM = [
  { name: 'backup power', plan: 25, deps: [], failed: [] },
  { name: 'network backbone', plan: 35, deps: ['backup power'], failed: ['current procedure'] },
  { name: 'identity service', plan: 40, deps: ['network backbone'], failed: ['authorized access'] },
  { name: 'primary database', plan: 60, deps: ['network backbone'],
    failed: ['readable backup', 'authorized access'] },
  { name: 'intake service', plan: 30, deps: ['identity service', 'primary database'],
    failed: ['current procedure'] },
  { name: 'validation service', plan: 25, deps: ['intake service'], failed: [] },
  { name: 'billing', plan: 45, deps: ['primary database', 'validation service'],
    failed: ['backup capacity'] },
  { name: 'work order service', plan: 30, deps: ['identity service', 'primary database'],
    failed: ['contact list', 'current procedure'] },
  { name: 'field application', plan: 20, deps: ['work order service'],
    failed: ['readable backup', 'contact list'] },
  { name: 'subscriber portal', plan: 35, deps: ['identity service', 'billing'], failed: [] },
  { name: 'monitoring & logging', plan: 20, deps: ['network backbone'], failed: ['authorized access'] },
  { name: 'report generation', plan: 40, deps: ['billing'], failed: ['backup capacity'] },
];
export const delay = (t) => t.reduce((a, v) => a + ASSUMPTION[v].min, 0);

// Latest finish in the dependency graph: duration function item -> minutes
export function finish(duration) {
  const f = new Map();
  const compute = (name) => {
    if (f.has(name)) return f.get(name);
    const item = ITEM.find((x) => x.name === name);
    const start = item.deps.reduce((a, d) => Math.max(a, compute(d)), 0);
    f.set(name, start + duration(item));
    return f.get(name);
  };
  for (const item of ITEM) compute(item.name);
  return f;
}
export const total = (duration) => Math.max(...finish(duration).values());
export const planDuration = (item) => item.plan;
export const measuredDuration = (closed = new Set()) => (item) =>
  item.plan + delay(item.failed.filter((v) => !closed.has(`${item.name}|${v}`)));

// Critical path: walk backward from the item whose finish time equals the total
export function criticalPath(duration) {
  const f = finish(duration), last = Math.max(...f.values());
  const path = []; let name = [...f.keys()].find((x) => f.get(x) === last);
  while (name) { path.unshift(name);
    const item = ITEM.find((x) => x.name === name);
    name = item.deps.slice().sort((p, q) => f.get(q) - f.get(p))[0]; }
  return path;
}
// Slack: minutes an item can slip without entering the critical path
export function slack(duration) {
  const f = finish(duration), last = Math.max(...f.values());
  const successors = new Map(ITEM.map((item) => [item.name, []]));
  for (const item of ITEM) for (const d of item.deps) successors.get(d).push(item.name);
  const late = new Map();
  const compute = (name) => {
    if (late.has(name)) return late.get(name);
    const s = successors.get(name);
    late.set(name, s.length ? Math.min(...s.map((x) =>
      compute(x) - duration(ITEM.find((z) => z.name === x)))) : last);
    return late.get(name);
  };
  return new Map(ITEM.map((item) => [item.name, compute(item.name) - f.get(item.name)]));
}
export const CANDIDATE = ITEM.flatMap((item) => item.failed.map((v) => ({ item: item.name, assumption: v,
  key: `${item.name}|${v}`, min: ASSUMPTION[v].min, days: ASSUMPTION[v].days })));

Reported Delay versus Reflected Delay

// continuity/drill.mjs — the gap between plan and reality: items that fail to hold, the gap
// between target and measured duration, and how much of that delay is reflected in recovery time
import { ITEM, TARGET_MIN, CANDIDATE, delay, total, criticalPath, slack,
  planDuration, measuredDuration } from './model.mjs';

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());
const planTotal = total(planDuration), measuredTotal = total(measuredDuration());
const pathPlan = criticalPath(planDuration), pathMeasured = new Set(criticalPath(measuredDuration()));
const itemSlack = slack(measuredDuration());

console.log(`${ITEM.length} recovery items, declared target ${TARGET_MIN} min`);
console.log(`plan critical path ${planTotal} min (${TARGET_MIN - planTotal} min under target), ` +
  `measured in drill ${measuredTotal} min (${measuredTotal - TARGET_MIN} min over target); difference ${measuredTotal - planTotal} min`);
console.log(`items carrying a failed assumption: ` +
  `${ITEM.filter((item) => item.failed.length).length}/${ITEM.length}, ` +
  `${CANDIDATE.length} assumptions total, total reported delay ` +
  `${ITEM.reduce((a, item) => a + delay(item.failed), 0)} min`);

const A = [-20, 7, 9, 10, 10, 9, 9];
console.log('\n1. plan, measured and slack per item');
y(A, 'item', 'plan', 'measured', 'delay', 'critical', 'slack', 'impact');
for (const item of ITEM) {
  const g = delay(item.failed);
  const solo = total(measuredDuration(new Set(item.failed.map((v) => `${item.name}|${v}`))));
  y(A, item.name, item.plan, item.plan + g, g, pathMeasured.has(item.name) ? 'yes' : 'no', itemSlack.get(item.name), measuredTotal - solo);
}
console.log(`\ncritical path (same in plan and drill): ${pathPlan.join(' -> ')}`);
console.log(`delay carried by items outside the critical path: ` +
  `${ITEM.filter((item) => !pathMeasured.has(item.name)).reduce((a, item) => a + delay(item.failed), 0)} min`);

12 recovery items, declared target 300 min
plan critical path 260 min (40 min under target), measured in drill 421 min (121 min over target); difference 161 min
items carrying a failed assumption: 9/12, 12 assumptions total, total reported delay 293 min

1. plan, measured and slack per item
item                   plan measured     delay  critical    slack   impact
backup power             25       25         0       yes        0        0
network backbone         35       53        18       yes        0       18
identity service         40       65        25        no       60        0
primary database         60      125        65       yes        0       60
intake service           30       48        18       yes        0       18
validation service       25       25         0       yes        0        0
billing                  45       75        30       yes        0       30
work order service       30       60        30        no       86        0
field application        20       72        52        no       86        0
subscriber portal        35       35         0        no       35        0
monitoring & logging     20       45        25        no      298        0
report generation        40       70        30       yes        0       30

critical path (same in plan and drill): backup power -> network backbone -> primary database -> intake service -> validation service -> billing -> report generation
delay carried by items outside the critical path: 132 min

The top lines show the plan hitting its target on paper: the critical path is 260 minutes, the declared target is 300 minutes, a forty-minute margin. Running the same graph with real durations in the drill produces 421 minutes — 121 minutes over the target. At least one assumption fails to hold in nine of the twelve items.

The real distinction is in the last two columns. The drill report shows 293 minutes of delay, but only 161 minutes of that delay are reflected in recovery time. The remaining 132 minutes sit in items outside the critical path. The monitoring & logging item has 298 minutes of slack; its 25-minute delay does not extend recovery time by even one minute. The field application has a 52-minute delay and 86 minutes of slack; its impact is likewise zero. A drill report’s total delay line does not say how much longer recovery took.

The impact column has its own trap. The five critical-path items’ individual impacts are 18, 60, 18, 30 and 30 minutes; they sum to 156. Yet fixing all of them at once yields a gain of 161 minutes. Each individual measurement is bounded by the second-longest path that takes over once that item is fixed; a list that evaluates items one at a time therefore shows a total gain that is smaller than it actually is.

It is also worth noting that the critical path stayed the same in the plan and in the drill. The delays were large, but not large enough to change the order; the plan’s assumption about which item was the bottleneck held, its assumption about durations did not. The two assumptions are tested separately and should be written up separately in the drill report.

Same Budget, Three Rankings

// continuity/fix.mjs — ranks the drill findings by three different measures and runs them
// against the same budget
import { TARGET_MIN, CANDIDATE, total, planDuration, measuredDuration } from './model.mjs';

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());
const measuredTotal = total(measuredDuration());
// ties are broken by a fixed item order, so the ranking does not depend on how the names are spelled
const TIE_ORDER = ['subscriber portal', 'network backbone', 'intake service', 'validation service', 'billing',
  'monitoring & logging', 'work order service', 'identity service', 'report generation', 'field application',
  'primary database', 'backup power'];
const rank = (name) => TIE_ORDER.indexOf(name);
const margin = (a, closed) => total(measuredDuration(closed)) - total(measuredDuration(new Set([...closed, a.key])));

const RULE = {
  'report delay': (a) => a.min,
  'critical path gain': (a, closed) => margin(a, closed),
  'gain/day': (a, closed) => margin(a, closed) / a.days,
};
function run(name, budget) {
  const rule = RULE[name], closed = new Set(); let remaining = budget, n = 0, list = [...CANDIDATE];
  while (remaining > 0 && list.length) {
    list.sort((p, q) => rule(q, closed) - rule(p, closed) || rank(p.item) - rank(q.item));
    const a = list.find((x) => x.days <= remaining);
    if (!a) break;
    remaining -= a.days; closed.add(a.key); n += 1; list = list.filter((x) => x !== a);
  }
  return { n, days: budget - remaining, duration: total(measuredDuration(closed)) };
}

console.log(`measured in drill ${measuredTotal} min, plan ${total(planDuration)} min, target ${TARGET_MIN} min; ` +
  `${CANDIDATE.length} fix candidates, closing all of them costs ${CANDIDATE.reduce((a, x) => a + x.days, 0)} person-days`);
const A = [-8, 20, 10, 7, 10, 11];
console.log('\n1. same budget, three different ranking measures');
y(A, 'budget', 'measure', 'fixes', 'days', 'time min', 'gain min');
for (const B of [6, 12, 20])
  for (const name of Object.keys(RULE)) {
    const r = run(name, B);
    y(A, B, name, r.n, r.days, r.duration, measuredTotal - r.duration);
  }

console.log('\n2. smallest budget needed to drop below the target');
for (const name of Object.keys(RULE)) {
  let g = 1; while (g <= 43 && run(name, g).duration > TARGET_MIN) g += 1;
  console.log(`   ${name.padEnd(20)} ${g} person-days`);
}
console.log('\n3. two largest delays in the report');
for (const a of [...CANDIDATE].sort((p, q) => q.min - p.min || rank(p.item) - rank(q.item)).slice(0, 2))
  console.log(`   ${a.item} / ${a.assumption}: reported ${a.min} min, ${a.days} person-days, ` +
    `reflected in recovery time ${margin(a, new Set())} min`);

measured in drill 421 min, plan 260 min, target 300 min; 12 fix candidates, closing all of them costs 43 person-days

1. same budget, three different ranking measures
budget               measure     fixes   days  time min   gain min
6               report delay         2      6       421          0
6         critical path gain         2      6       381         40
6                   gain/day         3      6       378         43
12              report delay         3     12       381         40
12        critical path gain         4     12       338         83
12                  gain/day         4     12       338         83
20              report delay         4     20       351         70
20        critical path gain         5     20       308        113
20                  gain/day         8     19       320        101

2. smallest budget needed to drop below the target
   report delay         32 person-days
   critical path gain   25 person-days
   gain/day             21 person-days

3. two largest delays in the report
   field application / readable backup: reported 40 min, 5 person-days, reflected in recovery time 0 min
   primary database / readable backup: reported 40 min, 5 person-days, reflected in recovery time 40 min

The first triplet of rows carries this lesson’s harshest result. When a six-person-day budget is spent by report-delay order, two fixes get done, the whole budget is gone, and recovery time stays at 421 minutes: zero gain. The same budget spent by critical-path-gain order yields 40 minutes, and by the order that divides gain by days it yields 43 minutes. All three rankings use the same twelve candidates, the same report and the same budget.

The reason is written in the third block. The report’s two largest delays are both forty minutes and both cost five person-days; the first one’s impact on recovery time is zero, the second one’s is 40 minutes. The number in the report does not give the fix’s value; the value only emerges once the item is placed on the dependency graph.

In the twenty-person-day row, the two smarter measures swap places. Up to twelve person-days, critical-path gain and gain/day give the same result (338 minutes); at twenty person-days gain/day stays at 320 minutes while critical-path gain drops to 308. This is the point where the ranking changes: as the budget grows, the measure that puts cheap items first runs out of budget before it gets around to the expensive item that gains more on its own. At a tight budget the efficiency measure leads; at a wide budget the direct-gain measure leads.

The last block ties the decision to a single number. Dropping below the target costs 32 person-days by report-delay order, 25 by critical-path gain, 21 by gain/day order. The findings are the same, the plan is the same, the team is the same — the 11-person-day gap comes from the ranking measure alone.

Summary

• The plan’s critical path stays under the declared 300-minute target at 260 minutes; the drill measures 421 minutes and overshoots the target by 121 minutes. At least one assumption fails to hold in nine of the twelve items.
• Only 161 of the drill report’s 293 minutes of delay are reflected in recovery time; the remaining 132 minutes sit in items that have slack (298 minutes of slack in the monitoring & logging item).
• The critical-path items’ individual impacts total 156 minutes; fixing all of them at once yields 161 minutes: evaluating them individually shows a smaller gain because of the second path that takes over.
• The critical path stayed the same in the plan and the drill: the bottleneck assumption held, the duration assumption did not. The two assumptions are tested separately and written up separately.
• When six person-days are spent by report-delay order the gain is zero; by critical-path-gain order it is 40 minutes, by gain/day order 43 minutes. The report’s two largest delays are both 40 minutes and 5 person-days; one has an impact of 40 minutes, the other 0.
• At twenty person-days the two measures swap places (308 versus 320 minutes); dropping below the target costs 32, 25 and 21 person-days — the 11-person-day gap comes from the ranking measure alone.

Next Step

Four lessons produced four different rankings, and the same pattern repeated in every one: a set of items, a measure, and an order that changes when the measure changes. This lesson’s report-delay column was the extreme case — it was a real number, correctly measured, written in the drill report, and it spent a six-person-day budget for zero gain. A column like that is not wrong; it just carries no decision. The final lesson separates every program metric through this lens: which metric is tied to a decision and what happens once it crosses a threshold, which one only reaches the report, and how the list gets shorter once the cost of collecting a metric is placed side by side with the number of decisions it triggers.