--- title: 'Disaster Recovery Drill' source: 'https://academia.sh/en/courses/resilience-patterns/disaster-recovery-drill' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:31+00:00' license: 'CC BY-SA 4.0' --- # Disaster Recovery Drill Testing the recovery plan with a controlled failure trial: comparing the plan's step durations against what the drill measures, surfacing a step the plan never wrote down, showing the recovery time objective holding by plan but breaking by drill, and counting the duplicated side effects and orphaned workflows that two delivery policies leave behind for workflows left half-done in the failover window. This topic built a recovery arrangement over five lessons: the failover threshold and failback policy, the health endpoint's content and interval, the recovery time and data loss objectives, zone and region placement, stamps and geo-replicas. Every lesson's numbers came from a model or arithmetic, and all of them assumed the same thing: **the plan will work as written.** This assumption is a guess until it is tested. A **disaster recovery drill** tests the plan with a controlled failure trial: a specific unit, at a chosen time, is deliberately taken offline, and how recovery actually works is measured. A drill's scope is defined by three things — which unit (replica, zone, region, stamp), which failure mode (stop, slowdown, partition), and how long. A trial run without a written scope cannot say what it measured either. ## The Gap Between Plan and Measurement A recovery plan is a list of steps, and each step is assigned a duration. The drill's first output is a comparison of these durations. The measured values in the calculation below are not made up: they come from previous lessons' parameters. The detection duration is the threshold the first lesson chose; the warming duration is the warming window in the second lesson's failure schedule. **KU8 — the routing record's lifetime is 3 rounds.** Rationale: routing to the new active replica does not complete until the old replica's record expires. **KU9 — the queue consumer's reconnection is 2 rounds.** Both are assumptions and are not added to K01's table. ```js // drill/plan.mjs — comparing the recovery plan's step durations against what the drill measured. // Measured values come from previous lessons' parameters; a round is an abstract step. const THRESHOLD = 3; // lesson 01: the selected trigger threshold const PROMOTION = 1; // lesson 01: promotion round const LIFETIME = 3; // KU8 (assumption): the routing record's lifetime (rounds) const WARMING = 8; // lesson 02 schedule: a replica's warming duration (rounds) const QUEUE = 2; // KU9 (assumption): the queue consumer's reconnection (rounds) const VALIDATION = 2; // the validation step written into the plan const ROUND_SEC = 2; // KU1 (assumption): one round is 2 seconds const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PLANNED = 15.0; // K01: monthly budget's planned-work share (minutes) const b = (x, n = 2) => x.toFixed(n); const STEP = [ ["detection", 1, THRESHOLD], ["promotion", 1, PROMOTION], ["routing", 1, LIFETIME], ["cache-warming", 0, WARMING], ["queue-reconnect", 1, QUEUE], ["validation", 2, VALIDATION], ]; console.log(`${"step".padEnd(17)}${"planned".padStart(11)}${"measured".padStart(12)}${"deviation".padStart(11)}`); let p = 0, o = 0; for (const [name, plan, measured] of STEP) { p += plan; o += measured; const d = measured - plan; console.log(`${name.padEnd(17)}${(plan + " round(s)").padStart(11)}${(measured + " round(s)").padStart(12)}` + `${((d >= 0 ? "+" : "") + d).padStart(11)}`); } console.log(`${"total".padEnd(17)}${(p + " round(s)").padStart(11)}${(o + " round(s)").padStart(12)}` + `${("+" + (o - p)).padStart(11)}`); console.log(`measured/planned ratio ${b(o / p)}x`); console.log(`\nplan ${p * ROUND_SEC} sec, measured in drill ${o * ROUND_SEC} sec (KU1: round = ${ROUND_SEC} sec)`); for (const target of [30, 60]) { const pl = p * ROUND_SEC <= target ? "holds" : "breaks", ol = o * ROUND_SEC <= target ? "holds" : "breaks"; console.log(` recovery time objective ${target} sec: ${pl} by plan, ${ol} by drill`); } console.log(` edge requests dropped in the drill window = ${b(o * ROUND_SEC * PEAK_EDGE)} (at peak load)`); console.log(`\nthe drill's own cost (one drill = ${b((o * ROUND_SEC) / 60)} min):`); console.log(`${"frequency".padEnd(14)}${"drills a month".padStart(16)}${"minutes a month".padStart(17)}` + `${"of planned share".padStart(19)}${"of monthly budget".padStart(19)}`); for (const [name, perMonth] of [["weekly", 4], ["monthly", 1], ["quarterly", 1 / 3]]) { const min = (perMonth * o * ROUND_SEC) / 60; console.log(`${name.padEnd(14)}${b(perMonth).padStart(16)}${b(min).padStart(17)}` + `${(b((100 * min) / PLANNED) + "%").padStart(19)}${(b((100 * min) / 43.2) + "%").padStart(19)}`); } console.log(`${STEP.filter(([, pl, ol]) => ol !== pl).length} of the plan's 6 steps measured a deviation; ` + `one (cache-warming) was not in the plan at all`); ``` ``` step planned measured deviation detection 1 round(s) 3 round(s) +2 promotion 1 round(s) 1 round(s) +0 routing 1 round(s) 3 round(s) +2 cache-warming 0 round(s) 8 round(s) +8 queue-reconnect 1 round(s) 2 round(s) +1 validation 2 round(s) 2 round(s) +0 total 6 round(s) 19 round(s) +13 measured/planned ratio 3.17x plan 12 sec, measured in drill 38 sec (KU1: round = 2 sec) recovery time objective 30 sec: holds by plan, breaks by drill recovery time objective 60 sec: holds by plan, holds by drill edge requests dropped in the drill window = 19527.82 (at peak load) the drill's own cost (one drill = 0.63 min): frequency drills a month minutes a month of planned share of monthly budget weekly 4.00 2.53 16.89% 5.86% monthly 1.00 0.63 4.22% 1.47% quarterly 0.33 0.21 1.41% 0.49% 4 of the plan's 6 steps measured a deviation; one (cache-warming) was not in the plan at all ``` The total row is the lesson's first result: the plan said 6 rounds, the drill measured 19 — 3.17 times as much. This ratio is a quantity derived independently of the run; it stays the same even if rounds are never converted to seconds. The deviation does not come from a single place. The detection and routing steps were written into the plan as one round each, yet the first depends on the selected threshold and the second on the routing record's lifetime — both are decisions **already made** in other lessons, and the plan never read them. The largest deviation, though, comes from a step that is not in the plan at all: the cache warming up. The new active replica is up, accepts writes, and passes the health check, but serves reads from a cold cache. **The plan's most expensive mistake is not a wrong estimate but a missing step.** The result hits the threshold. A thirty-second recovery time objective holds comfortably when read from the plan (12 seconds), and breaks when read from the drill (38 seconds). The sixty-second objective passes on both readings. Whether an objective holds cannot be read from the plan; the drill is the one step that turns an objective from a claim into a measured number. ## Scope and Frequency The drill's scope is a ladder, and each rung tests something different. The bottom rung is a paper review of the plan's steps; this can find a missing step but cannot measure duration — in the table above, cache warming being missing from the plan could have been caught this way, but its eight-round duration could not have been. The middle rung is taking a single component offline; it tests the failover threshold and the health endpoint's content. The top rung is the loss of an entire zone or region, and it is the only trial that tests the previous two lessons' placement decisions; only here is it seen whether idle capacity actually carries the real load. The frequency table ties the cost to frequency. A weekly drill takes 2.53 minutes a month — 16.89 percent of the planned-work share and 5.86 percent of the monthly budget. A monthly drill takes 0.63 minutes, a quarterly one 0.21 minutes. An infrequent drill is cheap, but the changes that accumulate between two drills age the plan again; the measured deviation grows from zero each time. This lesson gives the number, not the choice: the choice depends on how often the plan's steps change. ## The Second Problem the Drill Surfaces The drill's last step is validation: checking the state of the data and the in-progress work once recovery is done. Recovery time has been measured, but what happens to **workflows left half-done** during the failover window has not been counted. The setup below is a **model**: the end-of-day billing flow has eight steps, four of them have side effects (writing an invoice, posting a ledger entry, sending a notification, notifying the carrier), and the failover window is the 19 rounds just measured. ```js // drill/work.mjs — the drill's data-validation step is a MODEL: what the workflows left half-done // in the failover window leave behind under two policies. A round is an abstract step. const STEP = [ // steps of the billing workflow ["read-state", false], ["apply-tariff", false], ["compute-amount", false], ["write-invoice", true], ["post-ledger-entry", true], ["send-notification", true], ["notify-carrier", true], ["close", false], ]; const ROUND = 70, WORKFLOWS_PER_ROUND = 4, START_UNTIL = 40; // 4 workflows start each round, through round 40 const FAILOVER = 20, WINDOW = 19; // 19 rounds: the total measured in plan.mjs function run(policy) { const workflows = []; const s = { completed: 0, duplicatedSideEffects: 0, orphaned: 0, restarted: 0, discarded: 0 }; for (let t = 1; t <= ROUND; t += 1) { const frozen = t >= FAILOVER && t < FAILOVER + WINDOW; if (t === FAILOVER) { for (const w of workflows) { if (w.step === 0 || w.step >= STEP.length) continue; // not yet started or already finished if (policy === "at-least-once") { s.restarted += 1; s.duplicatedSideEffects += STEP.slice(0, w.step).filter(([, y]) => y).length; w.step = 0; // the workflow restarts from the beginning } else { s.discarded += 1; if (STEP.slice(0, w.step).some(([, y]) => y)) s.orphaned += 1; w.dead = true; } } } if (frozen) continue; // the workflow makes no progress during the window for (const w of workflows) { if (w.dead || w.step >= STEP.length) continue; w.step += 1; if (w.step === STEP.length) s.completed += 1; } if (t <= START_UNTIL) for (let i = 0; i < WORKFLOWS_PER_ROUND; i += 1) workflows.push({ step: 0, dead: false }); } return { ...s, total: workflows.length }; } const s = (x, n) => String(x).padStart(n); console.log(`${ROUND} rounds, ${WORKFLOWS_PER_ROUND} workflows start per round, each workflow has ${STEP.length} steps ` + `(${STEP.filter(([, y]) => y).length} of them have side effects). Failover at round ${FAILOVER}, ` + `window ${WINDOW} rounds.`); console.log(); console.log(`${"policy".padEnd(14)} | ${s("total workflows", 15)} | ${s("completed", 9)} | ${s("restarted", 9)} | ${s("discarded", 9)} | ${s("duplicated side effects", 24)} | ${s("orphaned", 8)}`); console.log(`${"-".repeat(14)}-|-${"-".repeat(15)}-|-${"-".repeat(9)}-|-${"-".repeat(9)}-|-${"-".repeat(9)}-|-${"-".repeat(24)}-|-${"-".repeat(8)}`); for (const p of ["at-least-once", "at-most-once"]) { const r = run(p); console.log(`${p.padEnd(14)} | ${s(r.total, 15)} | ${s(r.completed, 9)} | ` + `${s(r.restarted, 9)} | ${s(r.discarded, 9)} | ${s(r.duplicatedSideEffects, 24)} | ${s(r.orphaned, 8)}`); } ``` ``` 70 rounds, 4 workflows start per round, each workflow has 8 steps (4 of them have side effects). Failover at round 20, window 19 rounds. policy | total workflows | completed | restarted | discarded | duplicated side effects | orphaned ---------------|-----------------|-----------|-----------|-----------|--------------------------|--------- at-least-once | 84 | 84 | 28 | 0 | 40 | 0 at-most-once | 84 | 56 | 0 | 28 | 0 | 16 ``` The two rows show two bad outcomes, and there is no third option in between. In **at-least-once** delivery, every workflow completes: 84/84. In exchange, the 28 workflows that were half-done at the moment of failover run from the beginning, and side-effect steps already executed run a **second time** — 40 duplicated side effects. That means 40 extra invoice lines, ledger entries, or carrier notifications. In **at-most-once** delivery there is no duplication, but completed work drops from 84 to 56: 28 workflows are discarded, and 16 of them have already executed at least one side-effect step. That means an invoice was written for 16 shipments but no ledger entry was posted, or a notification was sent but the workflow never closed. These workflows were neither completed nor rolled back; they left a trace in the system, and nothing collects them. The length of the recovery window directly grows these numbers. Because the window is 19 rounds, 28 workflows were left half-done; with the plan's 6-round window, this number would have been smaller. **Shortening recovery time does not only protect the outage budget, it also shrinks the correctness damage** — two separate quantities measured by two lessons meet in the same place. ## The Numbers for Two Days **The failure-free day's cost** is the drill itself. A drill run once a month takes 38 seconds, that is, 0.63 minutes, and spends 4.22 percent of the 15.0 minutes K01 sets aside for planned work. If run at peak load, it drops 19,527.82 edge requests; this is why a drill's scope includes the time window too. The cost is real and paid voluntarily. **The failing day's gain** comes from reading both tables together. In an untested arrangement, the plan promised a 12-second recovery time; the real failure would have measured 38 seconds, and the 30-second objective would have broken on the first failure. The drill made this gap visible before the objective broke. It also surfaced a step that was not in the plan at all, and measured that four of six steps had the wrong duration written down. The same drill also counted a damage that has nothing to do with recovery time: 40 duplicated side effects, or 16 orphaned workflows. This comparison is the clearest example of this course's rule. The drill pays only cost on a failure-free day — 0.63 minutes and 19,527.82 requests. Its gain becomes visible only when a failure happens, and on that day it takes the form of not running into an unknown deviation. ## Summary - A drill tests the recovery plan with a controlled failure trial; its scope is defined by the questions of which unit, which failure mode, and how long. - Recovery written into the plan as 6 rounds measured 19 rounds in the drill (3.17 times); four of six steps had the wrong duration written down, and the largest deviation came from cache warming, which was not in the plan at all. - An objective cannot be tested from the plan: the 30-second recovery time objective passes by plan, and breaks against the 38 seconds measured in the drill. - The drill's cost depends on frequency: a monthly drill costs 0.63 minutes (4.22 percent of the planned share), a weekly one 2.53 minutes (16.89 percent); at peak load, every drill drops 19,527.82 edge requests. - The validation step surfaced a second damage: 28 workflows were left half-done in the 19-round window; in at-least-once delivery all of them completed but 40 side effects ran a second time, in at-most-once delivery there was no duplication but 16 workflows were left orphaned. - Shortening the recovery window does not only protect the outage budget; it also reduces correctness damage by shrinking the number of workflows left half-done. ## Next Step The drill validated the recovery arrangement and, at the same time, showed its limit. Failover completed, objectives were measured, placement was tested — but the last table left something behind that none of this topic's patterns solves. **The same work ran twice:** 40 side-effect steps ran again, because the new active replica did not know where the work had left off and started it from the beginning. The same problem arises not only at failover but also in the previous topic's retries; every resent request may already have been processed. At the other end, **the orphaned workflows were never rolled back:** 16 workflows stopped after leaving one side effect, and no step was designed to collect them. The next topic takes on both questions: making a request's second processing harmless, and compensating for steps that cannot be rolled back.