--- title: 'Recovery Objectives' source: 'https://academia.sh/en/courses/resilience-patterns/recovery-objectives' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:32+00:00' license: 'CC BY-SA 4.0' --- # Recovery Objectives Defining the recovery time objective and the data loss objective as two separate thresholds: converting the data loss objective into a number of lost events via the backup interval and replication lag, computing how many failures the recovery time objective fits into K01's monthly failure share, and counting the failure-free-day cost of driving data loss to zero. The previous two lessons counted how long a failure takes to notice and to hand over: threshold, interval, promotion, and failback rounds. What they measured was always the same quantity, the time for the service to come back. But the 3,888.80 writes lost in the first lesson's half-failed window are a different quantity that neither lesson measured: how much data is left behind when the system comes back? This lesson names two thresholds and separates them. The **recovery time objective** is the longest time allowed to pass from a failure's start to the service's return. The **data loss objective** is the most data loss accepted after recovery, written as a duration: "at most one minute of writes". Both are **thresholds**. The `time to recovery` from the Introduction to System Design course is a **measurement** — how long a failure that actually happened took; the objective here is the permitted upper bound, set in advance as a design decision. The same separation holds for service availability: the measured percentage and the targeted percentage are separate numbers, and one does not validate the other. ## Two Thresholds Held by Separate Mechanisms The recovery time objective is held by **failover**: the sum of detection, promotion, and routing rounds must stay under the target. The data loss objective is held by **data replication**: it determines how close the recovered replica's data is to the moment of failure. A system can fail over fast and lose a lot of data, or lose no data and recover slowly. A recovery claim only carries meaning with both numbers together: "failover completed in eight seconds" is incomplete without "and how many events were lost". Two inputs convert the data loss objective into a number. The **backup interval**: in the worst case, the failure happens right before the next backup, and a full interval's worth of writes is lost. **Replication lag**: under asynchronous replication, writes that left the source but never reached the target are lost. Replication's mechanics were established in the Scaling the Data Layer course and are not repeated here; the lag is only an input here. ```js // objective/loss.mjs — tying the data loss objective to the number of lost events via the backup // interval and replication lag. All arithmetic; the inputs are K01's computed values and this // lesson's assumptions. const DAY = 86_400; const EVENTS_DAY = 2_800_000; // K01: daily state events (V3 x V4 = 400,000 x 7) const EVENTS_SEC = EVENTS_DAY / DAY; // average write rate const PEAK = 97.22; // K01: peak write requests/s const EVENTS_SHIPMENT = 7; // K01: state events per shipment const GROWTH_GB = 0.976; // K01: daily data growth (976 MB) const FULL_GB = 712.48; // K01: stored data const b = (x, n = 2) => x.toFixed(n); console.log(`average write ${b(EVENTS_SEC)} events/s, peak ${PEAK} events/s`); console.log(`${"backup interval".padStart(18)}${"worst-case loss".padStart(17)}` + `${"lost events (avg)".padStart(19)}${"lost events (peak)".padStart(20)}${"lost shipments".padStart(16)}`); for (const [name, sec] of [["24 hr", 86_400], ["6 hr", 21_600], ["1 hr", 3600], ["15 min", 900], ["5 min", 300], ["1 min", 60]]) { const avg = EVENTS_SEC * sec, peak = PEAK * sec; console.log(`${name.padStart(18)}${(sec + " sec").padStart(17)}` + `${Math.round(avg).toLocaleString("en-US").padStart(19)}` + `${Math.round(peak).toLocaleString("en-US").padStart(20)}` + `${Math.round(avg / EVENTS_SHIPMENT).toLocaleString("en-US").padStart(16)}`); } console.log(`\n${"replication lag".padStart(19)}${"data loss objective".padStart(21)}` + `${"lost events (peak)".padStart(20)}${"lost shipments".padStart(16)}`); for (const L of [0, 0.5, 1, 5, 10]) { console.log(`${(b(L, 1) + " sec").padStart(19)}${(b(L, 1) + " sec").padStart(21)}` + `${b(PEAK * L).padStart(20)}${b((PEAK * L) / EVENTS_SHIPMENT).padStart(16)}`); } console.log(`\n${"backup interval".padStart(18)}${"backups/day".padStart(13)}` + `${"backups/30 days".padStart(17)}${"incremental GB/30 days".padStart(24)}${"full backup GB (weekly)".padStart(25)}`); for (const [name, sec] of [["24 hr", 86_400], ["1 hr", 3600], ["5 min", 300]]) { const perDay = DAY / sec; console.log(`${name.padStart(18)}${b(perDay, 0).padStart(13)}${b(perDay * 30, 0).padStart(17)}` + `${b(GROWTH_GB * 30).padStart(24)}${b(FULL_GB * 5).padStart(25)}`); } console.log(`stored backup data does not change with the interval: only the backup operation count changes`); ``` ``` average write 32.41 events/s, peak 97.22 events/s backup interval worst-case loss lost events (avg) lost events (peak) lost shipments 24 hr 86400 sec 2,800,000 8,399,808 400,000 6 hr 21600 sec 700,000 2,099,952 100,000 1 hr 3600 sec 116,667 349,992 16,667 15 min 900 sec 29,167 87,498 4,167 5 min 300 sec 9,722 29,166 1,389 1 min 60 sec 1,944 5,833 278 replication lag data loss objective lost events (peak) lost shipments 0.0 sec 0.0 sec 0.00 0.00 0.5 sec 0.5 sec 48.61 6.94 1.0 sec 1.0 sec 97.22 13.89 5.0 sec 5.0 sec 486.10 69.44 10.0 sec 10.0 sec 972.20 138.89 backup interval backups/day backups/30 days incremental GB/30 days full backup GB (weekly) 24 hr 1 30 29.28 3562.40 1 hr 24 720 29.28 3562.40 5 min 288 8640 29.28 3562.40 stored backup data does not change with the interval: only the backup operation count changes ``` ## Backups Alone Cannot Hold an Objective The first table shows a difference of orders of magnitude. Daily backups lose 2,800,000 state events in the worst case — a full day's shipment traffic; hourly backups lose 116,667 events and 16,667 shipments. Even at five minutes, 9,722 events are lost. These numbers reflect not a shortcoming but a **definition**: a backup is a copy of specific moments, and everything between two moments falls outside the copy. The second table gives the same quantity on the replication side, and the comparison is stark: a one-second lag loses 97.22 events at peak load — 300 times less than a five-minute backup interval's loss. **In a stream taking continuous writes, a meaningful data loss objective is held by replication, not backups**; a backup's job is covering a different failure mode — data overwritten with the wrong content, a case replication faithfully copies. The third table shows where tightening the backup interval costs nothing. With incremental backups, the data stored over thirty days is 29.28 GB at all three intervals, because what is stored is the change itself, independent of the interval. The 3,562.40 GB of weekly full backups is fixed too. The only number that changes is the backup operation count: 1 to 288 a day. Tightening the interval bills operation count and coordination, not storage. ## The Recovery Time Objective Looks at the Budget The recovery time objective cannot be chosen alone, because the objective and the monthly outage budget share the same resource. K01 established the budget: the 99.9% target gives 43.2 minutes a month, 15.0 minutes of which go to planned work, the remaining 28.2 to failure. The percentage arithmetic is not repeated here; these three numbers are inputs. ```js // objective/budget.mjs — comparing the recovery time objective against K01's monthly outage // budget, and the failure-free-day cost of driving the data loss objective to zero const FAILURE_SHARE = 28.2; // K01: the outage budget's monthly failure share (minutes) const MONTHLY = 43.2; // K01: monthly outage budget for 99.9% (minutes) const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PEAK_WRITE = 97.22; // K01: peak write requests/s const EVENTS_DAY = 2_800_000; // K01: daily state events const b = (x, n = 2) => x.toFixed(n); console.log(`${"recovery time objective".padStart(24)}${"fits in failure share".padStart(23)}` + `${"one failure's budget share".padStart(28)}${"dropped edge requests in window".padStart(33)}`); for (const [name, min] of [["8 sec (failover)", 8 / 60], ["5 min", 5], ["10 min", 10], ["30 min", 30], ["60 min", 60]]) { console.log(`${name.padStart(24)}${b(FAILURE_SHARE / min).padStart(23)}` + `${("%" + b((100 * min) / MONTHLY)).padStart(28)}` + `${Math.round(min * 60 * PEAK_EDGE).toLocaleString("en-US").padStart(33)}`); } console.log(`K01's recovery time assumption was 10 min and fit 2.82 failures into the failure share`); console.log(`\ncost of driving the data loss objective to zero (synchronous acknowledgment):`); console.log(` every write waits for one extra acknowledgment round -> ${EVENTS_DAY.toLocaleString("en-US")} extra rounds a day`); const WINDOW = 8; // lesson 01: failover window at threshold 3 (seconds) const STOPS_MONTH = 2; // KU2 (assumption): 2 real stops a month, per replica const LAG = 1; // KU6 (assumption): asynchronous replication lag, 1 second const sync = STOPS_MONTH * WINDOW * PEAK_WRITE; const async_ = STOPS_MONTH * LAG * PEAK_WRITE; console.log(` the write path binds to two replicas at once: in the standby replica's ${WINDOW}-sec window`); console.log(` ${b(sync)} writes visibly drop a month (the carrier can resend)`); console.log(` under asynchronous replication, ${b(async_)} events a month are silently lost (unrecoverable)`); console.log(` ratio ${b(sync / async_)}: the synchronous choice rejects more requests, loses less data`); ``` ``` recovery time objective fits in failure share one failure's budget share dropped edge requests in window 8 sec (failover) 211.50 %0.31 4,111 5 min 5.64 %11.57 154,167 10 min 2.82 %23.15 308,334 30 min 0.94 %69.44 925,002 60 min 0.47 %138.89 1,850,004 K01's recovery time assumption was 10 min and fit 2.82 failures into the failure share cost of driving the data loss objective to zero (synchronous acknowledgment): every write waits for one extra acknowledgment round -> 2,800,000 extra rounds a day the write path binds to two replicas at once: in the standby replica's 8-sec window 1555.52 writes visibly drop a month (the carrier can resend) under asynchronous replication, 194.44 events a month are silently lost (unrecoverable) ratio 8.00: the synchronous choice rejects more requests, loses less data ``` The table shows how the objective speaks to the budget. A ten-minute recovery time objective fits 2.82 failures into the failure share — K01's own number, placed beside the other objectives, not recalculated. A thirty-minute objective means 0.94 failures: **a single failure eats 69.44 percent of the monthly budget, and a second breaks the target.** At sixty minutes, one failure already holds 138.89 percent — the target is broken by a single failure. A recovery time objective cannot be chosen independent of the availability target; both are bound to the same budget. The top row puts the previous two lessons' gain on this scale. The eight-second window measured with failover fits 211.50 failures into the failure share, and a single failure holds only 0.31 percent of the budget. Automatic failover is therefore not a speed improvement but making the target **achievable**: manual recovery already breaks the target at three failures a month, while failover fits two hundred. ## The Numbers for Two Days The path to a zero data loss objective is synchronous acknowledgment: a write does not count as accepted until a second replica confirms it. **The failure-free day's cost** has two line items. The first is every write waiting for one extra acknowledgment round — 2,800,000 extra rounds a day, with no failure at all. The second, and sharper, is the write path binding to two replicas at once: now the **standby** replica's failure stops writes too. If the standby replica's eight-second window happens twice a month, 1,555.52 writes are rejected. **The failing day's gain** is in the same block's last two lines. Under the asynchronous setup, a one-second lag silently loses 194.44 state events a month, unrecoverable. The synchronous setup zeroes this out. The ratio is 8.00: the synchronous choice openly rejects eight requests for every one event it would otherwise lose. This trade holds up for the shipment tracking service, because a rejected status update can be resent by the carrier, while a lost one cannot be recovered by anyone. This is exactly where K01 split the two patterns: the same failure can be paid either as outage or as lost data. The decision is made per flow. A tracking query is a read, and the data loss objective does not concern it. A zero data loss objective is defensible for the state event. End-of-day billing, though, runs in a four-hour window and can be restarted; both its recovery time objective and its data loss objective are loose — as long as the state events feeding it are complete. ## Summary - The recovery time objective and the data loss objective are separate thresholds held by separate mechanisms: the first by failover, the second by data replication; a recovery claim is incomplete without both numbers. Both are thresholds, not measurements: K01's `time to recovery` is a failure's actual duration, the objective here is the permitted upper bound. - The backup interval directly determines data loss: daily backups lose up to 2,800,000 events in the worst case, hourly 116,667, five-minute 9,722. - A one-second replication lag loses 97.22 events — one three-hundredth of a five-minute backup interval's loss; in a stream taking continuous writes, the data loss objective is held by replication. - Tightening the backup interval does not bill storage: the incremental data stored over thirty days is 29.28 GB at all three intervals; the only number that changes is the backup operation count, rising from 1 to 288 a day. - The recovery time objective is bound to the budget: a 30-minute objective fits 0.94 failures into the failure share (one failure is 69.44 percent of the budget), while the 8-second window measured with failover fits 211.50. - Zeroing out data loss costs the failure-free day 2,800,000 extra acknowledgment rounds a day and 1,555.52 rejected writes a month; in exchange, 194.44 permanent losses a month disappear (ratio 8.00). ## Next Step Every calculation in this lesson carried one implicit assumption: there is a replica to recover, and that replica was untouched by the failure. The assumption fails if two replicas sit on the same rack, the same power line, or behind the same network switch — this is why K01's replica arithmetic was an upper bound, with the assumption that "replicas fail independently" explicitly flagged there. The next lesson takes on **where** replicas are placed: the placement unit's definition, the capacity that survives losing one unit, the unpropagated write that cross-unit replication loses, and how much moving redundancy up one level costs on a failure-free day.