Recovery Objectives

The previous two lessons counted how long a failure takes to notice and to hand over: threshold, interval, promotion, and failback rounds. What they measured was always the same quantity, the time for the service to come back. But the 3,888.80 writes lost in the first lesson’s half-failed window are a different quantity that neither lesson measured: how much data is left behind when the system comes back?

This lesson names two thresholds and separates them. The recovery time objective is the longest time allowed to pass from a failure’s start to the service’s return. The data loss objective is the most data loss accepted after recovery, written as a duration: “at most one minute of writes”.

Both are thresholds. The time to recovery from the Introduction to System Design course is a measurement — how long a failure that actually happened took; the objective here is the permitted upper bound, set in advance as a design decision. The same separation holds for service availability: the measured percentage and the targeted percentage are separate numbers, and one does not validate the other.

Two Thresholds Held by Separate Mechanisms

The recovery time objective is held by failover: the sum of detection, promotion, and routing rounds must stay under the target. The data loss objective is held by data replication: it determines how close the recovered replica’s data is to the moment of failure. A system can fail over fast and lose a lot of data, or lose no data and recover slowly. A recovery claim only carries meaning with both numbers together: “failover completed in eight seconds” is incomplete without “and how many events were lost”.

Two inputs convert the data loss objective into a number. The backup interval: in the worst case, the failure happens right before the next backup, and a full interval’s worth of writes is lost. Replication lag: under asynchronous replication, writes that left the source but never reached the target are lost. Replication’s mechanics were established in the Scaling the Data Layer course and are not repeated here; the lag is only an input here.

// objective/loss.mjs — tying the data loss objective to the number of lost events via the backup
// interval and replication lag. All arithmetic; the inputs are K01's computed values and this
// lesson's assumptions.
const DAY = 86_400;
const EVENTS_DAY = 2_800_000;          // K01: daily state events (V3 x V4 = 400,000 x 7)
const EVENTS_SEC = EVENTS_DAY / DAY;   // average write rate
const PEAK = 97.22;                    // K01: peak write requests/s
const EVENTS_SHIPMENT = 7;             // K01: state events per shipment
const GROWTH_GB = 0.976;               // K01: daily data growth (976 MB)
const FULL_GB = 712.48;                // K01: stored data
const b = (x, n = 2) => x.toFixed(n);

console.log(`average write ${b(EVENTS_SEC)} events/s, peak ${PEAK} events/s`);
console.log(`${"backup interval".padStart(18)}${"worst-case loss".padStart(17)}` +
  `${"lost events (avg)".padStart(19)}${"lost events (peak)".padStart(20)}${"lost shipments".padStart(16)}`);
for (const [name, sec] of [["24 hr", 86_400], ["6 hr", 21_600], ["1 hr", 3600], ["15 min", 900],
  ["5 min", 300], ["1 min", 60]]) {
  const avg = EVENTS_SEC * sec, peak = PEAK * sec;
  console.log(`${name.padStart(18)}${(sec + " sec").padStart(17)}` +
    `${Math.round(avg).toLocaleString("en-US").padStart(19)}` +
    `${Math.round(peak).toLocaleString("en-US").padStart(20)}` +
    `${Math.round(avg / EVENTS_SHIPMENT).toLocaleString("en-US").padStart(16)}`);
}

console.log(`\n${"replication lag".padStart(19)}${"data loss objective".padStart(21)}` +
  `${"lost events (peak)".padStart(20)}${"lost shipments".padStart(16)}`);
for (const L of [0, 0.5, 1, 5, 10]) {
  console.log(`${(b(L, 1) + " sec").padStart(19)}${(b(L, 1) + " sec").padStart(21)}` +
    `${b(PEAK * L).padStart(20)}${b((PEAK * L) / EVENTS_SHIPMENT).padStart(16)}`);
}

console.log(`\n${"backup interval".padStart(18)}${"backups/day".padStart(13)}` +
  `${"backups/30 days".padStart(17)}${"incremental GB/30 days".padStart(24)}${"full backup GB (weekly)".padStart(25)}`);
for (const [name, sec] of [["24 hr", 86_400], ["1 hr", 3600], ["5 min", 300]]) {
  const perDay = DAY / sec;
  console.log(`${name.padStart(18)}${b(perDay, 0).padStart(13)}${b(perDay * 30, 0).padStart(17)}` +
    `${b(GROWTH_GB * 30).padStart(24)}${b(FULL_GB * 5).padStart(25)}`);
}
console.log(`stored backup data does not change with the interval: only the backup operation count changes`);

average write 32.41 events/s, peak 97.22 events/s
   backup interval  worst-case loss  lost events (avg)  lost events (peak)  lost shipments
             24 hr        86400 sec          2,800,000           8,399,808         400,000
              6 hr        21600 sec            700,000           2,099,952         100,000
              1 hr         3600 sec            116,667             349,992          16,667
            15 min          900 sec             29,167              87,498           4,167
             5 min          300 sec              9,722              29,166           1,389
             1 min           60 sec              1,944               5,833             278

    replication lag  data loss objective  lost events (peak)  lost shipments
            0.0 sec              0.0 sec                0.00            0.00
            0.5 sec              0.5 sec               48.61            6.94
            1.0 sec              1.0 sec               97.22           13.89
            5.0 sec              5.0 sec              486.10           69.44
           10.0 sec             10.0 sec              972.20          138.89

   backup interval  backups/day  backups/30 days  incremental GB/30 days  full backup GB (weekly)
             24 hr            1               30                   29.28                  3562.40
              1 hr           24              720                   29.28                  3562.40
             5 min          288             8640                   29.28                  3562.40
stored backup data does not change with the interval: only the backup operation count changes

Backups Alone Cannot Hold an Objective

The first table shows a difference of orders of magnitude. Daily backups lose 2,800,000 state events in the worst case — a full day’s shipment traffic; hourly backups lose 116,667 events and 16,667 shipments. Even at five minutes, 9,722 events are lost. These numbers reflect not a shortcoming but a definition: a backup is a copy of specific moments, and everything between two moments falls outside the copy.

The second table gives the same quantity on the replication side, and the comparison is stark: a one-second lag loses 97.22 events at peak load — 300 times less than a five-minute backup interval’s loss. In a stream taking continuous writes, a meaningful data loss objective is held by replication, not backups; a backup’s job is covering a different failure mode — data overwritten with the wrong content, a case replication faithfully copies.

The third table shows where tightening the backup interval costs nothing. With incremental backups, the data stored over thirty days is 29.28 GB at all three intervals, because what is stored is the change itself, independent of the interval. The 3,562.40 GB of weekly full backups is fixed too. The only number that changes is the backup operation count: 1 to 288 a day. Tightening the interval bills operation count and coordination, not storage.

The Recovery Time Objective Looks at the Budget

The recovery time objective cannot be chosen alone, because the objective and the monthly outage budget share the same resource. K01 established the budget: the 99.9% target gives 43.2 minutes a month, 15.0 minutes of which go to planned work, the remaining 28.2 to failure. The percentage arithmetic is not repeated here; these three numbers are inputs.

// objective/budget.mjs — comparing the recovery time objective against K01's monthly outage
// budget, and the failure-free-day cost of driving the data loss objective to zero
const FAILURE_SHARE = 28.2;   // K01: the outage budget's monthly failure share (minutes)
const MONTHLY = 43.2;         // K01: monthly outage budget for 99.9% (minutes)
const PEAK_EDGE = 513.89;     // K01: peak requests/s at the edge
const PEAK_WRITE = 97.22;     // K01: peak write requests/s
const EVENTS_DAY = 2_800_000; // K01: daily state events
const b = (x, n = 2) => x.toFixed(n);

console.log(`${"recovery time objective".padStart(24)}${"fits in failure share".padStart(23)}` +
  `${"one failure's budget share".padStart(28)}${"dropped edge requests in window".padStart(33)}`);
for (const [name, min] of [["8 sec (failover)", 8 / 60], ["5 min", 5], ["10 min", 10], ["30 min", 30],
  ["60 min", 60]]) {
  console.log(`${name.padStart(24)}${b(FAILURE_SHARE / min).padStart(23)}` +
    `${("%" + b((100 * min) / MONTHLY)).padStart(28)}` +
    `${Math.round(min * 60 * PEAK_EDGE).toLocaleString("en-US").padStart(33)}`);
}
console.log(`K01's recovery time assumption was 10 min and fit 2.82 failures into the failure share`);

console.log(`\ncost of driving the data loss objective to zero (synchronous acknowledgment):`);
console.log(`  every write waits for one extra acknowledgment round -> ${EVENTS_DAY.toLocaleString("en-US")} extra rounds a day`);
const WINDOW = 8;             // lesson 01: failover window at threshold 3 (seconds)
const STOPS_MONTH = 2;        // KU2 (assumption): 2 real stops a month, per replica
const LAG = 1;                // KU6 (assumption): asynchronous replication lag, 1 second
const sync = STOPS_MONTH * WINDOW * PEAK_WRITE;
const async_ = STOPS_MONTH * LAG * PEAK_WRITE;
console.log(`  the write path binds to two replicas at once: in the standby replica's ${WINDOW}-sec window`);
console.log(`  ${b(sync)} writes visibly drop a month (the carrier can resend)`);
console.log(`  under asynchronous replication, ${b(async_)} events a month are silently lost (unrecoverable)`);
console.log(`  ratio ${b(sync / async_)}: the synchronous choice rejects more requests, loses less data`);

 recovery time objective  fits in failure share  one failure's budget share  dropped edge requests in window
        8 sec (failover)                 211.50                       %0.31                            4,111
                   5 min                   5.64                      %11.57                          154,167
                  10 min                   2.82                      %23.15                          308,334
                  30 min                   0.94                      %69.44                          925,002
                  60 min                   0.47                     %138.89                        1,850,004
K01's recovery time assumption was 10 min and fit 2.82 failures into the failure share

cost of driving the data loss objective to zero (synchronous acknowledgment):
  every write waits for one extra acknowledgment round -> 2,800,000 extra rounds a day
  the write path binds to two replicas at once: in the standby replica's 8-sec window
  1555.52 writes visibly drop a month (the carrier can resend)
  under asynchronous replication, 194.44 events a month are silently lost (unrecoverable)
  ratio 8.00: the synchronous choice rejects more requests, loses less data

The table shows how the objective speaks to the budget. A ten-minute recovery time objective fits 2.82 failures into the failure share — K01’s own number, placed beside the other objectives, not recalculated. A thirty-minute objective means 0.94 failures: a single failure eats 69.44 percent of the monthly budget, and a second breaks the target. At sixty minutes, one failure already holds 138.89 percent — the target is broken by a single failure. A recovery time objective cannot be chosen independent of the availability target; both are bound to the same budget.

The top row puts the previous two lessons’ gain on this scale. The eight-second window measured with failover fits 211.50 failures into the failure share, and a single failure holds only 0.31 percent of the budget. Automatic failover is therefore not a speed improvement but making the target achievable: manual recovery already breaks the target at three failures a month, while failover fits two hundred.

The Numbers for Two Days

The path to a zero data loss objective is synchronous acknowledgment: a write does not count as accepted until a second replica confirms it. The failure-free day’s cost has two line items. The first is every write waiting for one extra acknowledgment round — 2,800,000 extra rounds a day, with no failure at all. The second, and sharper, is the write path binding to two replicas at once: now the standby replica’s failure stops writes too. If the standby replica’s eight-second window happens twice a month, 1,555.52 writes are rejected.

The failing day’s gain is in the same block’s last two lines. Under the asynchronous setup, a one-second lag silently loses 194.44 state events a month, unrecoverable. The synchronous setup zeroes this out. The ratio is 8.00: the synchronous choice openly rejects eight requests for every one event it would otherwise lose. This trade holds up for the shipment tracking service, because a rejected status update can be resent by the carrier, while a lost one cannot be recovered by anyone. This is exactly where K01 split the two patterns: the same failure can be paid either as outage or as lost data.

The decision is made per flow. A tracking query is a read, and the data loss objective does not concern it. A zero data loss objective is defensible for the state event. End-of-day billing, though, runs in a four-hour window and can be restarted; both its recovery time objective and its data loss objective are loose — as long as the state events feeding it are complete.

Summary

• The recovery time objective and the data loss objective are separate thresholds held by separate mechanisms: the first by failover, the second by data replication; a recovery claim is incomplete without both numbers. Both are thresholds, not measurements: K01’s time to recovery is a failure’s actual duration, the objective here is the permitted upper bound.
• The backup interval directly determines data loss: daily backups lose up to 2,800,000 events in the worst case, hourly 116,667, five-minute 9,722.
• A one-second replication lag loses 97.22 events — one three-hundredth of a five-minute backup interval’s loss; in a stream taking continuous writes, the data loss objective is held by replication.
• Tightening the backup interval does not bill storage: the incremental data stored over thirty days is 29.28 GB at all three intervals; the only number that changes is the backup operation count, rising from 1 to 288 a day.
• The recovery time objective is bound to the budget: a 30-minute objective fits 0.94 failures into the failure share (one failure is 69.44 percent of the budget), while the 8-second window measured with failover fits 211.50.
• Zeroing out data loss costs the failure-free day 2,800,000 extra acknowledgment rounds a day and 1,555.52 rejected writes a month; in exchange, 194.44 permanent losses a month disappear (ratio 8.00).

Next Step

Every calculation in this lesson carried one implicit assumption: there is a replica to recover, and that replica was untouched by the failure. The assumption fails if two replicas sit on the same rack, the same power line, or behind the same network switch — this is why K01’s replica arithmetic was an upper bound, with the assumption that “replicas fail independently” explicitly flagged there. The next lesson takes on where replicas are placed: the placement unit’s definition, the capacity that survives losing one unit, the unpropagated write that cross-unit replication loses, and how much moving redundancy up one level costs on a failure-free day.