Failover Design

The previous topic contained the failure with eight patterns: the circuit breaker routed around a broken dependency, the bulkhead pattern separated resource pools, the timeout budget bounded the wait, graceful degradation kept the core function standing. All shared one limit: the failure was contained, not removed. The broken part is still broken, and how a replacement replica takes over was never designed.

The setup itself was measured in the Introduction to System Design course: in active–passive, writes drop until failover completes; in active–active, there is no failover window, but conflicting writes occur. That measurement is an input here, not repeated. This lesson’s question sits one layer higher: when failover triggers, what happens when the old active replica returns, why a replica that looks up but cannot do its job never triggers failover, and how many seconds these transitions cost from the outage budget.

The Three Decisions Left After the Setup

The setup choice says whether a replica accepts writes; it does not design the transition itself. Three decisions remain.

Triggering. How many consecutive failed health checks start the promotion. At threshold 1, the first failed check starts the transition; at threshold 4, four are awaited. What the check asks is the next lesson’s subject; here it only asks “does it respond”, which makes it shallow.

Failback. When the old active replica recovers, is activity handed back to it, or does the new replica stay active permanently. Failback is itself a transition with its own window.

Distinguishing. How a replica that responds but cannot do its job is seen. Since this replica passes the health check, no threshold catches it.

Two replicas both believing themselves active is a separate problem; the lease duration and the two-leader round were established in the Application Layer and Service Interaction course and are not repeated here.

The Setup

The setup below is a model: no real cluster, zone, or container is set up. A round is an abstract step — the failover model’s arrangement from the Introduction to System Design course continues. The failure schedule is a parameter: replica A briefly stops responding twice and recovers on its own, actually stops twice, and goes half-failed once (up, passing the shallow check, but unable to write to the state store). Replica B is always up in the model, a simplification that leaves both replicas failing together unaccounted for.

// failover/model.mjs — active-passive failover MODEL. A round is an abstract step (the
// arrangement from the Availability Patterns lesson, M19/K01); the detection threshold,
// promotion, and failback windows are parameters in rounds, not measured durations. No real
// cluster is set up.
export const ROUND = 130;
export const BRIEF = [[12, 12], [20, 21]];       // A does not respond but recovers on its own
export const STOPPED = [[30, 75], [90, 96]];     // A actually stopped
export const HALF_FAILED = [[100, 120]];         // A is up, passes the shallow check, but cannot write to the store
const PROMOTION = 1;                             // assumption: promotion takes one round
const FAILBACK_THRESHOLD = 3;                    // assumption: consecutive healthy checks required for failback

const within = (t, a) => a.some(([b, c]) => t >= b && t <= c);
export const isResponsive = (t) => !within(t, BRIEF) && !within(t, STOPPED);
export const isHealthy = (t) => isResponsive(t) && !within(t, HALF_FAILED);

export function run({ threshold, failback, round: N = ROUND }) {
  const s = { dropped: 0, silent: 0, failovers: 0, falseFailovers: 0, failbacks: 0, window: 0 };
  let active = "A", consecutive = 0, healthy = 0, doneAt = null, direction = null;

  for (let t = 1; t <= N; t += 1) {
    const responds = active === "A" ? isResponsive(t) : true;   // B is always up in the model

    if (doneAt !== null) {                                    // transition window is running
      s.window += 1; s.dropped += 1;
      if (t === doneAt) {
        // retroactive classification: the old active replica recovered on its own, the failover was unnecessary
        if (direction === "failover" && isResponsive(t + 1)) s.falseFailovers += 1;
        active = direction === "failover" ? "B" : "A";
        doneAt = null; direction = null; consecutive = 0; healthy = 0;
      }
      continue;
    }

    if (active === "A" && responds === false) {                // shallow check: response only
      consecutive += 1;
      s.dropped += 1;
      if (consecutive >= threshold) { doneAt = t + PROMOTION; direction = "failover"; s.failovers += 1; }
      continue;
    }
    consecutive = 0;

    if (active === "B" && failback === "automatic") {
      healthy = isResponsive(t) ? healthy + 1 : 0;
      if (healthy >= FAILBACK_THRESHOLD) { doneAt = t + PROMOTION; direction = "failback"; s.failbacks += 1; healthy = 0; continue; }
    }

    if (active === "A" && isHealthy(t) === false) s.silent += 1; // write accepted, never reached the store
  }
  return s;
}

export function noFailover(N = ROUND) {           // no failover: A is whatever it is
  let dropped = 0, silent = 0, accepted = 0;
  for (let t = 1; t <= N; t += 1) {
    if (isResponsive(t) === false) dropped += 1;
    else if (isHealthy(t) === false) silent += 1;
    else accepted += 1;
  }
  return { accepted, dropped, silent };
}

// failover/measure.mjs — measuring the detection threshold and failback policy in the same failure schedule
import { run, noFailover, ROUND, BRIEF, STOPPED, HALF_FAILED } from "./model.mjs";

const s = (x, n) => String(x).padStart(n);
const length = (a) => a.reduce((t, [b, c]) => t + (c - b + 1), 0);
console.log(`${ROUND} rounds. Replica A: ${length(BRIEF)} rounds briefly unresponsive, ` +
  `${length(STOPPED)} rounds stopped, ${length(HALF_FAILED)} rounds half-failed. B is always up in the model.`);
const nf = noFailover();
console.log(`no failover: accepted ${nf.accepted}, dropped ${nf.dropped}, silent lost update ${nf.silent}`);
console.log();

console.log("threshold | failovers | false failovers | failbacks | transition round | dropped | silent lost update");
console.log("----------|-----------|------------------|-----------|-------------------|---------|--------------------");
for (const threshold of [1, 2, 3, 4]) {
  const r = run({ threshold, failback: "automatic" });
  console.log(`${s(threshold, 9)} | ${s(r.failovers, 9)} | ${s(r.falseFailovers, 16)} | ${s(r.failbacks, 9)} | ` +
    `${s(r.window, 17)} | ${s(r.dropped, 7)} | ${s(r.silent, 19)}`);
}
console.log();

console.log("failback  | threshold | failovers | failbacks | transition round | dropped | active replica changes");
console.log("----------|-----------|-----------|-----------|-------------------|---------|------------------------");
for (const failback of ["automatic", "permanent"]) {
  const r = run({ threshold: 3, failback });
  console.log(`${failback.padEnd(9)} | ${s(3, 9)} | ${s(r.failovers, 9)} | ${s(r.failbacks, 9)} | ` +
    `${s(r.window, 17)} | ${s(r.dropped, 7)} | ${s(r.failovers + r.failbacks, 22)}`);
}

130 rounds. Replica A: 3 rounds briefly unresponsive, 53 rounds stopped, 21 rounds half-failed. B is always up in the model.
no failover: accepted 53, dropped 56, silent lost update 21

threshold | failovers | false failovers | failbacks | transition round | dropped | silent lost update
----------|-----------|------------------|-----------|-------------------|---------|--------------------
        1 |         4 |                2 |         4 |                 8 |      12 |                  20
        2 |         3 |                1 |         3 |                 6 |      13 |                  20
        3 |         2 |                0 |         2 |                 4 |      13 |                  20
        4 |         2 |                0 |         2 |                 4 |      15 |                  20

failback  | threshold | failovers | failbacks | transition round | dropped | active replica changes
----------|-----------|-----------|-----------|-------------------|---------|------------------------
automatic |         3 |         2 |         2 |                 4 |      13 |                      4
permanent |         3 |         1 |         0 |                 1 |       7 |                      1

Two Sides of the Threshold

The no-failover row is the baseline: without failover, 56 of 130 rounds accept no write. With threshold 3, dropped rounds in the same schedule fall to 13 — the pattern’s gain on the failing day.

The threshold column shows the trade-off in two directions. As the threshold shrinks, false failovers rise. Threshold 1 treated both self-recovering outages as transitions: 2 of 4 failovers are false, since the old active replica started responding again just as promotion finished. Threshold 2 caught one of these; thresholds 3 and 4 triggered on neither.

As the threshold grows, detection is delayed. The time from a real stop’s start to the transition’s completion lengthens with the threshold: dropped rounds are 13 at threshold 3, 15 at threshold 4. The total is not one-directional: rounds dropped at threshold 1 and threshold 3 come out close, 12 and 13, because the false failover’s window takes back the detection speed threshold 1 gained. The threshold is not a speed decision but a balance of two costs, and where it sits depends on the failure schedule.

The silent lost update column is 20 in all four rows. It does not change with the threshold, because the half-failed replica passes the shallow check and the counter never increases.

Failback Is a Second Window

The second table compares two failback policies at the same threshold. Under automatic failback, the active replica changes four times (2 failovers, 2 failbacks), a transition round count of 4; under the permanent policy, one change and one transition round. Dropped rounds fall from 13 to 7.

The difference comes from two places. First, every failback produces its own window. Second, and more important, automatic failback reconnects the system to the same replica: A recovers after its first stop, activity is handed back to it, and it stops a second time shortly after. Under the permanent policy this second stop produces no outage at all, since the active replica is now B.

Automatic failback’s payoff is operational: the replicas’ roles return to a known arrangement, bought with 6 extra dropped rounds.

The Replica That Responds But Cannot Write

The half-failed window is this model’s quietest part. The replica is up, passes the health check, and accepts requests — but the write it accepted never reaches the state store. Writes were accepted and lost over 21 rounds without the pattern, 20 rounds with it. The outage budget never sees these rounds, because the system was responding.

The Numbers for Two Days

Rounds convert to a budget only with a cycle-length assumption. This course’s own assumptions are not added to K01’s table; they are cited by their own names. KU1 — a round is one health-check interval, 2 seconds. KU2 — 2 real stops a month, since K01’s failure share tolerates 2.82 failures a month. KU3 — 20 transient unresponsive periods a month, since brief pauses are more frequent than real stops. Recovery time, peak write, and failure share are read from K01.

// failover/budget.mjs — converting the model's rounds into K01's outage budget and request counts
const ROUND_SEC = 2;         // KU1 (assumption): one round is one health-check interval, 2 seconds
const REAL_STOPS_MONTH = 2;  // KU2 (assumption): 2 real stops a month
const BRIEF_MONTH = 20;      // KU3 (assumption): 20 brief unresponsive periods a month
const RECOVERY_SEC = 600;    // K01: recovery time assumption, 10 minutes
const FAILURE_SHARE = 28.2;  // K01: the outage budget's monthly failure share (minutes)
const PEAK_WRITE = 97.22;    // K01: peak write requests/s
const PEAK_EDGE = 513.89;    // K01: peak requests/s at the edge
const PROMOTION = 1;
const b = (x, n = 2) => x.toFixed(n);

console.log(`${"threshold".padStart(9)}${"transition window".padStart(19)}${"real stops (month)".padStart(20)}` +
  `${"false failovers (month)".padStart(25)}${"total outage".padStart(15)}${"of failure share".padStart(18)}`);
for (const threshold of [1, 2, 3, 4]) {
  const window = (threshold + PROMOTION) * ROUND_SEC;
  const realStop = REAL_STOPS_MONTH * window;
  const falseFailover = threshold <= 2 ? BRIEF_MONTH * (window + PROMOTION * ROUND_SEC) : 0;  // threshold 3+ never triggers on a brief outage
  const total = realStop + falseFailover;
  console.log(`${String(threshold).padStart(9)}${(window + " sec").padStart(19)}${(b(realStop, 1) + " sec").padStart(20)}` +
    `${(b(falseFailover, 1) + " sec").padStart(25)}${(b(total / 60, 2) + " min").padStart(15)}` +
    `${("%" + b((100 * total) / 60 / FAILURE_SHARE, 2)).padStart(18)}`);
}

const noFailover = REAL_STOPS_MONTH * RECOVERY_SEC;
console.log(`\nno failover: ${REAL_STOPS_MONTH} stops/month x ${RECOVERY_SEC} sec = ${b(noFailover / 60, 1)} min, ` +
  `%${b((100 * noFailover) / 60 / FAILURE_SHARE, 1)} of failure share`);
const threshold3 = REAL_STOPS_MONTH * (3 + PROMOTION) * ROUND_SEC;
console.log(`same two stops with threshold 3: ${threshold3} sec, ${b(noFailover / threshold3, 1)}x shorter`);
console.log(`dropped in the window: ${b((3 + PROMOTION) * ROUND_SEC * PEAK_WRITE)} writes, ` +
  `${b((3 + PROMOTION) * ROUND_SEC * PEAK_EDGE)} edge requests (at peak load)`);

const SILENT_ROUND = 20;   // from the model: rounds accepted during the half-failed window but never reaching the store
console.log(`\nhalf-failed window: ${SILENT_ROUND} rounds x ${ROUND_SEC} sec = ${SILENT_ROUND * ROUND_SEC} sec, ` +
  `${b(SILENT_ROUND * ROUND_SEC * PEAK_WRITE)} writes silently lost`);
console.log(`this window is invisible in the outage budget: the system was responding`);

console.log();
for (const replicas of [2, 3]) {              // active-active: the failed replica carries 1/k of the pool
  console.log(`active-active, ${replicas} replicas: in the removal window, ` +
    `${b(((3 + PROMOTION) * ROUND_SEC * PEAK_WRITE) / replicas)} writes drop, ` +
    `in the half-failed window ${b((SILENT_ROUND * ROUND_SEC * PEAK_WRITE) / replicas)} writes are silently lost`);
}

threshold  transition window  real stops (month)  false failovers (month)   total outage  of failure share
        1              4 sec             8.0 sec                120.0 sec       2.13 min             %7.57
        2              6 sec            12.0 sec                160.0 sec       2.87 min            %10.17
        3              8 sec            16.0 sec                  0.0 sec       0.27 min             %0.95
        4             10 sec            20.0 sec                  0.0 sec       0.33 min             %1.18

no failover: 2 stops/month x 600 sec = 20.0 min, %70.9 of failure share
same two stops with threshold 3: 16 sec, 75.0x shorter
dropped in the window: 777.76 writes, 4111.12 edge requests (at peak load)

half-failed window: 20 rounds x 2 sec = 40 sec, 3888.80 writes silently lost
this window is invisible in the outage budget: the system was responding

active-active, 2 replicas: in the removal window, 388.88 writes drop, in the half-failed window 1944.40 writes are silently lost
active-active, 3 replicas: in the removal window, 259.25 writes drop, in the half-failed window 1296.27 writes are silently lost

The failing day’s gain is the first two lines compared. Without failover, the same two stops last as long as the recovery time: 20.0 minutes, 70.9 percent of the failure share — nearly the entire budget, and a third stop breaks the target that month. With threshold 3, the same two stops take 16 seconds, a 75x reduction, spending 0.95 percent of the failure share. This ratio does not depend on the machine, only on the ratio of recovery time to transition window.

The failure-free day’s cost is in the false-failover column. At threshold 1, 20 transient unresponsive periods a month produce 120 seconds of outage though none is a real failure; at threshold 2, 160 seconds — in both rows exceeding the failing day’s cost, 12 seconds at threshold 2. At threshold 3 it drops to zero, trading the real-stop window up from 4 to 8 seconds. A second failure-free-day cost never shows up in the arithmetic: the passive replica carries its full capacity and answers no requests at all.

The window’s one-time bill is also recorded: an eight-second transition drops 777.76 writes and 4,111.12 edge requests at peak load — a concentration an outage budget, which measures an average, does not show.

The last two lines tie the setup difference to a number. In the active–active setup, the transition window drops only the failed replica’s share, not the whole service: 388.88 writes with two replicas, 259.25 with three. The same division holds for the silent lost update — 1,296.27 instead of 3,888.80 with three replicas. Active–active does not make the failure cheaper, it divides it; K01’s conflicting writes are the price of that division.

Summary

• Three decisions remain after the setup choice: the threshold that triggers promotion, the failback policy toward the old active replica, and distinguishing a replica that responds but cannot do its job.
• The threshold balances two costs: threshold 1 turned both self-recovering outages into transitions (2 false failovers), threshold 4 delayed detection (15 dropped rounds); total dropped rounds stayed in a narrow 12–15 range.
• Failback produces its own window: 4 transitions and 13 dropped rounds under the automatic policy, 1 transition and 7 dropped rounds under the permanent policy.
• No threshold catches the half-failed replica: silent lost updates are 20 rounds at all four thresholds; the counter never increases because the shallow check gets a response.
• With the KU1–KU3 assumptions, the failing day’s gain is a drop from 20.0 minutes to 16 seconds: a 75x reduction, and 0.95 percent of the failure share instead of 70.9 percent.
• The failure-free day’s cost is 120 seconds of false-failover outage a month at threshold 1, 160 seconds at threshold 2; it zeroes out at threshold 3, in exchange for the real-stop window rising from 4 to 8 seconds.

Next Step

Every decision in this lesson rested on a single input: the yes or no the health check returns. When that input is wrong, both the threshold and the failback policy lose their meaning — in the model’s half-failed window, 3,888.80 writes were lost while the system looked healthy. The next lesson designs the check’s content: whether the endpoint looks at its dependencies, the risk that adding a dependency to the check marks all replicas unhealthy at once, separating being ready to take traffic from being up, and where the check interval sits between detection delay and false failover.