--- title: 'Failover Design' source: 'https://academia.sh/en/courses/resilience-patterns/failover-design' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:32+00:00' license: 'CC BY-SA 4.0' --- # Failover Design The three decisions failover leaves after the setup choice: choosing the consecutive-failed-check threshold that triggers promotion numerically between false failover and detection delay, the second transition window the failback policy produces, counting the writes a replica that responds but cannot do its job silently swallows, and converting all these transitions into the monthly outage budget's failure share. The previous topic contained the failure with eight patterns: the circuit breaker routed around a broken dependency, the bulkhead pattern separated resource pools, the timeout budget bounded the wait, graceful degradation kept the core function standing. All shared one limit: the failure was **contained**, not removed. The broken part is still broken, and how a replacement replica takes over was never designed. The setup itself was measured in the Introduction to System Design course: in active–passive, writes drop until failover completes; in active–active, there is no failover window, but conflicting writes occur. That measurement is an **input** here, not repeated. This lesson's question sits one layer higher: when failover triggers, what happens when the old active replica returns, why a replica that looks up but cannot do its job never triggers failover, and how many seconds these transitions cost from the outage budget. ## The Three Decisions Left After the Setup The setup choice says whether a replica accepts writes; it does not design the transition itself. Three decisions remain. **Triggering.** How many consecutive failed health checks start the promotion. At threshold 1, the first failed check starts the transition; at threshold 4, four are awaited. What the check **asks** is the next lesson's subject; here it only asks "does it respond", which makes it shallow. **Failback.** When the old active replica recovers, is activity handed back to it, or does the new replica stay active permanently. Failback is itself a transition with its own window. **Distinguishing.** How a replica that responds but cannot do its job is seen. Since this replica passes the health check, no threshold catches it. Two replicas both believing themselves active is a separate problem; the lease duration and the two-leader round were established in the Application Layer and Service Interaction course and are not repeated here. ## The Setup The setup below is a **model**: no real cluster, zone, or container is set up. A round is an abstract step — the failover model's arrangement from the Introduction to System Design course continues. The failure schedule is a parameter: replica `A` briefly stops responding twice and recovers on its own, actually stops twice, and goes **half-failed** once (up, passing the shallow check, but unable to write to the state store). Replica `B` is always up in the model, a simplification that leaves both replicas failing together unaccounted for. ```js // failover/model.mjs — active-passive failover MODEL. A round is an abstract step (the // arrangement from the Availability Patterns lesson, M19/K01); the detection threshold, // promotion, and failback windows are parameters in rounds, not measured durations. No real // cluster is set up. export const ROUND = 130; export const BRIEF = [[12, 12], [20, 21]]; // A does not respond but recovers on its own export const STOPPED = [[30, 75], [90, 96]]; // A actually stopped export const HALF_FAILED = [[100, 120]]; // A is up, passes the shallow check, but cannot write to the store const PROMOTION = 1; // assumption: promotion takes one round const FAILBACK_THRESHOLD = 3; // assumption: consecutive healthy checks required for failback const within = (t, a) => a.some(([b, c]) => t >= b && t <= c); export const isResponsive = (t) => !within(t, BRIEF) && !within(t, STOPPED); export const isHealthy = (t) => isResponsive(t) && !within(t, HALF_FAILED); export function run({ threshold, failback, round: N = ROUND }) { const s = { dropped: 0, silent: 0, failovers: 0, falseFailovers: 0, failbacks: 0, window: 0 }; let active = "A", consecutive = 0, healthy = 0, doneAt = null, direction = null; for (let t = 1; t <= N; t += 1) { const responds = active === "A" ? isResponsive(t) : true; // B is always up in the model if (doneAt !== null) { // transition window is running s.window += 1; s.dropped += 1; if (t === doneAt) { // retroactive classification: the old active replica recovered on its own, the failover was unnecessary if (direction === "failover" && isResponsive(t + 1)) s.falseFailovers += 1; active = direction === "failover" ? "B" : "A"; doneAt = null; direction = null; consecutive = 0; healthy = 0; } continue; } if (active === "A" && responds === false) { // shallow check: response only consecutive += 1; s.dropped += 1; if (consecutive >= threshold) { doneAt = t + PROMOTION; direction = "failover"; s.failovers += 1; } continue; } consecutive = 0; if (active === "B" && failback === "automatic") { healthy = isResponsive(t) ? healthy + 1 : 0; if (healthy >= FAILBACK_THRESHOLD) { doneAt = t + PROMOTION; direction = "failback"; s.failbacks += 1; healthy = 0; continue; } } if (active === "A" && isHealthy(t) === false) s.silent += 1; // write accepted, never reached the store } return s; } export function noFailover(N = ROUND) { // no failover: A is whatever it is let dropped = 0, silent = 0, accepted = 0; for (let t = 1; t <= N; t += 1) { if (isResponsive(t) === false) dropped += 1; else if (isHealthy(t) === false) silent += 1; else accepted += 1; } return { accepted, dropped, silent }; } ``` ```js // failover/measure.mjs — measuring the detection threshold and failback policy in the same failure schedule import { run, noFailover, ROUND, BRIEF, STOPPED, HALF_FAILED } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); const length = (a) => a.reduce((t, [b, c]) => t + (c - b + 1), 0); console.log(`${ROUND} rounds. Replica A: ${length(BRIEF)} rounds briefly unresponsive, ` + `${length(STOPPED)} rounds stopped, ${length(HALF_FAILED)} rounds half-failed. B is always up in the model.`); const nf = noFailover(); console.log(`no failover: accepted ${nf.accepted}, dropped ${nf.dropped}, silent lost update ${nf.silent}`); console.log(); console.log("threshold | failovers | false failovers | failbacks | transition round | dropped | silent lost update"); console.log("----------|-----------|------------------|-----------|-------------------|---------|--------------------"); for (const threshold of [1, 2, 3, 4]) { const r = run({ threshold, failback: "automatic" }); console.log(`${s(threshold, 9)} | ${s(r.failovers, 9)} | ${s(r.falseFailovers, 16)} | ${s(r.failbacks, 9)} | ` + `${s(r.window, 17)} | ${s(r.dropped, 7)} | ${s(r.silent, 19)}`); } console.log(); console.log("failback | threshold | failovers | failbacks | transition round | dropped | active replica changes"); console.log("----------|-----------|-----------|-----------|-------------------|---------|------------------------"); for (const failback of ["automatic", "permanent"]) { const r = run({ threshold: 3, failback }); console.log(`${failback.padEnd(9)} | ${s(3, 9)} | ${s(r.failovers, 9)} | ${s(r.failbacks, 9)} | ` + `${s(r.window, 17)} | ${s(r.dropped, 7)} | ${s(r.failovers + r.failbacks, 22)}`); } ``` ``` 130 rounds. Replica A: 3 rounds briefly unresponsive, 53 rounds stopped, 21 rounds half-failed. B is always up in the model. no failover: accepted 53, dropped 56, silent lost update 21 threshold | failovers | false failovers | failbacks | transition round | dropped | silent lost update ----------|-----------|------------------|-----------|-------------------|---------|-------------------- 1 | 4 | 2 | 4 | 8 | 12 | 20 2 | 3 | 1 | 3 | 6 | 13 | 20 3 | 2 | 0 | 2 | 4 | 13 | 20 4 | 2 | 0 | 2 | 4 | 15 | 20 failback | threshold | failovers | failbacks | transition round | dropped | active replica changes ----------|-----------|-----------|-----------|-------------------|---------|------------------------ automatic | 3 | 2 | 2 | 4 | 13 | 4 permanent | 3 | 1 | 0 | 1 | 7 | 1 ``` ## Two Sides of the Threshold The no-failover row is the baseline: without failover, 56 of 130 rounds accept no write. With threshold 3, dropped rounds in the same schedule fall to 13 — the pattern's gain on the failing day. The threshold column shows the trade-off in two directions. **As the threshold shrinks, false failovers rise.** Threshold 1 treated both self-recovering outages as transitions: 2 of 4 failovers are false, since the old active replica started responding again just as promotion finished. Threshold 2 caught one of these; thresholds 3 and 4 triggered on neither. **As the threshold grows, detection is delayed.** The time from a real stop's start to the transition's completion lengthens with the threshold: dropped rounds are 13 at threshold 3, 15 at threshold 4. The total is not one-directional: rounds dropped at threshold 1 and threshold 3 come out close, 12 and 13, because the false failover's window takes back the detection speed threshold 1 gained. **The threshold is not a speed decision but a balance of two costs**, and where it sits depends on the failure schedule. The silent lost update column is 20 in all four rows. It does not change with the threshold, because the half-failed replica passes the shallow check and the counter never increases. ## Failback Is a Second Window The second table compares two failback policies at the same threshold. Under automatic failback, the active replica changes four times (2 failovers, 2 failbacks), a transition round count of 4; under the permanent policy, one change and one transition round. Dropped rounds fall from 13 to 7. The difference comes from two places. First, every failback produces its own window. Second, and more important, automatic failback **reconnects the system to the same replica**: `A` recovers after its first stop, activity is handed back to it, and it stops a second time shortly after. Under the permanent policy this second stop produces no outage at all, since the active replica is now `B`. Automatic failback's payoff is operational: the replicas' roles return to a known arrangement, bought with 6 extra dropped rounds. ## The Replica That Responds But Cannot Write The half-failed window is this model's quietest part. The replica is up, passes the health check, and accepts requests — but the write it accepted never reaches the state store. Writes were **accepted and lost** over 21 rounds without the pattern, 20 rounds with it. The outage budget never sees these rounds, because the system was responding. ## The Numbers for Two Days Rounds convert to a budget only with a cycle-length assumption. This course's own assumptions are not added to K01's table; they are cited by their own names. **KU1 — a round is one health-check interval, 2 seconds.** **KU2 — 2 real stops a month**, since K01's failure share tolerates 2.82 failures a month. **KU3 — 20 transient unresponsive periods a month**, since brief pauses are more frequent than real stops. Recovery time, peak write, and failure share are read from K01. ```js // failover/budget.mjs — converting the model's rounds into K01's outage budget and request counts const ROUND_SEC = 2; // KU1 (assumption): one round is one health-check interval, 2 seconds const REAL_STOPS_MONTH = 2; // KU2 (assumption): 2 real stops a month const BRIEF_MONTH = 20; // KU3 (assumption): 20 brief unresponsive periods a month const RECOVERY_SEC = 600; // K01: recovery time assumption, 10 minutes const FAILURE_SHARE = 28.2; // K01: the outage budget's monthly failure share (minutes) const PEAK_WRITE = 97.22; // K01: peak write requests/s const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PROMOTION = 1; const b = (x, n = 2) => x.toFixed(n); console.log(`${"threshold".padStart(9)}${"transition window".padStart(19)}${"real stops (month)".padStart(20)}` + `${"false failovers (month)".padStart(25)}${"total outage".padStart(15)}${"of failure share".padStart(18)}`); for (const threshold of [1, 2, 3, 4]) { const window = (threshold + PROMOTION) * ROUND_SEC; const realStop = REAL_STOPS_MONTH * window; const falseFailover = threshold <= 2 ? BRIEF_MONTH * (window + PROMOTION * ROUND_SEC) : 0; // threshold 3+ never triggers on a brief outage const total = realStop + falseFailover; console.log(`${String(threshold).padStart(9)}${(window + " sec").padStart(19)}${(b(realStop, 1) + " sec").padStart(20)}` + `${(b(falseFailover, 1) + " sec").padStart(25)}${(b(total / 60, 2) + " min").padStart(15)}` + `${("%" + b((100 * total) / 60 / FAILURE_SHARE, 2)).padStart(18)}`); } const noFailover = REAL_STOPS_MONTH * RECOVERY_SEC; console.log(`\nno failover: ${REAL_STOPS_MONTH} stops/month x ${RECOVERY_SEC} sec = ${b(noFailover / 60, 1)} min, ` + `%${b((100 * noFailover) / 60 / FAILURE_SHARE, 1)} of failure share`); const threshold3 = REAL_STOPS_MONTH * (3 + PROMOTION) * ROUND_SEC; console.log(`same two stops with threshold 3: ${threshold3} sec, ${b(noFailover / threshold3, 1)}x shorter`); console.log(`dropped in the window: ${b((3 + PROMOTION) * ROUND_SEC * PEAK_WRITE)} writes, ` + `${b((3 + PROMOTION) * ROUND_SEC * PEAK_EDGE)} edge requests (at peak load)`); const SILENT_ROUND = 20; // from the model: rounds accepted during the half-failed window but never reaching the store console.log(`\nhalf-failed window: ${SILENT_ROUND} rounds x ${ROUND_SEC} sec = ${SILENT_ROUND * ROUND_SEC} sec, ` + `${b(SILENT_ROUND * ROUND_SEC * PEAK_WRITE)} writes silently lost`); console.log(`this window is invisible in the outage budget: the system was responding`); console.log(); for (const replicas of [2, 3]) { // active-active: the failed replica carries 1/k of the pool console.log(`active-active, ${replicas} replicas: in the removal window, ` + `${b(((3 + PROMOTION) * ROUND_SEC * PEAK_WRITE) / replicas)} writes drop, ` + `in the half-failed window ${b((SILENT_ROUND * ROUND_SEC * PEAK_WRITE) / replicas)} writes are silently lost`); } ``` ``` threshold transition window real stops (month) false failovers (month) total outage of failure share 1 4 sec 8.0 sec 120.0 sec 2.13 min %7.57 2 6 sec 12.0 sec 160.0 sec 2.87 min %10.17 3 8 sec 16.0 sec 0.0 sec 0.27 min %0.95 4 10 sec 20.0 sec 0.0 sec 0.33 min %1.18 no failover: 2 stops/month x 600 sec = 20.0 min, %70.9 of failure share same two stops with threshold 3: 16 sec, 75.0x shorter dropped in the window: 777.76 writes, 4111.12 edge requests (at peak load) half-failed window: 20 rounds x 2 sec = 40 sec, 3888.80 writes silently lost this window is invisible in the outage budget: the system was responding active-active, 2 replicas: in the removal window, 388.88 writes drop, in the half-failed window 1944.40 writes are silently lost active-active, 3 replicas: in the removal window, 259.25 writes drop, in the half-failed window 1296.27 writes are silently lost ``` **The failing day's gain** is the first two lines compared. Without failover, the same two stops last as long as the recovery time: 20.0 minutes, 70.9 percent of the failure share — nearly the entire budget, and a third stop breaks the target that month. With threshold 3, the same two stops take 16 seconds, a 75x reduction, spending 0.95 percent of the failure share. This ratio does not depend on the machine, only on the ratio of recovery time to transition window. **The failure-free day's cost** is in the false-failover column. At threshold 1, 20 transient unresponsive periods a month produce 120 seconds of outage though none is a real failure; at threshold 2, 160 seconds — in both rows exceeding the failing day's cost, 12 seconds at threshold 2. At threshold 3 it drops to zero, trading the real-stop window up from 4 to 8 seconds. A second failure-free-day cost never shows up in the arithmetic: the passive replica carries its full capacity and answers no requests at all. The window's one-time bill is also recorded: an eight-second transition drops 777.76 writes and 4,111.12 edge requests at peak load — a concentration an outage budget, which measures an average, does not show. The last two lines tie the setup difference to a number. In the active–active setup, the transition window drops only the failed replica's share, not the whole service: 388.88 writes with two replicas, 259.25 with three. The same division holds for the silent lost update — 1,296.27 instead of 3,888.80 with three replicas. Active–active does not make the failure cheaper, it **divides** it; K01's conflicting writes are the price of that division. ## Summary - Three decisions remain after the setup choice: the threshold that triggers promotion, the failback policy toward the old active replica, and distinguishing a replica that responds but cannot do its job. - The threshold balances two costs: threshold 1 turned both self-recovering outages into transitions (2 false failovers), threshold 4 delayed detection (15 dropped rounds); total dropped rounds stayed in a narrow 12–15 range. - Failback produces its own window: 4 transitions and 13 dropped rounds under the automatic policy, 1 transition and 7 dropped rounds under the permanent policy. - No threshold catches the half-failed replica: silent lost updates are 20 rounds at all four thresholds; the counter never increases because the shallow check gets a response. - With the KU1–KU3 assumptions, the failing day's gain is a drop from 20.0 minutes to 16 seconds: a 75x reduction, and 0.95 percent of the failure share instead of 70.9 percent. - The failure-free day's cost is 120 seconds of false-failover outage a month at threshold 1, 160 seconds at threshold 2; it zeroes out at threshold 3, in exchange for the real-stop window rising from 4 to 8 seconds. ## Next Step Every decision in this lesson rested on a single input: the yes or no the health check returns. When that input is wrong, both the threshold and the failback policy lose their meaning — in the model's half-failed window, 3,888.80 writes were lost while the system looked healthy. The next lesson designs the check's **content**: whether the endpoint looks at its dependencies, the risk that adding a dependency to the check marks all replicas unhealthy at once, separating being ready to take traffic from being up, and where the check interval sits between detection delay and false failover.