Automatic Failover

The previous lesson measured replication under the assumption that the primary node stayed up: a replica fell behind, overflowed its buffer, underwent a resync, but the node accepting writes always stayed the same. This lesson covers the moment that node stops, and asks two questions together: who decides it has stopped, and what remains once the decision is made.

Who Decides

Being unable to reach a node does not show that the node has stopped; the side that cannot reach it may itself be broken. This is why the decision is not left to a single observer: sentinel processes watch the primary independently, and failover begins only once a sentinel majority votes the same way. The majority is floor(g/2)+1 votes; the mechanics of this pattern, built up in the leader-election and recovery courses, are not repeated here.

This lesson’s question is the in-memory store’s own constraint. The store acknowledges a write from memory and does not wait for a replica’s apply; a backup is by definition behind the primary. The writes it does not have at the moment it takes over are loan records that were accepted and reported successful. Loss on failover is not a possibility, it is a guarantee; only its size remains to be measured.

Mechanism

The mechanism below is a model: no real sentinel process, network, or node is set up. A round is an abstract step; detection threshold, election, and load time are parameters counted in rounds, not measured time.

CU4 — the primary’s temporary pauses (2–7 rounds) and each sentinel’s own unreachability windows (3–6 rounds) come from separate generators and are independent of the setting. Rationale: the schedule is generated ahead of time, with a visible seed, so the same outage sequence can run across every setting; also, a sentinel’s own fault is indistinguishable from a pause in the node it is watching. CU5 — after a false failover, the old primary keeps accepting writes for 3 rounds. Rationale: learning that it has been displaced is not instantaneous; the records written during those rounds are split writes, discarded once the node is demoted to backup.

// failover/model.mjs — sentinel-driven failover is an IN-PROCESS MODEL. A round is an abstract
// step; detection threshold, election, and load time are parameters counted in rounds, not
// measured time. No real sentinel process, network, or node is set up. The previous lesson's
// constraint still holds: replication is asynchronous, so writes the backup has not applied are
// writes that were ACCEPTED but never arrived.
export const ENTRY_BYTES = 240, RECORD_BYTES = 96, KEYS = 2000;

export function generator(seed) {  // linear congruential generator; seed is visible
  let s = seed >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; };
}

export function run({ sentinels = 3, downSentinels = 0, detection = 3, election = 2, delay = 2,
                      cold = null, rounds = 300, writes = 40, outage = 200, splitRounds = 3,
                      pauseProb = 0.02, selfProb = 0.03, seed = 20260731 }) {
  const rnd = generator(seed);
  const schedule = (prob, minLen, variety) => {             // schedules are generated independent of setting
    const d = new Uint8Array(rounds + 2);
    for (let t = 1; t <= rounds; t += 1) {
      if (d[t] || rnd() >= prob) continue;
      for (let x = t, b = minLen + Math.floor(rnd() * variety); x < t + b && x <= rounds; x += 1) d[x] = 1;
    }
    return d;
  };
  const pause = schedule(pauseProb, 2, 6);                  // the primary's temporary pauses
  const own = [...Array(7)].map(() => schedule(selfProb, 3, 4));  // a sentinel's OWN unreachability
  const majority = Math.floor(sentinels / 2) + 1, voters = sentinels - downSentinels;
  const wait = election + (cold ? cold.load : 0), counter = Array(7).fill(0);
  let accepted = 0, applied = 0, outageActive = true, remaining = 0, cause = null, splitRemaining = 0;
  let writelessRounds = 0, realFailovers = 0, falseFailovers = 0, lostWrites = 0, splitWrites = 0,
      outageLoss = 0, outageDuration = null;

  for (let t = 1; t <= rounds; t += 1) {
    if (splitRemaining > 0) { splitRemaining -= 1; splitWrites += writes; }  // old primary still writing
    const unreachable = (outageActive && t >= outage) || pause[t] === 1;
    if (remaining > 0) {                                    // failover window: writes not accepted
      remaining -= 1; writelessRounds += 1;
      if (remaining > 0) continue;
      const floor = cold                                    // cold backup starts from the last snapshot
        ? Math.floor(accepted / (cold.interval * writes)) * cold.interval * writes : applied;
      lostWrites += accepted - floor;
      if (cause === "real") { realFailovers += 1; outageActive = false; outageDuration = t - outage + 1; outageLoss = accepted - floor; }
      else { falseFailovers += 1; splitRemaining = splitRounds; }
      accepted = floor; applied = floor; counter.fill(0);
      for (let x = t; x <= rounds && pause[x] === 1; x += 1) pause[x] = 0;  // new primary is not in that pause
      continue;
    }
    if (unreachable) writelessRounds += 1;
    else { accepted += writes; applied = Math.max(applied, accepted - delay * writes); }
    let votes = 0;                                          // each sentinel votes by its OWN view
    for (let j = 0; j < voters; j += 1) {
      counter[j] = unreachable || own[j][t] === 1 ? counter[j] + 1 : 0;
      if (counter[j] >= detection) votes += 1;
    }
    if (votes >= majority) { remaining = wait; cause = outageActive && t >= outage ? "real" : "false"; }
  }
  return { majority, voters, realFailovers, falseFailovers, lostWrites, outageLoss, splitWrites,
           writelessRounds, outageDuration,
           data: (cold ? 1 : 2) * KEYS * ENTRY_BYTES };      // hot backup keeps the data set in memory
}

// failover/measure.mjs — same outage across three settings: sentinel majority, detection threshold, backup kind
import { run, ENTRY_BYTES, RECORD_BYTES, KEYS } from "./model.mjs";

const s = (x, n) => String(x).padStart(n);
const e = (x, n) => String(x).padEnd(n);
const kib = (b) => (b / 1024).toFixed(1) + " KiB";
const row = (cells, widths) => cells.map((c, i) => s(c, widths[i])).join(" | ");

console.log("300 rounds; each round writes 40 loan records. Primary stops permanently at round 200. Seed");
console.log("20260731: temporary pauses of 2-7 rounds at the primary, each sentinel gets its own");
console.log("unreachability windows of 3-6 rounds. Delay 2 rounds, detection 3, election 2 rounds.\n");
const W1 = [9, 4, 8, 6, 16, 16, 9];
console.log(row(["sentinels", "down", "majority", "voters", "outage duration", "false failovers", "writeless"], W1));
console.log(W1.map((w) => "-".repeat(w)).join("-|-"));
for (const [g, d] of [[1, 0], [2, 0], [3, 0], [3, 1], [3, 2], [5, 0], [5, 2], [5, 3]]) {
  const r = run({ sentinels: g, downSentinels: d });
  console.log(row([g, d, r.majority, r.voters, r.outageDuration === null ? "n/a" : r.outageDuration + " rd",
    r.falseFailovers, r.writelessRounds + " rd"], W1));
}

console.log("\n3 sentinels, all up; detection threshold varies:");
const W2 = [9, 16, 12, 16, 9, 10];
console.log(row(["threshold", "false failovers", "split writes", "outage duration", "writeless", "total lost"], W2));
console.log(W2.map((w) => "-".repeat(w)).join("-|-"));
for (const a of [1, 2, 3, 4, 5, 6]) {
  const r = run({ detection: a });
  console.log(row([a, r.falseFailovers, r.splitWrites, r.outageDuration + " rd", r.writelessRounds + " rd",
    r.lostWrites], W2));
}

console.log("\nBackup kind. Temporary pauses off: a single-outage failover. Cold backup's data is not");
console.log("in memory; the last snapshot is 30 rounds old, loading it takes 12 rounds:");
const W3 = [13, 9, 16, 11, 10];
console.log(e("backup", W3[0]) + " | " + row(["data held", "outage duration", "lost writes", "lost bytes"], W3.slice(1)));
console.log(W3.map((w) => "-".repeat(w)).join("-|-"));
for (const g of [1, 2, 4, 8]) {
  const r = run({ delay: g, pauseProb: 0, selfProb: 0 });
  console.log(e("hot, delay " + g, W3[0]) + " | " +
    row([kib(r.data), r.outageDuration + " rd", r.outageLoss, r.outageLoss * RECORD_BYTES + " B"], W3.slice(1)));
}
const c = run({ pauseProb: 0, selfProb: 0, cold: { load: 12, interval: 30 } });
console.log(e("cold backup", W3[0]) + " | " +
  row([kib(c.data), c.outageDuration + " rd", c.outageLoss, c.outageLoss * RECORD_BYTES + " B"], W3.slice(1)));

console.log("\nquantities independent of the run:");
console.log("  sentinel majority   = floor(g/2)+1; sentinel loss tolerated = g - majority");
console.log("  g        : " + [1, 2, 3, 4, 5, 6, 7].map((n) => s(n, 4)).join(""));
console.log("  majority : " + [1, 2, 3, 4, 5, 6, 7].map((n) => s(Math.floor(n / 2) + 1, 4)).join(""));
console.log("  tolerated: " + [1, 2, 3, 4, 5, 6, 7].map((n) => s(n - (Math.floor(n / 2) + 1), 4)).join(""));
console.log("  lost per failover = delay x write rate (hot backup)");
console.log("  delay    : " + [1, 2, 4, 8].map((n) => s(n, 6)).join(""));
console.log("  lost     : " + [1, 2, 4, 8].map((n) => s(n * 40, 6)).join(""));
console.log("  outage duration      = detection threshold + election (+ load time for cold backup) rounds");
console.log("  worst-case loss for cold backup = snapshot interval x write rate = 30 x 40 = 1200");
console.log("  hot backup's cost = " + (KEYS * ENTRY_BYTES / 1024).toFixed(1) +
  " KiB, i.e. a second full copy of the data set (" + KEYS + " x " + ENTRY_BYTES + " bytes)");

300 rounds; each round writes 40 loan records. Primary stops permanently at round 200. Seed
20260731: temporary pauses of 2-7 rounds at the primary, each sentinel gets its own
unreachability windows of 3-6 rounds. Delay 2 rounds, detection 3, election 2 rounds.

sentinels | down | majority | voters |  outage duration |  false failovers | writeless
----------|------|----------|--------|------------------|------------------|----------
        1 |    0 |        1 |      1 |             6 rd |               12 |     40 rd
        2 |    0 |        2 |      2 |             5 rd |                6 |     29 rd
        3 |    0 |        2 |      3 |             5 rd |                9 |     35 rd
        3 |    1 |        2 |      2 |             5 rd |                6 |     29 rd
        3 |    2 |        2 |      1 |              n/a |                0 |    115 rd
        5 |    0 |        3 |      5 |             5 rd |                4 |     25 rd
        5 |    2 |        3 |      3 |             5 rd |                4 |     25 rd
        5 |    3 |        3 |      2 |              n/a |                0 |    115 rd

3 sentinels, all up; detection threshold varies:
threshold |  false failovers | split writes |  outage duration | writeless | total lost
----------|------------------|--------------|------------------|-----------|-----------
        1 |               13 |         1560 |             3 rd |     33 rd |        960
        2 |               11 |         1320 |             5 rd |     34 rd |        880
        3 |                9 |         1080 |             5 rd |     35 rd |        800
        4 |                4 |          480 |             6 rd |     28 rd |        400
        5 |                2 |          240 |             7 rd |     26 rd |        240
        6 |                2 |          240 |             8 rd |     28 rd |        240

Backup kind. Temporary pauses off: a single-outage failover. Cold backup's data is not
in memory; the last snapshot is 30 rounds old, loading it takes 12 rounds:
backup        | data held |  outage duration | lost writes | lost bytes
--------------|-----------|------------------|-------------|-----------
hot, delay 1  | 937.5 KiB |             5 rd |          40 |     3840 B
hot, delay 2  | 937.5 KiB |             5 rd |          80 |     7680 B
hot, delay 4  | 937.5 KiB |             5 rd |         160 |    15360 B
hot, delay 8  | 937.5 KiB |             5 rd |         320 |    30720 B
cold backup   | 468.8 KiB |            17 rd |         760 |    72960 B

quantities independent of the run:
  sentinel majority   = floor(g/2)+1; sentinel loss tolerated = g - majority
  g        :    1   2   3   4   5   6   7
  majority :    1   2   2   3   3   4   4
  tolerated:    0   0   1   1   2   2   3
  lost per failover = delay x write rate (hot backup)
  delay    :      1     2     4     8
  lost     :     40    80   160   320
  outage duration      = detection threshold + election (+ load time for cold backup) rounds
  worst-case loss for cold backup = snapshot interval x write rate = 30 x 40 = 1200
  hot backup's cost = 468.8 KiB, i.e. a second full copy of the data set (2000 x 240 bytes)

Two Faces of the Majority

When the majority cannot be reached, failover never starts. When two of three sentinels or three of five go down, the surviving sentinels correctly see that the primary has stopped; the vote count still falls short of the majority. The result is in the last column: 115 writeless rounds. This is the rule itself: a sentinel cannot decide until the others confirm what it sees.

The majority mainly guards against a sentinel’s own misjudgment. In the single-sentinel row, the majority is 1, and the sentinel’s own unreachability windows turn directly into decisions: 12 false failovers. At five sentinels, the majority rises to 3, and with the same schedule false failovers drop to 4. The primary’s outage schedule is identical across every row; what changes is whether one wrong sentinel is enough on its own.

The three-sentinel row looks backward at first glance: the majority is still 2, yet false failovers climb from 6 to 9. The reason is the voter count — at two sentinels, both must be wrong at once; at three, any two of them being wrong is enough, three times as many pairs. The confirmation is in the same table: when one of the three sentinels goes down, the row produces exactly the same numbers as the two-sentinel row.

The Price of the Threshold

The second table measures the detection threshold in both directions at once. At threshold 1, the real outage’s duration drops to 3 rounds; the same row shows 13 false failovers and 1,560 split-write records. At threshold 6, false failovers drop to 2 and split writes to 240; in exchange, the outage duration climbs to 8 rounds.

Split writes are the most expensive line item in this trade-off. In a false failover, the old primary is healthy and keeps accepting loan records for three rounds; these records never reach the new primary, and are discarded once the old node is demoted to backup. The difference between threshold 1 and threshold 6 in this line item is 1,320 records.

The total-lost column moves in the same direction (960 versus 240): every failover, right or wrong, produces its own loss. The writeless-round column is not one-directional (33, 34, 35, 28, 26, 28) — raising the threshold extends the outage on the real failure but closes the windows opened by false failovers.

The Backup’s Memory Cost

The third table turns off temporary pauses and isolates a single real outage. In the hot-backup rows, the loss is exactly delay × write rate: 40, 80, 160, and 320 loan records. The outage duration is 5 rounds across all four rows; delay grows the loss, not the outage.

The cold-backup row shows this course’s rule directly. When the backup’s data is not held in memory, data held drops from 937.5 KiB to 468.8 KiB. In exchange, the outage duration climbs from 5 rounds to 17, and the loss climbs from 80 records to 760 (72,960 bytes; up to 1,200 records in the worst case). The hot backup’s 468.8 KiB cost buys exactly this: a third of the outage and a tenth of the loss.

Summary

• The sentinel majority is floor(g/2)+1; when two of three sentinels or three of five went down, failover never started, and not a single loan record was accepted in the 115 rounds after the outage.
• The majority rule mainly guards against a sentinel’s own misjudgment: with the same outage schedule, 12 false failovers were measured at one sentinel, 4 at five.
• Adding sentinels without raising the majority ratio works in reverse: two sentinels (majority 2) gave 6 false failovers, three sentinels (majority still 2) gave 9.
• The detection threshold cuts both ways: at threshold 1, the outage is 3 rounds, false failovers 13, split writes 1,560 records; at threshold 6, the outage is 8 rounds, false failovers 2, split writes 240.
• Asynchronous replication guarantees a per-failover loss, and its size is delay × write rate: 40, 80, 160, 320 loan records — all of them records that were reported successful.
• A hot backup holds a second full copy of the data set, 468.8 KiB; in exchange it cuts the outage from 17 rounds to 5 and the loss from 760 records to 80.

Next Step

The two lessons so far carried the same implicit assumption: the entire data set fits on a single node. Replication produced copies of that set, failover promoted one of the copies into service; neither one split the set. When catalog and loan records outgrow a single node’s memory budget, only one path remains: splitting the key space. Its cost shows up immediately — if two keys touched by the same transaction land on different nodes, that transaction cannot run on a single node. The next lesson counts how a key is mapped to a partition, how many multi-key transactions run into the cluster’s cross-node constraint, and how much tagging keys into the same partition unbalances the bytes held per node.