--- title: 'Leader Election' source: 'https://academia.sh/en/courses/application-layer/leader-election' course: 'The Application Layer and Service Interaction' language: en updated: '2026-08-23T14:25:21+00:00' license: 'CC BY-SA 4.0' --- # Leader Election Assigning a task that requires singular responsibility to a single node: building lease-based leader election with an in-process model, measuring the handover round and the number of rounds two leaders run at once, the leaderless round growing as the lease lengthens, fencing removing double starts, the persistent trigger recovering the missed batch, and translating the results into K01's daily invoice lines. The previous lesson's supervisor was treated as a single party. In reality, the supervisor, and the scheduler that starts the end-of-day job, both run on more than one node — because a supervisor running on a single node becomes, when it goes down, exactly the kind of workflow no one notices. But the scheduler's job is singular: end-of-day billing must be started exactly once. If two nodes start it at the same time, two invoice lines are produced for the same seller-day, and that can only be fixed by hand, not by compensation. **Leader election** is the arrangement that gives responsibility to exactly one node from a set of nodes for a defined period. It is related to the handover measured in the Introduction to System Design course, but the question differs: there, handover was done to **sustain the service** — a replica accepting writes had to remain; here, election is done for **singular responsibility** — exactly one party must do the work. The first kind's failure is an outage; the second kind's is a double job. ## Lease and Fencing Token Responsibility is not granted indefinitely, because no one can know for certain that the node holding it has gone down. Instead, a **lease** is granted: the leader signs the shared durable record at regular intervals. If the record has not been refreshed for the length of the lease, another node takes over responsibility and increments the record's **fencing token** counter by one. The token says which ordinal a leadership was granted in; it has nothing to do with billing's thirty-day period. The lease does not distinguish a node that is down from one that is up; it only distinguishes one that **cannot write.** A leader that has paused, dropped off the network, or slowed down is still up, and keeps believing itself to be the leader. This is why the number that matters is not the handover round, but **the number of rounds in which two nodes believe themselves to be the leader at the same time.** The model has 5 nodes; the first leader is unreachable for five rounds starting on round 6, and the end-of-day batch must be started every six rounds. ```js // leader/election.mjs — lease-based leader election; a round-based in-process model. A round // is an abstract step; outage, lease, and renewal are interval parameters, not measured durations. export const TRIGGER = [6, 12, 18, 24]; // rounds at which the end-of-day batch must start export const START = 6; // first round the leader becomes unreachable export function run({ nodes = 5, lease = 3, renewal = 3, outage = 5, fencing = false, persistentTrigger = false, rounds = 24 }) { const end = START + outage - 1; const record = { leader: 0, lastRenewal: 0, token: 1 }; // shared durable record const believes = new Array(nodes).fill(false); believes[0] = true; const nodeToken = new Array(nodes).fill(0); nodeToken[0] = 1; const s = { started: 0, missed: 0, doubleStarts: 0, rejected: 0, delayedRounds: 0, twoLeaderRounds: 0, leaderlessRounds: 0, handoverRounds: 0, handovers: 0 }; let pending = 0, pendingRound = 0; const reachable = (d, t) => d !== 0 || t < START || t > end; for (let t = 1; t <= rounds; t++) { if (reachable(record.leader, t)) record.lastRenewal = t; // the leader renews its lease if (t - record.lastRenewal >= lease) { // lease expired: a node takes over const newLeader = [...Array(nodes).keys()].find((d) => d !== record.leader && reachable(d, t)); s.handoverRounds += t - record.lastRenewal; s.handovers += 1; record.leader = newLeader; record.lastRenewal = t; record.token += 1; believes[newLeader] = true; nodeToken[newLeader] = record.token; } const active = [...Array(nodes).keys()].filter((d) => believes[d] && reachable(d, t)); if (active.length > 1) s.twoLeaderRounds += 1; if (active.length === 0) s.leaderlessRounds += 1; let accepted = 0; for (const d of active) { if (fencing && nodeToken[d] < record.token) { s.rejected += 1; continue; } // stale token accepted += 1; } if (TRIGGER.includes(t)) { // the end-of-day batch must start if (accepted === 0) { if (persistentTrigger) { pending += 1; pendingRound = t; } else s.missed += 1; } else { s.started += 1; s.doubleStarts += accepted - 1; } } else if (pending > 0 && accepted > 0) { // persistent trigger: the new leader takes over s.started += 1; s.delayedRounds += t - pendingRound; pending = 0; } if (t % renewal === 0) for (const d of active) // node reads the record, drops belief if needed if (record.leader !== d) believes[d] = false; } return s; } ``` The model has two add-on options. **Fencing** is doing every piece of work together with this token: the record rejects the work of a node whose token has fallen behind. Fencing was established in the Relational Database Administration course; here it is applied to singular responsibility. The **persistent trigger** is recording a start request that arrives in a leaderless round, so that the next leader takes it over. ```js // leader/measure.mjs — a run without and with fencing, then a scan of the lease duration import { run, TRIGGER, START } from "./election.mjs"; const D = [["plain", run({})], ["fencing", run({ fencing: true })], ["fencing+trigger", run({ fencing: true, persistentTrigger: true })]]; const MEASURES = [["started", "batches started"], ["missed", "batches missed"], ["delayedRounds", "delayed start rounds"], ["doubleStarts", "double-started batches"], ["rejected", "rejected by fencing"], ["twoLeaderRounds", "two-leader rounds"], ["leaderlessRounds", "leaderless rounds"], ["handoverRounds", "handover rounds"]]; console.log(`5 nodes, 24 rounds, lease 3, renewal 3; the leader node is unreachable in rounds ` + `${START}-${START + 4}. End-of-day batch trigger: rounds ${TRIGGER.join(", ")}.`); console.log(); console.log(`${"measure".padEnd(24)}${D.map(([a]) => a.padStart(22)).join("")}`); for (const [k, label] of MEASURES) console.log(`${label.padEnd(24)}${D.map(([, r]) => String(r[k]).padStart(22)).join("")}`); const S = ["handoverRounds", "leaderlessRounds", "missed", "twoLeaderRounds", "doubleStarts"]; const heads = ["handover", "leaderless", "missed", "2-leader", "double"]; console.log(`\n${"".padStart(6)}${"long outage (5 rounds)".padStart(50)}${"short outage (2 rounds)".padStart(50)}`); console.log(`${"lease".padStart(6)}${heads.map((a) => a.padStart(11)).join("").repeat(2)}`); for (const lease of [2, 3, 5, 8]) { const row = [5, 2].flatMap((outage) => { const r = run({ lease, outage, fencing: false }); return S.map((k) => String(r[k]).padStart(11)); }); console.log(`${String(lease).padStart(6)}${row.join("")}`); } ``` ``` 5 nodes, 24 rounds, lease 3, renewal 3; the leader node is unreachable in rounds 6-10. End-of-day batch trigger: rounds 6, 12, 18, 24. measure plain fencing fencing+trigger batches started 3 3 4 batches missed 1 1 0 delayed start rounds 0 0 2 double-started batches 1 0 0 rejected by fencing 0 2 2 two-leader rounds 2 2 2 leaderless rounds 2 2 2 handover rounds 3 3 3 long outage (5 rounds) short outage (2 rounds) lease handover leaderless missed 2-leader double handover leaderless missed 2-leader double 2 2 1 1 2 1 2 1 1 2 0 3 3 2 1 2 1 0 2 1 0 0 5 5 4 1 2 1 0 2 1 0 0 8 0 5 1 0 0 0 2 1 0 0 ``` ## Reading the Numbers The plain run could start only three of the four triggers. On round 6 the leader became unreachable, and because the lease had not yet expired, no one took over: **the batch never started.** The handover took three rounds to complete, and two rounds passed leaderless. On round 11 the old leader came back and still believed itself to be the leader; both it and the new leader saw the trigger on round 12, and the batch was started twice. The plain arrangement's balance sheet is this: one batch was missed, one batch ran twice, and one ran correctly. Three rows must be kept separate. `two-leader rounds` is the number of rounds in which two nodes **believe themselves to be the leader**, and it is 2 in all three columns; fencing does not change this number, because fencing does not correct a node's belief. What changes is the `double-started batches` row: it drops from 1 to 0, and in exchange `rejected by fencing` rises from 0 to 2. The record rejected the work of the node whose token had fallen behind. This distinction between belief and effect is the core of the pattern: **leader election determines who is the leader; fencing determines who can write.** The first is not enough without the second. The third column closes the second failure. With the persistent trigger, missed batches drop to 0, batches started rise from 3 to 4, and the cost is a two-round delay. The trigger being momentary ties it to a point in time; recording it turns it into a job. This is the same rule as the previous lesson's scheduler: an intent that sits only in memory cannot survive a failure. The table below shows what the lease buys. As the lease rises from 2 to 5, the handover round climbs from 2 to 5 and leaderless rounds from 1 to 4; at lease 8, no handover happens at all, and 5 rounds pass leaderless across the whole outage. By contrast, the `2-leader` column is the same at lease 2, 3, and 5 — 2. **A longer lease does not narrow the two-leader window**, because that window arises not from the lease but from when the returning node next reads the record. The column reading 0 at lease 8 is not a win: no handover happened there, so a second leader never appeared, and the system stayed unaccountable for all five rounds. The block on the right shows the short outage: at a two-round slowdown, a lease of 3 or higher produces no handover at all, while a lease of 2 produces an unnecessary handover and two rounds of two leaders. A short lease mistakes a temporary slowdown for a failure; a long lease notices a real failure late. The correct setting sits between the two, and the measure for the decision is the sum of leaderless rounds and two-leader rounds. ## Back to the Estimate The model's rounds connect to K01's numbers in two steps. What the model gives is a **window share**: in a run with one handover, 2/24 of the rounds passed leaderless and 2/24 passed with two leaders. The rate at which the trigger falls into these windows is the same share as reading that the trigger is spread evenly across rounds. **KK4 — 6 handovers a year.** This assumption is not new: the same number was used when the Introduction to System Design course's Availability Patterns lesson converted the handover window into a request rate, and it carries forward here for the same reason. It is not added to K01's assumption table; its sensitivity is calculated at 12. One end-of-day batch produces 4,000 invoice lines by K01's accounting. ```js // leader/cost.mjs — applies the model's window ratios to K01's end-of-day batch import { run } from "./election.mjs"; const INVOICE_LINES = 4000; // K01: daily invoice lines (computed value) = one end-of-day batch const ROUNDS = 24; const plain = run({}), fenced = run({ fencing: true, persistentTrigger: true }); const leaderlessShare = plain.leaderlessRounds / ROUNDS, twoLeaderShare = plain.twoLeaderRounds / ROUNDS; console.log(`model window: leaderless ${plain.leaderlessRounds}/${ROUNDS} = ${leaderlessShare.toFixed(4)}, ` + `two-leader ${plain.twoLeaderRounds}/${ROUNDS} = ${twoLeaderShare.toFixed(4)}`); console.log(`one batch = ${INVOICE_LINES} invoice lines (K01, computed value)`); console.log(); console.log(`${"KK4".padStart(5)}${"missed batches/yr".padStart(21)}${"lines not produced".padStart(19)}` + `${"double batches/yr".padStart(18)}${"double lines".padStart(13)}`); for (const KK4 of [6, 12]) { // KK4: handovers per year (M19/K01 Availability Patterns) const missed = KK4 * leaderlessShare, double = KK4 * twoLeaderShare; console.log(`${String(KK4).padStart(5)}${missed.toFixed(2).padStart(21)}` + `${(missed * INVOICE_LINES).toFixed(0).padStart(19)}${double.toFixed(2).padStart(18)}` + `${(double * INVOICE_LINES).toFixed(0).padStart(13)}`); } console.log(`\nwith fencing + persistent trigger: missed batches ${fenced.missed}, ` + `double-started batches ${fenced.doubleStarts}, delayed start ${fenced.delayedRounds} rounds`); console.log(`with lease 8 (long): handover rounds ${run({ lease: 8 }).handoverRounds}, ` + `leaderless rounds ${run({ lease: 8 }).leaderlessRounds}, two-leader rounds ${run({ lease: 8 }).twoLeaderRounds}`); ``` ``` model window: leaderless 2/24 = 0.0833, two-leader 2/24 = 0.0833 one batch = 4000 invoice lines (K01, computed value) KK4 missed batches/yr lines not produced double batches/yr double lines 6 0.50 2000 0.50 2000 12 1.00 4000 1.00 4000 with fencing + persistent trigger: missed batches 0, double-started batches 0, delayed start 2 rounds with lease 8 (long): handover rounds 0, leaderless rounds 5, two-leader rounds 0 ``` At six handovers a year, the expected missed batch is 0.50 and the expected double batch is 0.50. Converted to invoice lines: once every two years a 4,000-line day is never produced, and once every two years one is produced twice; the annual expectation is 2,000 lines either way. If the handover count doubles, that becomes one event a year and 4,000 lines each. The size of these numbers says where the decision should be anchored. Measured in service availability, two leaderless rounds are nothing; they do not show up in K01's outage budget. Measured in invoice lines, the same two rounds mean 2,000 duplicate lines a year, each one posting the wrong amount to a seller. For work that requires singular responsibility, the unit of measure is not duration but **the job**. Fencing brings the duplicate line to zero, the persistent trigger brings the unproduced line to zero; together the two cost a two-round delay. Leader election on its own removes neither of these two failures. ## Summary - Leader election and the handover that sustains a service are separate questions: in one, the goal is that a replica accepting writes remains; in the other, that the work is done exactly once. - The lease distinguishes a node that cannot write, not one that is down; a paused leader still believes itself to be the leader when it comes back. - In the plain run, 3 of 4 triggers were started: 1 batch was missed (it arrived in a leaderless round), 1 batch was started twice (it arrived in a two-leader round), and the handover took 3 rounds to complete. - Fencing corrects effect, not belief: two-leader rounds stayed at 2, while double-started batches dropped from 1 to 0 and rejected work rose from 0 to 2. The persistent trigger brought missed batches to 0 and added a two-round delay. - As the lease lengthens, the handover round climbs from 2 to 5 and leaderless rounds from 1 to 4; the two-leader window does not change, because its source is not the lease but the moment the returning node re-reads the record. - Back to K01: at KK4 = 6 handovers a year, the expected missed batch is 0.50 and the double batch 0.50, that is, 2,000 unproduced and 2,000 duplicate invoice lines a year; at 12 handovers, these become 4,000 each. ## Next Step Every lesson in this course had the work run detached from the party that made the request: the state event was written to a queue, the workflow ran in the background, the end-of-day batch was started by a leader. One question always stayed outside — **how the requesting party learns the result.** When a seller requests a thirty-day billing report, the job is queued and finishes seconds later; but the seller's browser is waiting for a response, and the connection cannot be held open that long. The next lesson builds the arrangement that closes this gap: the request is answered immediately with a job id, the result is fetched from a separate address, and how many times the client asks before it sees the result is measured.