--- title: 'Redundancy and Failover' source: 'https://academia.sh/en/courses/switching-and-routing/redundancy-and-failover' course: 'Switching and Routing' language: en updated: '2026-08-17T18:07:21+00:00' license: 'CC BY-SA 4.0' --- # Redundancy and Failover The failover window in gateway redundancy is measured: when the upstream link breaks and nobody has heard, all forty flows fall into a black hole one hop later; when the gateway fails over, the black hole clears but the peer link carries forty crossings; when two gateways each believe themselves active, all forty flows enter a loop and the peer link carries eighty crossings. In a spine–leaf design, redundancy was structural: no device was the sole owner of a path, and the loss of a spine only changed how flows were distributed. On the host side, this is not the case. A host sends its outbound packet not to a path but to an address — the default gateway's address. The address is singular, and at any moment it sits on exactly one device. This singularity takes redundancy out of the structural category. When the device fails, a second device must take over the address, and everyone must hear about the takeover. This lesson's question is what happens to the packet during that short interval while the hearing is still in progress. ## The Virtual Address and Two Roles **Gateway redundancy** puts two devices behind a single address. This address is called the **virtual address**; hosts know it as their default gateway and are unaware that two devices exist. One of the devices is **active**, the other is **standby**. The active one answers for the virtual address and forwards packets upward; the standby one listens silently. When the active device's periodic advertisements stop arriving, the standby waits for a period and then takes over the address. ```text # taught transcript, not run gateway pair virtual address single address, hosts' default gateway g1 priority 110 active -> answers the virtual address, forwards upward g2 priority 100 standby -> listens silently, waits for advertisements peer link g1 - g2, carries advertisements and state timing advertisement interval active side advertises once per interval dead timer standby takes over if no advertisement heard in this time failover instant standby begins answering for the virtual address leaf table (for remote networks) destination next hop u1 u2 u3 the gateway believed to currently hold the virtual address ``` Service-level failover patterns — health checks, retries, the circuit breaker — were established in the Resilience and Reliability course and are not repeated here. There, what failed over was a service instance, and what was measured was the request's fate; here, what fails over is the **gateway**, and what is measured is the packet's fate. ## What the Window Is Failover is not instantaneous. The time it takes for the dead timer to expire, for the standby to announce the address, and for switches along the path to relearn where the address is — the sum of these is called the **failover window**. During the window, devices disagree about where the virtual address is. Inside the window there are two separate questions. First: **who answers for the virtual address?** Second: **can the answering device forward the packet?** The measurement counts exactly this distinction: a gateway whose upstream link has broken keeps collecting packets for as long as it holds the address. The assumptions the measurement rests on: - **TD17** — Nine leaves (`y1`-`y9`) connect to a gateway pair (`g1`, `g2`); the pair connects to a core (`c`), and the core connects to three remote networks (`u1`, `u2`, `u3`). The forty flows have a leaf as their source and a remote network as their destination; the traffic measured is entirely north-south. - **TD18** — The failure measured is the breaking of the `g1-c` upstream link. `g1` is healthy, remains connected to the leaves, and can continue to hold the virtual address. - **TD19** — Every leaf holds a belief: whether the virtual address is on `g1` or `g2`. A leaf that has heard says `g2`; one that has not says `g1`. - **TD20** — Every gateway also holds a belief. A gateway that believes itself active forwards the packet upward; one that believes its peer active hands the packet to its peer over the peer link. - **TD21** — The oracle is known because we built the setup ourselves: after `g1-c` breaks, the only path to the remote networks is through `g2`. - **TD22** — The regimes differ only in belief; the topology does not change except for the presence of the `g1-c` link. So the difference measured is a difference in information, not in structure. - **TD23** — The "peer link" column counts how many times the forty flows cross the `g1-g2` link in total. This link is sized for control traffic; it carrying user traffic is a symptom. - **TD24** — A packet is counted as a loop when it crosses the same link a second time or burns through the hop limit (12); a packet whose next hop is no longer a neighbor is counted as a black hole. - **TD25** — The hops column counts only the hops spent; a packet that falls into a black hole has spent the hops up to where it died. - **TD26** — In a set of forty flows, the smallest measurable difference is 1/40 = 0.025. ## The Measurement ```python """Gateway redundancy: the packet's fate during the failover window. Part 1 - regimes: converged, window, upstream-link tracking, split-brain. Part 2 - the change in fate as the number of leaves that have heard grows. """ SEED = 20260810 HOP_LIMIT = 12 LEAF = [f"y{i}" for i in range(1, 10)] REMOTE = ["u1", "u2", "u3"] NODES = LEAF + ["g1", "g2", "c"] + REMOTE LINKS = ([(y, g) for y in LEAF for g in ("g1", "g2")] + [("g1", "g2"), ("g1", "c"), ("g2", "c")] + [("c", u) for u in REMOTE]) BROKEN = [b for b in LINKS if b != ("g1", "c")] def make_rng(seed): state = seed % 2147483646 + 1 def rand(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return rand def neighbors(links): k = {u: set() for u in NODES} for x, y in links: k[x].add(y) k[y].add(x) return k def build_tables(leaf_belief, g1_belief, g2_belief): """The next hop each node holds for the remote networks.""" t = {} for y in LEAF: for u in REMOTE: t[(y, u)] = leaf_belief[y] for g, peer, belief in (("g1", "g2", g1_belief), ("g2", "g1", g2_belief)): for u in REMOTE: t[(g, u)] = "c" if belief == g else peer for u in REMOTE: t[("c", u)] = u return t def forward(source, dest, tab, links): k, u, crossed, hops = neighbors(links), source, [], 0 while u != dest: if hops >= HOP_LIMIT: return "loop", hops nxt = tab.get((u, dest)) if nxt is None or nxt not in k[u]: return "black hole", hops if (u, nxt) in crossed: return "loop", hops crossed.append((u, nxt)) u, hops = nxt, hops + 1 return "reached", hops def pairs(count=40, seed=SEED): rand, out = make_rng(seed), [] while len(out) < count: out.append((LEAF[rand(9)], REMOTE[rand(3)])) return out def peer_crossings(tab, links): """How many of the forty flows cross the g1-g2 peer link.""" k, count = neighbors(links), 0 for x, y in pairs(): u, crossed, hops = x, [], 0 while u != y and hops < HOP_LIMIT: v = tab.get((u, y)) if v is None or v not in k[u] or (u, v) in crossed: break if {u, v} == {"g1", "g2"}: count += 1 crossed.append((u, v)) u, hops = v, hops + 1 return count def measure(tab, links): counts, hops = {"reached": 0, "loop": 0, "black hole": 0}, 0 for x, y in pairs(): fate, a = forward(x, y, tab, links) counts[fate] += 1 hops += a return counts, hops def belief(heard): return {y: ("g2" if y in heard else "g1") for y in LEAF} ALL, NONE = set(LEAF), set() REGIME = [ ("converged, g1 active", LINKS, belief(NONE), "g1", "g1"), ("upstream link broke, nobody heard", BROKEN, belief(NONE), "g1", "g1"), ("g1 failed over, no leaves heard", BROKEN, belief(NONE), "g2", "g2"), ("g1 failed over, four leaves heard", BROKEN, belief(set(LEAF[:4])), "g2", "g2"), ("g1 failed over, all heard", BROKEN, belief(ALL), "g2", "g2"), ("both think themselves active", LINKS, belief(set(LEAF[:5])), "g1", "g2"), ("both think the peer active", LINKS, belief(set(LEAF[:5])), "g2", "g1"), ] print(f"{'regime':<35s} {'reached':>7s} {'loop':>6s} {'black hole':>10s} " f"{'hops':>5s} {'peer link':>9s}") for name, links, lb, a1, a2 in REGIME: tab = build_tables(lb, a1, a2) s, a = measure(tab, links) print(f"{name:<35s} {s['reached']:7d} {s['loop']:6d} {s['black hole']:10d} " f"{a:5d} {peer_crossings(tab, links):9d}") print() print(f"{'leaves heard':>12s} {'g1 did not fail over':>32s} {'g1 failed over':>32s}") print(f"{'':>12s} {'reached black hole hops peer':>32s}" f" {'reached black hole hops peer':>32s}") for n in range(0, 10): lb, columns = belief(set(LEAF[:n])), [] for g1b in ("g1", "g2"): tab = build_tables(lb, g1b, "g2") s, a = measure(tab, BROKEN) columns.append(f"{s['reached']:9d}{s['black hole']:11d}{a:5d}" f"{peer_crossings(tab, BROKEN):5d}") print(f"{n:12d} {columns[0]} {columns[1]}") print() print(f"nodes {len(NODES)}, links {len(LINKS)}, flows 40, " f"hop limit {HOP_LIMIT}") ``` ``` regime reached loop black hole hops peer link converged, g1 active 40 0 0 120 0 upstream link broke, nobody heard 0 0 40 40 0 g1 failed over, no leaves heard 40 0 0 160 40 g1 failed over, four leaves heard 40 0 0 146 26 g1 failed over, all heard 40 0 0 120 0 both think themselves active 40 0 0 120 0 both think the peer active 0 40 0 120 80 leaves heard g1 did not fail over g1 failed over reached black hole hops peer reached black hole hops peer 0 0 40 40 0 40 0 160 40 1 4 36 48 0 40 0 156 36 2 7 33 54 0 40 0 153 33 3 9 31 58 0 40 0 151 31 4 14 26 68 0 40 0 146 26 5 18 22 76 0 40 0 142 22 6 24 16 88 0 40 0 136 16 7 29 11 98 0 40 0 131 11 8 34 6 108 0 40 0 126 6 9 40 0 120 0 40 0 120 0 nodes 15, links 24, flows 40, hop limit 12 ``` ## Answering Is Not Forwarding The second row is the lesson's most important number. When `g1-c` breaks and nobody has heard, all forty of the forty flows fall into a **black hole**. The hops spent are **40** — one hop per flow. The packet leaves the leaf, arrives at the gateway holding the virtual address, and dies there. This is the shape of the failure: the device is up, its ports work, the virtual address answers, and the host finds its default gateway reachable. Only the packet never goes up. **Who owns the virtual address and who can forward the packet are separate questions**, and the host can only ask the first one. ## Failing Over Turns the Black Hole into the Peer Link The third row closes the failure: when `g1` notices its upstream link has broken and fails over, all forty of the forty flows reach, and the black hole drops to **0**. The leaves still have not heard anything; `g1` takes their packet and hands it to its peer. The cost shows up in two columns at once. Hops rise from **120** to **160** — one extra hop per flow. And the peer link column rises from **0** to **40**: the link set up for control traffic carries all of the user traffic. The bottom table shows how this cost dissolves. As the number of leaves that have heard climbs from zero to nine, the reached count in the `g1 failed over` column stays fixed at **40**; what changes is hops falling from **160** to **120**, and peer-link crossings falling from **40** to **0**. The failover window does not lose packets; it routes them over the peer link, and as the window closes, that load lifts. The left column shows the same window without failover: as the number of leaves that have heard rises, reached flows climb linearly from **0** to **40**, and black holes fall from **40** to **0**. Even with eight leaves already having heard, **6** flows still go to the dead gateway. ## Partial Information Produces a Loop Again The last row pays off the course's second claim for a second time. When `g1` and `g2` each believe the other active, all forty of the forty flows enter a **loop**: black holes are **0**, reached is **0**. This situation is not contrived. As `g1` was going down, it received the information "my peer is now active"; `g2`, when `g1` came back, received the information "my peer has the higher priority." **Each heard something true, but at a different moment.** What is wrong is what the two say **together**. Two numbers must be compared. In the regime with no information at all (nobody heard), forty packets fall into a black hole, **40** hops are spent, and the peer link carries **0** crossings: the failure is cheap and unmistakable. In the regime with partial information, forty packets enter a loop, **120** hops are spent, and the peer link carries **80** crossings — two crossings per flow. **The loop is three times as expensive as the black hole, and it does not look like a black hole — it looks like congestion:** the administrator does not see an idle link, but a full peer link. The third claim also reads here. The leaves behind the forty flows that entered the loop were split into two groups; five sent to `g1`, four to `g2`. Which group a flow belonged to made no difference — all forty entered the loop. The leaf's table was correct, and its packet still died. **No single device can put a packet into a loop by itself; a loop is the joint product of the pair.** The sixth row carries a warning. When both believe themselves active — that is, when the virtual address answers from two devices at once — the fate table looks spotless: **40 reached, 0 loop, 0 black hole, 120 hops.** This measurement cannot see the split-brain, because the packets genuinely arrive. Where it becomes visible is in a different table: when the same address is heard from two ports, the MAC address table oscillates between two records. That table is the subject of the MAC Address Tables lesson and is not repeated here. ## The Counted Surfaces and Their Narrowing This section writes how the failures **look** and how they are **narrowed**; it does not give a procedure. The attack side belongs to the **Wireless Networks and Network Security** course. **First surface — the length of the window.** In the measurement, without failover, even with eight leaves already having heard, **6** flows still die. **Narrowing:** shortening the advertisement interval and the dead timer narrows the window; its cost is that a transient glitch can trigger an unnecessary failover. A second narrowing is precomputing the backup path: the table is not computed at the moment of failover, it is kept ready. **Second surface — a gateway that answers but cannot forward.** In the measurement, **40** black holes and one hop per flow. **Narrowing: upstream link tracking** — the gateway's priority is tied to its upstream link's state; when the link drops, priority drops and the peer takes over. The measurement gives the full gain of this narrowing: black holes fall from **40** to **0**, at the cost of the peer link carrying **40** crossings. **Third surface — two gateways each believing the other active.** In the measurement, **40** loops and **80** crossings on the peer link. **Narrowing:** ranking priorities strictly and unequally, and turning off **preemption** — if the returning device does not reclaim the address, a two-sided standoff never forms. The hop limit also terminates the loop, but the limit is a ceiling, not a fix: its cost stands in the measurement at **120** hops. **Fourth surface — the peer link being singular.** The peer link carries both the advertisements and, during failover, all of the user traffic; in the measurement, this load is **40** crossings. When the link breaks, the two devices cannot see each other, and a split-brain is born. **Narrowing:** making the peer link itself redundant and confirming the peer's health over a second path — one link's silence is not treated as proof of death. **Fifth surface — advertisements accepted without verification.** A failover advertisement is a claim; without something to verify the claim, which device takes over the gateway is left to whoever announces it. **Narrowing:** tying advertisements to authentication and accepting them only on the ports where the gateway pair actually sits. ## Summary - Gateway redundancy puts two devices behind a single virtual address; the host knows the address, not the devices, and can only ask "is the address answering?" - When the upstream link breaks and nobody has heard, all forty of the forty flows fall into a black hole one hop later: **answering is not forwarding.** - When the gateway fails over, black holes fall to **0**, but hops rise from **120** to **160** and peer-link crossings from **0** to **40**; both recede as the number of leaves that have heard grows. - When two gateways each believe the other active, all forty flows enter a loop: **120** hops and **80** crossings on the peer link — three times the no-information regime, and it looks like congestion. - With leaves split between sending to one gateway and the other, all forty still entered the loop; a loop is the pair's joint product, and when both believe themselves active, the measurement sees nothing wrong. ## Next Step The three lessons so far measured topology as a graph: nodes, links, hops, rounds. Table row count appeared as a column in every measurement, and every time it came out of the topology — **272** in the three-layer design, **468** in the four-spine design. But there is a second thing that determines row count, and it has nothing to do with topology: how destinations are **named**. On the same topology, a plan that distributes addresses by a scheme and a plan that distributes them by creation order fill the same table with a different number of rows. The next lesson counts that difference and closes the course.