Redundancy and Failover

In a spine–leaf design, redundancy was structural: no device was the sole owner of a path, and the loss of a spine only changed how flows were distributed. On the host side, this is not the case. A host sends its outbound packet not to a path but to an address — the default gateway’s address. The address is singular, and at any moment it sits on exactly one device.

This singularity takes redundancy out of the structural category. When the device fails, a second device must take over the address, and everyone must hear about the takeover. This lesson’s question is what happens to the packet during that short interval while the hearing is still in progress.

The Virtual Address and Two Roles

Gateway redundancy puts two devices behind a single address. This address is called the virtual address; hosts know it as their default gateway and are unaware that two devices exist.

One of the devices is active, the other is standby. The active one answers for the virtual address and forwards packets upward; the standby one listens silently. When the active device’s periodic advertisements stop arriving, the standby waits for a period and then takes over the address.

# taught transcript, not run

gateway pair
  virtual address     single address, hosts' default gateway
  g1  priority 110  active   -> answers the virtual address, forwards upward
  g2  priority 100  standby  -> listens silently, waits for advertisements
  peer link          g1 - g2, carries advertisements and state

timing
  advertisement interval  active side advertises once per interval
  dead timer               standby takes over if no advertisement heard in this time
  failover instant          standby begins answering for the virtual address

leaf table (for remote networks)
  destination     next hop
  u1 u2 u3        the gateway believed to currently hold the virtual address

Service-level failover patterns — health checks, retries, the circuit breaker — were established in the Resilience and Reliability course and are not repeated here. There, what failed over was a service instance, and what was measured was the request’s fate; here, what fails over is the gateway, and what is measured is the packet’s fate.

What the Window Is

Failover is not instantaneous. The time it takes for the dead timer to expire, for the standby to announce the address, and for switches along the path to relearn where the address is — the sum of these is called the failover window. During the window, devices disagree about where the virtual address is.

Inside the window there are two separate questions. First: who answers for the virtual address? Second: can the answering device forward the packet? The measurement counts exactly this distinction: a gateway whose upstream link has broken keeps collecting packets for as long as it holds the address.

The assumptions the measurement rests on:

• TD17 — Nine leaves (y1-y9) connect to a gateway pair (g1, g2); the pair connects to a core (c), and the core connects to three remote networks (u1, u2, u3). The forty flows have a leaf as their source and a remote network as their destination; the traffic measured is entirely north-south.
• TD18 — The failure measured is the breaking of the g1-c upstream link. g1 is healthy, remains connected to the leaves, and can continue to hold the virtual address.
• TD19 — Every leaf holds a belief: whether the virtual address is on g1 or g2. A leaf that has heard says g2; one that has not says g1.
• TD20 — Every gateway also holds a belief. A gateway that believes itself active forwards the packet upward; one that believes its peer active hands the packet to its peer over the peer link.
• TD21 — The oracle is known because we built the setup ourselves: after g1-c breaks, the only path to the remote networks is through g2.
• TD22 — The regimes differ only in belief; the topology does not change except for the presence of the g1-c link. So the difference measured is a difference in information, not in structure.
• TD23 — The “peer link” column counts how many times the forty flows cross the g1-g2 link in total. This link is sized for control traffic; it carrying user traffic is a symptom.
• TD24 — A packet is counted as a loop when it crosses the same link a second time or burns through the hop limit (12); a packet whose next hop is no longer a neighbor is counted as a black hole.
• TD25 — The hops column counts only the hops spent; a packet that falls into a black hole has spent the hops up to where it died.
• TD26 — In a set of forty flows, the smallest measurable difference is 1/40 = 0.025.

The Measurement

"""Gateway redundancy: the packet's fate during the failover window.

Part 1 - regimes: converged, window, upstream-link tracking, split-brain.
Part 2 - the change in fate as the number of leaves that have heard grows.
"""
SEED = 20260810
HOP_LIMIT = 12
LEAF = [f"y{i}" for i in range(1, 10)]
REMOTE = ["u1", "u2", "u3"]
NODES = LEAF + ["g1", "g2", "c"] + REMOTE
LINKS = ([(y, g) for y in LEAF for g in ("g1", "g2")]
         + [("g1", "g2"), ("g1", "c"), ("g2", "c")]
         + [("c", u) for u in REMOTE])
BROKEN = [b for b in LINKS if b != ("g1", "c")]


def make_rng(seed):
    state = seed % 2147483646 + 1

    def rand(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return rand


def neighbors(links):
    k = {u: set() for u in NODES}
    for x, y in links:
        k[x].add(y)
        k[y].add(x)
    return k


def build_tables(leaf_belief, g1_belief, g2_belief):
    """The next hop each node holds for the remote networks."""
    t = {}
    for y in LEAF:
        for u in REMOTE:
            t[(y, u)] = leaf_belief[y]
    for g, peer, belief in (("g1", "g2", g1_belief), ("g2", "g1", g2_belief)):
        for u in REMOTE:
            t[(g, u)] = "c" if belief == g else peer
    for u in REMOTE:
        t[("c", u)] = u
    return t


def forward(source, dest, tab, links):
    k, u, crossed, hops = neighbors(links), source, [], 0
    while u != dest:
        if hops >= HOP_LIMIT:
            return "loop", hops
        nxt = tab.get((u, dest))
        if nxt is None or nxt not in k[u]:
            return "black hole", hops
        if (u, nxt) in crossed:
            return "loop", hops
        crossed.append((u, nxt))
        u, hops = nxt, hops + 1
    return "reached", hops


def pairs(count=40, seed=SEED):
    rand, out = make_rng(seed), []
    while len(out) < count:
        out.append((LEAF[rand(9)], REMOTE[rand(3)]))
    return out


def peer_crossings(tab, links):
    """How many of the forty flows cross the g1-g2 peer link."""
    k, count = neighbors(links), 0
    for x, y in pairs():
        u, crossed, hops = x, [], 0
        while u != y and hops < HOP_LIMIT:
            v = tab.get((u, y))
            if v is None or v not in k[u] or (u, v) in crossed:
                break
            if {u, v} == {"g1", "g2"}:
                count += 1
            crossed.append((u, v))
            u, hops = v, hops + 1
    return count


def measure(tab, links):
    counts, hops = {"reached": 0, "loop": 0, "black hole": 0}, 0
    for x, y in pairs():
        fate, a = forward(x, y, tab, links)
        counts[fate] += 1
        hops += a
    return counts, hops


def belief(heard):
    return {y: ("g2" if y in heard else "g1") for y in LEAF}


ALL, NONE = set(LEAF), set()
REGIME = [
    ("converged, g1 active", LINKS, belief(NONE), "g1", "g1"),
    ("upstream link broke, nobody heard", BROKEN, belief(NONE), "g1", "g1"),
    ("g1 failed over, no leaves heard", BROKEN, belief(NONE), "g2", "g2"),
    ("g1 failed over, four leaves heard", BROKEN, belief(set(LEAF[:4])), "g2", "g2"),
    ("g1 failed over, all heard", BROKEN, belief(ALL), "g2", "g2"),
    ("both think themselves active", LINKS, belief(set(LEAF[:5])), "g1", "g2"),
    ("both think the peer active", LINKS, belief(set(LEAF[:5])), "g2", "g1"),
]

print(f"{'regime':<35s} {'reached':>7s} {'loop':>6s} {'black hole':>10s} "
      f"{'hops':>5s} {'peer link':>9s}")
for name, links, lb, a1, a2 in REGIME:
    tab = build_tables(lb, a1, a2)
    s, a = measure(tab, links)
    print(f"{name:<35s} {s['reached']:7d} {s['loop']:6d} {s['black hole']:10d} "
          f"{a:5d} {peer_crossings(tab, links):9d}")

print()
print(f"{'leaves heard':>12s} {'g1 did not fail over':>32s} {'g1 failed over':>32s}")
print(f"{'':>12s} {'reached black hole hops peer':>32s}"
      f" {'reached black hole hops peer':>32s}")
for n in range(0, 10):
    lb, columns = belief(set(LEAF[:n])), []
    for g1b in ("g1", "g2"):
        tab = build_tables(lb, g1b, "g2")
        s, a = measure(tab, BROKEN)
        columns.append(f"{s['reached']:9d}{s['black hole']:11d}{a:5d}"
                        f"{peer_crossings(tab, BROKEN):5d}")
    print(f"{n:12d} {columns[0]} {columns[1]}")

print()
print(f"nodes {len(NODES)}, links {len(LINKS)}, flows 40, "
      f"hop limit {HOP_LIMIT}")

regime                              reached   loop black hole  hops peer link
converged, g1 active                     40      0          0   120         0
upstream link broke, nobody heard         0      0         40    40         0
g1 failed over, no leaves heard          40      0          0   160        40
g1 failed over, four leaves heard        40      0          0   146        26
g1 failed over, all heard                40      0          0   120         0
both think themselves active             40      0          0   120         0
both think the peer active                0     40          0   120        80

leaves heard             g1 did not fail over                   g1 failed over
                 reached black hole hops peer     reached black hole hops peer
           0         0         40   40    0        40          0  160   40
           1         4         36   48    0        40          0  156   36
           2         7         33   54    0        40          0  153   33
           3         9         31   58    0        40          0  151   31
           4        14         26   68    0        40          0  146   26
           5        18         22   76    0        40          0  142   22
           6        24         16   88    0        40          0  136   16
           7        29         11   98    0        40          0  131   11
           8        34          6  108    0        40          0  126    6
           9        40          0  120    0        40          0  120    0

nodes 15, links 24, flows 40, hop limit 12

Answering Is Not Forwarding

The second row is the lesson’s most important number. When g1-c breaks and nobody has heard, all forty of the forty flows fall into a black hole. The hops spent are 40 — one hop per flow. The packet leaves the leaf, arrives at the gateway holding the virtual address, and dies there.

This is the shape of the failure: the device is up, its ports work, the virtual address answers, and the host finds its default gateway reachable. Only the packet never goes up. Who owns the virtual address and who can forward the packet are separate questions, and the host can only ask the first one.

Failing Over Turns the Black Hole into the Peer Link

The third row closes the failure: when g1 notices its upstream link has broken and fails over, all forty of the forty flows reach, and the black hole drops to 0. The leaves still have not heard anything; g1 takes their packet and hands it to its peer.

The cost shows up in two columns at once. Hops rise from 120 to 160 — one extra hop per flow. And the peer link column rises from 0 to 40: the link set up for control traffic carries all of the user traffic.

The bottom table shows how this cost dissolves. As the number of leaves that have heard climbs from zero to nine, the reached count in the g1 failed over column stays fixed at 40; what changes is hops falling from 160 to 120, and peer-link crossings falling from 40 to 0. The failover window does not lose packets; it routes them over the peer link, and as the window closes, that load lifts.

The left column shows the same window without failover: as the number of leaves that have heard rises, reached flows climb linearly from 0 to 40, and black holes fall from 40 to 0. Even with eight leaves already having heard, 6 flows still go to the dead gateway.

Partial Information Produces a Loop Again

The last row pays off the course’s second claim for a second time. When g1 and g2 each believe the other active, all forty of the forty flows enter a loop: black holes are 0, reached is 0.

This situation is not contrived. As g1 was going down, it received the information “my peer is now active”; g2, when g1 came back, received the information “my peer has the higher priority.” Each heard something true, but at a different moment. What is wrong is what the two say together.

Two numbers must be compared. In the regime with no information at all (nobody heard), forty packets fall into a black hole, 40 hops are spent, and the peer link carries 0 crossings: the failure is cheap and unmistakable. In the regime with partial information, forty packets enter a loop, 120 hops are spent, and the peer link carries 80 crossings — two crossings per flow. The loop is three times as expensive as the black hole, and it does not look like a black hole — it looks like congestion: the administrator does not see an idle link, but a full peer link.

The third claim also reads here. The leaves behind the forty flows that entered the loop were split into two groups; five sent to g1, four to g2. Which group a flow belonged to made no difference — all forty entered the loop. The leaf’s table was correct, and its packet still died. No single device can put a packet into a loop by itself; a loop is the joint product of the pair.

The sixth row carries a warning. When both believe themselves active — that is, when the virtual address answers from two devices at once — the fate table looks spotless: 40 reached, 0 loop, 0 black hole, 120 hops. This measurement cannot see the split-brain, because the packets genuinely arrive. Where it becomes visible is in a different table: when the same address is heard from two ports, the MAC address table oscillates between two records. That table is the subject of the MAC Address Tables lesson and is not repeated here.

The Counted Surfaces and Their Narrowing

This section writes how the failures look and how they are narrowed; it does not give a procedure. The attack side belongs to the Wireless Networks and Network Security course.

First surface — the length of the window. In the measurement, without failover, even with eight leaves already having heard, 6 flows still die. Narrowing: shortening the advertisement interval and the dead timer narrows the window; its cost is that a transient glitch can trigger an unnecessary failover. A second narrowing is precomputing the backup path: the table is not computed at the moment of failover, it is kept ready.

Second surface — a gateway that answers but cannot forward. In the measurement, 40 black holes and one hop per flow. Narrowing: upstream link tracking — the gateway’s priority is tied to its upstream link’s state; when the link drops, priority drops and the peer takes over. The measurement gives the full gain of this narrowing: black holes fall from 40 to 0, at the cost of the peer link carrying 40 crossings.

Third surface — two gateways each believing the other active. In the measurement, 40 loops and 80 crossings on the peer link. Narrowing: ranking priorities strictly and unequally, and turning off preemption — if the returning device does not reclaim the address, a two-sided standoff never forms. The hop limit also terminates the loop, but the limit is a ceiling, not a fix: its cost stands in the measurement at 120 hops.

Fourth surface — the peer link being singular. The peer link carries both the advertisements and, during failover, all of the user traffic; in the measurement, this load is 40 crossings. When the link breaks, the two devices cannot see each other, and a split-brain is born. Narrowing: making the peer link itself redundant and confirming the peer’s health over a second path — one link’s silence is not treated as proof of death.

Fifth surface — advertisements accepted without verification. A failover advertisement is a claim; without something to verify the claim, which device takes over the gateway is left to whoever announces it. Narrowing: tying advertisements to authentication and accepting them only on the ports where the gateway pair actually sits.

Summary

• Gateway redundancy puts two devices behind a single virtual address; the host knows the address, not the devices, and can only ask “is the address answering?”
• When the upstream link breaks and nobody has heard, all forty of the forty flows fall into a black hole one hop later: answering is not forwarding.
• When the gateway fails over, black holes fall to 0, but hops rise from 120 to 160 and peer-link crossings from 0 to 40; both recede as the number of leaves that have heard grows.
• When two gateways each believe the other active, all forty flows enter a loop: 120 hops and 80 crossings on the peer link — three times the no-information regime, and it looks like congestion.
• With leaves split between sending to one gateway and the other, all forty still entered the loop; a loop is the pair’s joint product, and when both believe themselves active, the measurement sees nothing wrong.

Next Step

The three lessons so far measured topology as a graph: nodes, links, hops, rounds. Table row count appeared as a column in every measurement, and every time it came out of the topology — 272 in the three-layer design, 468 in the four-spine design. But there is a second thing that determines row count, and it has nothing to do with topology: how destinations are named. On the same topology, a plan that distributes addresses by a scheme and a plan that distributes them by creation order fill the same table with a different number of rows. The next lesson counts that difference and closes the course.