Link Aggregation

The previous lesson measured the first answer given to redundancy: an extra link is run, the protocol blocks it, and the blocked link waits until the moment of the break. In the measurement, two of the nine links carried no data at all. They have capacity, their cable is plugged in, and they sit idle.

A natural question follows: can two links not be used at the same time? The cycle ban seems to forbid this — two links mean two paths, two paths bring a flooded frame back, and a loop is born. This lesson’s question is how the ban is gotten around, and what new decision that requires.

A Single Logical Link

Link aggregation is presenting several physical ports between two switches as a single logical link. Spanning tree protocol sees the topology through these logical links: four ports are not four links, they are one link.

The consequence is direct. In the previous lesson’s measurement, the cycle came from two separate links between two switches; because aggregation reduces those two links to one, the tree sees no cycle and blocks neither. It is not a mechanism that violates the ban; it is a mechanism that removes the cycle the ban is about.

Flooding fits the same picture. A broadcast frame is handed to the logical link and goes out only one port; the far switch receives it as a single copy. Aggregation’s most important rule appears here: a frame coming in is never sent back out another port of the same aggregation. Without this rule, aggregation would reproduce within itself the multiplication the previous lesson measured.

Both ends have to know that the aggregation is set up. If the ports are aggregated on one end and counted separately on the other, one side sees a single link where the other sees several, and the tree’s blocking decision comes out different at the two ends. This is why aggregation is set up by agreement through a control message between the two ends; a port that cannot reach agreement is left out of the aggregation and kept as a separate link.

The Distribution Decision Looks at the Flow

Every frame handed to an aggregated link needs a decision: which port does it go out? The simplest answer is round-robin, and it is also the answer that balances load best. But there is a problem.

The ports are not on the same cable. Each has its own queue, its own length, its own instantaneous occupancy; the delay difference between two ports is not zero. If two frames of the same flow are sent out two separate ports, the second can arrive before the first.

The link layer cannot fix this. As seen in the previous lesson, the Ethernet frame carries no hop count; it carries no sequence number either. The receiving switch cannot tell, of the two frames it has, which was sent first, and it cannot reorder them. Keeping order intact is left to the sending side’s decision.

The cost of out-of-order delivery is paid at an upper layer. The mechanism established in the TCP lesson of the Network Models and Protocols course reorders segments that arrive out of order, but it does so by holding a buffer and, in some cases, by making unnecessary retransmissions; an application that never expects reordering counts the data as corrupted. This is why the common rule is: the distribution decision looks at the flow, not the frame. Every frame of the same flow goes out the same port.

What counts as a flow is a choice. A hash that looks only at the pair of hardware addresses counts all the traffic between two switches as a single flow and voids the aggregation; a hash that adds port numbers to the address pair makes the flow finer-grained and improves the balance. The measurement uses the coarsest form, because what is measured is not the hash’s fineness but what the decision looks at.

# taught transcript, not run

aggregation configuration (switch 1 <-> switch 2)
  logical link: t1
    port 0   agreed
    port 1   agreed
    port 2   agreed
    port 3   agreed
  what the tree sees: a SINGLE link

distribution decision
  per flow : port = hash(source, destination) mod port_count
  per frame: port = sequence_no mod port_count

rule: a frame coming in from an aggregation is never sent back
      out another port of the same aggregation.

The Measurement’s Assumptions

• ND30 — There is a single aggregation with four ports. Eight flows are drawn from the shared topology’s node names with a single generator and a single modulus; each of the forty frames is assigned to one of these eight flows with a separate generator. The oracle’s flow–frame mapping is known because we built it ourselves.
• ND31 — The flow hash is the sum of the character values of the names defining the flow, and the port number is this hash’s remainder with respect to the port count. The hash’s fineness is not the measurement’s subject.
• ND32 — Port delays are not equal, and are 1, 2, 1, 3 units respectively. Every port processes one frame per unit of time; a frame’s arrival instant is found by adding the delay to the port’s free-up instant.
• ND33 — A frame is counted out of order if it arrives after a frame of the same flow sent after it. Ordering cannot be fixed at the link layer, because the frame carries no sequence number.
• ND34 — Two distribution schemes are compared: per-flow and per-frame. The same forty frames are used for both.
• ND35 — In the break regime, port number 1 drops. Remapping is tried with two rules: modulus, which recomputes every flow because the port count has shrunk; in-place, which moves only the flows on the dropped port and leaves the rest untouched.
• ND36 — The break happens on frame 20, and the distribution mapping is updated five frames later. In this window, frames written to the dropped port are counted a black hole: the port does not exist, the frame does not arrive anywhere, and no one reports it.
• ND37 — The set’s resolution is forty frames and eight flows; the smallest measurable difference is $1/40 = 0{,}025$ in the frame set, $1/8 = 0{,}125$ in the flow set.

The Measurement

"""Link aggregation: distribution splits no flow, the dropped port moves flows."""
SEED = 20260810
NODES = ["a", "b", "c", "d", "e", "f", "g", "h"]
PORTS, DELAY = 4, (1, 2, 1, 3)
DROPPED, BREAK_FRAME, WINDOW = 1, 20, 5


def generator(seed):
    d = seed % 2147483646 + 1

    def r(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return r


def flows(count=8):
    r, result = generator(SEED), []
    while len(result) < count:
        x, y = NODES[r(8)], NODES[r(8)]
        if x != y and (x, y) not in result:
            result.append((x, y))
    return result


def frames(F, count=40):
    r = generator(SEED + 1)
    return [{"no": i, "flow": F[r(8)]} for i in range(count)]


def flow_hash(flow):
    return sum(ord(c) for c in flow[0] + flow[1])


def port(c, scheme, ports):
    n = len(ports)
    return ports[flow_hash(c["flow"]) % n if scheme == "flow" else c["no"] % n]


def arrival(C):
    """Every port processes one frame per unit time; delays are not equal."""
    free = {}
    for c in C:
        start = max(c["no"], free.get(c["port"], 0))
        free[c["port"]] = start + 1
        c["arrival"] = start + 1 + DELAY[c["port"]]
    return C


def out_of_order(C):
    last, count = {}, 0
    for c in sorted(C, key=lambda c: (c["arrival"], c["no"])):
        if c["flow"] in last and last[c["flow"]] > c["no"]:
            count += 1
        last[c["flow"]] = max(last.get(c["flow"], -1), c["no"])
    return count


F, ALL = flows(), list(range(PORTS))
print(f"flows {len(F)} | frames 40 | ports {PORTS} | dropped port {DROPPED}")
print()
print(f"{'scheme':<8s} {'port load':>18s} {'min':>5s} {'max':>6s} "
      f"{'flows split':>12s} {'unordered':>10s}")
for scheme in ("flow", "frame"):
    C = frames(F)
    for c in C:
        c["port"] = port(c, scheme, ALL)
    arrival(C)
    load = [sum(1 for c in C if c["port"] == p) for p in ALL]
    split = sum(1 for f in F
                if len({c["port"] for c in C if c["flow"] == f}) > 1)
    print(f"{scheme:<8s} {str(load):>18s} {min(load):5d} {max(load):6d} "
          f"{split:12d} {out_of_order(C):10d}")

print()
REMAINING = [p for p in ALL if p != DROPPED]
C = frames(F)
old = {f: port({"flow": f, "no": 0}, "flow", ALL) for f in F}
modulus = {f: port({"flow": f, "no": 0}, "flow", REMAINING) for f in F}
in_place = {f: (REMAINING[flow_hash(f) % len(REMAINING)] if old[f] == DROPPED else old[f])
            for f in F}
lost = sum(1 for c in C if BREAK_FRAME <= c["no"] < BREAK_FRAME + WINDOW
           and old[c["flow"]] == DROPPED)
print(f"flows on dropped port {sum(1 for f in F if old[f] == DROPPED)} | "
      f"frames lost in the {WINDOW}-frame window {lost}")
print()
print(f"{'remapping':<16s} {'flows moved':>12s} {'stayed in place':>16s} "
      f"{'new load':>14s}")
for name, new in (("modulus", modulus), ("in-place", in_place)):
    moved = sum(1 for f in F if new[f] != old[f])
    load = [sum(1 for c in C if new[c["flow"]] == p) for p in REMAINING]
    print(f"{name:<16s} {moved:12d} {len(F) - moved:16d} {str(load):>14s}")

flows 8 | frames 40 | ports 4 | dropped port 1

scheme            port load   min    max  flows split  unordered
flow         [0, 19, 5, 16]     0     19            0          0
frame      [10, 10, 10, 10]    10     10            7          2

flows on dropped port 3 | frames lost in the 5-frame window 2

remapping         flows moved  stayed in place       new load
modulus                     7                1    [26, 14, 0]
in-place                    3                5   [13, 11, 16]

Between Order and Balance

The table above puts the two distribution schemes side by side, and each gains one thing and loses another.

Per-flow distribution keeps its promise exactly: flows split, 0; frames arriving out of order, 0. None of the eight flows spreads across two ports, so even though port delays are not equal, order is never broken. Preserving order needs no extra mechanism — the decision already preserves it.

Its cost sits in the load column, and it is not small: the ports carry 0, 19, 5, and 16 frames respectively. In a four-port aggregation, one port is never used, and another pulls 19 of the forty frames on its own. The capacity aggregation promises is not a fourfold multiplication, but a distribution that depends on where the flows’ hash values happen to land.

Per-frame distribution equalizes load exactly: 10 frames per port. In return, 7 of the eight flows get split — the eighth is not split because its frames already happen to land on a single port. Splitting costs 2 out-of-order frames. This number is 0.050 in a set of forty frames and twice the resolution; it is within the measurement band but small, because the largest difference between port delays is two units. That this number would grow as the delay difference grows is read from the fiction, not from the measurement.

The pattern is this: the finer the distribution decision, the better the balance and the more fragile the order. No single rule gives both, because what preserves order is not splitting the flow, and what builds balance is splitting it.

Whose Capacity Is It

Reading the load column has a practical consequence, and it closes off the most common misconception about aggregation. A four-port aggregation’s total capacity is the sum of the four ports, but a single flow’s capacity is one port. A flow cannot be split, because it is not split; whichever port its hash value lands it on, that port’s capacity is the most data that flow can carry.

The measurement shows this directly: the port carrying 19 frames is, on its own, the ceiling for those flows, and the idle port adds nothing to them. Aggregation grows a link that has many flows; it does not grow a link that has one large flow. If the traffic between two switches consists of a single backup transfer, running four ports gives the same result as running one.

The second side of this is redundancy. The same mechanism, with the same cables, buys two separate things: capacity when there are many flows, staying up when a port drops. Which one dominates depends on the flow count, and this is something measurable.

Nor is it required that every port carry traffic at the same time. In an active–standby arrangement, one port carries all the traffic, and the others take over only when it drops. This removes the distribution decision entirely: no flow gets split, no frame arrives out of order, no balance problem exists. In return, the standby ports’ capacity is never used — the same situation as the previous lesson’s blocked link, this time by aggregation’s decision, not the tree’s.

When a Port Drops

The lower section looks at aggregation’s real reason for existing. When a port drops, the logical link stays up; spanning tree does not see this as a topology change and does not recompute. The rounds measured in the previous lesson are not spent here at all.

Rounds not spent do not mean a free failover. The dropped port carries 3 flows, and in the five-frame window before the distribution mapping is updated, 2 frames fall into a black hole. The port does not exist, the frame arrives nowhere, no one reports an error. This number grows linearly as the window lengthens.

The real difference shows up in the remapping rule. The modulus rule drops the port count from four to three and recomputes every hash, moving 7 of the eight flows. Yet only 3 flows need to move; the remaining four sat on healthy ports and changed place for no reason at all. Every flow that changes place means a new port in the matching table on the other side, and that table has to wait to be refreshed.

The in-place rule moves only the 3 flows on the dropped port, leaving five where they were. And the result is not just less work but a better distribution too: the modulus rule splits load into 26, 14, 0, while the in-place rule gives 13, 11, 16. Recomputing every flow did not improve the balance, it broke it — because hash values fall into a new pattern once taken modulo a different port count, and there is no guarantee that new pattern will be balanced.

In a set of eight flows, the difference between 7 and 3 is 0.500 and four times the flow set’s resolution of 0.125. The result lands on a general rule: a source dropping should not change mappings that do not depend on it. When it does, the price paid is relocating flows that have nothing to do with the failure at all.

Summary

• Link aggregation turns several ports into a single logical link; because spanning tree sees no cycle, it blocks no port, and aggregation’s own rule bans sending an incoming frame back out the same aggregation.
• The frame carries no sequence number; the receiver cannot reorder, so the obligation to preserve order falls on the distribution decision.
• Per-flow distribution splits 0 flows and produces 0 out-of-order frames, but spreads load as 0, 19, 5, 16; per-frame distribution equalizes load to 10, splits 7 flows, and produces 2 out-of-order frames.
• When a port drops, the tree does not recompute, but 2 frames fall into a black hole in the five-frame window before the mapping updates.
• Modulus remapping moves 7 flows instead of the 3 that need to move, and makes the load 26, 14, 0; in-place remapping moves 3 flows and gives 13, 11, 16.

Next Step

Every decision measured across this topic’s five lessons shared a single common assumption: the source and destination were within the same local network. The switch looked for the destination in its own table because the destination was expected to be behind one of its own ports; flooding served a purpose because a flooded frame had a chance of reaching the destination; segmentation produced black holes because the destination was in another domain and no path to it had been defined. In every case, the destination was either directly a neighbor, or not there at all.

The next topic lifts this assumption. If the destination is on another network, that destination’s address will never be found in the device’s table — because the destination is not behind any port at all. What does the decision look at then, and in which table does what it looks at reside?