Load Balancers

The previous course measured how a limit gets written: which power, which threshold, which rule sequence, which metric. No lesson asked what happens after the limit is written. A written limit sits in a living system; its load changes, its drift goes unseen by anyone. Writing a limit and operating what was written are separate jobs.

Every decision up to this point was for a single object: a packet, a flow, a client, a rule line. Operations does not work this way. The operator does not touch objects one by one; they touch a lever — a distribution rule, a sampling rate, a template — and that lever gets applied to forty subjects at once. This is this course’s question, and the first lever is the load balancer.

Lever and Subject

The same population is measured throughout the course: forty subjects. The subjects are not identical to each other. Their capacities spread between 22 and 99 units, their delays differ, each belongs to one of three service classes, and nine of them are special devices that a future lever has to leave out. We know what is in the population because we wrote it; the measurement’s oracle is the fiction itself.

What is measured is not the lever’s nature. An operations decision’s number is how many subjects it serves wrongly. The lever is single, the subjects are forty, and the minority carrying the difference between them is called, in this course, the lever’s tail. Tail has only this one sense in this course: the set of subjects the lever serves wrongly.

The tail has two separate numbers, and both are written at every measurement. The first is how many subjects stay in the tail. The second is how many of those in the tail do not show up in the lever’s own gauge. This lesson opens with the cleanest example of the second.

This is where it becomes visible why an average is not enough. A load split across forty subjects can stay below capacity on average while several subjects at the same time sit far above their capacity. An average folds both ends of the distribution into the same number: the overrunning subject’s excess and the idle subject’s shortfall cancel out in the sum. The tail measure refuses exactly this cancellation. This is why, in every table below, the overrunning subject count and the idle unit count are separate columns and are never reduced to a single number in any measurement.

What the Balancer Reads

A load balancer is an intermediary that splits incoming work across a set of subjects. Its role and the distinction by layer were established in the Traffic Layer course of the System Design and Distributed Systems curriculum; that account is not repeated here. The difference is one sentence: there the balancer’s design was measured, here the tail a single lever leaves on forty subjects with unequal capacity is measured.

The side of this distinction that matters for this lesson is what the balancer can read.

A balancer working at the transport layer sees the connection four-tuple: source and destination address, source and destination port. It does not open what is inside the connection. The unit of distribution is therefore the connection: it binds every incoming new connection to a subject and stays bound to it until that connection closes.

A balancer working at the application layer terminates the connection on itself, reads the request, and forwards it to subjects over connections it opens itself. The unit of distribution is the request. Because it can read the request’s path and header fields, it can sort subjects into classes and send each class to a separate pool.

The choice between the two is a configuration line:

# taught configuration transcript, not run

balancer transport {
  listen   *:443
  unit     connection
  rule     round-robin
  pool     subject-01 ... subject-40
}

balancer application {
  listen   *:443
  unit     request
  rule     round-robin
  match    /interactive/*  -> pool-interactive
  match    /batch/*        -> pool-batch
  match    /standby/*      -> pool-standby
}

This transcript is documentation, not a measurement. The numbers come from the measurement below.

A Health Check Is a Lever Too

The balancer has another lever besides distribution: the health check. An interval, a timeout, and how many consecutive failures are enough to pull a subject out of the pool are written down. These three numbers, too, get applied not to a single subject but to all forty at once.

Whether the check’s presence affects the service was measured in the Traffic Layer course of the System Design and Distributed Systems curriculum and is not repeated here. The difference is one sentence: there the check’s presence was measured, here the tail a single threshold leaves on forty subjects.

The threshold’s tail has two ends, and the ends do not close together. If the timeout is written loose, a subject that has slowed down but still responds is counted healthy and keeps having work routed to it; the tail is the requests going to that subject. If the timeout is written tight, subjects whose delay is already high fail the check and drop out of the pool; the tail this time is the remaining subjects that take on the dropped subjects’ load. Because the forty subjects’ delays are all different, a single timeout cannot close both ends at once.

It should also not be forgotten that what the check reads is the subject’s own announcement: the subject says it is healthy, the check believes it. The observation left by the Application Layer Protocols course holds here too — nothing verifies the announcement, only its absence not arriving is information.

The same duality shows up when a subject is pulled out of the pool. Connections that exist at the moment of removal are not cut; a draining period is granted. This period, too, is a single number applied to all forty subjects at once: written long, a failed subject’s work sits there for a long time; written short, half-finished work gets cut off. It is the same lever; only who makes up the tail changes.

The Lever’s Own Gauge

The only thing the transport-layer balancer can count is the connection. When it distributes with a round-robin rule, its own counter, by definition, shows the same value for all forty of the forty subjects. The gauge is flat, and being flat reads like proof the rule is correct.

Yet what the connections carry is not equal. A persistent connection can carry one request, or thirty. The balancer cannot see this, because what it cannot see is the connection’s inside. What gets distributed equally and what the subject is actually loaded with are not the same thing, and the difference between the two never shows up in the lever’s gauge.

The application-layer balancer closes this gap: it counts requests, distributes requests, and its gauge shows the real unit of work. The measurement asks how much of this it actually solves.

The measurement’s assumptions:

• TM1 — The forty subjects are the course’s fixed set; the capacity, delay, and class fields are not changed during the measurement. The oracle is this set itself.
• TM2 — Four hundred connections are generated, and each carries a variable number of requests. One-eighth of the connections are long-lived and carry between twenty and forty requests; the rest carry between one and four.
• TM3 — The balancer distributes with a round-robin rule. The rule is the same across all three measurements; the only thing that changes is the unit of distribution — connection, request, or request within a class pool.
• TM4 — The transport-layer balancer cannot read the number of requests a connection carries; the application-layer one can. The fiction codes this distinction directly.
• TM5 — Capacity is the number of requests a subject can serve at once. Every unit above this number is overrun; every unit below it is idle.
• TM6 — The measurement is not a network measurement; no connection is opened. What is counted is the distribution in the fiction itself.

The Measurement

"""Layer 4 vs layer 7 balancing: two levers on the same forty subjects."""
SEED = 20260812
CLASSES = ("interactive", "batch", "standby")


def generator(seed):
    d = seed % 2147483646 + 1

    def r(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return r


def subjects(count=40, seed=SEED):
    r, result = generator(seed), []
    for i in range(count):
        result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45),
                       "class": CLASSES[r(3)], "special": r(9) == 0})
    return result


def connections(count=400, seed=SEED + 11):
    r, result = generator(seed), []
    for i in range(count):
        long_lived = r(8) == 0
        result.append({"requests": 20 + r(21) if long_lived else 1 + r(4),
                       "class": CLASSES[r(3)]})
    return result


def layer4(conns, subs):
    """Distributes the connection; does not know how many requests it carries."""
    share = {s["no"]: 0 for s in subs}
    count = {s["no"]: 0 for s in subs}
    for i, c in enumerate(conns):
        s = subs[i % len(subs)]
        count[s["no"]] += 1
        share[s["no"]] += c["requests"]
    return share, count


def layer7(conns, subs, pooled):
    """Distributes the request; to the class pool if asked."""
    pool = {c: [s for s in subs if s["class"] == c] for c in CLASSES}
    count = {c: 0 for c in CLASSES}
    share, i = {s["no"]: 0 for s in subs}, 0
    for c in conns:
        p = pool[c["class"]] if pooled else subs
        for _ in range(c["requests"]):
            if pooled:
                share[p[count[c["class"]] % len(p)]["no"]] += 1
                count[c["class"]] += 1
            else:
                share[p[i % len(p)]["no"]] += 1
                i += 1
    return share


def overrun(share, subs):
    n = units = 0
    for s in subs:
        over = share[s["no"]] - s["capacity"]
        if over > 0:
            n, units = n + 1, units + over
    return n, units


subs, conns = subjects(), connections()
print(f"subjects {len(subs)} | total capacity {sum(s['capacity'] for s in subs)} | "
      f"smallest {min(s['capacity'] for s in subs)} largest {max(s['capacity'] for s in subs)}")
print(f"connections {len(conns)} | requests {sum(c['requests'] for c in conns)}")
for c in CLASSES:
    h = [s for s in subs if s["class"] == c]
    print(f"  {c:12s} subjects {len(h):2d} capacity {sum(s['capacity'] for s in h):5d} "
          f"requests {sum(x['requests'] for x in conns if x['class'] == c):5d}")

share4, count = layer4(conns, subs)
measurements = (("layer 4 (connection)", share4),
                 ("layer 7 (request)", layer7(conns, subs, False)),
                 ("layer 7 (class pool)", layer7(conns, subs, True)))
print()
print(f"{'lever':<24s} {'over':>5s} {'over units':>11s} {'idle':>7s} "
      f"{'min':>6s} {'max':>7s} {'no load at all':>16s}")
for name, share in measurements:
    n, units = overrun(share, subs)
    print(f"{name:<24s} {n:5d} {units:11d} "
          f"{sum(max(0, s['capacity'] - share[s['no']]) for s in subs):7d} "
          f"{min(share.values()):6d} {max(share.values()):7d} "
          f"{sum(1 for s in subs if share[s['no']] == 0):16d}")
print()
print(f"layer 4's own gauge — connection count: min {min(count.values())}, "
      f"max {max(count.values())}; {overrun(share4, subs)[0]} subjects over capacity "
      f"do not show up in this gauge")

subjects 40 | total capacity 2336 | smallest 22 largest 99
connections 400 | requests 2193
  interactive  subjects 11 capacity   580 requests   655
  batch        subjects 20 capacity  1346 requests   700
  standby      subjects  9 capacity   410 requests   838

lever                     over  over units    idle    min     max   no load at all
layer 4 (connection)        17         438     581     21     164                0
layer 7 (request)           16         310     453     54      55                0
layer 7 (class pool)        18         558     701     35      94                0

layer 4's own gauge — connection count: min 10, max 10; 17 subjects over capacity do not show up in this gauge

Where the Tail Is

The last line is this lesson’s core. The transport-layer balancer’s counter shows 10 for all forty of the forty subjects; there is no difference at all between min and max. At the same time, 17 of the forty subjects have overrun their capacity, and the real load spreads between 21 and 164. The lever is working correctly — by its own definition. The tail does not show up at the dimension the lever measures.

Moving to the application layer solves part of this. Once requests are distributed, load tightens to between 54 and 55; now the gauge actually shows what is being distributed. But overrunning subjects drop from 17 to 16, overrun units from 438 to 310. The gain across forty subjects is a single subject. In a set of forty subjects, the smallest measurable difference is 1/40 = 0.025; a one-subject gain is exactly at that threshold, and a smaller difference cannot be defended with this set.

The reason the gain stays small is clear. The application layer distributes equal requests, but the subjects are not equal: capacities run between 22 and 99. Dropping 54 requests on everyone is a threefold overrun for a subject with capacity 22, and roughly half-idle for one with capacity 99. Raising the lever’s resolution moved the tail, it did not remove it.

The third line says this more clearly. The class pool uses the real capability the application layer buys: reading the request’s path and picking the subject. The result is worse — overrunning subjects 18, overrun units 558. The reason sits in the three lines above: the standby class carries 9 subjects and 410 capacity but takes 838 requests; the batch class carries 20 subjects and 1346 capacity for 700 requests. Splitting into pools made subjects that had capacity unusable. Idle units climb from 453 to 701.

This is why the fourth column is part of the measurement. In all three levers, subjects with no load at all is 0: no subject is left out, no one is standing at the door outside. And yet 581, 453, and 701 units sit idle in the system. A gauge that counts only overrun never shows this column and never says a quarter of capacity went unused.

The only reason we can write this table is that we know the capacities. In operations, such a column does not sit ready-made: the balancer does not measure a subject’s capacity, it counts the work it gives it. The overrun column can only be computed once capacity is brought in from somewhere else — a separate per-subject measurement; without that, the gauge stays flat and stays no less wrong. In this lesson’s fiction we hold the oracle in hand, so we can put the two columns side by side, and this is exactly what the rest of the course asks: where is the difference between the lever’s gauge and the lever’s result read from? The next two topics count this difference — first in the traffic itself, then in telemetry.

Summary

• Operations’ unit is not a single object but a lever: the operator writes a rule, the rule gets applied to forty subjects at once, and what is measured is the minority the lever serves wrongly — the tail.
• The transport-layer balancer distributes connections and cannot see the requests a connection carries; its own gauge shows 10 for all forty subjects while the real load is between 21 and 164 and 17 subjects have overrun capacity.
• The application-layer balancer distributes requests and tightens load to a 54–55 range; overrunning subjects drop only to 16, overrun units to 310. The gain across forty subjects is one subject.
• Distributing equal requests is not equal load: because capacities run between 22 and 99, the same share is overrun for one subject and idle for another.
• Splitting into class pools grows the tail — overrunning subjects 18, idle units 701 — because pool capacity and the requests the pool receives do not match.
• A health check is also a single lever, and its tail has two ends: a loose threshold keeps sending work to a slowed subject, a tight threshold drops high-delay subjects and piles their load onto the rest; because the forty subjects’ delays all differ, a single threshold cannot close both ends at once.

Next Step

In this lesson, the rule was the same across all three measurements: round-robin distribution. The only thing that changed was the unit of distribution, and changing the unit moved the tail by about one-fortieth. So the real question is not the unit but the rule itself: does a rule that splits load by capacity, or picks the least-loaded subject at every step, actually remove the tail? The next lesson puts three more distribution rules side by side against round-robin on the same forty subjects at two separate loads, and counts what improving the rule does to the tail — whether it removes it, or only chooses who ends up in it.