--- title: 'Load Balancers' source: 'https://academia.sh/en/courses/network-operations/load-balancers' course: 'Network Operations and Automation' language: en updated: '2026-08-17T18:07:15+00:00' license: 'CC BY-SA 4.0' --- # Load Balancers A balancer working at the transport layer distributes connections, one working at the application layer distributes requests; on the same forty subjects, one overruns 17 subjects by 438 units, the other 16 subjects by 310 units, and the transport layer's own gauge shows the same number for all forty of the forty subjects. The previous course measured how a limit gets **written**: which power, which threshold, which rule sequence, which metric. No lesson asked what happens after the limit is written. A written limit sits in a living system; its load changes, its drift goes unseen by anyone. **Writing a limit and operating what was written are separate jobs.** Every decision up to this point was for **a single object**: a packet, a flow, a client, a rule line. Operations does not work this way. The operator does not touch objects one by one; they touch **a lever** — a distribution rule, a sampling rate, a template — and that lever gets applied to **forty subjects at once.** This is this course's question, and the first lever is the load balancer. ## Lever and Subject The same population is measured throughout the course: **forty subjects.** The subjects are not identical to each other. Their capacities spread between 22 and 99 units, their delays differ, each belongs to one of three service classes, and nine of them are special devices that a future lever has to leave out. We know what is in the population **because we wrote it**; the measurement's oracle is the fiction itself. What is measured is not the lever's nature. **An operations decision's number is how many subjects it serves wrongly.** The lever is single, the subjects are forty, and the minority carrying the difference between them is called, in this course, the **lever's tail.** Tail has only this one sense in this course: the set of subjects the lever serves wrongly. The tail has two separate numbers, and both are written at every measurement. The first is **how many subjects** stay in the tail. The second is **how many** of those in the tail **do not show up in the lever's own gauge.** This lesson opens with the cleanest example of the second. This is where it becomes visible why an average is not enough. A load split across forty subjects can stay below capacity on average while several subjects at the same time sit far above their capacity. An average folds **both ends** of the distribution into the same number: the overrunning subject's excess and the idle subject's shortfall cancel out in the sum. The tail measure refuses exactly this cancellation. This is why, in every table below, the overrunning subject count and the idle unit count are **separate columns** and are never reduced to a single number in any measurement. ## What the Balancer Reads A **load balancer** is an intermediary that splits incoming work across a set of subjects. Its role and the distinction by layer were established in the Traffic Layer course of the System Design and Distributed Systems curriculum; that account is not repeated here. The difference is one sentence: **there the balancer's design was measured, here the tail a single lever leaves on forty subjects with unequal capacity is measured.** The side of this distinction that matters for this lesson is what the balancer **can read.** A balancer working at the **transport layer** sees the connection four-tuple: source and destination address, source and destination port. It does not open what is inside the connection. The unit of distribution is therefore the **connection**: it binds every incoming new connection to a subject and stays bound to it until that connection closes. A balancer working at the **application layer** terminates the connection on itself, reads the request, and forwards it to subjects over connections it opens itself. The unit of distribution is the **request.** Because it can read the request's path and header fields, it can sort subjects into classes and send each class to a separate pool. The choice between the two is a configuration line: ```text # taught configuration transcript, not run balancer transport { listen *:443 unit connection rule round-robin pool subject-01 ... subject-40 } balancer application { listen *:443 unit request rule round-robin match /interactive/* -> pool-interactive match /batch/* -> pool-batch match /standby/* -> pool-standby } ``` This transcript is documentation, not a measurement. The numbers come from the measurement below. ## A Health Check Is a Lever Too The balancer has another lever besides distribution: the **health check.** An interval, a timeout, and how many consecutive failures are enough to pull a subject out of the pool are written down. These three numbers, too, get applied not to a single subject but to **all forty at once.** Whether the check's presence affects the service was measured in the Traffic Layer course of the System Design and Distributed Systems curriculum and is not repeated here. The difference is one sentence: **there the check's presence was measured, here the tail a single threshold leaves on forty subjects.** The threshold's tail has two ends, and the ends do not close together. If the timeout is written loose, a subject that has slowed down but still responds is counted healthy and keeps having work routed to it; the tail is the requests going to that subject. If the timeout is written tight, subjects whose delay is already high fail the check and drop out of the pool; the tail this time is the remaining subjects that take on the dropped subjects' load. Because the forty subjects' delays are all different, **a single timeout cannot close both ends at once.** It should also not be forgotten that what the check reads is the subject's **own announcement**: the subject says it is healthy, the check believes it. The observation left by the Application Layer Protocols course holds here too — nothing verifies the announcement, only its absence not arriving is information. The same duality shows up when a subject is pulled out of the pool. Connections that exist at the moment of removal are not cut; a **draining** period is granted. This period, too, is a single number applied to all forty subjects at once: written long, a failed subject's work sits there for a long time; written short, half-finished work gets cut off. It is the same lever; only who makes up the tail changes. ## The Lever's Own Gauge The only thing the transport-layer balancer can count is the connection. When it distributes with a round-robin rule, its own counter, by definition, shows **the same value** for all forty of the forty subjects. The gauge is flat, and being flat reads like proof the rule is correct. Yet what the connections carry is not equal. A persistent connection can carry one request, or thirty. The balancer cannot see this, because what it cannot see is the connection's inside. **What gets distributed equally and what the subject is actually loaded with are not the same thing**, and the difference between the two never shows up in the lever's gauge. The application-layer balancer closes this gap: it counts requests, distributes requests, and its gauge shows the real unit of work. The measurement asks how much of this it actually solves. The measurement's assumptions: - **TM1** — The forty subjects are the course's fixed set; the capacity, delay, and class fields are not changed during the measurement. The oracle is this set itself. - **TM2** — Four hundred connections are generated, and each carries a variable number of requests. One-eighth of the connections are long-lived and carry between twenty and forty requests; the rest carry between one and four. - **TM3** — The balancer distributes with a round-robin rule. The rule is the same across all three measurements; **the only thing that changes is the unit of distribution** — connection, request, or request within a class pool. - **TM4** — The transport-layer balancer cannot read the number of requests a connection carries; the application-layer one can. The fiction codes this distinction directly. - **TM5** — Capacity is the number of requests a subject can serve at once. Every unit above this number is overrun; every unit below it is idle. - **TM6** — The measurement is not a network measurement; no connection is opened. What is counted is the distribution in the fiction itself. ## The Measurement ```python """Layer 4 vs layer 7 balancing: two levers on the same forty subjects.""" SEED = 20260812 CLASSES = ("interactive", "batch", "standby") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, result = generator(seed), [] for i in range(count): result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45), "class": CLASSES[r(3)], "special": r(9) == 0}) return result def connections(count=400, seed=SEED + 11): r, result = generator(seed), [] for i in range(count): long_lived = r(8) == 0 result.append({"requests": 20 + r(21) if long_lived else 1 + r(4), "class": CLASSES[r(3)]}) return result def layer4(conns, subs): """Distributes the connection; does not know how many requests it carries.""" share = {s["no"]: 0 for s in subs} count = {s["no"]: 0 for s in subs} for i, c in enumerate(conns): s = subs[i % len(subs)] count[s["no"]] += 1 share[s["no"]] += c["requests"] return share, count def layer7(conns, subs, pooled): """Distributes the request; to the class pool if asked.""" pool = {c: [s for s in subs if s["class"] == c] for c in CLASSES} count = {c: 0 for c in CLASSES} share, i = {s["no"]: 0 for s in subs}, 0 for c in conns: p = pool[c["class"]] if pooled else subs for _ in range(c["requests"]): if pooled: share[p[count[c["class"]] % len(p)]["no"]] += 1 count[c["class"]] += 1 else: share[p[i % len(p)]["no"]] += 1 i += 1 return share def overrun(share, subs): n = units = 0 for s in subs: over = share[s["no"]] - s["capacity"] if over > 0: n, units = n + 1, units + over return n, units subs, conns = subjects(), connections() print(f"subjects {len(subs)} | total capacity {sum(s['capacity'] for s in subs)} | " f"smallest {min(s['capacity'] for s in subs)} largest {max(s['capacity'] for s in subs)}") print(f"connections {len(conns)} | requests {sum(c['requests'] for c in conns)}") for c in CLASSES: h = [s for s in subs if s["class"] == c] print(f" {c:12s} subjects {len(h):2d} capacity {sum(s['capacity'] for s in h):5d} " f"requests {sum(x['requests'] for x in conns if x['class'] == c):5d}") share4, count = layer4(conns, subs) measurements = (("layer 4 (connection)", share4), ("layer 7 (request)", layer7(conns, subs, False)), ("layer 7 (class pool)", layer7(conns, subs, True))) print() print(f"{'lever':<24s} {'over':>5s} {'over units':>11s} {'idle':>7s} " f"{'min':>6s} {'max':>7s} {'no load at all':>16s}") for name, share in measurements: n, units = overrun(share, subs) print(f"{name:<24s} {n:5d} {units:11d} " f"{sum(max(0, s['capacity'] - share[s['no']]) for s in subs):7d} " f"{min(share.values()):6d} {max(share.values()):7d} " f"{sum(1 for s in subs if share[s['no']] == 0):16d}") print() print(f"layer 4's own gauge — connection count: min {min(count.values())}, " f"max {max(count.values())}; {overrun(share4, subs)[0]} subjects over capacity " f"do not show up in this gauge") ``` ``` subjects 40 | total capacity 2336 | smallest 22 largest 99 connections 400 | requests 2193 interactive subjects 11 capacity 580 requests 655 batch subjects 20 capacity 1346 requests 700 standby subjects 9 capacity 410 requests 838 lever over over units idle min max no load at all layer 4 (connection) 17 438 581 21 164 0 layer 7 (request) 16 310 453 54 55 0 layer 7 (class pool) 18 558 701 35 94 0 layer 4's own gauge — connection count: min 10, max 10; 17 subjects over capacity do not show up in this gauge ``` ## Where the Tail Is The last line is this lesson's core. The transport-layer balancer's counter shows **10** for all forty of the forty subjects; there is no difference at all between min and max. At the same time, **17** of the forty subjects have overrun their capacity, and the real load spreads between **21** and **164**. The lever is working correctly — by its own definition. **The tail does not show up at the dimension the lever measures.** Moving to the application layer solves part of this. Once requests are distributed, load tightens to between **54** and **55**; now the gauge actually shows what is being distributed. But overrunning subjects drop from **17** to **16**, overrun units from **438** to **310**. The gain across forty subjects is **a single subject.** In a set of forty subjects, the smallest measurable difference is **1/40 = 0.025**; a one-subject gain is exactly at that threshold, and a smaller difference cannot be defended with this set. The reason the gain stays small is clear. The application layer distributes **equal requests**, but the subjects are not equal: capacities run between **22** and **99**. Dropping 54 requests on everyone is a threefold overrun for a subject with capacity 22, and roughly half-idle for one with capacity 99. Raising the lever's resolution **moved** the tail, it did not remove it. The third line says this more clearly. The class pool uses the real capability the application layer buys: reading the request's path and picking the subject. The result is **worse** — overrunning subjects **18**, overrun units **558**. The reason sits in the three lines above: the standby class carries **9** subjects and **410** capacity but takes **838** requests; the batch class carries **20** subjects and **1346** capacity for **700** requests. Splitting into pools made subjects that had capacity unusable. Idle units climb from **453** to **701**. This is why the fourth column is part of the measurement. In all three levers, **subjects with no load at all is 0**: no subject is left out, no one is standing at the door outside. And yet **581**, **453**, and **701** units sit idle in the system. A gauge that counts only overrun never shows this column and never says a quarter of capacity went unused. The only reason we can write this table is that we **know** the capacities. In operations, such a column does not sit ready-made: the balancer does not measure a subject's capacity, it counts the work it gives it. The overrun column can only be computed once capacity is brought in from somewhere else — a separate per-subject measurement; without that, the gauge stays flat and stays no less wrong. In this lesson's fiction we hold the oracle in hand, so we can put the two columns side by side, and this is exactly what the rest of the course asks: **where is the difference between the lever's gauge and the lever's result read from?** The next two topics count this difference — first in the traffic itself, then in telemetry. ## Summary - Operations' unit is not a single object but a **lever**: the operator writes a rule, the rule gets applied to forty subjects at once, and what is measured is the minority the lever serves wrongly — the **tail.** - The transport-layer balancer distributes **connections** and cannot see the requests a connection carries; its own gauge shows **10** for all forty subjects while the real load is between **21** and **164** and **17** subjects have overrun capacity. - The application-layer balancer distributes **requests** and tightens load to a **54–55** range; overrunning subjects drop only to **16**, overrun units to **310**. The gain across forty subjects is **one subject.** - Distributing equal requests is not equal load: because capacities run between **22** and **99**, the same share is overrun for one subject and idle for another. - Splitting into class pools grows the tail — overrunning subjects **18**, idle units **701** — because pool capacity and the requests the pool receives do not match. - A health check is also a single lever, and its tail has two ends: a loose threshold keeps sending work to a slowed subject, a tight threshold drops high-delay subjects and piles their load onto the rest; because the forty subjects' delays all differ, a single threshold cannot close both ends at once. ## Next Step In this lesson, the rule was the same across all three measurements: round-robin distribution. The only thing that changed was the unit of distribution, and changing the unit moved the tail by about one-fortieth. So the real question is not the unit but **the rule itself**: does a rule that splits load by capacity, or picks the least-loaded subject at every step, actually remove the tail? The next lesson puts three more distribution rules side by side against round-robin on the same forty subjects at two separate loads, and counts what improving the rule does to the tail — whether it removes it, or only chooses who ends up in it.