Load Balancers

The previous lesson filtered traffic: which request, from which source to which port, would pass was written rule by rule, and the ones that did not were counted. A request that passes the filter is now inside the network.

Which instance the passing request goes to is still undetermined. Six instances serve the measurement network’s dashboard; one is currently restarting, one is under high load, one just launched and is not ready. What binds the request to an instance is the load balancer.

The balancing algorithms — round robin, least connections, session affinity — and what a health check is were measured earlier; this lesson does not repeat them. The decision that stays with the renter in this layer is different: which layer the balancer runs at. The layer determines how much of the request the balancer can see, and how far it can route. Three things are measured here: the fields each layer sees and the rules it can apply, the round the application layer adds and the termination point it opens, and a sweep of the health check threshold.

• CN31. The fictional regional measurement network: two hosts (dashboard, asset), five content paths, six backend instances. The request flow is produced by our own generator; the seed 30631 is visible and gives the same sequence run to run. 1,200 requests.
• CN32. The operator’s intent is written in five rules: two path prefixes, one host, one management path, and one default. Rules are tried in order, the first match wins.
• CN33. The transport layer balancer sees only the address and the port; the application layer sees the address, the port, the path, the host, and the session field. Under the port-split layout, the client sends asset and management traffic to separate ports — this is the item that changes on the client side.
• CN34. The handshake round, termination point, and key copy counts are model constants: the transport layer passes the connection through (one round, the key on all six backends), the application layer terminates the connection and opens a new one to the back end (two rounds, the key on the balancer).
• CN35. The health check model is six instances and 300 check rounds; the third instance fails permanently at round 120. For a healthy instance, the probability of a transient deviation is independent at every check and is not affected by the interval; seed 41129.
• CN36. An instance is removed from traffic after as many consecutive failed checks as the threshold, and reinstated after two consecutive successful checks. The model does not measure time; the second values are the product of the round count and the check interval.

// load-balancer/tier.mjs — model: load balancer layer selection and the health check threshold.
// The fictional regional measurement network's request flow is produced by our own generator;
// the seed is visible. The balancing algorithm is not redefined; what is measured is the
// information the layer sees, the routing it can do with that information, and the surface it opens.
const SEED = 30631, REQUESTS = 1200, BACKENDS = 6;
// [path, host, weight, pool the operator wants]
const FLOW = [
  ["/map/region", "dashboard", 30, "map"], ["/counter/list", "dashboard", 25, "counter"],
  ["/nightly/digest", "dashboard", 15, "general"], ["/asset/logo.png", "asset", 20, "asset"],
  ["/management/node", "dashboard", 10, "management"],
];
const TOTAL = FLOW.reduce((a, x) => a + x[2], 0);
// [field examined, value, pool] — rules are tried in order, the last one is the default
const RULES = [["path", "/map", "map"], ["path", "/counter", "counter"],
  ["host", "asset", "asset"], ["path", "/management", "management"], ["port", "*", "general"]];
const SPLIT_POOLS = ["asset", "management"];  // pools moved to a separate port under port split
const generator = (t) => { let s = t >>> 0; return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; };
const pick = (u) => { let e = u * TOTAL; return FLOW.find((x) => (e -= x[2]) < 0) ?? FLOW[0]; };

function run({ fields, split }) {
  const rng = generator(SEED);
  let correct = 0, wrong = 0, examined = 0;
  for (let i = 0; i < REQUESTS; i += 1) {
    const [path, host, , target] = pick(rng());
    let routedTo = "general";
    if (fields.includes("path")) {               // application layer: the rule chain is run
      for (const [field, value, pool] of RULES) {
        examined += 1;
        if (field === "port" || (field === "path" ? path.startsWith(value) : host === value)) { routedTo = pool; break; }
      }
    } else {                                      // transport layer: only the port is looked at
      examined += 1;
      routedTo = split && SPLIT_POOLS.includes(target) ? target : "general";
    }
    if (routedTo === target) correct += 1; else wrong += 1;
  }
  const rules = RULES.filter(([field, , pool]) =>
    fields.includes(field) || (split && SPLIT_POOLS.includes(pool))).length;
  return { correct, wrong, examined, rules };
}

// [name, fields seen, port split, handshake round, termination point, key copy]
const LAYOUT = [
  ["transport/single port", ["address", "port"], false, 1, BACKENDS, BACKENDS],
  ["transport/port split", ["address", "port"], true, 1, BACKENDS, BACKENDS],
  ["application layer", ["address", "port", "path", "host", "session"], false, 2, BACKENDS + 1, 1],
];
const E = [22, 13, 16, 14, 13, 15];
const print = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(print(["layout", "fields seen", "rules applied", "correct pool", "wrong pool",
  "client item"]));
const O = [];
for (const [name, fields, split] of LAYOUT) {
  const o = run({ fields, split });
  O.push(o);
  console.log(print([name, fields.length, o.rules, o.correct, o.wrong, split ? SPLIT_POOLS.length : 0]));
}
const P = [22, 17, 18, 20, 16];
const print2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](P[i])).join("");
console.log("\n" + print2(["layout", "handshake round", "fields examined", "termination point",
  "key copy"]));
LAYOUT.forEach(([name, , , round, endpoint, keys], i) =>
  console.log(print2([name, round, O[i].examined, endpoint, keys])));

// --- Health check threshold ------------------------------------------------------
// Model: six instances, 300 check rounds; the third instance fails permanently at round 120.
// For a healthy instance, the deviation probability is independent at every check and is not affected by the interval.
const INSTANCES = 6, CHECKS = 300, FAIL_ROUND = 120, DEVIATION = 0.06, SEED2 = 41129, BROKEN = 3;
function health(threshold) {
  const rng = generator(SEED2);
  const consecutive = Array(INSTANCES).fill(0), evicted = Array(INSTANCES).fill(false);
  const successes = Array(INSTANCES).fill(0);
  let detectRound = 0, falseRemoval = 0, lostRounds = 0;
  for (let t = 1; t <= CHECKS; t += 1) {
    for (let inst = 0; inst < INSTANCES; inst += 1) {
      const permanent = inst === BROKEN && t >= FAIL_ROUND;
      const failed = (rng() < DEVIATION) || permanent;
      if (failed) { consecutive[inst] += 1; successes[inst] = 0; } else { consecutive[inst] = 0; successes[inst] += 1; }
      if (!evicted[inst] && consecutive[inst] >= threshold) {
        evicted[inst] = true;
        if (permanent) { if (detectRound === 0) detectRound = t; } else falseRemoval += 1;
      } else if (evicted[inst] && successes[inst] >= 2) evicted[inst] = false;
      if (evicted[inst] && !permanent) lostRounds += 1;
    }
  }
  return { detectRound, falseRemoval, lostRounds };
}
const S = [14, 11, 17, 16, 17, 20, 13];
const print3 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](S[i])).join("");
console.log("\n" + print3(["interval(s)", "threshold", "detect round", "detect(s)", "false removal",
  "lost capacity(s)", "checks/min"]));
for (const interval of [2, 5, 10]) {
  for (const threshold of [1, 2, 3]) {
    const result = health(threshold);
    console.log(print3([interval, threshold, result.detectRound, (result.detectRound - FAIL_ROUND + 1) * interval,
      result.falseRemoval, result.lostRounds * interval, (60 / interval) * INSTANCES]));
  }
}

layout                  fields seen   rules applied  correct pool   wrong pool    client item
transport/single port             2               1           176         1024              0
transport/port split              2               3           547          653              2
application layer                 5               5          1200            0              0

layout                  handshake round   fields examined   termination point        key copy
transport/single port                 1              1200                   6               6
transport/port split                  1              1200                   6               6
application layer                     2              3075                   7               1

interval(s)     threshold     detect round       detect(s)    false removal    lost capacity(s)   checks/min
2                       1              120               2               78                 330          180
2                       2              121               4                2                   8          180
2                       3              122               6                0                   0          180
5                       1              120               5               78                 825           72
5                       2              121              10                2                  20           72
5                       3              122              15                0                   0           72
10                      1              120              10               78                1650           36
10                      2              121              20                2                  40           36
10                      3              122              30                0                   0           36

Fields Seen, Rules That Can Be Applied

The first table’s first column is the number of fields the layer sees: two for the transport layer, five for the application layer. The second column is the number of rules that view allows.

Of the five rules the operator wrote, the transport layer can apply only one: the default. The other four look at the path or the host, fields the transport layer never sees. The result is in the third column: of the 1,200 requests, only 176 reached the pool the operator wanted, and 1,024 fell into the wrong pool — the only correct ones being nightly digest requests that already belonged to the default.

This is not a misconfiguration: the rule is written and correct, but the balancer cannot apply it because it does not see the field the rule looks at — a five-rule intent does not fit into a two-field view.

The second row measures the known fix for this squeeze: port split. Asset traffic is moved to one port, management traffic to another. Because the transport layer sees the port, it can now apply these two rules — applied rules rise from one to three, and correct-pool requests climb from 176 to 547.

The gain is paid for in the last column. Port split is not a change in the balancer but in the client: two client configuration items changed. Map and counter requests still arrive at the same host — there is no way to separate them by port, so 653 requests stay in the wrong pool.

In the third row, the application layer sees all five fields, applies all five rules, and both wrong-pool requests and changed client items fall to zero.

The Application Layer’s Bill

The second table counts the cost of the same three layouts.

The fields-examined column is computed: the transport layer looks at a single field on every request (the port), for a total of 1,200 looks. The application layer runs the rule chain until it matches — the map request matches at the first rule, the counter at the second, the asset at the third, the management at the fourth, and the nightly digest falls through to the fifth rule. The weighted total is 3,075 looks, two and a half times as many. The chain’s order is itself measurable: putting heavy traffic first lowers this number.

The handshake round column is sharper. The transport layer passes the connection through: the encrypted session is established between the client and the back end, the balancer only carries bytes, and the round count is one. The application layer must read the request’s path, which means decrypting it: the session terminates at the balancer, a second session is opened to the back end, and the round count is two.

The last two columns show two faces of this necessity, pointing in opposite directions.

Key copies number six under the transport layer: because the encryption terminates at the back end, the private key must sit on all six instances. Rotating the key means touching six machines. Under the application layer, key copies drop to one — the key lives only on the balancer.

The termination point moves the other way: from six to seven. The head of the request turns back into plaintext first at the balancer, then again at the back end. The added point is unlike the other six: it is the rented component. A component you do not operate, one that sees the path, the host, the session cookie, and the requesting address, has entered the chain. That component also logs these fields, and where that log stops is not this lesson’s decision.

The trade-off fits in one sentence: key copies from six to one, the termination point from six to seven, misrouting from 1,024 to zero.

The Health Check’s Three Numbers

The third table measures the balancer’s second job. How the health check itself works is the provider’s concern; three numbers stay with the renter: the interval, the threshold, and the check path. The sweep runs across two of these three numbers.

The detect round depends on the threshold and is independent of the interval: at threshold one, the broken instance is removed at round 120; at threshold three, at round 122. That same round count is two to six seconds at a two-second interval, and ten to thirty seconds at a ten-second interval. This is how long the broken instance keeps receiving traffic.

The false removal column runs the other way. At threshold one, a healthy instance was removed from traffic 78 times: a single failed check — a transient deviation — pulled it out. When the threshold rises to two, this number drops to two, and at three, to zero.

The lost capacity column is the bill for false removal. At threshold one and interval ten, 1,650 seconds: one instance was pulled out and put back repeatedly, leaving capacity short for roughly 28 minutes. In the same row, the broken instance had been detected in ten seconds. A ten-second gain cost 1,650 seconds of capacity.

The last column counts a third cost: shortening the interval multiplies check requests. At interval ten, 36 check requests a minute; at interval two, 180. These requests land on the back end, creating a measurable load if the check path is not cheap.

Together, the three columns show a pattern: the interval sets the speed and charges for the load, the threshold sets the accuracy and charges for the delay. Threshold three with interval two gives detection in six seconds, zero false removals, and 180 check requests a minute. Threshold one and interval two detect in two seconds but lose 330 seconds of capacity: a four-second difference costs 330 seconds.

The Decision That Stays

In a rented balancer, the provider’s items are clear: the distribution algorithm’s implementation, the balancer’s capacity and scaling, the connection queue’s limits, where the nodes sit. None of these were measured, because there is no decision left to make about them.

Nine items stay with the renter: the layer, the rule chain and its order, the definition of the back-end pools, the health check path, the interval, the threshold, the timeout, where the key sits, session affinity. Eight of them can change at any moment.

The ninth does not. The layer is fixed the moment the balancer is created. A transport layer balancer does not later turn into an application layer one. What has to be done is create a new balancer, wait for it to be given a new endpoint name, and move every client to that name. This is this lesson’s item that cannot be taken back.

The cost sits right in the table: a setup starting with the transport layer brings 1,024 misroutes down to 653 with port split, at a cost of two client items. Closing the remaining 653 means changing layers, which means changing the name the client connects to — the two items touched for port split get touched again.

Summary

• The transport layer sees two fields (address, port) and can apply only the default out of five rules; 1,024 of the 1,200 requests fell into the wrong pool. The application layer sees five fields, applies all five rules, and misrouting is zero.
• Port split gave the transport layer two more rules and raised correct routing from 176 to 547; the cost was two client items changed, and map and counter traffic arriving at the same host could not be separated.
• The application layer raised fields examined from 1,200 to 3,075 and the handshake round from one to two; it dropped key copies from six to one while raising the termination point from six to seven, and the added point is the rented component itself.
• In the health check, raising the threshold from one to three dropped false removals from 78 to zero and extended detection time by as many rounds as the threshold; the interval sets speed and charges for check load, the threshold sets accuracy and charges for delay.
• At threshold one and interval ten, the broken instance was detected in ten seconds, but 1,650 seconds of healthy capacity sat outside needlessly; that is the cost of the ten-second gain.
• Eight of the nine items that stay with the renter can change at any moment; the layer cannot. Changing the layer means a new balancer and a new endpoint name — this lesson’s item that cannot be taken back.

Next Step

This lesson set up the balancer and connected six instances behind it to traffic. Every measurement assumed one thing was fixed: the endpoint the client connects to.

That endpoint is not fixed. Its address changes with the instance, its name changes when the balancer is re-created, and every scaling-group addition arrives with a new address. What the client holds is a single address or name, written separately in the configuration file, the monitoring target, and the third party’s allow list.

The next lesson compares address stability across three layouts: the address going with the instance, moving as a separate resource, and binding to a name. What is measured: how many seconds the client takes to notice the change, how many items are updated by hand, and how many requests reach a stale address. The persistent address’s own cost is counted too: its count is limited, moving it is an operation, and an unused one is billed.