Layer 4 Balancing

The previous lesson’s balancer decided on every request, reading the request’s path to do it. Reading is a choice, not a requirement. A balancer can work without looking into the request: it decides once, when the connection is established, then passes everything that flows through it to the same replica.

This level is called layer 4 balancing. The layer model was established in the Network Models and Protocols course: layer 4 is the transport layer, and what is visible there is address, port, and byte stream; path, headers, and body are a layer up. This lesson builds that level, counts what it cannot know, and measures what not knowing does to distribution.

One Decision Per Connection

A layer 4 balancer accepts a connection, chooses a replica, opens its own connection to that replica, and relays bytes between the two directions as-is. The choice is made once and holds until the connection closes.

// l4/balancer.mjs — layer 4 balancer: the decision is made once when the connection is
// established, then both directions are relayed as-is. Usage: node l4/balancer.mjs <port> <counter.json> <replicas...>
import { createServer, connect } from "node:net";
import { writeFileSync } from "node:fs";

const port = Number(process.argv[2]);
const counterPath = process.argv[3];
const replicas = process.argv.slice(4).map(Number);
const counter = { decisions: 0, connections: Object.fromEntries(replicas.map((p) => [p, 0])), bytes: 0 };
let next = 0;

if (Number.isInteger(port) === false || replicas.length === 0) {
  console.log("usage: node l4/balancer.mjs <port> <counter.json> <replicas...>");
} else {
  const s = createServer((outer) => {
    const target = replicas[next++ % replicas.length];   // single decision: which replica this connection belongs to
    counter.decisions += 1;
    counter.connections[target] += 1;
    const inner = connect(target, "127.0.0.1");
    outer.on("data", (chunk) => { counter.bytes += chunk.length; });   // bytes passed are counted, content is not read
    outer.pipe(inner);
    inner.pipe(outer);
    for (const [a, b] of [[outer, inner], [inner, outer]]) {
      a.on("error", () => b.destroy());
      a.on("close", () => b.destroy());
    }
  });
  s.listen(port, "127.0.0.1");
  process.on("SIGTERM", () => { writeFileSync(counterPath, JSON.stringify(counter)); process.exit(0); });
}

The distinction the code carries is the distinction between node:net and node:http. There is no HTTP request object in this file, so there is no path, header, or tracking number either. The only content measure the counter can hold is the number of bytes passed — and even that is a count of bytes whose content is unknown.

The replicas run a layer up and count the requests they see by path. They report their counters through the /counter path; this path is queried directly, not through the balancer, so it does not interfere with the measurement.

// l4/replica.mjs — application replica: counts the requests it sees by path, reports via /counter
import { createServer } from "node:http";

const [port, name] = [Number(process.argv[2]), process.argv[3]];
const counter = { name, requests: 0, paths: {} };

if (Number.isInteger(port) === false) console.log("usage: node l4/replica.mjs <port> <name>");
else createServer((req, res) => {
  res.sendDate = false;
  const path = new URL(req.url, "http://local").pathname;
  if (path === "/counter") {
    res.writeHead(200, { "content-type": "application/json" }).end(JSON.stringify(counter));
    return;
  }
  counter.requests += 1;
  counter.paths[path] = (counter.paths[path] ?? 0) + 1;
  res.writeHead(200, { "content-type": "application/json", "x-replica": name });
  res.end(JSON.stringify({ state: "at transfer hub", zone: "35", step: 4 }));
}).listen(port, "127.0.0.1");

The client carries the setup’s real parameter: each client opens a single persistent connection and sends its share of requests through it. Whether the shares are even or uneven is the measurement’s input. Clients run in sequence, so the connection order — and hence the mapping round robin distribution produces — is deterministic.

// l4/client.mjs — each client opens a single persistent connection and sends its share of requests in sequence
import { Agent, request } from "node:http";

const port = Number(process.argv[2]);
const SHARES = (process.argv[3] ?? "1").split(",").map(Number);   // requests per client

function one(agent, no, path) {
  return new Promise((resolve, reject) => {
    const r = request({ port, path: `${path}?no=${no}`, agent }, (res) => {
      res.resume();
      res.on("end", () => resolve(res.headers["x-replica"] ?? "-"));
    });
    r.on("error", reject);
    r.end();
  });
}

if (Number.isInteger(port) === false) console.log("usage: node l4/client.mjs <port> <share,share,...>");
else {
  let total = 0;
  for (let i = 0; i < SHARES.length; i += 1) {          // clients run in sequence: connection order is deterministic
    const agent = new Agent({ keepAlive: true, maxSockets: 1 });
    for (let j = 0; j < SHARES[i]; j += 1) {
      await one(agent, `I${i}-${j}`, j % 2 === 0 ? "/tracking" : "/event");
      total += 1;
    }
    agent.destroy();
  }
  console.log(`${SHARES.length} clients, ${total} requests sent (shares: ${SHARES.join(",")})`);
}

The report script merges two sources: the balancer’s connection counters and the replicas’ request counters.

// l4/report.mjs — merges the balancer's counter file with the replicas' /counter output into a single table
import { existsSync, readFileSync } from "node:fs";

const [counterPath, ...ports] = process.argv.slice(2);
if (counterPath === undefined || existsSync(counterPath) === false) {
  console.log("usage: node l4/report.mjs <counter.json> <replica-ports...>");
  process.exit(0);
}
const data = JSON.parse(readFileSync(counterPath, "utf8"));

const rows = [];
for (const p of ports) {
  const r = await fetch(`http://127.0.0.1:${p}/counter`).then((res) => res.json());
  rows.push({ name: r.name, port: p, connections: data.connections[p], requests: r.requests,
    paths: Object.entries(r.paths).sort().map(([a, n]) => `${a.slice(1)}=${n}`).join(" ") });
}

const requests = rows.map((s) => s.requests);
console.log("replica  connections  requests  path distribution");
for (const s of rows) {
  console.log(`${s.name.padEnd(7)}  ${String(s.connections).padStart(11)}  ${String(s.requests).padStart(8)}  ${s.paths}`);
}
console.log(`decisions = ${data.decisions} (1 per connection), decisions per request = ` +
  `${(data.decisions / requests.reduce((a, b) => a + b, 0)).toFixed(3)}`);
console.log(`connection skew = ${(Math.max(...rows.map((s) => s.connections)) / Math.min(...rows.map((s) => s.connections))).toFixed(2)}, ` +
  `request skew = ${(Math.max(...requests) / Math.min(...requests)).toFixed(2)}`);
console.log(`request bytes through the balancer = ${data.bytes}`);

The setup runs twice. Both runs have six clients and three replicas; the only thing that changes is how requests are distributed among clients. Port numbers are environment-dependent.

# measure.sh — the same total load with two client mixes: even shares and uneven shares
for mix in "10,10,10,10,10,10" "1,2,4,8,16,32"; do
  node l4/replica.mjs 8871 k1 & K1=$!
  node l4/replica.mjs 8872 k2 & K2=$!
  node l4/replica.mjs 8873 k3 & K3=$!
  node l4/balancer.mjs 8841 counter.json 8871 8872 8873 & D=$!
  sleep 1
  node l4/client.mjs 8841 "$mix"
  kill -TERM $D; sleep 0.3
  node l4/report.mjs counter.json 8871 8872 8873
  kill $K1 $K2 $K3; sleep 0.3
  echo
done

6 clients, 60 requests sent (shares: 10,10,10,10,10,10)
replica  connections  requests  path distribution
k1                 2        20  event=10 tracking=10
k2                 2        20  event=10 tracking=10
k3                 2        20  event=10 tracking=10
decisions = 6 (1 per connection), decisions per request = 0.100
connection skew = 1.00, request skew = 1.00
request bytes through the balancer = 4710

6 clients, 63 requests sent (shares: 1,2,4,8,16,32)
replica  connections  requests  path distribution
k1                 2         9  event=4 tracking=5
k2                 2        18  event=9 tracking=9
k3                 2        36  event=18 tracking=18
decisions = 6 (1 per connection), decisions per request = 0.095
connection skew = 1.00, request skew = 4.00
request bytes through the balancer = 4975

Two Skews in One Table

These numbers are in the measurement class and are deterministic; since clients run in sequence and the distribution rule is round robin, another machine gives the same split.

Connection skew is 1.00 in both runs: the balancer split the six connections two-and-two across the three replicas, with no flaw in that split. It does the job it was given perfectly. In the first run, request skew is also 1.00: since the shares are even, balanced connections translate into balanced requests.

In the second run, that same perfect connection distribution produces 4.00 times the request skew. The replicas saw 9, 18, and 36 requests; the most loaded replica carried four times what the least loaded one did. The only thing that changed is how requests were distributed among connections. A layer 4 balancer balances connections, not requests; since the two are not the same thing, the measurement needs two separate skew columns.

The skew’s source is the persistent connection. If a connection carries one request, the two numbers are the same; if it carries thirty-two, one decision by the balancer ties thirty-two requests together. Because the decision is made once, it cannot be corrected: the balancer does not notice that 36 requests went to the same replica, because it never counts them.

What Is Not Known

The decision column is the measure of not knowing: 6 decisions for 63 requests, 0.095 decisions per request. The layer 4 balancer’s work is cheaper than that ratio suggests — choosing once per connection and then passing bytes through is less work than parsing and rewriting every request.

The bill paid shows up in the path distribution column. Every replica received both tracking and event requests, in the same ratio as the connection shares. Splitting some replicas off for reading and others for writing is impossible at this level: the decision is made when the connection is established, and no path has been sent yet at that point. The same constraint holds for finer distinctions too — no rule can look at a tracking number, a client type, or a header, because none of these are visible at layer 4. The 4975 bytes the balancer sees are a stream of unknown content.

Not knowing also has an upside. Because the layer 4 balancer does not know HTTP, it is not tied to HTTP: the same code works for another connection-based protocol too, and since it does not buffer the body, streamed data passes through without interference.

Back to the Calculation

The previous lesson divided the 513.89 req/s peak rate across three replicas and found 171.30 req/s per replica. That division assumed the distribution was perfect. The measured skew breaks the assumption. The calculation below derives an imbalance factor from the measured shares: the heaviest replica’s share multiplied by the replica count, which is 1 in perfect balance.

One more assumption is needed. Y4 — average requests per persistent connection: 10. Rationale: when a user opens the tracking page, several queries pass over the same connection; the number is this topic’s own assumption and is not added to K01’s table.

// l4/capacity.mjs — what the measured imbalance corresponds to at K01's peak rate, and the replicas it requires
const PEAK_EDGE = 513.89;          // K01 Back-of-the-Envelope Estimation: peak edge req/s
const SAFE_RATE = 200;             // lesson 01: Y1 x Y2 = 400 x 0.50
const REQ_PER_CONN = 10;           // assumption Y4: average requests per persistent connection

const MEASURED = { even: [20, 20, 20], uneven: [9, 18, 36] };   // lesson 02's measurement

const imbalanceFactor = (shares) => (Math.max(...shares) / shares.reduce((a, b) => a + b, 0)) * shares.length;
const required = (shares) => Math.ceil((imbalanceFactor(shares) * PEAK_EDGE) / SAFE_RATE);

console.log("mix      shares        heaviest share  imbalance  heaviest replica req/s  utilization  required replicas");
for (const [name, shares] of Object.entries(MEASURED)) {
  const heaviestShare = Math.max(...shares) / shares.reduce((a, b) => a + b, 0);
  const rate = heaviestShare * PEAK_EDGE;
  console.log(`${name.padEnd(8)} ${shares.join("/").padEnd(13)} ${heaviestShare.toFixed(4).padStart(15)} ` +
    `${imbalanceFactor(shares).toFixed(3).padStart(10)} ${rate.toFixed(2).padStart(23)} ` +
    `${(rate / 400).toFixed(3).padStart(12)} ${String(required(shares)).padStart(18)}`);
}

console.log(`\nreplicas in perfect balance = ${required(MEASURED.even)}, at measured imbalance = ` +
  `${required(MEASURED.uneven)}: difference ${required(MEASURED.uneven) - required(MEASURED.even)} replicas, ` +
  `purely for distribution skew`);

console.log(`\ndecision rate (1 decision per connection, Y4 = ${REQ_PER_CONN}): ` +
  `${(PEAK_EDGE / REQ_PER_CONN).toFixed(2)} decisions/s`);
console.log(`if a decision were made per request: ${PEAK_EDGE} decisions/s, ${REQ_PER_CONN} times`);
console.log(`sensitivity: if Y4 = 1 the two rates are equal, if Y4 = 100 the ratio becomes 100 times`);

mix      shares        heaviest share  imbalance  heaviest replica req/s  utilization  required replicas
even     20/20/20               0.3333      1.000                  171.30        0.428                  3
uneven   9/18/36                0.5714      1.714                  293.65        0.734                  5

replicas in perfect balance = 3, at measured imbalance = 5: difference 2 replicas, purely for distribution skew

decision rate (1 decision per connection, Y4 = 10): 51.39 decisions/s
if a decision were made per request: 513.89 decisions/s, 10 times
sensitivity: if Y4 = 1 the two rates are equal, if Y4 = 100 the ratio becomes 100 times

These numbers, in the computed class, correct the previous lesson’s result. In perfect balance the heaviest replica carries 171.30 req/s and 0.428 utilization; at the measured imbalance it carries 293.65 req/s and 0.734 utilization. The second number sits above the 0.50 target set by Y2, meaning the three-replica design does not hold the target at peak load. The replica count needed to hold the target is 5, not 3, and the extra two replicas are paid not for load growth but purely for distribution skew.

The decision-rate row also gives the next lesson’s price tag. With a decision per connection, the balancer produces 51.39 decisions a second; if it decided per request, it would produce 513.89, ten times as many. Correcting the skew means paying that tenfold price.

Summary

• Layer 4 balancing makes its decision once, when the connection is established; every request flowing through that connection goes to the same replica, and the decision is never corrected afterward.
• The same total load split 20–20–20 with even shares and 9–18–36 with uneven shares: connection skew was 1.00 in both runs, request skew was 1.00 against 4.00.
• Balancing connections is not balancing requests; the skew’s source is that persistent connections do not carry equal request counts, and the balancer cannot measure this because it does not count requests.
• The decision count is 6 for 63 requests (0.095 per request); this cheapness is bought by the path, headers, and tracking number staying invisible — splitting reads and writes across separate replicas is impossible at this level.
• The measured imbalance factor of 1.714 pushes the heaviest replica above target at the peak rate, to 293.65 req/s and 0.734 utilization; holding the target needs 5 replicas instead of 3.
• The cost of deciding per request, under the Y4 = 10 assumption, is 513.89 decisions/s instead of 51.39 — ten times as many.

Next Step

The measured 4.00 times skew and the two streams that cannot be split come from the same cause: the decision is made before the information it needs exists. That information sits one layer up — path, headers, method, and query string start flowing right after the connection is established. If a balancer chooses to read them, it can decide again for every request, route it to a different set of replicas by content, and pay the bill for producing ten times the decisions. The next lesson builds this level: it measures what the skew becomes when the same client mix passes through a content-aware balancer, how splitting reads and writes into separate pools changes the replica count, and what new responsibilities parsing the request hands the balancer.