Layer 7 Balancing

The previous lesson’s measured 4.00 times skew and the two streams that could not be split came from the same cause: the decision was made before the information it needed existed. That information sits one layer up. Path, method, headers, and query string start flowing right after the connection is established; if a balancer chooses to read them, it can decide again for every request.

This level is called layer 7 balancing. Layer 7 is the application layer, and HTTP is defined there; the protocol itself is the subject of the Network Models and Protocols course and is not retold here. This lesson’s job is to measure what deciding per request and by content gains and what it costs.

Deciding Per Request

A layer 7 balancer splits a connection into two parts: the one it holds with the client and the one it holds with the replica. In between, it parses the request, reads the path, chooses a pool, and forwards the request rewritten.

// l7/balancer.mjs — layer 7 balancer: the path is read on every request, a pool is chosen, round
// robin distribution is applied. Usage: node l7/balancer.mjs <port> <counter.json> <read> <write>
import { createServer, request, Agent } from "node:http";
import { writeFileSync } from "node:fs";

const port = Number(process.argv[2]);
const counterPath = process.argv[3];
const POOL = {
  read: (process.argv[4] ?? "").split(",").filter(Boolean).map(Number),
  write: (process.argv[5] ?? "").split(",").filter(Boolean).map(Number),
};
const next = { read: 0, write: 0 };
const agents = new Map();                     // one persistent inner connection per replica
const seen = new WeakSet();                   // unique count of inner connections
const counter = { decisions: 0, externalConnections: 0, internalConnections: 0, distribution: {} };

function count(replica, pool) {
  const entry = (counter.distribution[replica] ??= { read: 0, write: 0 });
  entry[pool] += 1;
}

if (Number.isInteger(port) === false || POOL.read.length === 0) {
  console.log("usage: node l7/balancer.mjs <port> <counter.json> <read> <write>");
} else {
  const s = createServer((outerReq, outerRes) => {
    const path = new URL(outerReq.url, "http://local").pathname;   // read on every request
    const pool = path === "/event" ? "write" : "read";
    const list = POOL[pool];
    const target = list[next[pool]++ % list.length];
    counter.decisions += 1;
    count(target, pool);
    if (agents.has(target) === false) agents.set(target, new Agent({ keepAlive: true, maxSockets: 1 }));
    const innerReq = request({ port: target, path: outerReq.url, method: outerReq.method,
      headers: { ...outerReq.headers, forwarded: `for="${outerReq.socket.remoteAddress}"` },
      agent: agents.get(target) }, (innerRes) => {
      outerRes.writeHead(innerRes.statusCode, innerRes.headers);
      innerRes.pipe(outerRes);
    });
    innerReq.on("socket", (socket) => {
      if (seen.has(socket) === false) { seen.add(socket); counter.internalConnections += 1; }
    });
    innerReq.on("error", () => outerRes.writeHead(502).end("replica unreachable"));
    outerReq.pipe(innerReq);
  });
  s.on("connection", () => { counter.externalConnections += 1; });
  s.listen(port, "127.0.0.1");
  process.on("SIGTERM", () => { writeFileSync(counterPath, JSON.stringify(counter)); process.exit(0); });
}

In this setup, the replicas report only their own name; counting now lives in the balancer, since it makes the decision and knows which request went where.

// l7/replica.mjs — application replica: only reports its own name, counting is kept in the balancer
import { createServer } from "node:http";

const [port, name] = [Number(process.argv[2]), process.argv[3]];

if (Number.isInteger(port) === false) console.log("usage: node l7/replica.mjs <port> <name>");
else createServer((req, res) => {
  res.sendDate = false;
  res.writeHead(200, { "content-type": "application/json", "x-replica": name });
  res.end(JSON.stringify({ state: "at transfer hub", zone: "35", step: 4 }));
}).listen(port, "127.0.0.1");

The client repeats the previous lesson’s mix exactly — six clients, uneven shares, one persistent connection per client — and adds one thing: a path mix. One in five requests is a status event, so the setup’s read/write ratio stays close to the one K01 calculated.

// l7/client.mjs — lesson 02's same mix: each client one persistent connection, uneven shares.
// The path mix approximately keeps K01's read/write ratio: one in five requests is a status event.
import { Agent, request } from "node:http";

const port = Number(process.argv[2]);
const SHARES = (process.argv[3] ?? "1").split(",").map(Number);

function one(agent, no, path) {
  return new Promise((resolve, reject) => {
    const r = request({ port, path: `${path}?no=${no}`, agent }, (res) => {
      res.resume();
      res.on("end", () => resolve(res.headers["x-replica"] ?? "-"));
    });
    r.on("error", reject);
    r.end();
  });
}

if (Number.isInteger(port) === false) console.log("usage: node l7/client.mjs <port> <share,share,...>");
else {
  let g = 0, tracking = 0, event = 0;
  for (let i = 0; i < SHARES.length; i += 1) {
    const agent = new Agent({ keepAlive: true, maxSockets: 1 });
    for (let j = 0; j < SHARES[i]; j += 1) {
      const path = g % 5 === 4 ? "/event" : "/tracking";
      if (path === "/event") event += 1; else tracking += 1;
      await one(agent, `I${i}-${j}`, path);
      g += 1;
    }
    agent.destroy();
  }
  console.log(`${SHARES.length} clients, ${g} requests: tracking ${tracking}, event ${event} ` +
    `(ratio ${(tracking / event).toFixed(2)}; K01's peak read/write = 4.29)`);
}

// l7/report.mjs — turns the balancer's counter file into a table
import { existsSync, readFileSync } from "node:fs";

const counterPath = process.argv[2];
if (counterPath === undefined || existsSync(counterPath) === false) {
  console.log("usage: node l7/report.mjs <counter.json>");
  process.exit(0);
}
const data = JSON.parse(readFileSync(counterPath, "utf8"));
const rows = Object.entries(data.distribution).sort();
const totals = rows.map(([, v]) => v.read + v.write);

console.log("replica  tracking  event  total");
for (const [name, v] of rows) {
  console.log(`${name.padEnd(7)}  ${String(v.read).padStart(8)}  ${String(v.write).padStart(5)}  ` +
    `${String(v.read + v.write).padStart(5)}`);
}
console.log(`decisions = ${data.decisions} (1 per request), external connections = ${data.externalConnections}, ` +
  `internal connections = ${data.internalConnections}`);
console.log(`request skew = ${(Math.max(...totals) / Math.min(...totals)).toFixed(2)}`);

The setup brings up the same four replicas twice. In the first run all four are in a single pool; in the second, three are in the read pool and one in the write pool. Port numbers are environment-dependent.

# measure.sh — the same client mix with two layouts: four replicas in a single pool, then read/write split
for layout in "8881,8882,8883,8884:8881,8882,8883,8884" "8881,8882,8883:8884"; do
  READ=${layout%%:*}; WRITE=${layout##*:}
  node l7/replica.mjs 8881 k1 & K1=$!
  node l7/replica.mjs 8882 k2 & K2=$!
  node l7/replica.mjs 8883 k3 & K3=$!
  node l7/replica.mjs 8884 k4 & K4=$!
  node l7/balancer.mjs 8845 counter.json "$READ" "$WRITE" & D=$!
  sleep 1
  echo "read pool = $READ   write pool = $WRITE"
  node l7/client.mjs 8845 "1,2,4,8,16,32"
  kill -TERM $D; sleep 0.3
  node l7/report.mjs counter.json
  kill $K1 $K2 $K3 $K4; sleep 0.3
  echo
done

read pool = 8881,8882,8883,8884   write pool = 8881,8882,8883,8884
6 clients, 63 requests: tracking 51, event 12 (ratio 4.25; K01's peak read/write = 4.29)
replica  tracking  event  total
8881           13      3     16
8882           13      3     16
8883           13      3     16
8884           12      3     15
decisions = 63 (1 per request), external connections = 6, internal connections = 4
request skew = 1.07

read pool = 8881,8882,8883   write pool = 8884
6 clients, 63 requests: tracking 51, event 12 (ratio 4.25; K01's peak read/write = 4.29)
replica  tracking  event  total
8881           17      0     17
8882           17      0     17
8883           17      0     17
8884            0     12     12
decisions = 63 (1 per request), external connections = 6, internal connections = 4
request skew = 1.42

The Skew Disappearing

These numbers are in the measurement class and are deterministic. The comparison is made against the previous lesson’s table: the same six clients, the same uneven shares, the same total request count.

At layer 4, request skew was 4.00. At layer 7, it is 1.07. Why the skew disappears is visible in the table: the decision count is 63, that is, 1 per request. The client that sent thirty-two requests over a single connection is now subject to thirty-two separate decisions, and those requests spread across four replicas. The remaining 1.07 comes from not dividing evenly — 63 requests split four ways come out 16, 16, 16, and 15.

The second run shows a separate gain. Because the same balancer reads the path, it sends /tracking requests to the read pool and /event requests to the write pool: the read replicas got 17 tracking requests each, the write replica got 12 event requests, with no cross-leak. The split that was impossible at layer 4 has been reduced here to a list assignment. Total skew rising to 1.42 is not a flaw: since the pools now serve different streams, comparing them with a single skew number is meaningless — each pool is sized to its own stream’s rate.

The connection column carries a third difference. External connections: 6, internal: 4. The balancer collapses the clients’ six connections into one persistent connection per replica; this kind of collapse was impossible at layer 4, since every external connection had a matching internal one there. The number of connections replicas see becomes independent of the client count.

The Cost Paid

Every gain is priced on the same line: 63 decisions. Layer 4 passed the same load with 6. The work done per decision has also grown — the balancer parses the request line, copies the headers, adds its own forwarded header, and rewrites the request. This work’s duration is not measured, since parsing cost depends on the implementation and the machine; what is measured is how many times the work is done.

The second cost is a transfer of responsibility. The moment the balancer parses the request, it becomes the owner of some of the decisions about it: which headers get forwarded, whether the body gets buffered, whether a request gets retried on another replica while one is not answering. None of these questions existed at the layer 4 balancer, since there was no concept of a request there. A content-aware balancer becomes part of the application.

The third cost is dependency. Layer 7 rules are tied to the protocol and to path names. When the /event path is renamed to /status-event, the balancer’s configuration must change too; a layer 4 balancer would never be affected by that change.

Back to the Calculation

The previous two lessons assumed a single pool and divided the 513.89 req/s peak rate. Content-aware routing opens the option of splitting that division in two, and K01 had already calculated the two stream rates separately: peak read 416.67 req/s, peak write 97.22 req/s.

// l7/pool.mjs — splitting K01's stream rates from a single pool into separate pools, and the effect of write overflow
const READ = 416.67;               // K01 Back-of-the-Envelope Estimation: peak read req/s
const WRITE = 97.22;               // K01: peak write req/s
const SATURATION = 400;            // lesson 01 assumption Y1
const SAFE_RATE = 200;             // lesson 01: Y1 x Y2

const replicas = (rate) => Math.ceil(rate / SAFE_RATE);
const utilization = (rate, n) => rate / (n * SATURATION);

const single = replicas(READ + WRITE);
const readReplicas = replicas(READ), writeReplicas = replicas(WRITE);
console.log(`single pool: ${single} replicas, utilization ${utilization(READ + WRITE, single).toFixed(3)}`);
console.log(`split pools: read ${readReplicas} + write ${writeReplicas} = ${readReplicas + writeReplicas} replicas, ` +
  `utilization read ${utilization(READ, readReplicas).toFixed(3)}, write ${utilization(WRITE, writeReplicas).toFixed(3)}`);
console.log(`cost of splitting = ${readReplicas + writeReplicas - single} replica(s)\n`);

console.log("write multiplier   single pool utilization   split pool read   split pool write   write pool overflowed");
for (const c of [1, 2, 3, 4]) {
  const writeRate = WRITE * c;
  console.log(`${`x${c}`.padStart(16)}   ${utilization(READ + writeRate, single).toFixed(3).padStart(23)}   ` +
    `${utilization(READ, readReplicas).toFixed(3).padStart(16)}   ${utilization(writeRate, writeReplicas).toFixed(3).padStart(17)}   ` +
    `${String(writeRate / writeReplicas > SAFE_RATE).padStart(20)}`);
}

console.log(`\ndecision rate: layer 4 at ${((READ + WRITE) / 10).toFixed(2)} decisions/s (Y4 = 10), ` +
  `layer 7 at ${(READ + WRITE).toFixed(2)} decisions/s`);
console.log(`sensitivity: once the write rate passes ${(SAFE_RATE * writeReplicas).toFixed(0)} req/s the write pool grows to ` +
  `${writeReplicas + 1} replicas, the read pool does not change`);

single pool: 3 replicas, utilization 0.428
split pools: read 3 + write 1 = 4 replicas, utilization read 0.347, write 0.243
cost of splitting = 1 replica(s)

write multiplier   single pool utilization   split pool read   split pool write   write pool overflowed
              x1                     0.428              0.347               0.243                  false
              x2                     0.509              0.347               0.486                  false
              x3                     0.590              0.347               0.729                   true
              x4                     0.671              0.347               0.972                   true

decision rate: layer 4 at 51.39 decisions/s (Y4 = 10), layer 7 at 513.89 decisions/s
sensitivity: once the write rate passes 200 req/s the write pool grows to 2 replicas, the read pool does not change

The first two lines give the bill for the split as a computed value: a single pool needs 3 replicas, split pools need 3 plus 1, 4. Splitting costs one replica, because each pool rounds up separately to its own rate, and the write pool’s 97.22 req/s leaves half of one replica’s 200 req/s share empty. Content-aware routing has no capacity gain on its own; on the contrary, it needs resources.

What it gets in return is in the second table. When the status event stream from the carrier triples — a transfer hub sending accumulated scans in bulk — the single pool’s utilization climbs from 0.428 to 0.590, and this rise affects tracking queries too, because the same replicas serve both. In split pools, read utilization stays fixed at 0.347 across every multiplier; the overflow happens only in the write pool, and there, from the third row on, growing that pool’s replica count is enough. What the split buys is not capacity but isolation: one stream overflowing does not change the other’s utilization.

The last lines carry the lesson’s price tag to K01’s scale. At layer 4, the balancer would make 51.39 decisions a second; at layer 7, it makes 513.89. Ten times the decisions, one extra replica, and one stream’s isolation — the three are parts of the same decision.

Summary

• Layer 7 balancing decides again for every request, deciding by the request’s content; the connection held with the replica is separate from the one held with the client.
• In the same client mix, request skew is 4.00 at layer 4 and 1.07 at layer 7; the remaining skew comes from 63 requests not dividing evenly by four.
• Reading the path makes it possible to split streams into separate pools: the read replicas got 17 tracking requests each, the write replica got 12 event requests, with no cross-leak.
• The balancer collapsed six external connections into four internal ones; the connection count replicas see became independent of the client count.
• The cost is the decision count: 63 instead of 6, 513.89 decisions/s instead of 51.39 at K01’s scale; parsing the request also makes the balancer the owner of header, buffering, and retry decisions.
• Splitting streams costs one replica (4 instead of 3) but buys isolation: when the write stream triples, the single pool’s utilization climbs from 0.428 to 0.590, while split-pool read utilization stays at 0.347.

Next Step

The three lessons so far kept the distribution rule fixed: replicas were chosen in turn. Round robin assumes replicas are each other’s equal and that every request does the same work. Neither is always true — a replica may have slowed down, one request may cost ten times another, and which replica a request lands on may need to stay stable in a way that serves the cache. The next lesson changes the rule itself: it compares round robin against a rule that looks at open connection count and one that looks at a key, under the same load, and counts how many keys move when the third one’s replica count changes.