---
title: 'Role of the Load Balancer'
source: 'https://academia.sh/en/courses/traffic-layer/role-of-the-load-balancer'
course: 'The Traffic Layer'
language: en
updated: '2026-08-23T14:25:25+00:00'
license: 'CC BY-SA 4.0'
---

# Role of the Load Balancer

The three responsibilities of the layer that distributes requests among replicas doing the same work: measuring distribution with local processes, comparing the answered-request rate between setups with and without health checking when a replica goes down, treating TLS termination as a placement decision, and dividing the peak request rate computed in the introductory course by the replica count.

The previous topic managed to keep some requests from reaching the application **at all**: the
edge cache met nine-tenths of tracking responses at the boundary, the separated content path
pulled away almost all of a page load's requests, and the DNS layer chose which zone the request
would go to. What remains are the requests that genuinely need to reach the application — 513.89
of them per second at the edge. One question is still unanswered: once the zone is chosen, which
of the several replicas doing the same work does a request land on, and if that replica is down,
who notices.

This lesson ties those two questions to a single component. The **load balancer** sits in front
of replicas doing the same work and routes an incoming request to one of them. The lesson
enumerates this device's responsibilities, gets each one's code-level measurable result, and
shows what dividing the peak request rate computed in the Introduction to System Design course by
the replica count yields.

## Three Responsibilities

The balancer's jobs can be named separately, each requiring its own decision.

**Distribution.** Choosing which replica an incoming request goes to. The selection rule is the
subject of later lessons; here, in its plainest form, it is **round robin** — replicas chosen in
turn.

**Health checking.** Independently testing whether a replica can still accept requests, and
removing the ones that cannot from the pool. The check request is separate from the user's
request: the balancer polls the replicas at fixed intervals and drops one from distribution when
it gets no answer.

**TLS termination.** Choosing where the encrypted connection ends. If it ends at the balancer, the
balancer sees the request's content; if it stays encrypted all the way to the replica, it does
not.

Some of these jobs are also done by the **reverse proxy** from the Server-Side Fundamentals
course; their overlap and distinction is this topic's fifth lesson. Here the balancer keeps a
single role: sharing requests among replicas doing the same work.

## Setup

The measurement is set up with local processes. Three replicas are each a `node:http` server that
writes its own name into its response; the balancer does round robin distribution and can run
with health checking on or off. No real load balancer, cloud environment, or container is set up.

```js
// balance/replica.mjs — an application replica: returns the tracking response, writes its own name to the header
import { createServer } from "node:http";

const [port, name] = [Number(process.argv[2]), process.argv[3]];

const BODY = (no) => JSON.stringify({
  no, state: "at transfer hub", zone: "35", step: 4,
  route: ["accepted", "departed", "transfer-34"],
});

if (Number.isInteger(port) === false) console.log("usage: node balance/replica.mjs <port> <name>");
else createServer((req, res) => {
  const url = new URL(req.url, "http://local");
  if (url.pathname === "/health") { res.writeHead(200, { "x-replica": name }).end("up"); return; }
  const body = BODY(url.searchParams.get("no") ?? "-");
  res.writeHead(200, { "content-type": "application/json", "x-replica": name });
  res.end(body);
}).listen(port, "127.0.0.1");
```

The balancer does two of the three jobs: it chooses a replica in turn and checks the replicas'
health on a separate timer. The check interval is 200 milliseconds, and a single failed round
counts a replica as down; both are this topic's assumptions, referred to below as Y3.

```js
// balance/balancer.mjs — round robin distribution + health check. Usage:
//   node balance/balancer.mjs <port> <on|off> <replica-ports...>
import { createServer, request, get } from "node:http";

const INTERVAL = 200;                // assumption Y3: health check interval (ms)
const THRESHOLD = 1;                 // assumption Y3: consecutive failures counted as down

const port = Number(process.argv[2]);
const healthOn = process.argv[3] === "on";
const replicas = process.argv.slice(4).map((p) => ({ port: Number(p), up: true, failures: 0 }));

let next = 0, unreachable = 0;

function check(r) {
  const req = get({ port: r.port, path: "/health", timeout: 100 }, (res) => {
    res.resume();
    if (res.statusCode === 200) { r.failures = 0; r.up = true; } else fail(r);
  });
  req.on("error", () => fail(r));
  req.on("timeout", () => req.destroy());
}
function fail(r) { r.failures += 1; if (r.failures >= THRESHOLD) r.up = false; }

if (Number.isInteger(port) === false || replicas.length === 0) {
  console.log("usage: node balance/balancer.mjs <port> <on|off> <replica-ports...>");
} else {
  if (healthOn) setInterval(() => replicas.forEach(check), INTERVAL).unref();
  createServer((outerReq, outerRes) => {
    if (outerReq.url === "/manage/status") {
      outerRes.writeHead(200, { "content-type": "application/json" });
      outerRes.end(JSON.stringify({ unreachable,
        up: replicas.filter((r) => r.up).map((r) => r.port) }));
      return;
    }
    const pool = replicas.filter((r) => r.up);
    if (pool.length === 0) { outerRes.writeHead(503).end("pool empty"); return; }
    const chosen = pool[next++ % pool.length];
    const innerReq = request({ port: chosen.port, path: outerReq.url,
      method: outerReq.method, headers: outerReq.headers }, (innerRes) => {
      outerRes.writeHead(innerRes.statusCode, innerRes.headers);
      innerRes.pipe(outerRes);
    });
    innerReq.on("error", () => { unreachable += 1; outerRes.writeHead(502).end("replica unreachable"); });
    outerReq.pipe(innerReq);
  }).listen(port, "127.0.0.1");
}
```

The client sends requests **in sequence**; keeping concurrency at 1 makes the distribution counts
deterministic — otherwise the same setup would give a different split on every run.

```js
// balance/client.mjs — sends requests in sequence, counts which replica answered each one
import { Agent, request } from "node:http";

const [port, count, label] = [Number(process.argv[2]), Number(process.argv[3]), process.argv[4]];
const agent = new Agent({ keepAlive: true, maxSockets: 1 });

function one(no) {
  return new Promise((resolve, reject) => {
    const r = request({ port, path: `/tracking?no=${no}`, agent }, (res) => {
      res.resume();
      res.on("end", () => resolve({ code: res.statusCode, replica: res.headers["x-replica"] ?? "-" }));
    });
    r.on("error", reject);
    r.end();
  });
}

if (Number.isInteger(port) === false) console.log("usage: node balance/client.mjs <port> <count> <label>");
else {
  const counts = new Map();
  let succeeded = 0, failed = 0;
  for (let i = 0; i < count; i += 1) {
    const s = await one(`G${i}`);
    if (s.code === 200) succeeded += 1; else failed += 1;
    const key = s.code === 200 ? s.replica : `code ${s.code}`;
    counts.set(key, (counts.get(key) ?? 0) + 1);
  }
  const distribution = [...counts].sort().map(([a, n]) => `${a}=${n}`).join("  ");
  console.log(`${label.padEnd(30)} answered ${String(succeeded).padStart(4)}/${count}` +
    `  failed ${String(failed).padStart(3)}  |  ${distribution}`);
  agent.destroy();
}
```

The driver script puts two balancers in front of the same three replicas, one with health
checking on and the other off, then kills replica `k2` and sends the same load to both. Port
numbers are environment-dependent and should change if already in use.

```bash
# measure.sh — three replicas, two balancers (health check on and off); replica k2 is dropped
node balance/replica.mjs 8861 k1 & K1=$!
node balance/replica.mjs 8862 k2 & K2=$!
node balance/replica.mjs 8863 k3 & K3=$!
node balance/balancer.mjs 8851 on  8861 8862 8863 & BON=$!
node balance/balancer.mjs 8852 off 8861 8862 8863 & BOFF=$!
sleep 1

node balance/client.mjs 8851 300 "three replicas up"
kill $K2; sleep 1
node balance/client.mjs 8851 300 "k2 down, health check on"
node balance/client.mjs 8852 300 "k2 down, health check off"
echo "pool (health check on) : $(curl -s localhost:8851/manage/status)"
echo "pool (health check off): $(curl -s localhost:8852/manage/status)"
kill $K1 $K3 $BON $BOFF
```

```
three replicas up              answered  300/300  failed   0  |  k1=100  k2=100  k3=100
k2 down, health check on       answered  300/300  failed   0  |  k1=150  k3=150
k2 down, health check off      answered  200/300  failed 100  |  code 502=100  k1=100  k3=100
pool (health check on) : {"unreachable":0,"up":[8861,8863]}
pool (health check off): {"unreachable":100,"up":[8861,8862,8863]}
```

## Measured Effect

These numbers are in the **measurement** class: they come from local processes, and they are
deterministic, since requests are sent in sequence and the distribution rule is round robin —
another machine gives the same split, though its port numbers will differ.

The first row shows the distribution: with three replicas up, 300 requests split 100–100–100.
The balancer alone produces no gain, only a three-way division — which becomes a capacity
calculation in the next section.

The second and third rows are the lesson's real measurement. Same failure, same load, same
replicas. With health checking on, the balancer answered 300 of 300 requests and split the load
150–150 across the remaining two; the pool status shows `k2`'s port dropped from the list. With
it off, the balancer answered only 200 of 300: round robin sent one in three requests to the dead
replica, and 100 came back with 502. The pool status still counts all three replicas as up.

This result changes the balancer's definition. **Distribution alone does not produce service
availability.** Without a check that notices the drop, adding replicas only dilutes the failure:
the system does not stop, but a third of requests keep dropping, and it never announces itself as
a mass outage. Health checking is what turns replication into a service availability tool.

## Where TLS Termination Ends

Where the encrypted connection ends is a **placement decision**, and the result measured here is
not a duration but a visibility. The balancer above reads `outerReq.url` to tell apart the
`/manage/status` path — that line exists only because the connection terminates at the balancer.
If the encrypted stream instead runs all the way to the replica, the balancer has only bytes in
hand, not a path, headers, or a body. TLS and the handshake were established in the Network
Models and Protocols course and are not retold here; this decision's result for this lesson is:
**termination is the precondition for a content-aware distribution rule.**

The decision's second result is countable. The number of endpoints holding the certificate is 1
when termination happens at the balancer, and equal to the replica count when it happens at the
replicas; adding a replica costs 0 certificate work in the first case, 1 in the second. The
processor cost of encryption is not measured here — it depends on the cipher suite and hardware,
and this course writes down no unmeasured number.

## Back to the Calculation

The Introduction to System Design course calculated the peak request rate at the edge as 513.89
req/s. That number is not the load a single replica carries; it is divided by the replica count.
The division needs this topic's own assumptions, **not added** to K01's V1–V13 table:

**Y1 — saturation throughput per replica: 400 req/s.** Rationale: the upper bound measured on a
bare endpoint in K01's Latency and Throughput lesson was 4268 req/s; the real tracking endpoint
looks up the store, serializes, and authorizes, so the planning number sits well below the
measured one. **Y2 — target utilization 0.50.** Rationale: the same lesson's queuing model showed
utilization at 50 percent multiplying duration by 1.4, and at 90 percent by 7. **Y3 — health
check interval 200 ms, threshold 1 round.** Rationale: the values used in the setup.

```js
// balance/capacity.mjs — K01's calculation divided by replica count, and the health check's effect on the budget
const PEAK_EDGE = 513.89;          // K01 Back-of-the-Envelope Estimation: peak edge req/s
const DAILY_REQUESTS = 12_000_000 + 2_800_000;   // K01: daily tracking query + status event
const SATURATION = 400;            // assumption Y1: saturation throughput per replica (req/s)
const TARGET_UTILIZATION = 0.5;    // assumption Y2: replica's target utilization
const CHECK_MS = 200;              // assumption Y3: health check interval, threshold 1 round
const RECOVERY_MIN = 10;           // K01 Availability in Numbers: recovery time
const FAILURES_MONTH = 2.82;       // K01: failures per month that fit the budget

const safeRate = SATURATION * TARGET_UTILIZATION;
const minReplicas = Math.ceil(PEAK_EDGE / safeRate);
console.log(`safe rate per replica = ${safeRate} req/s (${SATURATION} x ${TARGET_UTILIZATION})`);
console.log(`min replicas = ceil(${PEAK_EDGE} / ${safeRate}) = ${minReplicas}\n`);

console.log("replicas  req/s per replica  utilization  req/s if one drops  utilization then");
for (const n of [minReplicas, minReplicas + 1, minReplicas + 2]) {
  const share = PEAK_EDGE / n, afterDrop = PEAK_EDGE / (n - 1);
  console.log(`${String(n).padStart(8)}  ${share.toFixed(2).padStart(17)}  ` +
    `${(share / SATURATION).toFixed(3).padStart(11)}  ${afterDrop.toFixed(2).padStart(18)}  ` +
    `${(afterDrop / SATURATION).toFixed(3).padStart(16)}`);
}

const average = DAILY_REQUESTS / 86_400;
console.log(`\naverage edge rate = ${average.toFixed(2)} req/s; peak / ${minReplicas} replicas = ` +
  `${(PEAK_EDGE / minReplicas).toFixed(2)} req/s (same number, since the peak multiplier is ${minReplicas})`);

const monthlyRequests = DAILY_REQUESTS * 30;
console.log(`\nwindow             dropped req/failure   monthly dropped req   request based availability`);
for (const [name, sec] of [["health check", CHECK_MS / 1000], ["manual recovery", RECOVERY_MIN * 60]]) {
  const perFailure = (PEAK_EDGE / minReplicas) * sec, monthly = perFailure * FAILURES_MONTH;
  console.log(`${name.padEnd(18)} ${perFailure.toFixed(2).padStart(17)} ${monthly.toFixed(0).padStart(19)}   ` +
    `${(100 * (1 - monthly / monthlyRequests)).toFixed(5)}%`);
}

const checkLoad = (minReplicas * 1000) / CHECK_MS;
console.log(`\ncheck load = ${checkLoad} req/s, ${((checkLoad / PEAK_EDGE) * 100).toFixed(1)}% of the peak edge rate`);
console.log(`sensitivity: if Y1 doubles, min replicas = ${Math.ceil(PEAK_EDGE / (2 * SATURATION * TARGET_UTILIZATION))}, ` +
  `if halved = ${Math.ceil(PEAK_EDGE / (0.5 * SATURATION * TARGET_UTILIZATION))}`);
```

```
safe rate per replica = 200 req/s (400 x 0.5)
min replicas = ceil(513.89 / 200) = 3

replicas  req/s per replica  utilization  req/s if one drops  utilization then
       3             171.30        0.428              256.94             0.642
       4             128.47        0.321              171.30             0.428
       5             102.78        0.257              128.47             0.321

average edge rate = 171.30 req/s; peak / 3 replicas = 171.30 req/s (same number, since the peak multiplier is 3)

window             dropped req/failure   monthly dropped req   request based availability
health check                   34.26                  97   99.99998%
manual recovery            102778.00              289834   99.93472%

check load = 15 req/s, 2.9% of the peak edge rate
sensitivity: if Y1 doubles, min replicas = 2, if halved = 6
```

All these numbers are in the **computed** class. The minimum replica count is three, each
carrying 171.30 req/s. The second table's first column shows what happens the **moment** a
replica drops: the remaining two rise to 256.94 req/s and 0.642 utilization — a three-replica
design overshoots the target on a single failure. With four replicas, utilization in the same
situation stays at 0.428. So "minimum replica count" and "replica count that still holds the
target if one drops" are two separate numbers, one replica apart.

The average row records a coincidence: the peak divided by three replicas is 171.30 req/s, and
the system's average edge rate is also 171.30 req/s. The two numbers match because the peak
multiplier is 3 and the replica count was also chosen as 3; this is not a relationship, it is a
coincidence the assumptions produce. When the replica count changes, the equality breaks.

The last table is health checking's counterpart in the outage budget. K01's Availability in
Numbers lesson set aside 28.2 of the monthly 43.2-minute budget for failures, counting 2.82
failures a month at a ten-minute recovery time — assuming every failure was a **total outage**.
In a pool of three replicas, one dropping is not: with health checking, 34.26 requests drop per
failure, 97 a month. Without it, the failure lasts until recovery: 102,778 requests per failure,
289,834 a month — 2989 times as many. Checking's own bill: 15 req/s to the replicas, 2.9 percent
of the peak edge load.

The request-based column carries a warning. The unchecked setup's ratio, 99.93472 percent, sits
above the 99.9 percent target, yet the same failures spend the entire failure share of the
time-based budget. The same failure looks like two different magnitudes on the two bases, since
only a third of requests drop. K01 kept these bases separate; the numbers here show why.

## Summary

- The load balancer carries three responsibilities: distribution, health checking, and TLS
  termination — the first shares the load, the second drops a failed replica from the pool, the
  third decides where the encrypted connection ends.
- In the same failure, health checking on answered 300/300 requests and split the load 150–150;
  off, it answered 200/300 and 100 requests came back with 502.
- Distribution alone does not produce service availability: without a check that notices the
  drop, a third of requests keep failing without looking like a mass outage.
- TLS termination's measured result here is visibility: without it ending at the balancer, the
  path and headers cannot be read, so no content-aware rule can be built.
- The 513.89 req/s peak, with a 200 req/s safe rate per replica (Y1, Y2), needs a minimum of 3
  replicas at 171.30 req/s each; losing one pushes the rest to 256.94 req/s and 0.642
  utilization — over target, unlike with four replicas.
- Health checking cuts dropped requests per failure from 102,778 to 34.26 (289,834 monthly versus
  97, 2989 times); its own load is 15 req/s, 2.9 percent of the peak edge.

## Next Step

This lesson's balancer decided on every request, reading the request's path to do it — the
`/manage/status` distinction depended on that. Reading is a choice, not a requirement. A balancer
can work without looking into the request: deciding once, at connection setup, then passing
everything on that connection to the same replica. Less work, less knowledge. The next lesson
builds this level: what connection-level distribution cannot know, how persistent connections
break it, and what skew a once-per-connection decision produces in the per-request split.