Load Balancer and Reverse Proxy

The previous three lessons built the balancer as a routing device: it takes a request, applies a rule, forwards it to a process, and writes the response back. The same definition was given for another component in the Server-Side Fundamentals course. A reverse proxy also takes a request, forwards it to one of several backend servers by a rule, and writes the response back. The two definitions do not come apart.

This lesson starts by accepting the overlap and looks for the distinction somewhere else. Neither is redefined; what is measured is what changes when the two roles are gathered into the same process versus split across separate processes.

Same Program, Two Rule Tables

The shortest proof of the overlap is a single program. The relay below takes a rule table: a path prefix, a target list. If the list has one element, the work done is routing — which component. If the list holds several copies of the same component, the work done is distribution — which copy. The two can sit side by side in the same table.

// proxy/relay.mjs — a single relay program. The rule table describes both roles:
//   "/assets=8891;*=8850"            -> routing to separate components by path (reverse proxy)
//   "*=8892,8893,8894"               -> distributing across copies of the same component (load balancer)
//   "/assets=8891;*=8892,8893,8894"  -> both at once (a combined process)
// Usage: node proxy/relay.mjs <port> <name> <rule>
import { createServer, request } from "node:http";

const [port, name, ruleText] = [Number(process.argv[2]), process.argv[3], process.argv[4] ?? ""];
const RULES = ruleText.split(";").filter(Boolean)
  .map((p) => { const [prefix, list] = p.split("="); return { prefix, targets: list.split(",").map(Number), turn: 0 }; });

if (Number.isInteger(port) === false || RULES.length === 0) {
  console.log("usage: node proxy/relay.mjs <port> <name> <prefix=target[,target];...>");
} else createServer((clientReq, clientRes) => {
  clientRes.sendDate = false;
  const path = new URL(clientReq.url, "http://local").pathname;
  const rule = RULES.find((r) => r.prefix !== "*" && path.startsWith(r.prefix)) ?? RULES.find((r) => r.prefix === "*");
  if (rule === undefined) { clientRes.writeHead(404).end("no rule"); return; }
  const target = rule.targets[rule.turn++ % rule.targets.length];
  const previous = clientReq.headers["x-hop"];
  const backendReq = request({ port: target, path: clientReq.url, method: clientReq.method,
    headers: { ...clientReq.headers, "x-hop": previous === undefined ? name : `${previous} > ${name}` } },
  (backendRes) => { clientRes.writeHead(backendRes.statusCode, backendRes.headers); backendRes.pipe(clientRes); });
  backendReq.on("error", () => clientRes.writeHead(502, { "x-hop": `${name} > unreachable` }).end("none"));
  clientReq.pipe(backendReq);
}).listen(port, "127.0.0.1");

The processes at the end of the chain only report their names; the x-hop header accumulates through the request and carries the hops the request touched.

// proxy/endpoint.mjs — the process at the end of the chain: adds its name to the x-hop header
import { createServer } from "node:http";

const [port, name] = [Number(process.argv[2]), process.argv[3]];

if (Number.isInteger(port) === false) console.log("usage: node proxy/endpoint.mjs <port> <name>");
else createServer((req, res) => {
  res.sendDate = false;
  const previous = req.headers["x-hop"];
  res.writeHead(200, { "content-type": "application/json",
    "x-hop": previous === undefined ? name : `${previous} > ${name}` });
  res.end(JSON.stringify({ state: "at transit hub", zone: "35" }));
}).listen(port, "127.0.0.1");

// proxy/client.mjs — tries both paths, prints the touched hop chain and the answered request count
import { request } from "node:http";

const [port, label] = [Number(process.argv[2]), process.argv[3] ?? "-"];
const PATHS = ["/assets/style.css", "/tracking"];
const COUNT = 6;

function one(path) {
  return new Promise((resolve) => {
    const r = request({ port, path }, (res) => {
      res.resume();
      res.on("end", () => resolve({ code: res.statusCode, hop: res.headers["x-hop"] ?? "-" }));
    });
    r.on("error", () => resolve({ code: 0, hop: "no connection" }));
    r.end();
  });
}

if (Number.isInteger(port) === false) console.log("usage: node proxy/client.mjs <port> <label>");
else for (const path of PATHS) {
  const chain = new Map();
  let answered = 0;
  for (let i = 0; i < COUNT; i += 1) {
    const s = await one(path);
    if (s.code === 200) answered += 1;
    chain.set(s.hop, (chain.get(s.hop) ?? 0) + 1);
  }
  const d = [...chain].sort().map(([a, n]) => `${a} (${n})`).join("  |  ");
  console.log(`${label.padEnd(22)} ${path.padEnd(17)} ${String(answered)}/${COUNT}  ${d}`);
}

The setup runs the same job under two layouts. In the separate layout there are two relay processes: rp splits the content path from the application path, lb distributes across the copies. In the combined layout, a single rplb process does both. In each layout, the process holding the replica list is killed and which path stays up is counted.

# measure.sh — the same job under two layouts: two separate relay processes, then one combined process.
# In the second stage, the process holding the replica list is killed and which path stays up is counted.
node proxy/endpoint.mjs 8891 static & S=$!
node proxy/endpoint.mjs 8892 k1 & K1=$!
node proxy/endpoint.mjs 8893 k2 & K2=$!
node proxy/endpoint.mjs 8894 k3 & K3=$!

node proxy/relay.mjs 8860 rp "/assets=8891;*=8850" & RP=$!
node proxy/relay.mjs 8850 lb "*=8892,8893,8894" & LB=$!
sleep 1
node proxy/client.mjs 8860 "separate: rp + lb"
kill $LB; sleep 0.5
node proxy/client.mjs 8860 "separate: lb killed"
kill $RP; sleep 0.3

node proxy/relay.mjs 8861 rplb "/assets=8891;*=8892,8893,8894" & COMBINED=$!
sleep 1
node proxy/client.mjs 8861 "combined: rplb"
kill $COMBINED; sleep 0.5
node proxy/client.mjs 8861 "combined: killed"

kill $S $K1 $K2 $K3

separate: rp + lb      /assets/style.css 6/6  rp > static (6)
separate: rp + lb      /tracking         6/6  rp > lb > k1 (2)  |  rp > lb > k2 (2)  |  rp > lb > k3 (2)
separate: lb killed    /assets/style.css 6/6  rp > static (6)
separate: lb killed    /tracking         0/6  rp > unreachable (6)
combined: rplb         /assets/style.css 6/6  rplb > static (6)
combined: rplb         /tracking         6/6  rplb > k1 (2)  |  rplb > k2 (2)  |  rplb > k3 (2)
combined: killed       /assets/style.css 0/6  no connection (6)
combined: killed       /tracking         0/6  no connection (6)

Two Numbers, Two Directions

These numbers are in the measurement class and are deterministic. The two layouts rank in opposite order on two separate columns.

Hop count. In the separate layout, a tracking request touches three hops: rp > lb > k1. In the combined layout, two hops: rplb > k1. The content path is two hops in both. Every hop means a network jump, a parse, and a rewrite; the separate layout puts the same request through one extra pair of hands.

Isolation. When the process holding the replica list is killed, the content path in the separate layout keeps answering 6/6, and only the tracking path drops to 0/6 — and even that drop comes back through rp as a 502, meaning the client gets a response. In the combined layout both drop to 0/6 and the client gets no response at all, because there is no port left to connect to. A change to the replica list is a configuration change and a common one; in the combined process, that change also stops the content path.

The two numbers are two faces of the same decision: splitting adds a hop, merging adds a failure surface.

Where the Distinction Lies

Since the same program can take on both roles, the distinction is not in the program. It is in two other places.

Responsibility. The reverse proxy answers “which component”: a path starting with /assets goes to the content process, everything else to the application. The prefixes in the rule table point to components doing different jobs, and their target lists have one element. The load balancer answers “which copy”: its target list is made of processes doing the same job, and which one gets picked does not change the outcome of the request. In the first question, a wrong choice is wrong behavior; in the second, it is only a different copy.

Location. The reverse proxy sits at the application boundary: it presents the system’s outer face as a single path space and hides the component split behind it. The load balancer sits at the replica boundary: it hides how many copies a component runs on. The two forms of hiding operate at different scales, which is why the two can sit at different points in the chain — the rp > lb order in the measurement is an expression of that; the reverse order would not make sense.

A single process taking on both does not remove this distinction, it only writes both into the same configuration file. What keeps the distinction visible is the shape of the rule table: which row’s prefix is a component split, and which row’s target list is a set of copies.

Back to the Computation

The bill for splitting showed up in the hop count; that number’s counterpart in service availability was already established in C01. The Availability in Numbers lesson took the tracking path as a four-component series chain and, at 99.9 percent per component, found a composite availability of 99.6006 percent and a monthly outage of 172.5 minutes. The relay processes are new components added to that chain.

// proxy/chain.mjs — the cost of adding a hop in the C01 composite availability calculation
const MONTH = 30 * 24 * 60;        // minutes
const BASE = 4;                    // C01 Availability in Numbers: four-component tracking path
const COMPONENT = 0.999;           // C01 assumption: service availability per component

const series = (a, n) => a ** n;
const replicated = (a, k) => 1 - (1 - a) ** k;
const minutes = (a) => (1 - a) * MONTH;

console.log("layout                   hops  chain components  composite availability  monthly outage  vs base");
const base = series(COMPONENT, BASE);
for (const [name, hops, replicas] of [
  ["no hops (C01)", 0, 1], ["combined process", 1, 1], ["two separate processes", 2, 1],
  ["separate, 2 replicas each", 2, 2], ["combined, 2 replicas", 1, 2]]) {
  const a = base * series(replicated(COMPONENT, replicas), hops);
  console.log(`${name.padEnd(26)}${String(hops).padStart(6)}${String(BASE + hops).padStart(17)}  ` +
    `${(a * 100).toFixed(4).padStart(22)}%  ${minutes(a).toFixed(1).padStart(11)} min  ` +
    `${(minutes(a) - minutes(base)).toFixed(1).padStart(9)} min`);
}

const combined = base * series(COMPONENT, 1), separate = base * series(COMPONENT, 2);
console.log(`\nbill for splitting the roles into two processes = ${(minutes(separate) - minutes(combined)).toFixed(1)} min/month`);
console.log(`same bill when the hops have 2 replicas each = ` +
  `${(minutes(base * series(replicated(COMPONENT, 2), 2)) - minutes(base * series(replicated(COMPONENT, 2), 1))).toFixed(2)} min/month`);
console.log(`C01 monthly outage budget (99.9%) = ${((1 - 0.999) * MONTH).toFixed(1)} min; ` +
  `separate layout's outage is ${(minutes(separate) / ((1 - 0.999) * MONTH)).toFixed(2)}x the budget`);

layout                   hops  chain components  composite availability  monthly outage  vs base
no hops (C01)                  0                4                 99.6006%        172.5 min        0.0 min
combined process               1                5                 99.5010%        215.6 min       43.0 min
two separate processes         2                6                 99.4015%        258.6 min       86.0 min
separate, 2 replicas each      2                6                 99.6004%        172.6 min        0.1 min
combined, 2 replicas           1                5                 99.6005%        172.6 min        0.0 min

bill for splitting the roles into two processes = 43.0 min/month
same bill when the hops have 2 replicas each = 0.04 min/month
C01 monthly outage budget (99.9%) = 43.2 min; separate layout's outage is 5.99x the budget

These numbers are in the computation class. The first three rows give the bill for splitting: with single-copy hops, separating the roles into two processes raises the monthly outage from 215.6 minutes to 258.6 minutes — 43.0 minutes a month. That number is nearly equal to the entire monthly budget C01 set aside for the 99.9 percent target (43.2 minutes). So the decision to separate the roles, by itself, costs about a month’s outage budget.

The fourth and fifth rows show where the bill comes from. When the hops have two replicas each, the same split drops to 0.04 minutes and the two layouts become indistinguishable. The rule from C01’s replica table holds here too: outage share is raised to a power across parallel replicas. The bill for adding a hop does not come from adding the hop — it comes from leaving that hop single-copy.

The last row closes the table with one measure. The separate layout’s monthly outage is 258.6 minutes, 5.99 times the 99.9 percent budget. This does not say the layout is bad; it says a 99.9 percent target cannot be met with 99.9 percent per component — C01 found the same result with four components, and adding two hops sharpens it. When an edge layer is being designed, hop count is not an architectural preference; it is a direct availability line item.

Summary

• The load balancer and the reverse proxy can be written as the same program: in the rule table, a single-element target list means routing, and a list of copies of the same component means distribution.
• In the separate layout the tracking request touched three hops (rp > lb > k1); in the combined layout, two (rplb > k1); splitting puts the request through one extra pair of hands.
• When the process holding the replica list was killed, the content path in the separate layout kept answering 6/6; in the combined layout both paths dropped to 0/6 and the client got no response at all.
• The distinction is responsibility and location: the reverse proxy answers “which component” at the application boundary, the load balancer answers “which copy” at the replica boundary; a wrong choice is wrong behavior in the first case and only a different copy in the second.
• Separating the roles with single-copy hops raises monthly outage from 215.6 to 258.6 minutes: 43.0 minutes a month, the entire monthly budget for the 99.9 percent target.
• With two replicas per hop the same bill drops to 0.04 minutes; the cost of adding a hop comes from leaving it single-copy.

Next Step

The five lessons so far never questioned one thing: the balancer’s choice is free. Round robin, least connections, and both layouts all assumed a request could go to any copy; even consistent hashing, while it wanted the choice to stay stable, did not make it mandatory. There is a case where the replicas stop being each other’s equals: if state left over from a client’s previous request sits in a particular replica’s memory, that client’s next request can no longer go to just any replica. Choice stops being a preference and becomes a constraint. The next lesson names and measures that constraint: it counts how much distribution breaks down, how much state is lost when a replica drops, and why adding a replica does not lower the peak load.