Web Server Roles

The previous course made the system visible: measurement points were placed, logs were collected in one place, alerts reached an on-call person, and who closed an incident and how was recorded. Once visibility was established, what could be seen changed too — dashboards show the application’s behavior, but say nothing about the machine the application runs on.

That machine was set up one day and still stands with whatever it was given that day. The package was unpacked, the process was started, requests began flowing, and once it worked no one looked again. This lesson turns that look into a number: when nothing is done after installation, how many items stay open.

The unit of measure is the open surface item. An item is a single thing reachable from outside or inside: a listening port, a served path, a piece of information visible in a response, a management endpoint that asks for no identity. The item count is not a security note; it is a count, and no hardening step can be justified without that count.

The Three Roles Separately

A web server does one of three jobs, or all three at once.

Static serving answers a request with a file from disk. The input is a path, the output is a file; there is no application code in between. The reverse proxy takes the request, produces nothing itself, and forwards it to an upstream behind it; it carries the response back. Load distribution splits the same work across multiple upstream members; it tracks which member is up and drops the ones that fall out of the pool. What each of the three roles is was already established; this lesson does not redefine the roles, it counts the items they leave behind after setup.

The count is done on a fictional system: a regional water-meter measurement network. Field crews read meters with handheld terminals, a collector takes in the readings, a verifier checks them, billing and work-order services take over. There are three environments and a nightly batch job runs. Both the system and the numbers are fictional; what is measured is the ratio between the numbers.

• WS1. The item list is not read from a product’s default configuration file; it is a fictional list of the things typically left open after installation. What is measured is not the list but the ratio that comes out of it.
• WS2. If an item is “external,” it is reachable from the public port; if “internal,” only from the internal network. In separate processes, this distinction is preserved by the process boundary.
• WS3. A legitimate workflow represents a real piece of work that stops when an item is closed; there are six workflows and each one leans on a single item.
• WS4. Items with the same sharing key collapse into a single item when merged: the plaintext port all three roles listen on is one port, the version header all three stamp is one header.

// roles/surface.mjs — model: the surface left open after the default installation of three web
// server roles. Not a real product's default file, but the fictional measurement network's
// post-installation item list; every number is derived from this list.

// [role, item, type, access, sharing key]
const ITEM = [
  ["static",       "plaintext-port",       "listening port", "external", "port-80"],
  ["static",       "asset-root",           "served path",    "external", null],
  ["static",       "directory-listing",    "served path",    "external", null],
  ["static",       "backup-files",         "served path",    "external", null],
  ["static",       "sample-page",          "served path",    "external", null],
  ["static",       "version-header",       "visible info",   "external", "version"],

  ["proxy",        "plaintext-port",       "listening port", "external", "port-80"],
  ["proxy",        "encrypted-port",       "listening port", "external", null],
  ["proxy",        "upstream-port",        "listening port", "internal", null],
  ["proxy",        "version-header",       "visible info",   "external", "version"],
  ["proxy",        "error-page-trace",     "visible info",   "external", null],
  ["proxy",        "proxy-status-endpoint","management endpoint", "internal", null],

  ["distribution", "plaintext-port",       "listening port", "external", "port-80"],
  ["distribution", "version-header",       "visible info",   "external", "version"],
  ["distribution", "member-address-header","visible info",   "external", null],
  ["distribution", "session-cookie-member","visible info",   "external", null],
  ["distribution", "health-endpoint",      "management endpoint", "internal", null],
  ["distribution", "member-mgmt-endpoint", "management endpoint", "internal", null],
];

// Items born of merging into a single process, not found alone in any single role.
const MERGE = [
  ["merged", "cross-root",             "served path",           "external"],
  ["merged", "shared-mgmt-port",       "management endpoint",   "external"],
  ["merged", "single-process-identity","combined authority",    "external"],
];

// Legitimate workflows: which item each workflow leans on.
const FLOW = [
  ["handheld-terminal-read", "static/plaintext-port"],
  ["field-asset-serving",    "static/asset-root"],
  ["collector-upload",       "proxy/encrypted-port"],
  ["on-call-health-call",    "distribution/health-endpoint"],
  ["rollout-flow",           "distribution/member-mgmt-endpoint"],
  ["session-stickiness",     "distribution/session-cookie-member"],
];

const id = (r, k) => `${r}/${k}`;
const roles = ["static", "proxy", "distribution"];

const W = [14, 7, 11, 9, 9, 9, 6, 5];
const row = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](W[i])).join("");
const measure = (s) => [s.length, ...["listening port", "served path", "visible info", "management endpoint"]
  .map((t) => s.filter((x) => x[2] === t).length),
  s.filter((x) => x[3] === "external").length, s.filter((x) => x[3] === "internal").length];

console.log(row(["role", "items", "listening", "served", "visible", "mgmt", "ext", "int"]));
for (const role of roles) console.log(row([role, ...measure(ITEM.filter((x) => x[0] === role))]));
console.log(row(["total", ...measure(ITEM)]));

// Merging: items with the same sharing key collapse into one item.
const seen = new Set();
const unique = [];
for (const [role, item, type, access, share] of ITEM) {
  const key = share ?? id(role, item);
  if (seen.has(key)) continue;
  seen.add(key);
  unique.push([role, item, type, access, share]);
}
const merged = [...unique.map((x) => [...x]), ...MERGE.map((x) => [...x, null])];
console.log(`\nthree separate processes: ${ITEM.length} items -> ${unique.length} unique` +
  `  (ext ${unique.filter((x) => x[3] === "external").length}, int ${unique.filter((x) => x[3] === "internal").length})`);
// A single process runs on a single listener: even an "internal"-tagged item is now reachable from the public port.
for (const k of merged) k[3] = "external";
console.log(`single process:  ${unique.length} + ${MERGE.length} new = ${merged.length} items` +
  `  (ext ${merged.length}, int 0)`);

// Hardening steps: each step is a predicate, the items it closes, and the new items it introduces.
const STEP = [
  ["close visible info", (x) => x[2] === "visible info", []],
  ["narrow served paths", (x) => x[2] === "served path" && x[1] !== "asset-root",
    [["allowlist-file", "maintenance surface"]]],
  ["move mgmt endpoints off the public port", (x) => x[2] === "management endpoint",
    [["separate-mgmt-listener", "listening port"], ["mgmt-identity", "secret"]]],
  ["close the plaintext port", (x) => x[1] === "plaintext-port", []],
];

let open = merged.map((x) => [...x]);
const added = [];
console.log("\nstep                                        closed  wrongly closed  new surface  still open");
for (const [name, pred, introduces] of STEP) {
  const dropped = open.filter(pred);
  open = open.filter((x) => pred(x) === false);
  const broken = FLOW.filter(([, k]) => dropped.some((d) => id(d[0], d[1]) === k));
  added.push(...introduces);
  console.log(name.padEnd(42) + String(dropped.length).padStart(8) +
    String(broken.length).padStart(16) + String(introduces.length).padStart(13) +
    String(open.length + added.length).padStart(12) +
    (broken.length ? `   ${broken.map((b) => b[0]).join(", ")}` : ""));
}
console.log(`\nitems that cannot be closed (${open.length}): ${open.map((x) => x[1]).join(", ")}`);
console.log(`items closing introduces (${added.length}): ${added.map((x) => x[0]).join(", ")}`);

role            items  listening   served  visible     mgmt   ext  int
static              6          1        4        1        0     6    0
proxy               6          3        0        2        1     4    2
distribution        6          1        0        3        2     4    2
total              18          5        4        6        3    14    4

three separate processes: 18 items -> 14 unique  (ext 10, int 4)
single process:  14 + 3 new = 17 items  (ext 17, int 0)

step                                        closed  wrongly closed  new surface  still open
close visible info                               4               1            0          13   session-stickiness
narrow served paths                              4               0            1          10
move mgmt endpoints off the public port          4               2            2           8   on-call-health-call, rollout-flow
close the plaintext port                         1               1            0           7   handheld-terminal-read

items that cannot be closed (4): asset-root, encrypted-port, upstream-port, single-process-identity
items closing introduces (3): allowlist-file, separate-mgmt-listener, mgmt-identity

Counting is deterministic: the list is fixed, there is no randomness, no duration is measured.

The Role’s Own Surface

The first table lines up the three roles side by side. All three have six items, but the items’ type differs.

Four of static serving’s six items are served paths, and all four depend on a single setting: the root directory. Whatever is in the root directory gets served. The asset root is legitimate — the field crews’ browser is fed from there. The directory listing, the sample page that ships with the installation, and a configuration backup someone once dropped inside the root are things nobody decided on. All six of the static role’s items are externally reachable: this role has no internal surface, because it has no management endpoint of its own.

Three of the reverse proxy’s six items are listening ports, and one detail matters here: the third port is the upstream’s own port. When the proxy is set up, the application behind it starts listening on a port, and in most setups that port binds to all of the machine’s interfaces. The proxy does not open a port; the proxy’s existence opens a second port behind it, and that port can be reached without ever passing through the proxy. None of the proxy’s settings ever see a request that arrives directly at the port behind it.

Three of load distribution’s six items are visible info, and all three carry the pool’s internal structure outward: the member address in the response header, the member identity in the session cookie, the member list the health endpoint dumps. This role is set up to make internal structure visible — knowing which member answered is necessary when troubleshooting — and that same visibility is exposed outward as well.

The three roles total eighteen items: five listening ports, four served paths, six visible info items, three management endpoints. Fourteen are external, four are internal.

The asymmetry in the distribution says something. In the static role the surface sits in the file system: closing an item means deciding what goes in a directory. In the proxy role the surface sits in the network topology: closing an item means deciding which interface a port binds to. In the distribution role the surface sits in the response itself: closing an item means stripping a header or a cookie. This is why a single hardening procedure cannot be written for all three roles; even though the count is equal, the closing work happens in three separate places.

What Merging Does

The second measurement puts the three roles into a single process. This is the arrangement most commonly seen on one machine: the same process serves static files, forwards requests to the back end, and splits traffic across members.

Merging first produces a reduction. The plaintext port all three roles listen on collapses into one port, the version header all three stamp collapses into one header. Eighteen items drop to fourteen unique items. Someone looking only at the item count could stop here and say merging narrows the surface.

Two things break that reading.

First, merging spawns its own items. The static root is now also reachable through the proxy’s public port; in separate processes there was a boundary between these two paths, in a single process there is not. The three roles’ management endpoints collect on a single listener. The single process carries all three roles’ permissions at once: the same identity reads the static root, connects to the upstream, and changes pool membership. Three new items carry fourteen up to seventeen.

Second, and heavier, the access label falls away. In three separate processes, ten of the fourteen items were externally reachable; four — two management endpoints, the health endpoint, and the upstream port — sat behind the process boundary. A single process runs on a single listener, and that listener is the public port. All seventeen of the seventeen items become externally reachable.

This is the measure of merging: while the item count drops from eighteen to seventeen, the externally reachable item count climbs from ten to seventeen. Counted by items, merging looks good; counted by access, the surface has grown by 70%.

Four Steps and What They Stop

The third table applies four hardening steps to the seventeen-item surface. Each step produces two numbers: how many items closed and how many legitimate workflows stopped.

The first step closes visible info: the version header, the error-page trace, the member address header, and the member cookie. Four items close and one workflow stops — session stickiness. The field crew’s multi-step reading form relied on later steps landing on whichever member answered the first step, and it read that from the member cookie. Once the cookie is gone, the form falls onto a different member on the second step. This step gets defended under the heading “close the leaking info,” and the defense is incomplete: the fourth item it closed was not an information leak, it was the carrying mechanism.

The second step narrows the served paths: the directory listing, the backup files, the sample page, and the cross-root that merging spawned. Four items close, no workflow stops. The cost is elsewhere: closing is done with an allowlist file, and that file is now a place that needs watching. When a new asset directory is added and not added to the list, it is not served; this is the source of a future failure.

The third step takes the management endpoints off the public port. Four items close and two workflows stop: the on-call health call and the rollout flow’s member-add call. Both are legitimate and both went through those endpoints. For the step to be completable it introduces two new items: a separate management listener reachable only from the internal network, and a management identity that enters that listener. The second item is a secret, and secrets have their own lifetime, their own exposure window.

The fourth step closes the plaintext port. One item closes, one workflow stops: the old handheld terminals in the field only speak plaintext. This is the smallest closure and the most visible break — the day it stops, the entire field crew calls.

The four steps closed thirteen items, stopped four legitimate workflows, and introduced three new items. The ratio is thirteen to four: for every three items closed, roughly one workflow stopped. That ratio is nowhere near even across the steps — the second step closes four items with zero stoppage, while the fourth step closes a single item with one stoppage. This is what decides the order: cheap step first, expensive step last. An ordering that looked only at the surface-item count would not have put the fourth step last, because that step closes the most talked-about item and gives the smallest number in return.

The Four That Cannot Be Closed

At the end of the four steps, four items remain, and all four are items that cannot be closed.

The asset root and the encrypted port are the machine’s reason for existing; closing them is shutting down the system. The upstream port cannot be closed because the proxy’s connection to the back end passes through it; it can be narrowed — bound only to the loopback interface — but it stays open. The single-process identity closes only by undoing the merge, that is, by splitting the three roles back into three processes; no configuration line closes it.

The final table shows seven open items: four that cannot be closed and three new ones. Compared with the starting seventeen, this is a gain. In a report that does not count the new surface, the gain would be written as dropping from seventeen items to four; counted, it is dropping to seven.

Summary

• The default installation of three web server roles left eighteen items open: five listening ports, four served paths, six visible info items, three management endpoints; fourteen externally reachable, four internally.
• The roles’ item counts were equal, their types were not: four of static serving’s items were served paths, three of the reverse proxy’s were listening ports, three of load distribution’s were visible info.
• Merging into a single process reduced eighteen items to fourteen unique ones, spawned three items, and carried four internally reachable items out into the open: the externally reachable item count climbed from ten to seventeen.
• Four hardening steps closed thirteen items, stopped four legitimate workflows, and introduced three new items; the step that closed the management endpoints alone stopped two workflows and added two new items.
• Four items that cannot be closed remained; one of them, the single-process identity, closes not through any setting but only by splitting the roles back into separate processes.

Next Step

This lesson counted the surface from the outside: how many ports are listening, how many paths are served, how many endpoints are open. The whole count was about the request arriving at the server. What happens after the request arrives was never examined.

When the reverse proxy takes a request and forwards it, the request does not pass through unchanged. The connection ends at the proxy, and a new connection opens to the back end; at the other end of that new connection stands the proxy, not the client. The client’s address, whether the request arrived encrypted, which name the client asked for, and the request’s original path — all four traveled in the connection itself, and that connection ended. The next lesson sets up a two-hop proxy chain with node:http, passes real requests through it, counts one by one the four pieces of information lost while passing through the chain, shows the wrong decision each loss produces in the application, and measures the surface opened by trusting the header added to close the loss.