---
title: 'Servers, Networking and Hardening'
source: 'https://academia.sh/en/courses/servers-and-hardening'
language: en
updated: '2026-09-13T05:08:29+00:00'
license: 'CC BY-SA 4.0'
lessons: 16
outcomes:
    - 'Writes the reverse proxy and load balancer configuration'
    - 'Manages certificates and domain records automatically'
    - 'Hardens the server and access layer'
---

# Servers, Networking and Hardening

Counting and closing the surface left open after installation: the static-serving, proxy, and load-distribution roles of a web server, upstream definition and header forwarding in a reverse proxy, TLS termination and certificate automation, edge and origin caching, traffic management with sidecar proxies, the zone file and record time to live, sender verification and policy records, mail delivery problems, a default-deny firewall, key-based remote access and the bastion host, a central secret store and dynamic credentials, the patch window, backup scope, and the restore drill.

## Web Servers and Proxies

The layer a request passes through before it reaches the application: separating the static-serving, proxy, and load-distribution roles, upstream definition in a reverse proxy and the information header forwarding loses when it is done wrong, TLS termination and protocol selection, automatic certificate renewal and the window a failed renewal opens, the hit-and-freshness balance between the edge cache and the origin cache, and the cost of managing traffic with sidecar proxies.

### Web Server Roles

Source: https://academia.sh/en/courses/servers-and-hardening/web-server-roles

The surface the default installation of the static-serving, reverse-proxy, and load-distribution roles leaves open: a role-by-role count of eighteen items, merging the three roles into a single process raising fourteen unique items to seventeen and carrying the externally reachable item count from ten to seventeen, four hardening steps closing thirteen items against the four legitimate workflows they stop, and the four items left over that cannot be closed.

The previous course made the system visible: measurement points were placed, logs were collected
in one place, alerts reached an on-call person, and who closed an incident and how was recorded.
Once visibility was established, what could be seen changed too — dashboards show the
application's behavior, but say nothing about the **machine the application runs on**.

That machine was set up one day and still stands with whatever it was given that day. The package
was unpacked, the process was started, requests began flowing, and once it worked no one looked
again. This lesson turns that look into a number: when nothing is done after installation,
**how many items stay open**.

The unit of measure is the **open surface item**. An item is a single thing reachable from
outside or inside: a listening port, a served path, a piece of information visible in a response,
a management endpoint that asks for no identity. The item count is not a security note; it is a
count, and no hardening step can be justified without that count.

#### The Three Roles Separately

A web server does one of three jobs, or all three at once.

**Static serving** answers a request with a file from disk. The input is a path, the output is a
file; there is no application code in between. **The reverse proxy** takes the request, produces
nothing itself, and forwards it to an upstream behind it; it carries the response back. **Load
distribution** splits the same work across multiple upstream members; it tracks which member is up
and drops the ones that fall out of the pool. What each of the three roles is was already
established; this lesson does not redefine the roles, it counts **the items they leave behind
after setup**.

The count is done on a fictional system: a regional water-meter measurement network. Field crews
read meters with handheld terminals, a collector takes in the readings, a verifier checks them,
billing and work-order services take over. There are three environments and a nightly batch job
runs. Both the system and the numbers are fictional; what is measured is the ratio between the
numbers.

- **WS1.** The item list is not read from a product's default configuration file; it is a fictional
  list of the things typically left open after installation. What is measured is not the list but
  the ratio that comes out of it.
- **WS2.** If an item is "external," it is reachable from the public port; if "internal," only from
  the internal network. In separate processes, this distinction is preserved by the process boundary.
- **WS3.** A legitimate workflow represents a real piece of work that stops when an item is closed;
  there are six workflows and each one leans on a single item.
- **WS4.** Items with the same sharing key collapse into a single item when merged: the plaintext
  port all three roles listen on is one port, the version header all three stamp is one header.

```js
// roles/surface.mjs — model: the surface left open after the default installation of three web
// server roles. Not a real product's default file, but the fictional measurement network's
// post-installation item list; every number is derived from this list.

// [role, item, type, access, sharing key]
const ITEM = [
  ["static",       "plaintext-port",       "listening port", "external", "port-80"],
  ["static",       "asset-root",           "served path",    "external", null],
  ["static",       "directory-listing",    "served path",    "external", null],
  ["static",       "backup-files",         "served path",    "external", null],
  ["static",       "sample-page",          "served path",    "external", null],
  ["static",       "version-header",       "visible info",   "external", "version"],

  ["proxy",        "plaintext-port",       "listening port", "external", "port-80"],
  ["proxy",        "encrypted-port",       "listening port", "external", null],
  ["proxy",        "upstream-port",        "listening port", "internal", null],
  ["proxy",        "version-header",       "visible info",   "external", "version"],
  ["proxy",        "error-page-trace",     "visible info",   "external", null],
  ["proxy",        "proxy-status-endpoint","management endpoint", "internal", null],

  ["distribution", "plaintext-port",       "listening port", "external", "port-80"],
  ["distribution", "version-header",       "visible info",   "external", "version"],
  ["distribution", "member-address-header","visible info",   "external", null],
  ["distribution", "session-cookie-member","visible info",   "external", null],
  ["distribution", "health-endpoint",      "management endpoint", "internal", null],
  ["distribution", "member-mgmt-endpoint", "management endpoint", "internal", null],
];

// Items born of merging into a single process, not found alone in any single role.
const MERGE = [
  ["merged", "cross-root",             "served path",           "external"],
  ["merged", "shared-mgmt-port",       "management endpoint",   "external"],
  ["merged", "single-process-identity","combined authority",    "external"],
];

// Legitimate workflows: which item each workflow leans on.
const FLOW = [
  ["handheld-terminal-read", "static/plaintext-port"],
  ["field-asset-serving",    "static/asset-root"],
  ["collector-upload",       "proxy/encrypted-port"],
  ["on-call-health-call",    "distribution/health-endpoint"],
  ["rollout-flow",           "distribution/member-mgmt-endpoint"],
  ["session-stickiness",     "distribution/session-cookie-member"],
];

const id = (r, k) => `${r}/${k}`;
const roles = ["static", "proxy", "distribution"];

const W = [14, 7, 11, 9, 9, 9, 6, 5];
const row = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](W[i])).join("");
const measure = (s) => [s.length, ...["listening port", "served path", "visible info", "management endpoint"]
  .map((t) => s.filter((x) => x[2] === t).length),
  s.filter((x) => x[3] === "external").length, s.filter((x) => x[3] === "internal").length];

console.log(row(["role", "items", "listening", "served", "visible", "mgmt", "ext", "int"]));
for (const role of roles) console.log(row([role, ...measure(ITEM.filter((x) => x[0] === role))]));
console.log(row(["total", ...measure(ITEM)]));

// Merging: items with the same sharing key collapse into one item.
const seen = new Set();
const unique = [];
for (const [role, item, type, access, share] of ITEM) {
  const key = share ?? id(role, item);
  if (seen.has(key)) continue;
  seen.add(key);
  unique.push([role, item, type, access, share]);
}
const merged = [...unique.map((x) => [...x]), ...MERGE.map((x) => [...x, null])];
console.log(`\nthree separate processes: ${ITEM.length} items -> ${unique.length} unique` +
  `  (ext ${unique.filter((x) => x[3] === "external").length}, int ${unique.filter((x) => x[3] === "internal").length})`);
// A single process runs on a single listener: even an "internal"-tagged item is now reachable from the public port.
for (const k of merged) k[3] = "external";
console.log(`single process:  ${unique.length} + ${MERGE.length} new = ${merged.length} items` +
  `  (ext ${merged.length}, int 0)`);

// Hardening steps: each step is a predicate, the items it closes, and the new items it introduces.
const STEP = [
  ["close visible info", (x) => x[2] === "visible info", []],
  ["narrow served paths", (x) => x[2] === "served path" && x[1] !== "asset-root",
    [["allowlist-file", "maintenance surface"]]],
  ["move mgmt endpoints off the public port", (x) => x[2] === "management endpoint",
    [["separate-mgmt-listener", "listening port"], ["mgmt-identity", "secret"]]],
  ["close the plaintext port", (x) => x[1] === "plaintext-port", []],
];

let open = merged.map((x) => [...x]);
const added = [];
console.log("\nstep                                        closed  wrongly closed  new surface  still open");
for (const [name, pred, introduces] of STEP) {
  const dropped = open.filter(pred);
  open = open.filter((x) => pred(x) === false);
  const broken = FLOW.filter(([, k]) => dropped.some((d) => id(d[0], d[1]) === k));
  added.push(...introduces);
  console.log(name.padEnd(42) + String(dropped.length).padStart(8) +
    String(broken.length).padStart(16) + String(introduces.length).padStart(13) +
    String(open.length + added.length).padStart(12) +
    (broken.length ? `   ${broken.map((b) => b[0]).join(", ")}` : ""));
}
console.log(`\nitems that cannot be closed (${open.length}): ${open.map((x) => x[1]).join(", ")}`);
console.log(`items closing introduces (${added.length}): ${added.map((x) => x[0]).join(", ")}`);
```

```
role            items  listening   served  visible     mgmt   ext  int
static              6          1        4        1        0     6    0
proxy               6          3        0        2        1     4    2
distribution        6          1        0        3        2     4    2
total              18          5        4        6        3    14    4

three separate processes: 18 items -> 14 unique  (ext 10, int 4)
single process:  14 + 3 new = 17 items  (ext 17, int 0)

step                                        closed  wrongly closed  new surface  still open
close visible info                               4               1            0          13   session-stickiness
narrow served paths                              4               0            1          10
move mgmt endpoints off the public port          4               2            2           8   on-call-health-call, rollout-flow
close the plaintext port                         1               1            0           7   handheld-terminal-read

items that cannot be closed (4): asset-root, encrypted-port, upstream-port, single-process-identity
items closing introduces (3): allowlist-file, separate-mgmt-listener, mgmt-identity
```

Counting is deterministic: the list is fixed, there is no randomness, no duration is measured.

#### The Role's Own Surface

The first table lines up the three roles side by side. All three have six items, but the items'
**type** differs.

Four of static serving's six items are **served paths**, and all four depend on a single setting:
the root directory. Whatever is in the root directory gets served. The asset root is legitimate —
the field crews' browser is fed from there. The directory listing, the sample page that ships with
the installation, and a configuration backup someone once dropped inside the root are things
nobody decided on. All six of the static role's items are **externally reachable**: this role has
no internal surface, because it has no management endpoint of its own.

Three of the reverse proxy's six items are **listening ports**, and one detail matters here: the
third port is the upstream's own port. When the proxy is set up, the application behind it starts
listening on a port, and in most setups that port binds to all of the machine's interfaces. The
proxy does not open a port; the proxy's existence opens **a second port behind it**, and that port
can be reached without ever passing through the proxy. None of the proxy's settings ever see a
request that arrives directly at the port behind it.

Three of load distribution's six items are **visible info**, and all three carry the pool's
internal structure outward: the member address in the response header, the member identity in the
session cookie, the member list the health endpoint dumps. This role is set up to make internal
structure visible — knowing which member answered is necessary when troubleshooting — and that
same visibility is exposed outward as well.

The three roles total eighteen items: five listening ports, four served paths, six visible info
items, three management endpoints. Fourteen are external, four are internal.

The asymmetry in the distribution says something. In the static role the surface sits **in the
file system**: closing an item means deciding what goes in a directory. In the proxy role the
surface sits **in the network topology**: closing an item means deciding which interface a port
binds to. In the distribution role the surface sits **in the response itself**: closing an item
means stripping a header or a cookie. This is why a single hardening procedure cannot be written
for all three roles; even though the count is equal, the closing work happens in three separate
places.

#### What Merging Does

The second measurement puts the three roles into **a single process**. This is the arrangement
most commonly seen on one machine: the same process serves static files, forwards requests to the
back end, and splits traffic across members.

Merging first produces a **reduction**. The plaintext port all three roles listen on collapses
into one port, the version header all three stamp collapses into one header. Eighteen items drop
to fourteen unique items. Someone looking only at the item count could stop here and say merging
**narrows** the surface.

Two things break that reading.

First, merging **spawns** its own items. The static root is now also reachable through the proxy's
public port; in separate processes there was a boundary between these two paths, in a single
process there is not. The three roles' management endpoints collect on a single listener. The
single process carries all three roles' permissions at once: the same identity reads the static
root, connects to the upstream, and changes pool membership. Three new items carry fourteen up to
seventeen.

Second, and heavier, **the access label falls away**. In three separate processes, ten of the
fourteen items were externally reachable; four — two management endpoints, the health endpoint,
and the upstream port — sat behind the process boundary. A single process runs on a single
listener, and that listener is the public port. **All seventeen** of the seventeen items become
externally reachable.

This is the measure of merging: while the item count drops from eighteen to seventeen, the
externally reachable item count climbs from ten to seventeen. Counted by items, merging looks
good; counted by access, the surface has grown by 70%.

#### Four Steps and What They Stop

The third table applies four hardening steps to the seventeen-item surface. Each step produces two
numbers: how many items closed and **how many legitimate workflows stopped**.

The first step closes visible info: the version header, the error-page trace, the member address
header, and the member cookie. Four items close and one workflow stops — session stickiness. The
field crew's multi-step reading form relied on later steps landing on whichever member answered
the first step, and it read that from the member cookie. Once the cookie is gone, the form falls
onto a different member on the second step. This step gets defended under the heading "close the
leaking info," and the defense is incomplete: the fourth item it closed was not an information
leak, it was the carrying mechanism.

The second step narrows the served paths: the directory listing, the backup files, the sample
page, and the cross-root that merging spawned. Four items close, no workflow stops. The cost is
elsewhere: closing is done with an allowlist file, and that file is now a place that needs
watching. When a new asset directory is added and not added to the list, it is not served; this is
the source of a future failure.

The third step takes the management endpoints off the public port. Four items close and **two
workflows stop**: the on-call health call and the rollout flow's member-add call. Both are
legitimate and both went through those endpoints. For the step to be completable it introduces two
new items: a separate management listener reachable only from the internal network, and a
management identity that enters that listener. The second item is a secret, and secrets have their
own lifetime, their own exposure window.

The fourth step closes the plaintext port. One item closes, one workflow stops: the old handheld
terminals in the field only speak plaintext. This is the smallest closure and the most visible
break — the day it stops, the entire field crew calls.

The four steps closed thirteen items, stopped four legitimate workflows, and introduced three new
items. The ratio is thirteen to four: for every three items closed, roughly one workflow stopped.
That ratio is nowhere near even across the steps — the second step closes four items with zero
stoppage, while the fourth step closes a single item with one stoppage. This is what decides the
order: cheap step first, expensive step last. An ordering that looked only at the surface-item
count would not have put the fourth step last, because that step closes the most talked-about item
and gives the smallest number in return.

#### The Four That Cannot Be Closed

At the end of the four steps, four items remain, and all four are items that cannot be closed.

**The asset root** and **the encrypted port** are the machine's reason for existing; closing them
is shutting down the system. **The upstream port** cannot be closed because the proxy's connection
to the back end passes through it; it can be narrowed — bound only to the loopback interface — but
it stays open. **The single-process identity** closes only by undoing the merge, that is, by
splitting the three roles back into three processes; no configuration line closes it.

The final table shows seven open items: four that cannot be closed and three new ones. Compared
with the starting seventeen, this is a gain. In a report that does not count the new surface, the
gain would be written as dropping from seventeen items to four; counted, it is dropping to seven.

#### Summary

- The default installation of three web server roles left eighteen items open: five listening
  ports, four served paths, six visible info items, three management endpoints; fourteen
  externally reachable, four internally.
- The roles' item counts were equal, their types were not: four of static serving's items were
  served paths, three of the reverse proxy's were listening ports, three of load distribution's
  were visible info.
- Merging into a single process reduced eighteen items to fourteen unique ones, spawned three
  items, and carried four internally reachable items out into the open: the externally reachable
  item count climbed from ten to seventeen.
- Four hardening steps closed thirteen items, stopped four legitimate workflows, and introduced
  three new items; the step that closed the management endpoints alone stopped two workflows and
  added two new items.
- Four items that cannot be closed remained; one of them, the single-process identity, closes not
  through any setting but only by splitting the roles back into separate processes.

#### Next Step

This lesson counted the surface **from the outside**: how many ports are listening, how many
paths are served, how many endpoints are open. The whole count was about the request **arriving**
at the server. What happens after the request arrives was never examined.

When the reverse proxy takes a request and forwards it, the request does not pass through
unchanged. The connection ends at the proxy, and a new connection opens to the back end; at the
other end of that new connection stands the proxy, not the client. The client's address, whether
the request arrived encrypted, which name the client asked for, and the request's original path —
all four traveled in the connection itself, and that connection ended. The next lesson sets up a
two-hop proxy chain with `node:http`, passes real requests through it, counts one by one the four
pieces of information lost while passing through the chain, shows the wrong decision each loss
produces in the application, and measures the surface opened by **trusting** the header added to
close the loss.

### Reverse Proxy Configuration

Source: https://academia.sh/en/courses/servers-and-hardening/reverse-proxy-configuration

Counting the four pieces of information lost by passing a real request through a two-hop proxy chain: client address, protocol, server name, and the request's original path; the wrong decision each loss produces in the application (three members recorded under a single identity, an absolute link pointing at an internal address, a link that returns 404 once the prefix is stripped), the second hop erasing the first hop's forwarding, and the claimed-identity surface opened by trusting the forwarded header.

The previous lesson counted the surface from outside the server: how many ports are listening,
how many paths are served, how many endpoints are open. The whole count was about the request
**arriving**. This lesson measures what happens after the request arrives.

The reverse proxy takes the request and forwards it. The word "forwards" hides what actually
happens: the proxy does not carry the request, it **rebuilds** it. The connection the client
opened ends at the proxy; the proxy opens a new connection of its own to the back end. At the
other end of the new connection stands the proxy, not the client. Whatever traveled in the
connection itself is erased at that point. What a reverse proxy is and how it is configured when
taking an application to production were measured before; what is measured here is the **item of
lost information** and the surface that loss opens.

The fictional regional measurement network has two proxies in front of it: an outward-facing edge
proxy and an internal proxy that selects the environment. Field handheld terminals connect to the
edge proxy with the `/api` prefix, the edge proxy connects to the internal proxy, and the internal
proxy connects to the measurement app. Two hops, three connections.

- **WS7.** The app reads **the leftmost entry** in the relay header; when the relay chain grows
  longer with commas, the value read does not change. This is the common way a forwarded identity
  is read.
- **WS8.** On a single machine, every client shares the same loopback address. The fictional
  member address is derived at the edge proxy from the source-port block; in a real deployment
  this value comes straight from the connection.
- **WS9.** The edge proxy and the internal proxy are **the same file**; every difference between
  them lives in the config string.
- **WS10.** The app's internal summary endpoint can only be called from `10.9.`-prefixed
  addresses; the access decision rests on **the resolved address**.
- **WS11.** The measurement does not measure time; the request counts, source-port blocks, and
  expected values are constants independent of the run.

```js
// chain/app.mjs — the measurement app. Resolves the request's four pieces of info first from
// the "x-relay-*" headers, else from the connection itself, and produces a decision from each.
import { createServer } from "node:http";

const port = Number(process.argv[2]);
const INTERNAL_NET = "10.9.";                           // internal-network prefix
const log = { address: new Set(), allowed: 0, rejected: 0 };

const resolve = (i) => ({
  // The relay chain grows with commas; the app reads the leftmost entry.
  address: String(i.headers["x-relay-address"] ?? i.socket.remoteAddress).split(",")[0].trim(),
  scheme: i.headers["x-relay-scheme"] ?? "plaintext",
  name: i.headers["x-relay-name"] ?? i.headers.host,
  prefix: i.headers["x-relay-prefix"] ?? "",
});

if (Number.isInteger(port) === false) console.log("usage: node chain/app.mjs <port>");
else createServer((req, res) => {
  res.sendDate = false;
  const path = req.url;
  const c = resolve(req);
  if (path === "/dump") {
    res.writeHead(200).end(JSON.stringify({ ...log, identities: log.address.size, address: [...log.address] }));
    return;
  }
  if (path === "/internal/summary") {                    // endpoint meant to be called only from the internal network
    if (c.address.startsWith(INTERNAL_NET)) { log.allowed += 1; res.writeHead(200).end("internal summary"); }
    else { log.rejected += 1; res.writeHead(403).end("from outside"); }
    return;
  }
  const parts = path.split("/");
  if (parts[1] !== "measurement" || parts.length > 4) { res.writeHead(404).end("no path"); return; }
  if (parts.length === 4 && parts[3] !== "detail") { res.writeHead(404).end("no path"); return; }
  log.address.add(c.address);                            // identity written to the audit log
  res.writeHead(200, {
    "x-address": c.address, "x-scheme": c.scheme, "x-name": c.name, "x-prefix": c.prefix || "none",
    "x-decision": c.scheme === "encrypted" ? "serve" : "forward",
    "x-link": `${c.scheme}://${c.name}${c.prefix}${path}/detail`,
  }).end("measurement");
}).listen(port, "127.0.0.1");
```

The proxy takes six settings from a single string. In a product, each of these would be a
configuration line; here it is one string field.

```js
// chain/proxy.mjs — the reverse proxy. The same file runs as both the edge and internal proxy;
// the config string decides behavior. Config: address,scheme,name,prefixhdr = on|off; prefix=<path>; clear=on;
// chain=write|append|keep; map=on (edge proxy only: connection -> fictional client address).
import { createServer, request } from "node:http";

const [port, target, cfgText] = [Number(process.argv[2]), Number(process.argv[3]), process.argv[4] ?? ""];
const CFG = Object.fromEntries(cfgText.split(",").filter(Boolean).map((p) => p.split("=")));
// WS8: on a single machine every client shares the same loopback address. The fictional member
// address is derived from the source-port block; in a real deployment the value comes straight from the connection.
const ADDR_BOOK = { 512: "203.0.113.11", 513: "203.0.113.12", 514: "203.0.113.13" };
const source = (s) => (CFG.map === "on" ? ADDR_BOOK[Math.floor(s.remotePort / 100)] : undefined) ?? s.remoteAddress;

if (Number.isInteger(port) === false || Number.isInteger(target) === false) {
  console.log("usage: node chain/proxy.mjs <port> <target> <config>");
} else createServer((frontReq, frontRes) => {
  frontRes.sendDate = false;
  const hdrs = { ...frontReq.headers };
  if (CFG.clear === "on") for (const k of Object.keys(hdrs)) if (k.startsWith("x-relay-")) delete hdrs[k];
  const existing = hdrs["x-relay-address"];
  if (CFG.address === "on") {
    const own = source(frontReq.socket);
    if (CFG.chain === "keep" && existing) hdrs["x-relay-address"] = existing;
    else if (CFG.chain === "append" && existing) hdrs["x-relay-address"] = `${existing}, ${own}`;
    else hdrs["x-relay-address"] = own;
  }
  if (CFG.scheme === "on") hdrs["x-relay-scheme"] = "encrypted";
  if (CFG.name === "on") hdrs["x-relay-name"] = hdrs["x-relay-name"] ?? frontReq.headers.host;
  let path = frontReq.url;
  if (CFG.prefix) {
    if (path.startsWith(CFG.prefix) === false) { frontRes.writeHead(404).end("no path"); return; }
    path = path.slice(CFG.prefix.length) || "/";              // prefix is stripped; the back end never sees it
    if (CFG.prefixhdr === "on") hdrs["x-relay-prefix"] = CFG.prefix;
  }
  hdrs.host = `127.0.0.1:${target}`;                          // requested name is replaced with the upstream's name
  const backReq = request({ port: target, path, method: frontReq.method, headers: hdrs }, (backRes) => {
    frontRes.writeHead(backRes.statusCode, backRes.headers);
    backRes.pipe(frontRes);
  });
  backReq.on("error", () => { if (frontRes.headersSent === false) frontRes.writeHead(502).end("cut off"); });
  frontReq.pipe(backReq);
}).listen(port, "127.0.0.1");
```

The client connects three members from separate source-port blocks, compares the four resolved
values against the expected ones, follows the link the application generates, and reads the
identity count in the audit log.

```js
// chain/client.mjs — three members connect to the chain from separate source-port blocks.
// Usage: node chain/client.mjs <edge> <app> <scenario> <label> <shift>
import { request } from "node:http";

const [edge, app, scenario, label, shift] =
  [Number(process.argv[2]), Number(process.argv[3]), process.argv[4], process.argv[5] ?? "-", Number(process.argv[6] ?? 0)];
const NAME = "measurement.example.invalid";                // fictional name
const EXPECTED = { address: "203.0.113.11", scheme: "encrypted", name: NAME, prefix: "/api" };

const call = ({ path, source, port = edge, headers = {} }) => new Promise((resolve) => {
  const r = request({ port, path, agent: false, localPort: source, headers: { host: NAME, ...headers } },
    (y) => { let g = ""; y.on("data", (d) => { g += d; }); y.on("end", () => resolve({ code: y.statusCode, h: y.headers, body: g })); });
  r.on("error", (e) => resolve({ code: 0, h: {}, body: e.code }));
  r.end();
});

if (scenario === "lost") {
  let first = null;
  for (let i = 0; i < 3; i += 1) {
    const y = await call({ path: "/api/measurement/7", source: 51200 + i * 100 + shift * 5 });
    first = first ?? y;
  }
  const dump = JSON.parse((await call({ path: "/dump", port: app, source: 51700 + shift })).body);
  const link = first.h["x-link"] ?? "-";
  const linkPath = link.slice(link.indexOf("://") + 3).replace(/^[^/]*/, "");
  const followed = await call({ path: linkPath, source: 51800 + shift });
  const correct = ["address", "scheme", "name", "prefix"].filter((k) => first.h[`x-${k}`] === EXPECTED[k]).length;
  console.log(`${label.padEnd(18)}resolved ${correct}/4  log ${dump.identities} identities  ` +
    `decision ${String(first.h["x-decision"]).padEnd(10)}link followed ${followed.code}`);
  console.log(`${" ".repeat(18)}address ${String(first.h["x-address"]).padEnd(15)} name ${String(first.h["x-name"]).padEnd(30)} prefix ${first.h["x-prefix"]}`);
} else if (scenario === "spoofed") {
  let entered = 0;
  for (let i = 0; i < 4; i += 1) {
    const y = await call({ path: "/api/internal/summary", source: 51500 + i + shift * 10, headers: { "x-relay-address": "10.9.0.5" } });
    if (y.code === 200) entered += 1;
  }
  const dump = JSON.parse((await call({ path: "/dump", port: app, source: 51900 + shift })).body);
  console.log(`${label.padEnd(18)}entered internal endpoint ${entered}/4  app rejected ${dump.rejected}`);
} else console.log("usage: node chain/client.mjs <edge> <app> <scenario> <label> <shift>");
```

```bash
# run.sh — the same request chain with different relay configs. All three hops restart on every run.
R0="prefix=/api,map=on"                   # same on every run: prefix is stripped, source map at the edge
run() {                                   # run <edge-extra> <internal-cfg> <scenario> <label> <shift>
  node chain/app.mjs 8942 & A=$!
  node chain/proxy.mjs 8941 8942 "$2" & I=$!
  node chain/proxy.mjs 8940 8941 "$R0$1" & E=$!
  sleep 0.5
  node chain/client.mjs 8940 8942 "$3" "$4" "$5"
  kill $E $I $A 2>/dev/null; wait $E $I $A 2>/dev/null; sleep 0.2
}
D=",clear=on,address=on,scheme=on,name=on,prefixhdr=on"      # all four headers are forwarded

run ",clear=on"                     "" lost    "no forwarding"   0
run ",clear=on,address=on,scheme=on" "" lost    "address+scheme"  1
run "$D"                                "" lost    "four headers"    2
run "$D"       "address=on,chain=write"    lost    "second hop"     3
run ",address=on,scheme=on,name=on,prefixhdr=on,chain=append" "" spoofed "no clearing" 4
run "$D"                                "" spoofed "clearing"       5
```

```
no forwarding     resolved 0/4  log 1 identities  decision forward   link followed 404
                  address 127.0.0.1       name 127.0.0.1:8942                 prefix none
address+scheme    resolved 2/4  log 3 identities  decision serve     link followed 404
                  address 203.0.113.11    name 127.0.0.1:8942                 prefix none
four headers      resolved 4/4  log 3 identities  decision serve     link followed 200
                  address 203.0.113.11    name measurement.example.invalid    prefix /api
second hop        resolved 3/4  log 1 identities  decision serve     link followed 200
                  address 127.0.0.1       name measurement.example.invalid    prefix /api
no clearing       entered internal endpoint 4/4  app rejected 0
clearing          entered internal endpoint 0/4  app rejected 4
```

#### From Zero to Four

The first row is the default state: both proxies forward the request, neither relays anything.
The app loses **all four** pieces of information and produces four separate wrong decisions.

**Client address.** Three separate members sent three separate requests; the audit log has
**one** identity. The log is not missing anything, it is full — the address it wrote is just the
internal proxy's address. When asked which handheld terminal an event came from, the log answers,
and the answer is wrong.

**Protocol.** The connection reaching the app is plaintext, because encryption ended at the edge
proxy. The app decides it sees a plaintext request and says `forward`. The decision itself is a
correctly written rule; its input is wrong.

**Server name.** The client asked for the name `measurement.example.invalid`. When the proxy
connected to the back end, it replaced the requested name with the upstream's own name — this is
not a bug, it is how the connection is set up. The app writes `127.0.0.1:8942` into the absolute
link it generates. If this link were shown on a page or put in a notification email, the internal
address would leak outward.

**Original path.** The edge proxy stripped the `/api` prefix. Since the app has no idea the
prefix existed, it does not put the prefix into the link it generates. When the link is followed,
the edge proxy does not match it against its own routing rule and returns **404**. The link in the
notification email sent to the field crew is a link that does not work the day it is opened.

The second row turns on two headers. The identity count in the audit log climbs not to one but to
**three**, and the decision becomes `serve`. Two items closed; the name and the prefix are still
lost and the link is still 404. This row shows the most misleading part of partial relaying:
because the log looks correct, the configuration is assumed to be complete, when in fact both
closed items were items visible inside the log. The two items invisible in the log cannot be found
by looking at the log.

The third row turns on all four headers: all four pieces of information resolve, the log has three
identities, and the generated link returns 200 when followed.

#### What the Second Hop Erases

In the fourth row, the edge proxy's configuration is **identical** to the third row's. The only
thing that changes is that the internal proxy also turns on address relaying.

The result: resolved information drops from four to three, and the identity count in the audit
log falls from three to one. The internal proxy overwrote the header the edge proxy wrote with the
address it sees itself; the address it sees is the edge proxy's address. Even though relaying is
turned on at both hops, the value that reaches the app is not the client's address.

This is a failure that is harder to find than the first row's. In the first row there was no
relaying at all, and a configuration review catches that. In the fourth row, **relaying is turned
on at both proxies**; a review says "relaying is on" and moves past it. The difference is in a
single word — the second hop overwrites the incoming value instead of preserving it.

Chain length becomes a multiplier here: for relaying to work correctly, **every** hop in the chain
must preserve the incoming value or append to it. Two hops mean two decisions, four hops mean four
decisions, and all of them must be right; one being wrong is enough.

#### Trusting the Forwarded Header

The last two rows are the other side of the measurement. The app now **bases a decision** on the
relayed address: the internal summary endpoint can only be called from internal-network addresses.
The moment the header that closes the loss closes it, that header turns into an authorization
input.

In the fifth row, the edge proxy does not clear the incoming `x-relay-*` headers and appends to
the chain. The client writes `10.9.0.5` into its own request. The edge proxy appends the address
it sees on top of that, the header ends up with two entries, and the app reads the leftmost entry:
what the client wrote. **All four** of the four requests get into the internal endpoint, the app
rejects none of them, and the four requests sit in the logs as successful internal calls.

In the sixth row, the edge proxy clears the headers. Zero of the four requests get in, and the app
rejects all four.

The name of the surface this opens is **the claimed-identity surface**: every relay header the app
trusts is an input under the client's control unless every hop in front of it strips it. This
surface opens in two ways. The first is forgetting to clear it — the fifth row. The second is
**the upstream port** the previous lesson counted: if the internal proxy's and the app's port
listens on all of the machine's interfaces, a request that never passes through the edge proxy at
all can write the header itself, and clearing never runs. The edge proxy's single-line clearing
rule only counts as a rule for as long as the chain has a single entry point.

#### Four Items, Two Directions

Each of the four pieces of information produces two separate items: a wrong decision while closed,
a trust point while open.

| Information | Wrong decision produced while closed | Surface born while open |
|---|---|---|
| Client address | three members collapse into a single identity in the log | the address-based access decision falls to whatever value the client writes |
| Protocol | an encrypted request is counted as plaintext, decision becomes `forward` | the app never verifies encryption itself, it trusts the header |
| Server name | the generated link points at the internal address | a rule that distinguishes by name can be bypassed with a spoofed name |
| Original path | the prefix drops, the generated link gets 404 | a per-prefix authorization rule can be bypassed with a spoofed prefix |

The table's right column is the lesson's real result. The only way to close the loss is to carry
the information in a header, and a header is not provable the way the connection itself is. While
relaying is off, the app behaves correctly with wrong information; while it is on, it behaves with
correct information, and that information's correctness now depends on every hop in the chain
being disciplined. The four lost items were closed; four trust points that need auditing took
their place.

#### Summary

- Four pieces of information get lost passing through the proxy chain: client address, protocol,
  requested server name, and the request's original path; with relaying off, the app resolved all
  four incorrectly.
- All four losses produced a visible decision: three members collapsed into a single identity in
  the log, the decision became `forward`, the generated link pointed at the internal address, and
  following it returned 404 because the prefix was missing.
- Relaying two headers fixed the log but not the link: the log showed three identities, the link
  still returned 404; the remaining two items cannot be found by looking at the log.
- Even with relaying on at both proxies, when the second hop overwrote the incoming value instead
  of preserving it, resolved information dropped from four to three and the identity count in the
  log fell from three to one.
- Once a decision was based on the relayed header, the claimed-identity surface opened: with
  clearing off, all four of the four spoofed-header requests got into the internal endpoint; with
  clearing on, zero got in.

#### Next Step

Every measurement in this lesson assumed encryption **ends somewhere**: the edge proxy decrypts,
and everything behind it carries plaintext. That termination happens at the edge proxy was never
questioned, yet this is a decision, and it can be made in three places.

When the termination point changes, three numbers change at once: how many legs of the chain stay
plaintext, how many separate places the certificate sits in, and how many points get touched
during renewal. The three do not move in the same direction; the option that drops the plaintext
legs to zero increases both the number of places the certificate lives and the cost of renewing
it. The next lesson compares the three termination options with these three numbers, models the
accepted protocol versions and cipher suites as a set of items, and measures how many old options
the default leaves open and how many clients a restriction drops.

### TLS Termination

Source: https://academia.sh/en/courses/servers-and-hardening/tls-termination

Comparing three options for where termination happens: the plaintext-leg count, certificate-location count, renewal-touch-point count, and the work the edge loses when terminating at the edge, at the proxy, and at the app; modeling protocol version and cipher suite as an item set, the twenty-eight items the default leaves open, and the device and read counts a restriction drops.

Every measurement in the previous lesson took one thing as given: encryption ends somewhere. The
edge proxy decrypts, everything behind it carries plaintext, and the lost information was handed
back through headers. "It ends at the edge proxy" is not an assumption, it is a decision, and
there are three places it can be made.

This lesson measures that decision with three numbers: how many legs of the chain stay plaintext,
how many separate places the certificate sits in, how many points get touched during renewal. The
three do not move in the same direction. The option that drops the plaintext legs to zero triples
the other two numbers.

Which security headers go into the response and how many do not arrive by default were measured
before, in the **Security Headers and TLS** lesson; that measurement is not repeated here. What is
measured here is not the header itself, but **where the encryption ends**.

The fictional regional measurement network's request path passes through four components: the
client, the edge proxy, the internal proxy that selects the environment, and the measurement app.
There are three legs.

- **WS13.** The component copy counts are fictional: the edge proxy has two copies, the internal
  proxy three, the app six.
- **WS14.** The number of points touched during renewal is taken as equal to the number of places
  the certificate sits in; the renewal flow's own steps are measured in the next lesson.
- **WS15.** Version names are fictional: `s1` is oldest, `s4` is newest. Cipher suites are named
  `t01`–`t12`, and each suite can be used from its own minimum version onward.
- **WS16.** An item is an accepted **(version, suite)** pair. The default policy sets no floor and
  accepts every suite.
- **WS17.** A client population drops if the highest version it supports is below the policy's
  floor, or if no suite it can use remains in the accepted list.
- **WS18.** The model does not measure time; no real handshake is performed, and the item and
  device counts are derived from the lists.

```js
// termination/point.mjs — model: the three options for TLS termination. The chain is the
// fictional measurement network's request path; the copy counts are fictional. The model counts
// the plaintext legs, how many places the certificate sits in, and how many points get touched during renewal.
const CHAIN = ["client", "edge", "internal-proxy", "app"];
const COPIES = { edge: 2, "internal-proxy": 3, app: 6 };  // instance count per component
// What the edge can do: things that need plaintext, and the one job visible during the handshake.
const CAPABILITY = [
  ["path-based routing", "plain"], ["edge cache", "plain"],
  ["per-path rate limit", "plain"], ["body-size check", "plain"],
  ["name-based routing", "handshake"],
];

function measure(point) {
  const i = CHAIN.indexOf(point);
  const legs = CHAIN.slice(0, -1).map((_, b) => (b < i ? "encrypted" : "plaintext"));
  const seesPlain = CHAIN.slice(i);                         // everything after termination sees plaintext
  const edgePlain = seesPlain.includes("edge");
  return {
    plaintextLegs: legs.filter((x) => x === "plaintext").length,
    certLocations: COPIES[point],
    renewalPoints: COPIES[point],
    plainViewers: seesPlain.length,
    edgeCapability: CAPABILITY.filter(([, k]) => k === "handshake" || edgePlain).length,
  };
}

const E = [16, 16, 16, 18, 17, 18];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["termination", "plaintext legs", "cert locations", "renewal points",
  "sees plaintext", "edge capability"]));
for (const n of ["edge", "internal-proxy", "app"]) {
  const o = measure(n);
  console.log(write([n, o.plaintextLegs, o.certLocations, o.renewalPoints, o.plainViewers,
    `${o.edgeCapability}/${CAPABILITY.length}`]));
}

// --- Protocol and cipher suite: item set ---------------------------------
// Version names are fictional; s1 is oldest, s4 is newest. A suite can only be used from
// its own minimum version onward.
const SUITE = [
  ["t01", 1, false], ["t02", 1, false], ["t03", 2, false], ["t04", 2, false],
  ["t05", 2, false], ["t06", 3, true],  ["t07", 3, true],  ["t08", 3, false],
  ["t09", 3, true],  ["t10", 4, true],  ["t11", 4, true],  ["t12", 4, true],
];
// Fictional client population: [name, device count, highest version supported, daily reads]
const CLIENT = [
  ["old-field-handheld", 40, 1, 1200], ["billing-client", 25, 2, 400],
  ["new-field-handheld", 180, 3, 5400], ["work-order-app", 95, 3, 950],
  ["collector", 60, 4, 28800], ["browser-dashboard", 300, 4, 900],
];

function policy(minVersion, strongOnly) {
  const acceptedSuites = SUITE.filter(([, , g]) => (strongOnly ? g : true));
  const items = [];
  for (let s = minVersion; s <= 4; s += 1)
    for (const [name, min, strong] of acceptedSuites) if (min <= s) items.push([s, name, strong]);
  const connected = CLIENT.filter(([, , maxV]) =>
    maxV >= minVersion && acceptedSuites.some(([, min]) => min <= maxV));
  const dropped = CLIENT.filter((x) => connected.includes(x) === false);
  return {
    items: items.length, weak: items.filter((k) => k[2] === false).length,
    devices: connected.reduce((t, x) => t + x[1], 0),
    droppedDevices: dropped.reduce((t, x) => t + x[1], 0),
    droppedReads: dropped.reduce((t, x) => t + x[3], 0),
    droppedNames: dropped.map((x) => x[0]),
  };
}

const P = [13, 15, 16, 13, 18, 15, 20];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](P[i])).join("");
console.log("\n" + write2(["min version", "suite list", "accepted items", "weak items",
  "connected dev", "dropped dev", "dropped reads/day"]));
for (const [min, strong] of [[1, false], [2, false], [3, false], [3, true], [4, true]]) {
  const r = policy(min, strong);
  console.log(write2([`s${min}`, strong ? "strong only" : "all", r.items, r.weak, r.devices,
    r.droppedDevices, r.droppedReads]) + (r.droppedNames.length ? `   ${r.droppedNames.join(", ")}` : ""));
}
```

```
termination       plaintext legs  cert locations    renewal points   sees plaintext   edge capability
edge                           2               2                 2                3               5/5
internal-proxy                 1               3                 3                2               1/5
app                            0               6                 6                1               1/5

min version       suite list  accepted items   weak items     connected dev    dropped dev   dropped reads/day
s1                       all              28           19               700              0                   0
s2                       all              26           17               660             40                1200   old-field-handheld
s3                       all              21           12               635             65                1600   old-field-handheld, billing-client
s3               strong only               9            0               635             65                1600   old-field-handheld, billing-client
s4               strong only               6            0               360            340                7950   old-field-handheld, billing-client, new-field-handheld, work-order-app
```

#### Three Options, Three Numbers

The first table lines up the three options side by side, and no row beats the others in every
column.

**Terminating at the edge** is the most common arrangement and the one that asks for the least
maintenance: the certificate sits in two places, renewal touches two points. Its cost is two
plaintext legs. On the path from the edge proxy to the app, meter readings, subscriber identities,
and invoice amounts travel as plaintext. This path is called the "internal network," and treating
the internal network as trustworthy is not a measurement, it is a habit. The number of components
that can see plaintext is three.

**Terminating at the proxy** reduces the edge proxy to a carrier: the encrypted connection passes
through it unchanged and ends at the internal proxy. The plaintext leg drops to one, the number of
components that see plaintext drops to two. In return, the certificate starts sitting in three
places and renewal touches three points.

**Terminating at the app** drops the plaintext leg to zero. Only the app sees plaintext. The
certificate sits in six places and renewal touches six points. The number has tripled, and what is
growing is not a security setting, it is **a maintenance surface**: a key distributed to six
places is six separate copies, six separate file permissions, six separate chances for renewal to
fail.

The three rows show a single trend: as the plaintext leg shrinks, the number of places the key
sits in grows. Zero plaintext legs does not mean zero risk; it moves the risk from the transport
path to **key distribution**.

The reason the two numbers do not move in the same direction is structural. Encrypting a leg
requires the component at that leg's receiving end to be able to decrypt it; being able to decrypt
requires holding the key. The deeper encryption reaches into the chain, the more copies the key
lands in. The component count also grows toward the end of the chain — the edge has two copies,
the internal proxy three, the app six — and this growth is not a coincidence: the layer that
scales is the app, not the edge.

#### What the Edge Loses

The last column shows the decision's other cost. The edge proxy does five jobs, and four of them
require reading the request's content: path-based routing, the edge cache, the per-path rate
limit, the body-size check. The fifth rests on the server name visible during the handshake, and
it can be done without decrypting.

When termination happens at the edge, all five jobs can be done. When termination moves away from
the edge, **all four jobs drop at once**, and the second and third rows give the same value in
this column: `1/5`.

This is the course's "wrong closure" column, and the wrong closure here is large. The step that
closes the plaintext leg closes the edge cache — every bit of edge hit rate the next lesson
measures drops to zero. It closes the per-path rate limit — an abusive client can now only be
stopped at the app, and by the time it gets there it has crossed the entire chain. It closes the
body-size check — a large body now travels all the way to one of the six app copies.

The question to ask when deciding among the three options is not "which is more secure"; the
numbers do not give that question a single answer. The question is: accept two plaintext legs and
do four jobs at the edge, or close the leg and move those four jobs into the app. When the second
path is chosen, the app has to carry its own rate limit, its own body check, and its own cache,
and that is the new surface closing brings.

#### The Surface Closing Brings

Closing the plaintext leg spawns new items, and they do not show up in the table.

While the certificate sits in two places, a renewal has either been applied to both copies or to
neither. While it sits in six places, a **partial state** is possible: four copies renewed, two
still standing on the old key. This in-between state does not produce a failure — as requests are
distributed, they are met with different certificates depending on which copy they land on, and a
client watching from outside only sees this when it lands on the copy whose certificate has
expired. As the renewal-point count grows, the failure's **probability of showing up** does not
grow, its **time to be found** grows longer.

The second new item is key distribution. A file that can be placed by hand into two copies needs a
distribution path to reach six copies, and that path is a new authority: some component has to be
able to read the key and write it to six places. The key sitting in six places is not one item, it
is seven items — six copies and the single path that writes them.

Third is the certificate becoming tied to the app's version. While termination happens at the app,
renewing the certificate requires reloading the app; two separate calendars — the release calendar
and the renewal calendar — touch the same process.

#### Protocol as an Item Set

The second table measures another default. Once the termination point is decided, the terminating
component decides **which handshakes to accept**, and in most setups no one makes that decision.

The default row is the first row: no floor, every suite accepted. Twenty-eight items are open, and
**nineteen of them are weak**. All seven hundred devices can connect, no reads drop. This is
exactly the default's appeal: zero breakage. No one complains, and the nineteen weak items sit in
a list no one looks at, because no one complains.

Setting the floor to `s2` drops the item count to twenty-six: two items closed. The cost is forty
old handheld terminals and twelve hundred reads a day. The `s3` floor drops it to twenty-one
items, sixty-five devices drop, sixteen hundred reads a day are cut — and **twelve weak items are
still open**.

This is the measure of why the version floor is not enough by itself. The floor closes an item by
looking at its version; a weak suite keeps being accepted under a newer version too. Suite number
eight can be used starting from `s3`, and it is weak; the version floor never closes it.

#### Twelve Items Closed for Free

The fourth row is the measurement's sharpest result. The floor is held fixed at `s3`, and only the
suite list is restricted to the strong ones.

The accepted item count drops from twenty-one to **nine**, the weak item count drops from twelve
to **zero**, and the dropped-device count does not change at all: sixty-five. The dropped-read
count stays the same too. Twelve items closed with no additional breakage.

This is the difference between the two levers. The version floor is expensive: every step drops a
client population. The suite list is cheap: removing weak suites drops no one as long as strong
suites remain that clients speaking the same version can use. The hardening order should follow
this — the suite list first, the version floor second.

Reversing the order does measurable damage. A team that raises the floor first drops sixty-five
devices, the phones ring, the step gets rolled back, and the weak suites stay open too — breakage
happened, no item closed. If the same team had narrowed the suite list first, it would have closed
twelve items with zero breakage, and would then be having the floor argument without those twelve
items in the way.

The fifth row shows where the limit sits. When the floor is raised to `s4`, the item count drops
to six, but three hundred forty devices drop and seventy-nine hundred fifty reads a day are cut.
The price paid to go from nine items to six is losing half the system. The fourth row closed
twelve items with zero additional devices lost; the fifth row closes three items with two hundred
seventy-five additional devices lost. This difference between two steps in the same table shows
why treating the item count alone as a target is misleading.

The item that cannot be closed appears here by name: **forty old handheld terminals**. These
devices cannot speak anything above `s1`, and until they are replaced, the floor cannot rise above
`s1`. The only way out is opening a separate listener for them — accepting the old version only on
that listener. This is not a fix, it is a relocation: the surface item does not close, it changes
address, and one more item gets added to the **listening port** list the previous lesson counted.

#### Summary

- As termination moved from the edge toward the app, the plaintext-leg count dropped from two to
  zero, while the certificate-location count and the renewal-touch-point count climbed from two
  to six.
- The number of components able to see plaintext dropped from three to one; zero plaintext legs
  moved the risk from the transport path to key distribution.
- When termination moved away from the edge, four of the edge proxy's five jobs dropped:
  path-based routing, the edge cache, the per-path rate limit, and the body-size check.
- The default policy accepted twenty-eight items and nineteen were weak; because no device
  dropped, the default was never questioned.
- Setting the version floor to `s3` closed seven items, dropped sixty-five devices, and left
  twelve weak items open; restricting the suite list to the strong ones closed all twelve of the
  remaining weak items **with no additional device loss**.
- Forty old handheld terminals kept the floor from rising above `s1`; opening a separate listener
  for them does not close the item, it adds one to the listening-port list.

#### Next Step

This lesson counted **how many places** the certificate sits in and took the renewal-touch-point
count as equal to that number. Renewal itself stayed a black box: it happens some day, it
finishes, the certificate is refreshed.

Inside that box is a state machine with two settings: the certificate's lifetime and how many days
before expiry renewal starts. If the threshold is set too early, unnecessary renewals run; if it
is set too late, renewal may not finish before expiry, raising the question of **how many hours of
exposure window** are left in between. There is a validation step in the middle of the renewal
flow, and that step can fail; how many attempts are made, on which attempt the certificate
expires, and when the alert should arrive are separate numbers. The next lesson writes the renewal
flow as a state machine in `node`, sweeps lifetime against threshold, models validation failure
and measures the exposure window in hours, and counts how many items manual renewal gets forgotten
on over a period.

### Certificate Automation

Source: https://academia.sh/en/courses/servers-and-hardening/certificate-automation

Writing the renewal flow as a state machine and sweeping lifetime against threshold: the twelve unneeded renewals an early threshold produces, the thirty-eight-hour exposure window a late threshold leaves, five validation-failure attempts and the moment the alert must fire, and manual renewal getting forgotten on nine items over a three-year period, leaving seventeen hundred hours of exposure window.

The previous lesson counted how many places the certificate sits in and took the
renewal-touch-point count as equal to that number. Renewal itself stayed a black box: it happens
some day, it finishes, the certificate is refreshed.

Inside that box is a **state machine**. A request is sent, control of the domain is validated, a
signature is obtained, the new certificate is distributed to the copies. Each of the four steps
takes time, and the second one can fail. Two settings sit outside this machine and decide the
outcome: the certificate's **lifetime** and the **threshold** that says how many hours before
expiry renewal starts.

The measurement runs on the fictional regional measurement network's twelve certificates, over a
three-year period. There is no CA; the steps, their durations, and the certificate list are
fictional.

- **WS19.** Step durations are fixed in hours: request 1, validation 2, signing 1, distribution 2.
  If validation fails, the backoff sequence is `1, 2, 4, 8, 16` hours, and automation gives up
  after the fifth attempt.
- **WS20.** When automation gives up, a human takes over and finishes the job in eight hours;
  every manually performed renewal also takes eight hours.
- **WS21.** The **time to notice** a forgotten manual renewal depends on visibility: an
  external-facing item is noticed in two hours, an internal item in ninety-six hours.
- **WS22.** Forgetting and validation error are produced by our own generator; the seed `4711` is
  visible and gives the same sequence run to run. The forget rate is taken as `0.25`, the
  validation error rate as `0.30`.
- **WS23.** The model does not measure time; every hour value is derived arithmetically from the
  constants above.
- **WS24.** The exposure window is the number of hours by which the renewal flow's duration
  exceeds the threshold; the certificate stands expired for those hours.

```js
// renewal/flow.mjs — model: the certificate renewal flow is a state machine. Duration unit is
// hours, the period is three years. There is no CA; the steps and durations are fictional, every number is derived from them.
const PERIOD = 26280;                      // 3 years, hours
const MANUAL = 8;                          // human takeover and finish (hours)
const BACKOFF = [1, 2, 4, 8, 16];          // wait after a validation failure (hours)
const STEP = { request: 1, validate: 2, sign: 1, distribute: 2 };

// State machine: request -> validate -> (fail -> wait -> validate)* -> sign -> distribute.
// "bad" says which attempt succeeds on; BACKOFF.length means it never succeeds.
function run(bad) {
  let t = STEP.request;
  for (let a = 0; a < BACKOFF.length; a += 1) {
    t += STEP.validate;
    if (a === bad) return { duration: t + STEP.sign + STEP.distribute, result: "done" };
    t += BACKOFF[a];
  }
  return { duration: t + MANUAL, result: "manual takeover" };
}

const DURATION = [0, 4, BACKOFF.length].map((k) => run(k).duration);   // 0 failures, 4 failures, gave up
const window = (threshold, duration) => Math.max(0, duration - threshold);
const renewalCount = (lifetime, threshold) => Math.floor(PERIOD / (lifetime - threshold));
const required = (lifetime) => Math.ceil(PERIOD / lifetime) - 1;

const E = [13, 14, 17, 10, 12, 12, 19, 15];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["lifetime(h)", "threshold(h)", "renewals/period", "unneeded", "window(0)",
  "window(4)", "window(gaveup)", "alert"]));
for (const lifetime of [2160, 8760])
  for (const threshold of [1080, 720, 336, 48, 12]) {
    const n = renewalCount(lifetime, threshold);
    console.log(write([lifetime, threshold, n, n - required(lifetime), ...DURATION.map((s) => window(threshold, s)),
      DURATION[2] > threshold ? "at first fail" : "at giveup"]));
  }

// --- Manual renewal vs. automatic renewal, same period --------------------------
// The fictional measurement network's certificates: [name, lifetime(h), visibility]. Visibility
// decides the time to notice a miss: an external item in hours, an internal item days later.
const NOTICE = { external: 2, internal: 96 };
const CERT = [
  ["edge-proxy-external", 2160, "external"], ["internal-proxy-staging", 2160, "internal"],
  ["internal-proxy-production", 2160, "internal"], ["measurement-app", 8760, "external"],
  ["collector-client", 8760, "internal"], ["billing-client", 8760, "internal"],
  ["mgmt-listener", 8760, "internal"], ["notification-gateway", 2160, "internal"],
  ["backup-target", 8760, "internal"], ["work-order-app", 2160, "external"],
  ["meter-network-gateway", 17520, "internal"], ["dashboard", 2160, "external"],
];
// Our own generator; the seed is visible and gives the same sequence run to run.
const rng = (seed) => { let s = seed >>> 0; return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; };
const SEED = 4711, FORGET_RATE = 0.25, ERROR_RATE = 0.30;

function manual() {
  const r = rng(SEED);
  let events = 0, missed = 0, hours = 0; const items = new Set();
  const breakdown = { external: [0, 0], internal: [0, 0] };  // [missed events, open hours]
  for (const [name, lifetime, vis] of CERT)
    for (let k = 0; k < Math.floor(PERIOD / lifetime); k += 1) {
      events += 1;
      if (r() < FORGET_RATE) {
        missed += 1; hours += NOTICE[vis] + MANUAL; items.add(name);
        breakdown[vis][0] += 1; breakdown[vis][1] += NOTICE[vis] + MANUAL;
      }
    }
  return { events, missed, hours, items: items.size, breakdown };
}

function automatic(threshold) {
  const r = rng(SEED);
  let events = 0, missed = 0, hours = 0; const items = new Set();
  for (const [name, lifetime] of CERT)
    for (let k = 0; k < renewalCount(lifetime, threshold); k += 1) {
      events += 1;
      let a = 0;
      while (a < BACKOFF.length && r() < ERROR_RATE) a += 1;      // which attempt succeeds
      const p = window(threshold, run(a).duration);
      if (a === BACKOFF.length) missed += 1;
      if (p > 0) { hours += p; items.add(name); }
    }
  return { events, missed, hours, items: items.size };
}

const Y = [27, 8, 22, 12, 16];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](Y[i])).join("");
console.log("\n" + write2(["method", "events", "missed/gaveup", "open hours", "affected items"]));
const E0 = manual();
for (const [name, r] of [["manual (no threshold)", E0], ["automatic (threshold 168)", automatic(168)],
  ["automatic (threshold 12)", automatic(12)]])
  console.log(write2([name, r.events, r.missed, r.hours, r.items]));
console.log(`\nbreakdown of manual misses: external ${E0.breakdown.external[0]} events / ${E0.breakdown.external[1]} hours` +
  `   internal ${E0.breakdown.internal[0]} events / ${E0.breakdown.internal[1]} hours`);
```

```
lifetime(h)    threshold(h)  renewals/period  unneeded   window(0)   window(4)     window(gaveup)          alert
2160                   1080               24        12           0           0                  0      at giveup
2160                    720               18         6           0           0                  0      at giveup
2160                    336               14         2           0           0                  0      at giveup
2160                     48               12         0           0           0                  2  at first fail
2160                     12               12         0           0          17                 38  at first fail
8760                   1080                3         1           0           0                  0      at giveup
8760                    720                3         1           0           0                  0      at giveup
8760                    336                3         1           0           0                  0      at giveup
8760                     48                3         1           0           0                  2  at first fail
8760                     12                3         1           0          17                 38  at first fail

method                       events         missed/gaveup  open hours  affected items
manual (no threshold)            88                    29        1700               9
automatic (threshold 168)        94                     0           0               0
automatic (threshold 12)         88                     0          56               8

breakdown of manual misses: external 14 events / 140 hours   internal 15 events / 1560 hours
```

#### The Two Ends of the Threshold

The two ends of the first table produce two separate kinds of failure, and neither looks like a
failure.

**The early end.** For a certificate with a ninety-day lifetime, setting the threshold to
forty-five days starts renewal while the certificate is only halfway through its life. Over the
period, twenty-four renewals run; the required number is twelve. Twelve renewals are
**unneeded**: none of them close an exposure window, and each one produces a request, a
validation, a signature, and a distribution. When the threshold drops to twenty-one days, six
unneeded renewals remain; at fourteen days, two remain.

The cost of an unneeded renewal is paid in two places. The first is the validation step itself;
every renewal is an outside call and an outside dependency. The second is distribution: every
renewal touches every place the certificate sits in, and in the six-copy arrangement the previous
lesson counted, every touch carries a risk of partial state. Twenty-four renewals are twelve
unneeded chances at partial state.

**The late end.** When the threshold drops to forty-eight hours, zero unneeded renewals remain.
This column of the measurement is now clean; the other columns open up. When validation fails
four times, the flow takes twenty-nine hours and fits inside the forty-eight-hour threshold —
window zero. When automation fails all five of five attempts and hands off to a human, the total
takes fifty hours and opens a window of **two hours**.

When the threshold drops to twelve hours, two columns open at once: **seventeen hours** at four
failures, **thirty-eight hours** at giveup. Thirty-eight hours is how long the certificate stands
expired, and every connecting client gets a warning or a rejection during that time.

#### How Lifetime Devalues the Threshold

The table's second half repeats the same five thresholds at a one-year lifetime, and the
`unneeded` column stops moving: one, at all five of the five thresholds. As lifetime stretches
out, the threshold loses its effect on the unneeded-renewal count.

The reason is ratio: a forty-five-day threshold is half of a ninety-day lifetime, an eighth of a
one-year lifetime. At a short lifetime the threshold splits the cycle in two; at a long lifetime
it just clips the end. This does not mean a long lifetime is easier for automation — the `window`
columns are **identical** across the two halves. The exposure window does not depend on lifetime,
only on the threshold and the flow's duration; a twelve-hour threshold on a one-year certificate
leaves the same thirty-eight-hour exposure window as on a ninety-day one.

The real difference between the two halves is in the renewal count: at a ninety-day lifetime,
twelve to twenty-four renewals run over the period; at a one-year lifetime, three. The rarer
renewal is, the less the flow gets tested. A flow that runs three times in three years is only
found broken on its fourth run, and that run is twelve hours before the certificate expires.

#### When the Alert Must Fire

The last column gives not a number but a rule, and the rule is decided by the threshold itself.

Automation's worst-case total duration is forty-two hours: five validation attempts and
thirty-one hours of backoff between them. A human taking over and finishing takes eight hours.
The two together are fifty hours.

If the threshold is greater than fifty hours, automation can try all the way to the end; even
when it gives up, enough time remains for a human, and the alert can fire **at the moment of
giveup**. The first three threshold values are in this situation. If the threshold is smaller than
fifty hours, the situation flips: waiting for automation to try to the end pushes the moment a
human takes over past the certificate's expiry. In this case the alert must fire **at the first
validation failure**.

The rule comes down to this: the alert threshold is not a preference, it is comparing the sum of
`worst-case automatic duration + human duration` against the threshold. At the forty-eight- and
twelve-hour thresholds, automation staying silent is itself a failure — automation runs, tries,
backs off, and no one knows. Silent automation is noticed later than manual renewal is.

#### Manual Renewal's Bill for the Period

The second table counts the same twelve certificates as if renewed manually for three years.

There are eighty-eight renewal events over the period. Twenty-nine were forgotten — one in four.
The misses spread across nine separate items: nine of the twelve certificates expired at least
once. The total exposure window is **seventeen hundred hours**, more than seventy days over three
years.

The breakdown row gives the real finding. External-facing items had fourteen misses, leaving a
total of one hundred forty hours of exposure window. Internal items had fifteen misses and left
**fifteen hundred sixty hours**. The event counts are nearly equal; the hours differ by a factor
of eleven.

The difference comes from a single place: time to notice. When an external-facing certificate
expires, field crews call within two hours. When an internal certificate expires, what happens —
the collector cannot reach the verifier, the nightly batch job writes an alert, the notification
gateway cannot send email — and none of that calls anyone. Ninety-six hours is how long it takes
for the failure to reach a human's eyes.

This shows where the "manual renewal is good enough" defense breaks. The defense is made by
looking at the external-facing certificate, and it is right there: a two-hour window is
acceptable. When the same discipline is applied to internal items, the window stretches
forty-eight times longer, and because no one measures it, that stretch never shows up in any
report.

#### What Automation Closes and Introduces

The last two rows apply automation to the same period.

Automation with a threshold of one hundred sixty-eight hours runs ninety-four renewals — six more
than manual, because the threshold shortens the cycle. No renewal is given up on, the exposure
window is **zero**, affected items zero. The seventeen-hundred-hour window dropped to zero.

Automation with a threshold of twelve hours runs the same eighty-eight events and leaves
fifty-six hours of exposure window; eight items are affected. Again, no renewal is given up on —
all five of five attempts failing at once is a rare event — but renewals left to two or three
attempts do not fit inside the twelve-hour threshold. Automation itself is working correctly; the
threshold is wrong.

Read together, the two rows make clear what automation closed: it closed **forgetting**, not
error. In manual renewal, all seventeen hundred hours came from forgetting. In automation, all of
the remaining fifty-six hours come from validation error, and that error can be absorbed with a
threshold setting.

The new surface closing introduces can also be counted. Automation is a component: it runs with
an identity, it has write authority to every place the certificate sits in, it depends on an
outside validation step, and it carries a threshold in its own configuration. Four items. Manual
renewal had none of these four items; in their place it had a habit that got forgotten on nine
items. Automation **did not change the item count**, it changed the item's type: it turned a task
that had to be remembered into a component that has to be monitored.

#### Summary

- When the threshold was set early, twenty-four renewals ran over the period and twelve were
  unneeded; dropping the threshold to fourteen days brought unneeded renewals to two, and to
  forty-eight hours brought them to zero.
- When the threshold was set late, the window opened: at a twelve-hour threshold, four validation
  failures left a seventeen-hour window, and automation giving up left a thirty-eight-hour window.
- The alert's moment is derived from the threshold: if the sum of the forty-two-hour worst-case
  automatic duration and the eight-hour human duration exceeds the threshold, the alert must fire
  at the first validation failure; if it is smaller, the moment of giveup is enough.
- Over the three-year period, twenty-nine of manual renewal's eighty-eight events were forgotten,
  nine of the twelve certificates expired, and a total of seventeen hundred hours of exposure
  window remained.
- Half the misses were on external-facing items, but only one hundred forty of the open hours
  came from there; the internal items' fifteen misses left fifteen hundred sixty hours — the
  difference is time to notice.
- Automation with a threshold of one hundred sixty-eight hours brought the exposure window to
  zero and introduced four new items in its place: an identity, a write authority, an outside
  dependency, and a threshold setting.

#### Next Step

This lesson and the two before it built and encrypted the request's path: who arrived was
relayed, encryption terminated somewhere, the certificate refreshes itself. The whole path now
runs start to finish for every request — for every request.

The measurement network's dashboard asks for the same regional map, the same meter list, and the
same nightly summary file every time it opens. Most of these requests produce the same response,
and each one travels from the edge proxy to the internal proxy, on to the app, and down to the
data layer. The response can be held somewhere — but "somewhere" is two separate places: at the
edge, close to the client; at the origin, close to the app. When both are put in place, how the
hit rate splits, how many more seconds the old response stays at the edge after content changes,
and how the freshness setting changes that window are separate numbers. The next lesson measures
a two-layer cache with these three numbers and names the surface bad caching opens.

### Cache Server

Source: https://academia.sh/en/courses/servers-and-hardening/cache-server

How the hit rate splits between the edge and the origin in a two-layer cache: splitting the edge into three nodes raising the requests reaching the app from fifty to one hundred forty-three and the origin cache bringing that back down to ten, invalidation's propagation window dropping from three hundred sixty seconds to eleven seconds, the freshness setting's effect on that window, and the shared-key surface opened when a personal response lands in the shared cache.

The last three lessons built and encrypted the request's path: who arrived was relayed,
encryption terminated somewhere, the certificate refreshes itself. The whole path runs start to
finish for every request.

The measurement network's dashboard asks for the same regional map, the same meter list, and the
same nightly summary file every time it opens. Most of these requests produce the same response.
The response can be held somewhere — but "somewhere" is two separate places. **The edge cache**
sits close to the client and never sends the request across the network at all. **The origin
cache** sits in front of the app and keeps the app from generating the same response twice.

What a cache is, how the hit rate is calculated, and what freshness means were measured before;
this lesson does not repeat that mechanics. What is measured here is three things: how the hit
rate **splits** between the two layers, how many more seconds the old response stays at the edge
after a piece of content changes, and the surface a wrong cache key opens.

- **WS25.** The request stream is produced by our own generator; the seed `20260` is visible and
  gives the same sequence run to run. Two thousand requests are spread across six hundred
  seconds.
- **WS26.** There are five pieces of content and their weights are fixed; one (`/invoice/summary`)
  is personal, meaning its response varies by the subscriber making the request.
- **WS27.** A subscriber always lands on the same edge node: the node is the remainder of the
  subscriber's ID divided by the node count.
- **WS28.** The default cache key is **the path alone**; identity does not enter the key. Two
  more options are measured: adding identity to the key, and never storing personal content at
  all.
- **WS29.** The time for the invalidation broadcast to reach the three edge nodes is fictional:
  two, five, and eleven seconds. The window is computed against the slowest node.
- **WS30.** The model does not measure time; the second values are derived from the freshness
  settings and the fictional delays, and the count of requests receiving a stale response is
  computed from the changed content's share of traffic.

```js
// cache/layer.mjs — model: a two-layer cache (edge + origin). The fictional measurement network's
// request stream comes from our own generator; the seed is visible. The cache mechanism is not
// redefined; what is measured is the hit split between the two layers and the invalidation window.
const PERIOD = 600, REQUESTS = 2000, SUBSCRIBERS = 6, SEED = 20260;
// [path, weight, type]
const CONTENT = [
  ["/region/map", 40, "shared"], ["/meter/list", 25, "shared"],
  ["/nightly/summary", 15, "shared"], ["/asset/logo", 10, "shared"],
  ["/invoice/summary", 10, "personal"],
];
const TOTAL = CONTENT.reduce((a, x) => a + x[1], 0);
const rng = (seed) => { let s = seed >>> 0; return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; };
const pick = (u) => { let e = u * TOTAL; return CONTENT.find((x) => (e -= x[1]) < 0) ?? CONTENT[0]; };

function simulate({ node, hasOrigin, edgeFreshness, originFreshness, key }) {
  const r = rng(SEED);
  const edge = Array.from({ length: node }, () => new Map());
  const origin = new Map();
  const s = { edgeHits: 0, originHits: 0, app: 0, leakedRequests: 0 };
  const leakedSubscribers = new Set();
  for (let i = 0; i < REQUESTS; i += 1) {
    const t = Math.floor((i * PERIOD) / REQUESTS);
    const [path, , type] = pick(r());
    const subscriber = Math.floor(r() * SUBSCRIBERS);
    const personal = type === "personal";
    if (personal && key === "personal-bypass") { s.app += 1; continue; }   // personal content is never cached
    const cacheKey = key === "path+id" && personal ? `${path}#${subscriber}` : path;
    const edgeNode = edge[subscriber % node];
    const record = (entry) => {                         // whose response answered the hit
      if (personal && entry.subscriber !== subscriber) { s.leakedRequests += 1; leakedSubscribers.add(entry.subscriber); }
    };
    const hit = edgeNode.get(cacheKey);
    if (hit && t - hit.t < edgeFreshness) { s.edgeHits += 1; record(hit); continue; }
    const originHit = hasOrigin ? origin.get(cacheKey) : undefined;
    if (originHit && t - originHit.t < originFreshness) {
      s.originHits += 1; record(originHit); edgeNode.set(cacheKey, { t, subscriber: originHit.subscriber }); continue;
    }
    s.app += 1;                                          // neither layer hits: the app generates it
    if (hasOrigin) origin.set(cacheKey, { t, subscriber });
    edgeNode.set(cacheKey, { t, subscriber });
  }
  return { ...s, leakedSubscribers: leakedSubscribers.size };
}

const E = [8, 9, 18, 12, 14, 14, 13, 13];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["node", "origin", "key", "edge hits", "origin hits", "reaches app",
  "leaked req", "leaked sub"]));
const SETTING = [
  [1, false, "path"], [3, false, "path"], [3, true, "path"],
  [3, true, "path+id"], [3, true, "personal-bypass"],
];
for (const [node, hasOrigin, key] of SETTING) {
  const o = simulate({ node, hasOrigin, edgeFreshness: 60, originFreshness: 300, key });
  console.log(write([node, hasOrigin ? "yes" : "no", key, o.edgeHits, o.originHits,
    o.app, o.leakedRequests, o.leakedSubscribers]));
}

// --- Invalidation's propagation window -------------------------------------
// Content at the origin changed at t=0. The window is how many seconds the edge keeps the old
// content. Broadcast delay is fictional: it reaches three edge nodes at these seconds, in order.
const DELAY = [2, 5, 11];
const CHANGED = CONTENT[0];                             // changed content: /region/map
const RATE = (REQUESTS / PERIOD) * (CHANGED[1] / TOTAL);   // req/second arriving for that content
function window(method, edgeFreshness, originFreshness) {
  if (method === "none") return edgeFreshness + originFreshness;      // the two freshnesses stack
  if (method === "at-origin") return edgeFreshness;                    // the edge waits out its own duration
  if (method === "every-layer") return Math.max(...DELAY);
  return edgeFreshness;                                                // one node missed the broadcast
}
const P = [18, 16, 18, 12, 24];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](P[i])).join("");
console.log("\n" + write2(["invalidation", "edge freshness", "origin freshness", "window(s)",
  "stale responses served"]));
for (const [y, ef, of] of [["none", 60, 300], ["none", 10, 60], ["at-origin", 60, 300],
  ["at-origin", 10, 300], ["every-layer", 60, 300], ["one-node-missed", 60, 300]]) {
  const p = window(y, ef, of);
  console.log(write2([y, ef, of, p, Math.round(p * RATE)]));
}
```

```
node       origin               key   edge hits   origin hits   reaches app   leaked req   leaked sub
1              no              path        1950             0            50          169            5
3              no              path        1857             0           143           95            6
3             yes              path        1857           133            10          179            2
3             yes           path+id        1837           143            20            0            0
3             yes   personal-bypass        1664           107           229            0            0

invalidation        edge freshness  origin freshness   window(s)  stale responses served
none                            60               300         360                     480
none                            10                60          70                      93
at-origin                       60               300          60                      80
at-origin                       10               300          10                      13
every-layer                     60               300          11                      15
one-node-missed                 60               300          60                      80
```

#### The Cost of Splitting the Edge

The first two rows pass the same two thousand requests through the same freshness settings. The
only thing that changes is the edge node count.

With a single node, the edge hit count is nineteen hundred fifty, and requests reaching the app
are fifty. When the edge splits into three nodes, hits drop to eighteen hundred fifty-seven, and
requests reaching the app climb from **fifty to one hundred forty-three**. Roughly triple.

The reason is direct: every node has its own cache and requests the same content three times,
separately. When a piece of content refreshes, it is no longer one request but three that reach
the app. The edge node count is a scaling decision — more nodes means a response closer to the
client — and it **multiplies the load reaching the app by the node count**.

This is the two-layer cache's reason for existing. The third row adds the origin cache: the edge
hit count does not change (eighteen hundred fifty-seven), one hundred thirty-three requests are
met at the origin cache, and requests reaching the app drop to **ten**. The origin cache absorbs
nearly all of the request the three nodes multiplied.

Read together, the numbers make the origin cache's job visible: it does not reduce the request
coming from the edge itself, it keeps that request from **reaching the app**. The third row's app
load is even lower than the single-node first row's — the origin cache absorbs not just the node
multiplication but a single node's own freshness refills too.

#### The Propagation Window

The second table measures another question. The regional map changed; when does the new map
become visible?

The first row is the default state: no invalidation, both layers wait out their own freshness
duration. The window is sixty plus three hundred, that is, **three hundred sixty seconds**. The
two freshnesses add up because they stack: when the edge pulls a copy from the origin, that copy
may have already waited up to three hundred seconds at the origin, and the edge holds it for
sixty seconds more. Over the window, four hundred eighty requests for the changed content get the
old map.

The second row shortens the freshness settings: ten and sixty instead of sixty and three hundred.
The window drops to seventy seconds, stale responses served drop to ninety-three. The cost does
not show up in this table, but it can be read from the first table: as freshness shortens, the
hit rate drops and requests reaching the app rise. Shortening freshness is the **expensive** way
to close the window.

The third row sends invalidation to the origin. The origin cache clears immediately, the edge
waits out its own sixty seconds: the window is sixty seconds, stale responses served are eighty.
A gain from three hundred sixty down to sixty is achieved with a single message sent to the
origin.

The fourth row sends invalidation to the origin while also dropping the edge freshness to ten:
the window is ten seconds. The fifth row broadcasts invalidation to every layer: the window drops
to the slowest edge node's delay, **eleven seconds**, and edge freshness no longer determines the
window at all.

The order matters here. Shortening the freshness setting closes the window and drops the hit
rate; broadcasting invalidation closes the window and does not touch the hit rate. The fifth row
runs with a sixty-second edge freshness — that is, the first row's hit rate — and its window is
eleven seconds.

#### Where the Broadcast Misses

The sixth row shows what the fifth row cannot close. Invalidation broadcasts to every layer, but
one edge node does not get the message: a network outage, the node restarting, a queue
overflowing. On that node, the window falls back to the edge freshness: **sixty seconds**.

This is the new surface invalidation introduces. The freshness duration provides an **upper
bound**, and it holds even if nothing else is done. The invalidation broadcast does not provide
an upper bound; it provides **a hope**, and when the hope does not hold, the system silently
falls back to the upper bound. The system's state is eleven seconds on two nodes, sixty seconds
on one node, and no one watching from outside knows which node they landed on.

The conclusion is that freshness must be **kept** alongside the invalidation broadcast. The
broadcast provides the fast path, freshness provides the fallback bound. Stretching freshness to
infinity and pinning everything on the broadcast turns a single missed message into indefinitely
stale content.

#### The Shared-Key Surface

The last columns are the lesson's surface measurement. The default cache key is the path alone,
and `/invoice/summary` is personal: every subscriber gets their own invoice summary.

In the first row, one hundred sixty-nine requests received **another subscriber's** invoice
summary; five subscribers' data went to someone else. In the three-node second row, leaked
requests drop to ninety-five, but the leaked-subscriber count climbs to six — splitting does not
reduce the leak, it spreads it.

The third row is the sharpest. When the origin cache is added, leaked requests climb to one
hundred seventy-nine, and the leaked-subscriber count drops to **two**. A single shared origin
entry distributes two subscribers' invoices to all three edge nodes at once. The origin cache
concentrates the leak: fewer people's data, to more people.

The name of this surface is **the shared-key surface**. Opening it does not require a bug; it is
enough for the default key to include the path and not the identity. The knowledge that a
response is personal lives in the app, not in the cache, and neither layer asks for that
knowledge.

The fourth row adds identity to the key: leaked requests zero, leaked subscribers zero. Requests
reaching the app climb from ten to twenty — since six separate entries are held for six separate
subscribers, freshness refills multiply. For a cost of ten requests, one hundred seventy-nine
leaks closed.

The fifth row never stores personal content at all. The leak is zero again, but requests reaching
the app climb to **two hundred twenty-nine**. There is a twentyfold difference between the two
safe options, and both close the same surface. Setting up the key correctly is twenty times
cheaper than turning caching off.

#### The Cache's Own Surface

The two layers dropped the app's load from fifty to ten, and in doing so introduced four new
items.

The first is **the stored response itself**. Already-generated responses now sit at the edge and
the origin; they live somewhere on disk or in memory, and anyone able to read that place reads
the app's responses without authentication. The second is **the invalidation channel**: an
endpoint that can send a message and delete an entry adds two items to the management-endpoint
list the previous lessons counted — one at the edge, one at the origin. If this endpoint has no
identity, an invalidation stream sent from outside can flush the entire cache, and requests
reaching the app climb from ten to two thousand.

The third is **the key rule itself**. As the fifth row showed, the rule is a configuration line,
and getting it wrong throws no error; it just returns another subscriber's response. The fourth
is **the freshness setting being split in two**: the window's real value comes out of two
separate numbers on two separate layers, and no single layer knows that value by itself.

None of the four items show up in the sentence "the cache is on." The course's rule holds here
too: a decision cannot be defended unless the load it closes is written down alongside the
surface it opens. If a step that reduces the requests reaching the app by forty items adds four
items of open surface, both numbers belong on the same line of the report.

#### Summary

- Splitting the edge into three nodes dropped edge hits from nineteen hundred fifty to eighteen
  hundred fifty-seven and raised requests reaching the app from fifty to one hundred forty-three;
  the node count multiplies the app's load.
- The origin cache met one hundred thirty-three requests and dropped requests reaching the app to
  ten — lower even than the single-node arrangement; the origin layer's job is absorbing the
  request the edge multiplied.
- With no invalidation, the window is the sum of the two freshnesses: three hundred sixty seconds
  and four hundred eighty stale responses. Invalidation to the origin dropped the window to
  sixty; broadcasting to every layer dropped it to eleven seconds.
- Shortening freshness closes the window and drops the hit rate; broadcasting invalidation closes
  the window and does not touch the hit rate — but on a node that misses the broadcast, the
  window silently falls back to the freshness duration.
- Because the default key contained only the path, one hundred seventy-nine requests received
  another subscriber's invoice summary; the origin cache concentrated the leak — two subscribers'
  data went out in one hundred seventy-nine requests.
- Adding identity to the key zeroed the leak and raised the app load to twenty, not ten; never
  storing personal content also zeroed the leak but raised the load to two hundred twenty-nine.

#### Next Step

This lesson set two rules on traffic at the edge and the origin: what gets stored and for how
long it stays fresh. Both rules sat **in the request's path** and applied to traffic coming from
outside.

Services also send requests to each other. The verifier connects to billing, billing to work
orders, work orders to the notification gateway, and none of that traffic passes through the edge
proxy. Timeout, retry, which service can call which — today these rules sit in each service's own
code. The same rules can be moved to a proxy placed next to each service. The next lesson compares
the two arrangements on the same rule change: how many services get touched when a rule changes,
how many hops and how many milliseconds get added to every request, what the sidecar proxy itself
consumes, and what new surface it introduces. In a three-service system, the point where the cost
outweighs the gain is shown with numbers.

### Service Mesh

Source: https://academia.sh/en/courses/servers-and-hardening/service-mesh

Comparing the same traffic rules applied in application code and in a sidecar proxy: the number of services touched and the annual work items when a rule changes, measuring a real sidecar hop, the hop and latency added to every request, sidecar memory and core consumption, the new surface items they introduce, and the break-even point in a three-service system where the cost outweighs the gain.

The previous lesson set two rules on traffic at the edge and the origin. Both sat in the
request's path and applied to traffic coming from outside.

Services also send requests to each other. In the fictional measurement network, the verifier
connects to billing, billing to work orders, work orders to the notification gateway, and none of
that traffic passes through the edge proxy. These calls have rules too: how many seconds the
timeout is, how many times to retry, which service can call which, whether the call travels
encrypted.

These rules can go in two places. **In application code**: each service configures its own
client. **In a sidecar proxy**: a proxy is placed next to each service instance, all outbound and
inbound traffic passes through that proxy, and the rules are written into the proxy. The name of
the second arrangement is **service mesh**.

This lesson compares the two arrangements on the same rule change and asks a single question: is
what the sidecar gains bigger than what it costs?

- **WS31.** Every service has two instances, and one sidecar goes to each instance.
- **WS32.** A "work item" is a fictional unit: a change, a review, or a release. In the code
  arrangement, a rule change costs three work items per service; in the mesh arrangement, two
  work items total.
- **WS33.** The mesh arrangement's fixed annual cost is forty work items (the control plane), and
  every sidecar instance adds two work items a year (version upgrade, identity rotation). Six
  traffic rules change per year.
- **WS34.** A sidecar instance consumes sixty-four megabytes of memory and a tenth of a core; the
  latency a hop adds is fictionally ten times forty microseconds, that is, `0.4` milliseconds.
- **WS35.** The real measurement block does not print duration: the expected hop and connection
  counts are written as derived from the input, and the measured extra latency is written as
  `yes` or `no` based only on whether it falls between zero and fifty milliseconds.
- **WS36.** The new-surface-item count is this: one management endpoint and one identity per
  sidecar instance, plus the control plane and the rule store.

#### The Hop Itself

The first thing a sidecar adds to every request is a hop. The block below does not model this, it
actually runs it: an app server and a sidecar proxy placed in front of it are set up, and both
paths are tested with the same number of requests.

```js
// sidecar/hop.mjs — real measurement: the hop a sidecar proxy adds to a request.
// Servers are set up in this file and closed when the measurement ends. Raw duration is not
// printed: expected values are derived from the input, the measured extra latency only goes through a band check.
import { createServer, request } from "node:http";

const REQUESTS = 200, WARMUP = 50, BAND = [0, 50];         // band: milliseconds
const APP = 8951, SIDECAR = 8950;
const app = createServer((i, y) => { y.sendDate = false; y.end("measurement"); }).listen(APP, "127.0.0.1");
const sidecar = createServer((inReq, inRes) => {           // one job: carry the request one hop onward
  inRes.sendDate = false;
  const outReq = request({ port: APP, path: inReq.url, method: inReq.method, headers: inReq.headers },
    (outRes) => { inRes.writeHead(outRes.statusCode, outRes.headers); outRes.pipe(inRes); });
  outReq.on("error", () => inRes.writeHead(502).end("cut off"));
  inReq.pipe(outReq);
}).listen(SIDECAR, "127.0.0.1");

const call = (port) => new Promise((resolve) => {
  request({ port, path: "/measurement", agent: false }, (y) => { y.resume(); y.on("end", resolve); }).end();
});
async function measure(port, n) {                          // ms per request
  const b = process.hrtime.bigint();
  for (let i = 0; i < n; i += 1) await call(port);
  return Number(process.hrtime.bigint() - b) / 1e6 / n;
}

await measure(APP, WARMUP); await measure(SIDECAR, WARMUP);   // warmup: both paths run once
const extra = (await measure(SIDECAR, REQUESTS)) - (await measure(APP, REQUESTS));
app.close(); sidecar.close();

console.log(`requests / path                    ${REQUESTS}`);
console.log(`expected hops                     direct 1, sidecar 2   (diff 1)`);
console.log(`expected connections / request     direct 1, sidecar 2   (diff 1)`);
console.log(`extra latency ${BAND[0]}-${BAND[1]} ms band          ${extra >= BAND[0] && extra <= BAND[1] ? "yes" : "no"}`);
```

```
requests / path                    200
expected hops                     direct 1, sidecar 2   (diff 1)
expected connections / request     direct 1, sidecar 2   (diff 1)
extra latency 0-50 ms band          yes
```

The measured duration does not go into the output, because it varies machine to machine and
cannot be compared. Two numbers derived from the input do not change: on the sidecar path, every
request spends **two hops** and **two connections**, on the direct path, one. This is for one
direction of a call; in a call between two services, since a sidecar stands at both ends, the
extra-hop count climbs to two.

#### What a Rule Change Touches

The second block keeps the ledger for the two arrangements.

```js
// sidecar/cost.mjs — model: applying the same traffic rules in two places. "Work item" is a
// fictional unit (a change, a review, a release); every number is derived from the constants
// below. There is no product, there is an arrangement.
const INSTANCES = 2;        // instance count per service
const CODE_ITEM = 3;        // per service in the code arrangement: change + review + release
const MESH_ITEM = 2;        // per rule in the mesh arrangement, total: config + rollout check
const CONTROL_PLANE = 40;   // control plane's fixed annual work items
const SIDECAR_UPKEEP = 2;   // per sidecar instance, annual: version upgrade + identity rotation
const RULE = 6;             // traffic rules changed per year
const HOP_MS = 0.4;         // fictional: latency a sidecar hop adds
const MEMORY = 64, CORES = 0.1;   // per sidecar instance

const SERVICE = [3, 5, 8, 20, 40];
const codeWork = (n) => RULE * n * CODE_ITEM;
const meshWork = (n) => RULE * MESH_ITEM + CONTROL_PLANE + n * INSTANCES * SIDECAR_UPKEEP;

const E = [9, 11, 17, 14, 15, 14, 12, 15, 10];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["service", "instances", "code touched", "code work/yr", "mesh touched",
  "mesh rule/yr", "mesh fixed", "mesh total/yr", "winner"]));
for (const n of SERVICE)
  console.log(write([n, n * INSTANCES, n, codeWork(n), 0, RULE * MESH_ITEM,
    CONTROL_PLANE + n * INSTANCES * SIDECAR_UPKEEP, meshWork(n), codeWork(n) < meshWork(n) ? "code" : "mesh"]));
const breakeven = [...Array(60).keys()].map((i) => i + 1).find((n) => codeWork(n) >= meshWork(n));
console.log(`break-even service count: ${breakeven}  (below this count, the code arrangement is cheaper)`);

// Per-request cost and the new surface the sidecar introduces.
const P = [9, 20, 12, 19, 16, 12, 20];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](P[i])).join("");
console.log("\n" + write2(["service", "internal calls/req", "extra hops", "extra latency(ms)",
  "sidecar MB", "cores", "new surface items"]));
for (const n of SERVICE) {
  const calls = Math.min(n - 1, 4), hops = 2 * (calls + 1), instances = n * INSTANCES;
  console.log(write2([n, calls, hops, (hops * HOP_MS).toFixed(1), instances * MEMORY,
    (instances * CORES).toFixed(1), instances * 2 + 2]));
}
```

```
service    instances     code touched  code work/yr   mesh touched  mesh rule/yr  mesh fixed  mesh total/yr    winner
3                  6                3            54              0            12          52             64      code
5                 10                5            90              0            12          60             72      mesh
8                 16                8           144              0            12          72             84      mesh
20                40               20           360              0            12         120            132      mesh
40                80               40           720              0            12         200            212      mesh
break-even service count: 4  (below this count, the code arrangement is cheaper)

service    internal calls/req  extra hops  extra latency(ms)      sidecar MB       cores   new surface items
3                           2           6                2.4             384         0.6                  14
5                           4          10                4.0             640         1.0                  22
8                           4          10                4.0            1024         1.6                  34
20                          4          10                4.0            2560         4.0                  82
40                          4          10                4.0            5120         8.0                 162
```

#### The Number of Services Touched

The first table's real column is the third one. A traffic rule — say, the timeout drops from
five seconds to two — touches **each service separately** in the code arrangement. Three touches
in a three-service system, forty in a forty-service one. Every touch is a change, a review, and a
release; in a forty-service system, a single rule change costs one hundred twenty work items, and
because the releases are independent of each other, **there is no moment when the rule is in
effect everywhere at once**.

In the mesh arrangement, the number of services touched is **zero**. The rule is written once,
distributed to the sidecars, and the application code does not change. Two work items per rule
change, independent of the service count. In a forty-service system, the annual seven hundred
twenty work items drop to two hundred twelve.

Looking at this one column alone, the service mesh wins at every scale. The other columns break
that.

#### Three Services: Cost Outweighs Gain

The first row is where the measurement breaks. In a three-service system, the code arrangement
costs fifty-four work items a year, the mesh arrangement sixty-four. The mesh is **more
expensive**, and the reason is the fixed cost: the control plane's forty work items are bigger
than three services' total rule-change cost.

The break-even service count is four. Below that, the work gained from rule changes is smaller
than the control plane's upkeep.

The second table reads the same row from the other direction. In a three-service system, the
sidecars consume three hundred eighty-four megabytes of memory and six-tenths of a core, add six
hops and two-point-four milliseconds to every request, and introduce **fourteen new surface
items**. None of these three costs show up in the work-item ledger.

This is the case where the service mesh gains nothing, and it is not rare: three-service systems
are the majority. Where the decision is made with the sentence "a service mesh is good practice,"
the price paid is not the return on six gained work items — it is minus ten work items plus
fourteen surface items.

#### What the Central Rule Wrongly Closes

The zero in the `mesh touched` column depends on one condition: the rule fitting **every service
the same way**. That condition does not hold for most rules.

The rule that drops the timeout to two seconds is correct for thirty-eight services; it is not
correct for the service that calls the nightly batch job, because that call is already slow. The
rule written as a single line at the center cuts off all of that service's calls at two seconds.
This problem never arises in the code arrangement — the rule is already written separately in
each service, and that service keeps its own value. In the central arrangement, the fix is an
exception, and an exception costs as many work items as the rule itself: by the model's
constants, every exception costs two work items a year, just like a rule does.

This is the service mesh's counterpart to the course's "wrong closure" column. The step that
drops the number of services touched by a rule change to zero does not remove the question of
**how well the rule fits which service**; it moves that question from the codebase to a
configuration file, where its visibility drops. A service owning its own exception in its own
code gets seen in a review; an exception in a central rule set depends on the attention of
whoever reads the set.

In numbers, the result is this: as the exception count rises, the mesh side's annual work items
grow the way the rule count does, and the break-even service count shifts up from four. In a
system with ten exceptions, the mesh side costs twenty more work items a year, and the break-even
point climbs to six services.

#### What Gets Paid Per Request

The middle of the second table does not **saturate** as scale grows. The internal-call count
settles at four — chain length does not grow without limit with the service count — and extra
hops settle at ten, extra latency at four milliseconds. This is the service mesh's good news: the
per-request cost does not grow with scale.

Four milliseconds looks small. The measurement network's nightly batch job makes hundreds of
thousands of internal calls on every run, and there four milliseconds turns into hours. On a
dashboard request that passes through the edge cache, it never shows up at all. What the extra
latency means is decided by the call count, not by the number itself.

Resource consumption, though, grows **linearly** with scale. In a forty-service system, the
sidecars hold five gigabytes of memory and eight cores. That is the resources of eight app
instances; the service mesh sets aside 15% to 20% of the system's hardware just to run, and this
line never comes up in any rule discussion.

#### The Sidecar's Own Surface

The last column is the course's unit of measure. Every sidecar instance introduces two items.

The first is **the management endpoint**. It shows the sidecar's configuration, dumps its
metrics, reloads itself. This endpoint listens next to the app, in the same network namespace;
the "management endpoint" item the first lesson counted now exists not per machine but **per
instance**. Eighty management endpoints in a forty-service system.

The second is **identity**. Sidecars speak to each other encrypted, and each one has a
certificate. This is the extreme case of the third lesson's "terminating at the app" row: the
certificate now sits in eighty places, not six. The fourth lesson's automation becomes mandatory
here — renewing eighty certificates by hand gets forgotten on eighty items, not nine.

The third is a single item and the heaviest: **the control plane**. The component that
distributes rules to every sidecar is a single place with write authority to every service.
Together with the rule store, that is two items, and neither can be closed — they are the
service mesh's reason for existing.

This count, climbing from fourteen items to one hundred sixty-two in a forty-service system, has
to be placed next to the seventeen items the first lesson counted on a single machine. The
service mesh collected the traffic rules in one place and spread the management endpoints
everywhere.

#### Summary

- On a rule change, the code arrangement touches every service (three for three services, forty
  for forty), the mesh arrangement touches none; in a forty-service system, the annual work-item
  count drops from seven hundred twenty to two hundred twelve.
- The real measurement confirmed that the sidecar path adds one hop and one connection to every
  request; the measured extra latency stayed inside the zero-to-fifty-millisecond band, and raw
  duration was not printed.
- In a three-service system, the mesh arrangement is more expensive than the code arrangement:
  sixty-four work items against fifty-four. The break-even service count is four, and the
  control plane's fixed forty work items decide that threshold.
- The per-request cost does not saturate with scale — ten hops, four milliseconds — but resource
  consumption grows linearly: five gigabytes of memory and eight cores at forty services.
- The sidecar adds two surface items to every instance (management endpoint and identity);
  fourteen items at three services, one hundred sixty-two at forty. The control plane and the
  rule store are items that cannot be closed.

#### Next Step

This topic built the request's path from start to finish. The path passes through three roles,
the four pieces of information lost passing through the proxy chain get relayed, encryption
terminates at one point and its certificate refreshes itself, the response gets stored in two
layers, and inter-service traffic gets bound to a single rule set. All of these measurements took
one thing as given.

The client connected somewhere. Where it connected to had **a name**, and that name resolved to
an address. The second lesson used a name, `measurement.example.invalid`; the third lesson
assumed the certificate was issued for that name; the fifth lesson never asked how a client
landing on an edge node got there. The name itself, which address that name points to, how long
the old one stays valid after it changes, and who that name is registered to were not measured
once in this topic. The next topic takes on the name itself: the default state of the record set,
the consequences of treating the lifetime as a decision, and who owns the record that carries the
name.

## Domain and Email Records

Verifying the name and the sender: the zone file, record types, and the effect of time to live on the change window, sender verification through the sending policy and signature records, tightening the policy declaration in stages for non-compliant messages, and measuring reputation when delivery goes wrong.

### DNS Management

Source: https://academia.sh/en/courses/servers-and-hardening/dns-management

Counting the zone file as an inventory, the takeover surface dangling records leave behind, and measuring time to live as a decision: the propagation window, rollback speed, and query load.

The request's path is set up: the connection terminated at some point, headers were
forwarded, the response came back from a cache. Across this whole chain, one question never
gets asked — which name did the request arrive **at**, and who owns that name?

The name comes before the machine. A client knows the name, not the address; it learns the
address from someone else and keeps the answer it learned for a while. These two sentences
turn domain records from a configuration detail into a surface worth measuring. This lesson
counts that surface along two axes: the zone file is an **inventory**, and the inventory's
drift from reality leaves a countable opening; **time to live**, meanwhile, is not a default
left over from setup but a decision that sets three values at once.

What the record types are — address, canonical name, mail exchange, name server, text — was
established in the Computer Networks course. Here the types are referred to by name and not
redefined; what gets measured is not the type's definition but the record's **upkeep** and
**lifespan**.

#### A Zone Is an Inventory

A zone is the full set of records an authoritative server is responsible for. On the
operations side, it is something more: every target written into the zone points at
something once assumed to exist. Systems get migrated, trials get shut down, contracts with
outside services end — the record stays.

The program below builds the regional measurement network's zone as a **model**. The network
is fictional: a collector that gathers water-meter readings, a validator, and billing and
work-order services. The model keeps two lists apart: the targets written in the zone and the
addresses that are genuinely up.

**DR1.** The zone is under a single authoritative set and records are managed by hand.
**DR2.** The live inventory is kept independent of the zone; no process reconciles the two lists.

```javascript
// Model: the fictional zone of the regional measurement network. Not a real
// zone; a data structure carrying each record's name, type, TTL, and target.
const ZONE = [
  { name: 'measurenet.invalid',             type: 'NS',    ttl: 86400, target: 'ns1.measurenet.invalid' },
  { name: 'measurenet.invalid',             type: 'NS',    ttl: 86400, target: 'ns2.measurenet.invalid' },
  { name: 'measurenet.invalid',             type: 'A',     ttl: 3600,  target: '198.51.100.10' },
  { name: 'measurenet.invalid',             type: 'MX',    ttl: 3600,  target: 'mail.measurenet.invalid' },
  { name: 'measurenet.invalid',             type: 'TXT',   ttl: 3600,  target: '(text)' },
  { name: 'www.measurenet.invalid',         type: 'CNAME', ttl: 3600,  target: 'measurenet.invalid' },
  { name: 'mail.measurenet.invalid',        type: 'A',     ttl: 3600,  target: '198.51.100.20' },
  { name: 'ns1.measurenet.invalid',         type: 'A',     ttl: 86400, target: '198.51.100.2' },
  { name: 'ns2.measurenet.invalid',         type: 'A',     ttl: 86400, target: '198.51.100.3' },
  { name: 'collector.measurenet.invalid',   type: 'A',     ttl: 300,   target: '198.51.100.11' },
  { name: 'validator.measurenet.invalid',   type: 'A',     ttl: 300,   target: '198.51.100.12' },
  { name: 'billing.measurenet.invalid',     type: 'A',     ttl: 3600,  target: '198.51.100.13' },
  { name: 'workorder.measurenet.invalid',   type: 'A',     ttl: 3600,  target: '198.51.100.14' },
  { name: 'field.measurenet.invalid',       type: 'CNAME', ttl: 300,   target: 'workorder.measurenet.invalid' },
  { name: 'dashboard.measurenet.invalid',   type: 'CNAME', ttl: 3600,  target: 'legacy-dashboard.measurenet.invalid' },
  { name: 'rehearsal.measurenet.invalid',   type: 'A',     ttl: 86400, target: '198.51.100.71' },
  { name: 'trial.measurenet.invalid',       type: 'A',     ttl: 86400, target: '198.51.100.72' },
  { name: 'report.measurenet.invalid',      type: 'CNAME', ttl: 3600,  target: 'temp.external.invalid' },
];

// Live inventory: addresses that are genuinely up right now.
const LIVE_ADDRESSES = new Set([
  '198.51.100.2', '198.51.100.3', '198.51.100.10', '198.51.100.11',
  '198.51.100.12', '198.51.100.13', '198.51.100.14', '198.51.100.20',
]);
const IN_ZONE = 'measurenet.invalid';

function resolve(name, depth = 0) {
  if (depth > 8) return { status: 'loop' };
  if (!name.endsWith(IN_ZONE)) return { status: 'out-of-zone' };
  const address = ZONE.find((k) => k.name === name && k.type === 'A');
  if (address) {
    return LIVE_ADDRESSES.has(address.target)
      ? { status: 'live', address: address.target }
      : { status: 'dead-address', address: address.target };
  }
  const alias = ZONE.find((k) => k.name === name && k.type === 'CNAME');
  if (alias) return resolve(alias.target, depth + 1);
  return { status: 'no-record' };
}

const typeDistribution = {};
const ttlDistribution = {};
for (const k of ZONE) {
  typeDistribution[k.type] = (typeDistribution[k.type] || 0) + 1;
  ttlDistribution[k.ttl] = (ttlDistribution[k.ttl] || 0) + 1;
}
console.log('total records    :', ZONE.length);
console.log('distinct names   :', new Set(ZONE.map((k) => k.name)).size);
console.log('type distribution:', Object.entries(typeDistribution).map(([t, n]) => `${t}=${n}`).join(' '));
console.log('ttl distribution :', Object.keys(ttlDistribution).map(Number).sort((a, b) => a - b)
  .map((t) => `${t}s=${ttlDistribution[t]}`).join(' '));

const targeted = ZONE.filter((k) => k.type !== 'TXT');
const dangling = [];
for (const k of targeted) {
  const s = /^\d/.test(k.target)
    ? (LIVE_ADDRESSES.has(k.target) ? { status: 'live' } : { status: 'dead-address' })
    : resolve(k.target);
  if (s.status !== 'live') dangling.push({ ...k, status: s.status });
}

console.log('\ndangling records:', dangling.length, '/', targeted.length);
for (const k of dangling) {
  console.log(`  ${k.name.padEnd(30)} ${k.type.padEnd(5)} -> ${k.target.padEnd(37)} ${k.status}`);
}
const claimable = dangling.filter((k) => k.status === 'no-record' || k.status === 'out-of-zone');
console.log('\nempty target someone else could claim:', claimable.length);
console.log('of these, targets outside the zone   :', dangling.filter((k) => k.status === 'out-of-zone').length);
```

```
total records    : 18
distinct names   : 14
type distribution: NS=2 A=10 MX=1 TXT=1 CNAME=4
ttl distribution : 300s=3 3600s=9 86400s=6

dangling records: 4 / 17
  dashboard.measurenet.invalid   CNAME -> legacy-dashboard.measurenet.invalid   no-record
  rehearsal.measurenet.invalid   A     -> 198.51.100.71                         dead-address
  trial.measurenet.invalid       A     -> 198.51.100.72                         dead-address
  report.measurenet.invalid      CNAME -> temp.external.invalid                 out-of-zone

empty target someone else could claim: 2
of these, targets outside the zone   : 1
```

This is the first item the default leaves open: four of the seventeen targeted records point
at something that no longer exists. Two point at a dead address; those are just noise —
resolution ends and no connection is made. The other two are different, and they are the ones
that actually earn the name **dangling record**: the record points at an empty **name**, and
that name belongs to no one. Whoever claims the name first, a subdomain under
`measurenet.invalid` starts showing their content. This is the record being **taken over**;
from the browser's point of view, the content is still under the measurement network's name.
A record pointing outside the zone is heavier still, because claiming that name requires
touching none of the measurement network's own systems.

The zone itself shows a second surface in the count: **fourteen distinct names**. Every name
written in the zone makes the network's internal structure readable — which services exist,
which environment stands apart. Record names are a reconnaissance resource, and this item
cannot be closed; a zone exists to be published.

#### Time to Live Is a Decision

Every record has a time to live, and most records go untouched during setup. Yet this single
number sets three separate things at once: a change's **propagation window** (how long the
stale answer stays in circulation), a bad change's **rollback speed**, and the **query load**
landing on the authoritative server. The three don't improve in the same direction — lower
the TTL and the first two win while the third loses.

**DR3.** Two hundred forty resolver caches request the record continuously; at the moment of
change, the age of each cached copy is uniformly distributed within the TTL.
**DR4.** The load landing on the authoritative server comes only from cache refreshes.

```javascript
// Model: a TTL sweep. At the moment of change, the age of each cached copy
// is uniform on [0, ttl); the stale answer stays in circulation for the rest.
const CACHES = 240;            // fictional resolver cache count
const SEED = 4242;             // generator seed, visible on purpose

function prng(seed) {
  let x = seed % 2147483647;
  return () => { x = (x * 48271) % 2147483647; return (x - 1) / 2147483646; };
}

function window(ttl) {
  const rand = prng(SEED);
  const remaining = [];
  for (let i = 0; i < CACHES; i++) remaining.push(Math.round(ttl * (1 - rand())));
  remaining.sort((a, b) => a - b);
  const pct = (p) => remaining[Math.min(remaining.length - 1, Math.floor(remaining.length * p))];
  const staleAfter = (t) => remaining.filter((k) => k > t).length;
  return {
    p50: pct(0.50), p95: pct(0.95), longest: remaining[remaining.length - 1],
    stillOld5min: staleAfter(300), stillOld1h: staleAfter(3600),
    dailyQueries: Math.round(CACHES * (86400 / ttl)),
  };
}

const duration = (s) => (s < 90 ? `${s}s` : s < 5400 ? `${(s / 60).toFixed(0)}min` : `${(s / 3600).toFixed(1)}h`);

console.log('cache count:', CACHES, '| seed:', SEED);
console.log('ttl      propagation p50  propagation p95  longest   still old after 5min  still old after 1h  daily queries');
for (const ttl of [60, 300, 900, 3600, 21600, 86400]) {
  const o = window(ttl);
  console.log(
    duration(ttl).padEnd(9) + duration(o.p50).padEnd(17) + duration(o.p95).padEnd(17) +
    duration(o.longest).padEnd(10) + `${o.stillOld5min}/${CACHES}`.padEnd(22) +
    `${o.stillOld1h}/${CACHES}`.padEnd(20) + String(o.dailyQueries),
  );
}
```

```
cache count: 240 | seed: 4242
ttl      propagation p50  propagation p95  longest   still old after 5min  still old after 1h  daily queries
60s      29s              55s              60s       0/240                 0/240               345600
5min     2min             5min             5min      0/240                 0/240               69120
15min    7min             14min            15min     156/240               0/240               23040
60min    29min            55min            60min     215/240               0/240               5760
6.0h     2.9h             5.5h             6.0h      233/240               202/240             960
24.0h    11.6h            22.0h            23.9h     239/240               230/240             240
```

The table reads as one line of trade-off. When an address change happens within a one-day
TTL, 230 of the caches still return the stale answer an hour later; the longest wait runs to
roughly twenty-four hours. At a five-minute TTL, the same change finishes within five
minutes, at a cost of 69,120 queries a day instead of 240 — **288 times as many**. The values
in between are linear: halve the TTL and the window halves, the query count doubles.

The rule that follows from this isn't the TTL itself but the **record's likelihood of
changing**. Name server records don't change for years; a long TTL is the right call for them
and it lowers the query load. Address records are subject to being moved. In the model's
zone, the TTL distribution doesn't reflect this distinction at all: nine records sit at one
hour, six sit at one day, and among them are both name server records that never change and
movable trial addresses.

#### Before a Planned Change

The TTL is lowered **before** a change. Why this works, and why it stops working when done
halfway, can both be counted.

**DR5.** The field request rate is 8 per second and is spread evenly across caches.
**DR6.** At the moment of change, the new answer is served from all authoritative servers at
once; replication delay between authoritative servers is not counted in this model.

```javascript
// Model: the cost of lowering the TTL before a planned address change. On the
// time axis, 0 is the moment of change; leadWindow is the lead time, in seconds.
const CACHES = 240, SEED = 4242, OLD_TTL = 3600, LOW_TTL = 300, REQUEST_RATE = 8;

function prng(seed) {
  let x = seed % 2147483647;
  return () => { x = (x * 48271) % 2147483647; return (x - 1) / 2147483646; };
}

function scenario(name, leadWindow, ttlLowered) {
  const rand = prng(SEED);
  const remaining = [];
  for (let i = 0; i < CACHES; i++) {
    let untilExpiry = -leadWindow + rand() * OLD_TTL;
    while (untilExpiry <= 0) untilExpiry += ttlLowered ? LOW_TTL : OLD_TTL;
    remaining.push(Math.round(untilExpiry));
  }
  remaining.sort((a, b) => a - b);
  const pct = (p) => remaining[Math.floor(remaining.length * p)] ?? remaining[remaining.length - 1];
  return {
    name, leadWindow, p50: pct(0.5), p95: pct(0.95),
    misrouted: Math.round(remaining.reduce((a, b) => a + b, 0) * (REQUEST_RATE / CACHES)),
    extraQueries: ttlLowered ? Math.round(CACHES * leadWindow * (1 / LOW_TTL - 1 / OLD_TTL)) : 0,
    rollbackTtl: ttlLowered ? LOW_TTL : OLD_TTL,
  };
}

const duration = (s) => (s < 90 ? `${s}s` : s < 5400 ? `${Math.round(s / 60)}min` : `${(s / 3600).toFixed(1)}h`);
const scenarios = [
  scenario('ttl untouched', 0, false),
  scenario('lowered 30min early', 1800, true),
  scenario('lowered 24h early', 86400, true),
];

console.log('request rate:', REQUEST_RATE, 'req/s | caches:', CACHES, '| seed:', SEED);
console.log('scenario              lead window  extra queries  propagation p50  propagation p95  misrouted        rollback');
for (const s of scenarios) {
  console.log(
    s.name.padEnd(22) + (s.leadWindow ? duration(s.leadWindow) : '-').padEnd(13) +
    (s.extraQueries ? String(s.extraQueries) : '-').padEnd(15) + duration(s.p50).padEnd(17) +
    duration(s.p95).padEnd(17) + String(s.misrouted).padEnd(17) + duration(s.rollbackTtl),
  );
}

const neverRestoredDaily = Math.round(CACHES * 86400 * (1 / LOW_TTL - 1 / OLD_TTL));
console.log('\nextra daily authoritative queries if the ttl is never raised back:', neverRestoredDaily);
console.log('unplanned rollback window (p95) if lowering the ttl is forgotten:', duration(scenarios[0].p95));
```

```
request rate: 8 req/s | caches: 240 | seed: 4242
scenario              lead window  extra queries  propagation p50  propagation p95  misrouted        rollback
ttl untouched         -            -              32min            57min            14812            60min
lowered 30min early   30min        1320           5min             27min            4292             5min
lowered 24h early     24.0h        63360          2min             5min             1142             5min

extra daily authoritative queries if the ttl is never raised back: 63360
unplanned rollback window (p95) if lowering the ttl is forgotten: 57min
```

The middle row is the real lesson. When the TTL is lowered thirty minutes ahead, the job is
only half done: part of the caches refreshed within those thirty minutes and picked up the
short TTL, the rest still hold the one-hour copy. The result is a win at p50 but a
**twenty-seven-minute tail at p95**. The short TTL only settles everywhere once a full period
of the old TTL has passed; if the lead window is shorter than the old TTL, the lowering stays
on paper.

The cost has two items. A twenty-four-hour lead window brings 63,360 extra authoritative
queries, and those queries are paid whether the change happens or not. The second is
sneakier: if the TTL isn't raised back after the change, the same 63,360 queries keep getting
paid **every day** — and no one treats it as an incident, because nothing breaks.

The cost of forgetting to lower it shows up in the unplanned case. When a bad record gets
published, the rollback speed is whatever TTL is in effect at that moment: with a one-hour
TTL, 14,812 requests go to the wrong address; with a five-minute TTL, 1,142. The 13,670
requests in between are the result not of a domain setting but of a decision nobody made.

#### The New Surface Closing Brings

Both steps carry their own surface. Cleaning up dangling records closes four items, but if
some job still calls the rehearsal or trial names from somewhere, it stops two legitimate
flows; that is why cleanup calls for a query count before deleting, and that count lives not
in the zone but in the resolver logs. If an automation that can write to the zone is set up
to keep the inventory reconciled continuously, a new **privileged identity** appears:
whatever can write a record to the zone is the authority for that name.

Lowering the TTL has a measurable surface too. A five-minute TTL leaves only a five-minute
buffer when the authoritative server becomes unreachable; at a one-day TTL, the same outage
would have gone unnoticed for hours. Closing the propagation window makes the authoritative
server's availability critical.

#### Summary

- The zone file is an inventory; four of the model's seventeen targeted records point at a
  target that no longer exists, and two point at an empty name and so are open to takeover.
- The fourteen distinct names written in the zone are a reconnaissance surface, and this item
  cannot be closed; a zone exists to be published.
- Time to live alone sets three things: the propagation window, the rollback speed, and the
  query load. Dropping from a day to five minutes pulls the window from 22 hours to 5 minutes
  and multiplies the query count by 288.
- When the lead window is shorter than the old TTL, the lowering stays half-done: with a
  thirty-minute lead window, p95 is still 27 minutes.
- Forgetting to raise the TTL back means 63,360 extra queries a day; forgetting to lower it
  means 14,812 misrouted requests in an unplanned rollback.
- Closing has its own surface: inventory automation brings a new identity with write access
  to the zone, and a short TTL brings only a five-minute buffer against an
  authoritative-server outage.

#### Next Step

This lesson measured where the name points. The same name does one more job beyond pointing:
messages get sent under it. The regional measurement network sends notification emails to
field teams, and those messages carry the name `measurenet.invalid`. Nothing says the message
carrying that name actually came from that name — by default, sending a message in a name's
name doesn't require owning that name. The next lesson measures sender verification as a
surface: the authorized-sender list and the signature record are run against a fictional set
of messages, and the count is taken of how many forged messages are caught, how many get
through, and how many legitimate messages are mistakenly dropped.

### Sender Verification Records

Source: https://academia.sh/en/courses/servers-and-hardening/sender-verification-records

Running the sender policy record and the signature record against a fictional set of messages: the counts of forged messages caught, forged messages that pass, and legitimate messages mistakenly dropped, and the surface opened by overly broad authorization.

The previous lesson measured where the name points. The same name does a second job: the
regional measurement network sends notification emails to field teams, and those messages
carry `measurenet.invalid`. The network is fictional; a collector that gathers meter
readings, a validator, and billing and work-order services all produce messages every night.

By default, nothing says those messages actually came from that name. The from field is just
text written inside the message; writing it does not require owning the name. When nothing is
done after setup, the open item is single and large: sending a message in the name's name is
open to anyone, and the more familiar the name, the more the forgery works. When a billing
notification is forged, the field team cannot tell the difference.

Two records close this gap, and they answer **separate questions**. The **sender policy
record** is the list of machines authorized to send messages in this name's name; its
question is "which machine did this message come from." The **signature record** publishes a
public key; the sending machine signs the message with the matching private key, and the
receiving side verifies the signature with the key in the record. Its question is "did this
content come from whoever holds the key, and did it change on the way." Neither substitutes
for the other, and as shown below, neither is sufficient alone.

#### Two Checks, One Run

The program below actually runs both checks. Signatures are generated and verified with
`node:crypto`; keys are created at run time, and no key body is ever written into the text.

**DR7.** The message set is 16 fictional messages: 10 legit notifications, 6 forged. The
`kind` field exists only for counting; the checker never sees that field.
**DR8.** The signature is taken over the from–subject–body triple; a real mail message's
header set is not modeled.
**DR9.** The receiving side runs both checks and sees their results; what it does with the
results is not this lesson's subject.

```javascript
import { generateKeyPairSync, sign, verify } from 'node:crypto';

// Model: sender verification. Keys are generated at run time; no key body
// is ever written down or printed. The message set and records are fictional.
const KEYS = {
  k1: generateKeyPairSync('ed25519'),   // the measurement network's own key
  k2: generateKeyPairSync('ed25519'),   // the third-party sender's key
};
const FORGED_KEY = generateKeyPairSync('ed25519');   // the attacker's key

// The two records in the zone. The sender policy record is the list of
// authorized senders; the signature record carries one public key per tag.
const POLICY_RECORD = {
  singles: ['198.51.100.11', '198.51.100.12', '198.51.100.13', '198.51.100.14', '198.51.100.20'],
  networkBlocks: [{ prefix: '203.0.113.', size: 256 }],   // the third-party sender's whole egress network
};
const SIGNATURE_RECORD = { k1: KEYS.k1.publicKey, k2: KEYS.k2.publicKey };

const serialize = (m) => `${m.from}|${m.subject}|${m.body}`;

function signMessage(m, tag, privateKey) {
  m.tag = tag;
  m.signature = sign(null, Buffer.from(serialize(m)), privateKey);
  return m;
}

function policyCheck(source) {
  if (POLICY_RECORD.singles.includes(source)) return true;
  return POLICY_RECORD.networkBlocks.some((b) => source.startsWith(b.prefix));
}

function signatureCheck(m) {
  if (!m.signature || !SIGNATURE_RECORD[m.tag]) return false;
  return verify(null, Buffer.from(serialize(m)), SIGNATURE_RECORD[m.tag], m.signature);
}

// 16 fictional messages: 10 legit notifications, 6 forged.
const MESSAGES = [
  { id: 'm01', source: '198.51.100.11', from: 'collector@measurenet.invalid', subject: 'meter batch', body: 'nightly job done', kind: 'legit', key: 'k1' },
  { id: 'm02', source: '198.51.100.12', from: 'validator@measurenet.invalid', subject: 'deviation alert', body: 'zone 4 deviation', kind: 'legit', key: 'k1' },
  { id: 'm03', source: '198.51.100.13', from: 'billing@measurenet.invalid', subject: 'period close', body: 'period 7 closed', kind: 'legit', key: 'k1' },
  { id: 'm04', source: '198.51.100.14', from: 'workorder@measurenet.invalid', subject: 'field work order', body: '5 orders for team 2', kind: 'legit', key: 'k1' },
  { id: 'm05', source: '198.51.100.20', from: 'notify@measurenet.invalid', subject: 'weekly summary', body: 'summary attached', kind: 'legit', key: 'k1' },
  { id: 'm06', source: '203.0.113.40', from: 'notify@measurenet.invalid', subject: 'bulk announcement', body: 'maintenance window', kind: 'legit', key: 'k2' },
  { id: 'm07', source: '203.0.113.41', from: 'notify@measurenet.invalid', subject: 'bulk announcement', body: 'maintenance window', kind: 'legit', key: 'k2' },
  { id: 'm08', source: '198.51.100.13', from: 'billing@measurenet.invalid', subject: 'nightly report', body: 'batch job output', kind: 'legit', key: null },
  { id: 'm09', source: '192.0.2.77', from: 'alert@measurenet.invalid', subject: 'threshold exceeded', body: 'new alert service', kind: 'legit', key: 'k1' },
  { id: 'm10', source: '198.51.100.30', from: 'workorder@measurenet.invalid', subject: 'field work order', body: '2 orders for team 3', kind: 'legit', key: 'k1', tampered: ' -- forwarded' },
  { id: 'f01', source: '203.0.113.200', from: 'billing@measurenet.invalid', subject: 'payment request', body: 'account changed', kind: 'forged', key: null },
  { id: 'f02', source: '192.0.2.200', from: 'workorder@measurenet.invalid', subject: 'urgent work order', body: 'unplug the meter', kind: 'forged', key: null },
  { id: 'f03', source: '192.0.2.201', from: 'notify@measurenet.invalid', subject: 'password renewal', body: 'follow the link', kind: 'forged', key: 'forged' },
  { id: 'f04', source: '203.0.113.201', from: 'billing@measurenet.invalid', subject: 'period close', body: 'new account number', kind: 'forged', key: 'forged' },
  { id: 'f05', source: '198.51.100.13', from: 'billing@measurenet.invalid', subject: 'period close', body: 'compromised machine', kind: 'forged', key: 'k1' },
  { id: 'f06', source: '192.0.2.202', from: 'collector@measurenet.invalid', subject: 'meter batch', body: 'nightly job done', kind: 'forged', key: 'k1', tampered: ' extra line' },
];

for (const m of MESSAGES) {
  if (m.key === 'forged') signMessage(m, 'k1', FORGED_KEY.privateKey);
  else if (m.key) signMessage(m, m.key, KEYS[m.key].privateKey);
  if (m.tampered) m.body += m.tampered;   // body changes after signing
}

const evaluate = () => MESSAGES.map((m) => ({
  id: m.id, kind: m.kind, policy: policyCheck(m.source), signature: signatureCheck(m),
}));
const RULES = {
  'policy only': (s) => s.policy,
  'signature only': (s) => s.signature,
  either: (s) => s.policy || s.signature,
  both: (s) => s.policy && s.signature,
};
const legit = MESSAGES.filter((m) => m.kind === 'legit').length;
const forged = MESSAGES.length - legit;

function table(label) {
  const results = evaluate();
  console.log(`\n[${label}]  rule              forged caught    forged passed  legit dropped`);
  for (const [name, rule] of Object.entries(RULES)) {
    const passed = results.filter((s) => s.kind === 'forged' && rule(s)).length;
    const dropped = results.filter((s) => s.kind === 'legit' && !rule(s)).length;
    console.log('  '.padEnd(label.length + 5) + name.padEnd(18) + `${forged - passed}/${forged}`.padEnd(17) +
      String(passed).padEnd(15) + `${dropped}/${legit}`);
  }
}

console.log('message   kind     policy    signature');
for (const s of evaluate()) {
  console.log(`${s.id.padEnd(9)} ${s.kind.padEnd(8)} ${(s.policy ? 'passed' : 'dropped').padEnd(9)} ${s.signature ? 'passed' : 'dropped'}`);
}
console.log(`\nlegit messages: ${legit}  forged messages: ${forged}`);
table('wide');

const broadAuthorized = POLICY_RECORD.singles.length +
  POLICY_RECORD.networkBlocks.reduce((a, b) => a + b.size, 0);
const used = [...new Set(MESSAGES.filter((m) => m.kind === 'legit' && policyCheck(m.source))
  .map((m) => m.source))];
console.log('\nauthorized addresses:', broadAuthorized, '| actually sending:', used.length,
  '| authorized but unused:', broadAuthorized - used.length);

POLICY_RECORD.networkBlocks = [];       // network block removed
POLICY_RECORD.singles = used;           // only the used addresses remain
table('narrow');
```

```
message   kind     policy    signature
m01       legit    passed    passed
m02       legit    passed    passed
m03       legit    passed    passed
m04       legit    passed    passed
m05       legit    passed    passed
m06       legit    passed    passed
m07       legit    passed    passed
m08       legit    passed    dropped
m09       legit    dropped   passed
m10       legit    dropped   dropped
f01       forged   passed    dropped
f02       forged   dropped   dropped
f03       forged   dropped   dropped
f04       forged   passed    dropped
f05       forged   passed    passed
f06       forged   dropped   dropped

legit messages: 10  forged messages: 6

[wide]  rule              forged caught    forged passed  legit dropped
         policy only       3/6              3              2/10
         signature only    5/6              1              2/10
         either            3/6              3              1/10
         both              5/6              1              3/10

authorized addresses: 261 | actually sending: 7 | authorized but unused: 254

[narrow]  rule              forged caught    forged passed  legit dropped
           policy only       5/6              1              2/10
           signature only    5/6              1              2/10
           either            5/6              1              1/10
           both              5/6              1              3/10
```

#### Reading the Numbers

No row catches all six of the six forgeries, and no row passes every legitimate message. The
decision is made between two columns.

**Policy only** misses three forgeries. The reason shows up in the column: `f01` and `f04`
came from inside the authorized network block, and `f05` came directly from the legitimate
machine itself. The policy record asks about the address, not the content — it passes
anything sent from an authorized address.

What this check asks about determines how much it closes. The policy record tests the
address of the connecting machine, not the from field written inside the message. There is no
mandatory link between the two, and because the check only sees the address, a message sent
from an authorized address passes no matter what name it carries inside. What the record
closes is sending a message in this name's name from anywhere on that network — not a small
thing, but it says nothing about the content.

**Signature only** catches five forgeries, and the only one it misses is `f05`. In exchange it
drops two legitimate messages: `m08` is not signed (it does not go through the nightly batch
job's signing step), and `m10`'s body changed after signing — an intermediate server appended
a line while forwarding the message. The signature asks about content, not source.

**Combining the two is a third decision.** The loose combination (accept if either passes)
brings the dropped-legitimate count down to 1 but keeps the passed-forged count at 3; the
strict combination (both must pass) brings the passed-forged count down to 1, at the cost of
dropping three legitimate messages. The same two records give a threefold difference in
outcome depending on the combination rule. Who declares this rule, and how, is the next
lesson's subject.

#### The Record's Drift from Reality

The previous lesson counted the divergence between the zone file and the live inventory. The
same drift exists here too, and it moves faster, because the set of senders grows
independently of the set of records.

In the model, legitimate messages come from nine separate addresses, and the policy record
recognizes seven of them. The two it does not recognize speak for themselves: `m09` is a
newly set-up threshold-alert service that started sending messages but was never added to the
record; `m10` comes from a different machine as a result of a redirect. Adding to the record
is not part of any setup step — a service gets written, deployed, runs, sends messages, and no
one looks at the zone. The drift is one-directional and silent: a missing record does not
produce an error, it just makes a message lose one check.

Measuring this drift means comparing two lists: the addresses authorized in the record and
the addresses that actually sent messages in a recent period. Both directions of the
difference are items. An address that is in the record but does not send is unnecessary
surface; an address that sends but is not in the record is a legitimate message that is a
candidate for being dropped. Where these two numbers get read from is the reason the next
lesson's report flow exists; for now, both are visible only in this run, not in production.

#### Overly Broad Authorization

Authorizing a network block instead of a single address in the policy record is a common
shortcut; it is done so the record does not break when the third-party sender changes its
egress address. The cost shows up in the numbers:

**DR10.** The network block in the policy record carries 256 addresses; the third-party
sender uses two of them.

The record authorizes **261 addresses**, and the number actually sending is **7**. The 254
addresses in between are ones no one uses but that are authorized to send messages in the
measurement network's name. These may be allocated to the third-party sender's other
customers; when one of those customers is compromised, the authority to send messages in
`measurenet.invalid`'s name changes hands along with it. That is exactly what `f01` and `f04`
are in the run.

When the block is narrowed, the numbers settle into place: policy only goes from 3/6 to
**5/6**, passed-forged drops from 3 to 1, and **dropped-legitimate does not increase at all**.
This is a step that closes an open item without stopping any legitimate work, and it is the
only example of that in this topic. In exchange, a maintenance burden is created: if the
record is not updated when the third-party sender changes address, legitimate announcements
get dropped, and that drop is silent.

The second layer of authorization must be counted separately too. The third-party sender is
authorized not only by the address block but also by the second tag in the signature record:
messages `m06` and `m07` are signed with its key and pass verification. That means this party
is a second party able to produce a valid signature in the measurement network's name. Even
if the address block is narrowed, this authority stands; it is only revoked by removing the
tag from the record. The two layers are two separate items, and unless both are counted
separately, the sentence "sender verification is set up" carries no number.

#### The Item That Cannot Be Closed, and the New Surface

`f05` is caught by no rule.

**DR11.** A compromised legitimate machine also has access to the signing key sitting on that
machine.

This item cannot be closed, because both records ask whether the machine is authorized, not
whose hands it is currently in. The outcome even runs backward: a message that passes
verification looks **more** trustworthy on the receiving side than one that does not. `f05`
is not just uncaught — because it passes both checks, it is the most convincing of the forged
messages. Setting up sender verification raises the value of this one message; in exchange
for the five items it closes, the one item left behind gets heavier. Sender verification
prevents the name from being impersonated; it does not prevent the name from being abused
from its own machine. The measure of that is not in this topic but in hardening the machine.

`m10` is also among the items that cannot be closed: when a message is forwarded, the source
address changes and the body can change. Both checks treat this like an attack. No one can
manage forwarding from the zone file.

**DR12.** While the key is being rotated, the old tag stays in the record for a while longer;
during that time, signatures produced with either key still verify.

Closing this item has its own surface, and it sits where the key lives. The signature record
publishes a **public** key, but the matching private key sits on a machine and is a secret
there. Because the record carries a key per tag, more than one tag can be published at once —
a new tag is added while the key is rotated, and the old tag stays for a while. During that
time, **two keys are valid at once**, and whoever compromises the old key still produces a
valid signature. Key rotation is a window opened in exchange for an item closed; the length
of that window depends on the time to live measured in the previous lesson.

#### Summary

- By default, sending a message in a name's name does not require owning that name; the open
  item after setup is single, and it grows with how recognizable the name is.
- The sender policy record answers "which machine," and the signature record answers "did it
  come from the key holder and did it change"; neither substitutes for the other.
- In the 16-message fictional set, policy only catches 3/6 forgeries and signature only
  catches 5/6; the signature drops two legitimate messages (an unsigned batch job and a
  notification whose body changed while being forwarded).
- The combination rule determines the outcome: the loose rule brings dropped-legitimate down
  to 1 but passes 3 forgeries, the strict rule passes 1 forgery but drops 3 legitimate
  messages.
- The network block in the policy record authorizes 261 addresses, and only 7 actually send;
  the 254 extra authorized addresses are a surface, and it closes without any
  legitimate-message loss when the block is narrowed.
- A message sent from a compromised legitimate machine is caught by no rule; key rotation,
  meanwhile, opens a window in which two keys are valid at once.

#### Next Step

The two checks now produce one outcome: two bits per message. The real gap this lesson leaves
is that nothing says what to do with those two bits. Whether the receiving side delivers,
quarantines, or rejects a message that fails a check is currently the receiver's decision, not
the name owner's, and receivers behave differently. The next lesson covers declaring this
decision from the zone: the policy declaration is tightened in stages — report only,
quarantine, reject — and at each stage the count of dropped legitimate messages and blocked
forged messages is taken separately. The report flow itself is also a measurement tool: how
many unauthorized senders get found through it.

### Domain Policy Record

Source: https://academia.sh/en/courses/servers-and-hardening/domain-policy-record

Tightening the policy declaration for noncompliant messages in stages: the count of dropped legitimate messages and blocked forged messages at each stage, the report flow's rate of finding unauthorized senders, and the cost of rejecting outright.

The previous lesson ran both checks and produced two outcomes for every message. The
question left open there was what the outcome is for: when a message fails a check, does the
receiving side deliver it, separate it, or never accept it at all? By default, the name owner
does not make this decision. Each receiver acts on its own criteria; some deliver a failed
message as is, some apply their own estimate. The result is that the same forged message
reaches an inbox in one place and goes nowhere in another.

**The policy declaration** moves this decision back to the name owner. It is a record
published in the zone, and it says: apply this action to a message that arrives in this
name's name but fails the checks. It has three stages — **report only**, **quarantine**,
**reject** — and the difference between them is who gets billed for the gaps in the sending
infrastructure.

Jumping between stages is one of the most expensive countable mistakes in this topic. This
lesson runs one period and derives two numbers together at each stage: **how many legitimate
messages get dropped** and **how many forged messages get blocked**.

#### The Fields the Declaration Carries

The record is written into the zone as text and declares four things: the action to apply to
a noncompliant message, what percentage of messages the action applies to, the address
reports get sent to, and whether a separate action is defined for subdomains. The first two
are the two axes this lesson measures; the third is the source of the measurement.

The declaration's scope is also a decision. The action applies only to messages that carry
this name in the **from field**; the check looks for the name that passed verification to
match the name in the from field. Without this requirement, the declaration would not work: a
message properly signed under its own name could carry a different name in the from field and
appear to pass the checks. The overlap requirement is the link that ties the surface the two
records close to the declaration's scope.

The reason a separate action can be defined for subdomains is that subdomains mostly do not
send messages. In the model's zone there are fourteen names, and only a few of them send; for
the subdomains that do not send, the strictest action can be declared directly, because there
is no legitimate message to lose there. This is the first and costless step of tightening,
and it stays outside the count below.

#### One Period, Four Stages

**DR13.** The period is one month and consists of 414 fictional messages; flow counts are
taken as fixed.
**DR14.** The compliance criterion is that at least one of the two checks passes aligned with
the sender's name.
**DR15.** A legitimate message put into quarantine counts as delivered; the probability it
gets read is not counted in this model.

```javascript
// Model: one period's message flow and the policy declaration's stages. Flows,
// message counts, and the report rate are fictional; compliance means at
// least one of the two checks passes aligned with the sender's name.
const SEED = 7311;
const REPORT_RATE = 0.65;      // the rate at which receivers send reports

function prng(seed) {
  let x = seed % 2147483647;
  return () => { x = (x * 48271) % 2147483647; return (x - 1) / 2147483646; };
}

// sources: [address, message count]; the policy/signature fields are check results
const FLOWS = [
  { name: 'nightly-summary',        kind: 'legit',  policy: true,  signature: true,  sources: [['198.51.100.11', 120]] },
  { name: 'deviation-alert',        kind: 'legit',  policy: true,  signature: true,  sources: [['198.51.100.12', 60]] },
  { name: 'period-close',           kind: 'legit',  policy: true,  signature: false, sources: [['198.51.100.13', 40]] },
  { name: 'bulk-announcement',      kind: 'legit',  policy: true,  signature: true,  sources: [['203.0.113.40', 50], ['203.0.113.41', 40]] },
  { name: 'threshold-alert',        kind: 'legit',  policy: false, signature: false, sources: [['192.0.2.77', 25]] },
  { name: 'redirected-order',       kind: 'legit',  policy: false, signature: false, sources: [['198.51.100.30', 18]] },
  { name: 'billing-impersonation',  kind: 'forged', policy: false, signature: false, sources: [['192.0.2.200', 30], ['198.18.0.5', 2]] },
  { name: 'workorder-impersonation', kind: 'forged', policy: false, signature: false, sources: [['192.0.2.201', 15], ['198.18.0.9', 1], ['198.18.0.12', 1]] },
  { name: 'in-block-impersonation', kind: 'forged', policy: true,  signature: false, sources: [['203.0.113.200', 12]] },
];

const compliant = (a) => a.policy || a.signature;
const count = (a) => a.sources.reduce((t, [, n]) => t + n, 0);
const sum = (predicate) => FLOWS.filter(predicate).reduce((t, a) => t + count(a), 0);

const legit = sum((a) => a.kind === 'legit');
const forged = sum((a) => a.kind === 'forged');
console.log('period total:', legit + forged, '| legit:', legit, '| forged:', forged);
console.log('noncompliant legit messages:', sum((a) => a.kind === 'legit' && !compliant(a)),
  '| compliant forged messages:', sum((a) => a.kind === 'forged' && compliant(a)));

console.log('unprepared legit flows:',
  FLOWS.filter((a) => a.kind === 'legit' && !compliant(a)).map((a) => `${a.name}=${count(a)}`).join(' '));

const STAGES = ['no declaration', 'report only', 'quarantine', 'reject'];
console.log('\nstage           inbox (legit)         separated legit  inbox (forged)        blocked forged');
for (const stage of STAGES) {
  const separates = stage === 'quarantine' || stage === 'reject';
  const legitSeparated = separates ? sum((a) => a.kind === 'legit' && !compliant(a)) : 0;
  const forgedSeparated = separates ? sum((a) => a.kind === 'forged' && !compliant(a)) : 0;
  console.log(stage.padEnd(16) + String(legit - legitSeparated).padEnd(22) +
    `${legitSeparated}${stage === 'reject' ? ' (lost)' : stage === 'quarantine' ? ' (quarantine)' : ''}`.padEnd(17) +
    String(forged - forgedSeparated).padEnd(22) + String(forgedSeparated));
}

// Report flow: each message independently enters the report with REPORT_RATE odds.
// A source counts as found if at least one of its messages was reported.
const rand = prng(SEED);
const unauthorized = [];
for (const a of FLOWS) {
  if (a.policy) continue;                       // already in the policy record
  for (const [address, n] of a.sources) {
    let reported = 0;
    for (let i = 0; i < n; i++) if (rand() < REPORT_RATE) reported++;
    unauthorized.push({ address, flow: a.name, kind: a.kind, n, found: reported > 0 });
  }
}
const detected = unauthorized.filter((k) => k.found);
console.log('\nreport rate:', REPORT_RATE, '| seed:', SEED);
console.log('unauthorized sources:', unauthorized.length, '| found via reports:', detected.length,
  `(${Math.round((detected.length / unauthorized.length) * 100)}%)`);
for (const k of unauthorized.filter((x) => !x.found)) {
  console.log(`  missed source: ${k.address.padEnd(15)} ${k.flow.padEnd(18)} messages=${k.n}`);
}
console.log('message share of found sources:',
  `${detected.reduce((t, k) => t + k.n, 0)}/${unauthorized.reduce((t, k) => t + k.n, 0)}`);

// Prep: the new service is added to the policy record, the batch job is signed.
FLOWS.find((a) => a.name === 'threshold-alert').policy = true;
console.log('\nlegit messages lost in the reject stage after prep:',
  sum((a) => a.kind === 'legit' && !compliant(a)));
```

```
period total: 414 | legit: 353 | forged: 61
noncompliant legit messages: 43 | compliant forged messages: 12
unprepared legit flows: threshold-alert=25 redirected-order=18

stage           inbox (legit)         separated legit  inbox (forged)        blocked forged
no declaration  353                   0                61                    0
report only     353                   0                61                    0
quarantine      310                   43 (quarantine)  12                    49
reject          310                   43 (lost)        12                    49

report rate: 0.65 | seed: 7311
unauthorized sources: 7 | found via reports: 6 (86%)
  missed source: 198.18.0.12     workorder-impersonation messages=1
message share of found sources: 91/92

legit messages lost in the reject stage after prep: 18
```

#### Reading the Stages

The first two rows carry the same numbers, and that is not a mistake. The **report-only**
stage blocks no message at all; sixty-one forged messages keep reaching the inbox. The only
thing this stage produces is a report, and a report is not a defense but a **measurement
tool**.

Tightening begins at the **quarantine** row: forty-nine forged messages are separated from
the inbox, and in exchange forty-three legitimate messages are separated too. The ratio is
almost one to one — meaning that alongside the closed open item, the wrong closure is of the
same magnitude. The only difference is that a quarantined legitimate message still sits
somewhere; the recipient finds it if they look.

**Reject** gives the same numbers as quarantine, but the column's meaning changes: the
forty-three legitimate messages are now nowhere at all. Two stages at the same strictness
diverge on the legitimate-message side in terms of **reversibility**. This is why the order
matters: quarantine is a rehearsal for rejecting, and it keeps the loss recoverable.

The last column shows the item that never closes. Twelve forged messages reach the inbox at
every stage, because they come from inside the network block authorized in the policy record
and count as compliant. The policy declaration says what to do with a noncompliant message; it
does not see a forgery that looks compliant. The overly broad authorization from the previous
lesson comes back here directly as an item: tightening the declaration without narrowing the
block does not touch these twelve messages.

#### The Report's Value and Its Limit

The report flow is receivers telling the name owner: a message came from this source in this
name's name, and the check resulted in this. Its value is that the drift visible only in the
previous lesson's run becomes visible in production.

**DR16.** A portion of receivers send reports; each message independently has a 0.65
probability of entering a report.
**DR17.** A report shows the source, not whether the message was legitimate or forged.

The number is good: six of seven unauthorized sources are found, the find rate is **86%**,
and the found sources cover **91** of the messages. The one source that got away sent only a
single message during the period. The rule here is that the report flow finds a high-volume
source almost for certain and misses a sparse one. That means it is the right tool for
preparation — legitimate flows that would be dropped unprepared are, by definition, regular
and high-volume.

The limit is in DR17. The report says 25 noncompliant messages came from `192.0.2.77`; it
does not say whether that is a newly set-up threshold-alert service or an impersonation. What
makes that distinction is not the report but the **inventory** — comparison against the list
of services that actually send. The report flow produces the question, not the answer.

#### The Cost of Rejecting, and Preparation

The cost of moving straight to rejecting is written by name in the output: threshold-alert,
25 messages; redirected-order, 18 messages. The first is a new service waiting to be added to
the record; its loss is a preparation gap. The second is a redirect field teams set up to
their own addresses, and it is not under the name owner's control.

The preparation step turns this distinction into a number: after the new service is added to
the policy record, the legitimate messages lost in the reject stage drop from **43 to 18**.
The remaining 18 messages, however, cannot be closed with a record — the redirect makes the
message come from a different machine and changes its body.

How long preparation needs to run also comes from the report. A single-period report shows
only the sources that sent during that period; a close-out job that runs at the start of the
quarter, or a notification sent once a year, never shows up in a one-period window. The one
source that got away having sent a single message during the period is the small-scale
example of this. The preparation window must cover the period of the sparsest sending flow;
kept short, tightening drops that flow the first day it runs.

This is the justification for staged tightening. In the report stage, sources are found,
records are completed, and the magnitude of the remaining loss is measured; only after that
does the move go to quarantine, then to rejecting. If the order is skipped, the loss ends up
the same size, but it is experienced first and learned about afterward.

#### Tightening by Percentage

A stage is not a single switch. The declaration can also say the action applies to only a
percentage of messages; tightening is thereby graduated within a stage too.

```javascript
// Model: the declaration can apply to only a percentage of noncompliant
// messages. Input is the prior counts: two unprepared legit flows, 49 forged.
const SEED = 9013;
const LEGIT_FLOWS = [['threshold-alert', 25], ['redirected-order', 18]];
const NONCOMPLIANT_FORGED = 49;
const NOTICE_THRESHOLD = 10;   // this many losses in a flow per period produce a complaint

function prng(seed) {
  let x = seed % 2147483647;
  return () => { x = (x * 48271) % 2147483647; return (x - 1) / 2147483646; };
}

const countHits = (rand, n, rate) => {
  let k = 0;
  for (let i = 0; i < n; i++) if (rand() < rate) k++;
  return k;
};

console.log('seed:', SEED, '| notice threshold:', NOTICE_THRESHOLD, 'messages/period');
console.log('rate   threshold-alert  redirected-order    total legit lost    forged blocked    flows past threshold');
for (const rate of [0.05, 0.1, 0.25, 0.5, 1.0]) {
  const rand = prng(SEED);
  const losses = LEGIT_FLOWS.map(([, n]) => countHits(rand, n, rate));
  const forgedBlocked = countHits(rand, NONCOMPLIANT_FORGED, rate);
  console.log(
    `${Math.round(rate * 100)}%`.padEnd(7) +
    String(losses[0]).padEnd(17) + String(losses[1]).padEnd(20) +
    String(losses[0] + losses[1]).padEnd(20) + String(forgedBlocked).padEnd(18) +
    String(losses.filter((k) => k >= NOTICE_THRESHOLD).length),
  );
}
```

```
seed: 9013 | notice threshold: 10 messages/period
rate   threshold-alert  redirected-order    total legit lost    forged blocked    flows past threshold
5%     0                2                   2                   2                 0
10%    4                3                   7                   4                 0
25%    9                6                   15                  12                0
50%    14               12                  26                  26                2
100%   25               18                  43                  49                2
```

Two things read at once in the table. First, a low percentage blocks forged messages at a low
rate too; at 10%, only four of the forty-nine forgeries get stopped. The percentage grades
**risk**, not defense.

Second, and the actually useful one, is the last column. Up to 25%, no flow crosses the
notice threshold; there is loss, but no one complains. This shows the percentage's real
function: a low rate is not a defense setting but a tool for **finding preparation gaps
cheaply**. A loss of fifteen messages shows up in reports and not with the field team; the
same flow, opened to 100%, loses twenty-five messages at once and work stops. The cost of
running tightening by percentage is that forged messages also get through at the same rate
every period that passes.

#### The Remaining Item and the New Surface

**DR18.** In this model, the declaration is applied identically by every receiver.

This assumption overreaches reality and shows the part of this topic that **cannot be
measured**: the declaration is a request, not something enforceable. The receiving side may
not apply it, may apply it partially, or may layer its own criteria on top. Here the name
owner only declares a decision; someone else applies it, and whether it gets applied is
visible, incompletely, only through the report flow.

The new surface closing brings is the report flow itself. An address is written into the
zone, and continuous data arrives at that address from outside parties: how many messages
came from which source, which check failed. This is a flow that makes the measurement
network's sending structure readable from outside, and the address it goes to is written
openly in the zone for anyone. It is also a volume surface: the report address is a data
intake that comes from outside and must be processed.

#### Summary

- The policy declaration moves the decision of what to do with a noncompliant message from
  the receiver to the name owner; without a declaration, the same forged message reaches an
  inbox at one receiver and goes nowhere at another.
- The report-only stage blocks no message: 61 forged messages keep getting through; this
  stage is a measurement tool, not a defense.
- Quarantine and reject give the same numbers (49 forged blocked, 43 legitimate separated);
  the difference is in the reversibility of the legitimate message.
- 12 forged messages coming from inside the authorized network block reach the inbox at every
  stage; the declaration separates the noncompliant but does not see a forgery that looks
  compliant.
- The report flow finds six of seven unauthorized sources (86%) and covers 91 of the
  messages; the source it misses sent a single message during the period.
- Preparation brings the legitimate loss in the reject stage down from 43 to 18; the
  remaining 18 messages come from a redirect and cannot be closed with any record.

#### Next Step

The name now also defines who is permitted to send messages in its own name, and declares
what happens to a noncompliant message. All of this is a measure of the sending side. A
message can fail to arrive even after passing every check: the receiving side also decides
based on the sender's history — how many of its messages bounced, how many people complained,
whether the sending volume spiked suddenly. The next lesson models deliverability as a ratio,
counts the factors that degrade it separately, and runs one period: how many messages get
delayed when reputation drops, how many never arrive at all, and how many periods recovery
takes.

### Mail Delivery Issues

Source: https://academia.sh/en/courses/servers-and-hardening/mail-delivery-issues

Modeling deliverability as a ratio: counting bounced messages, complaints, sudden volume spikes, and missing verification records separately, and measuring how many periods blacklisting and recovery take.

The previous three lessons established, at the record level, where the name points, who can
send messages in that name's name, and what happens to a noncompliant message. All three are
measures of the **sending side**. Where this lesson starts, messages are now compliant,
signed, and the policy is declared — and they may still not arrive.

The receiving side does not accept a message by looking only at the records. It also looks at
the sender's **history**: how many of its messages bounced in the recent period, how many
people complained, whether volume spiked suddenly. The summary of this history is called
**reputation**, and it behaves like a number: invisible while high, and it determines
everything once it drops. The regional measurement network's work-order notifications to
field teams fall under this measure, and the network is fictional.

#### Deliverability Is a Ratio

When a message is sent, one of three outcomes happens: it reaches the inbox, it gets delayed
and delivered later, or it never arrives. **Deliverability** is the ratio of these three
outcomes — not a yes/no.

The factors that degrade the ratio can be counted separately, and all four pull in the same
direction:

- **Bounce rate.** Sending to an address that no longer exists shows the list is not
  maintained.
- **Complaint rate.** A recipient marking a message as unwanted is the heaviest single factor.
- **Sudden volume spike.** Volume from the same address jumping to several times its usual
  level is a deviation from established behavior.
- **Missing verification record.** If the previous two lessons' records are missing, it does
  not produce an incident by itself; it only lowers the highest reputation reachable.

The last item is the link connecting this topic: when sender verification is not set up, the
resulting harm does not look like an incident — it behaves like a **ceiling**.

#### A One-Period Run

**DR19.** A period is one week and twenty periods are run; volume, rates, and thresholds are
fictional.
**DR20.** Reputation is modeled as a single number.
**DR21.** Reputation falls fast and climbs slowly: it drops by at most 35 points in one period
and rises by at most 5.
**DR22.** Blacklisting happens below 35 points; exiting requires two periods above 55 points.

```javascript
// Model: deliverability is a ratio, determined via reputation. Periods,
// factors, and thresholds are fictional; not a real service's measurement.
const STARTING_REPUTATION = 88;
const BLACKLIST_ENTRY = 35;   // the sending address is blacklisted below this value
const BLACKLIST_EXIT = 55;    // two periods are waited above this value

// Period inputs: volume, bounce rate, complaint rate, whether the verification record is missing
const PERIODS = [
  { d: 1,  volume: 400,  bounceRate: 0.012, complaintRate: 0.0004, recordsMissing: true },
  { d: 2,  volume: 410,  bounceRate: 0.011, complaintRate: 0.0005, recordsMissing: true },
  { d: 3,  volume: 395,  bounceRate: 0.013, complaintRate: 0.0004, recordsMissing: true },
  { d: 4,  volume: 420,  bounceRate: 0.012, complaintRate: 0.0006, recordsMissing: true },
  { d: 5,  volume: 405,  bounceRate: 0.010, complaintRate: 0.0004, recordsMissing: true },
  { d: 6,  volume: 3200, bounceRate: 0.140, complaintRate: 0.0040, recordsMissing: true },   // legacy list added
  { d: 7,  volume: 3100, bounceRate: 0.135, complaintRate: 0.0052, recordsMissing: true },
  { d: 8,  volume: 2900, bounceRate: 0.120, complaintRate: 0.0048, recordsMissing: true },
  { d: 9,  volume: 900,  bounceRate: 0.040, complaintRate: 0.0020, recordsMissing: false },  // list cleaned up
  { d: 10, volume: 450,  bounceRate: 0.015, complaintRate: 0.0008, recordsMissing: false },
  { d: 11, volume: 430,  bounceRate: 0.011, complaintRate: 0.0005, recordsMissing: false },
  { d: 12, volume: 425,  bounceRate: 0.009, complaintRate: 0.0004, recordsMissing: false },
  { d: 13, volume: 415,  bounceRate: 0.008, complaintRate: 0.0003, recordsMissing: false },
  { d: 14, volume: 420,  bounceRate: 0.008, complaintRate: 0.0003, recordsMissing: false },
  { d: 15, volume: 430,  bounceRate: 0.007, complaintRate: 0.0002, recordsMissing: false },
  { d: 16, volume: 425,  bounceRate: 0.007, complaintRate: 0.0002, recordsMissing: false },
  { d: 17, volume: 440,  bounceRate: 0.006, complaintRate: 0.0002, recordsMissing: false },
  { d: 18, volume: 435,  bounceRate: 0.006, complaintRate: 0.0002, recordsMissing: false },
  { d: 19, volume: 430,  bounceRate: 0.005, complaintRate: 0.0001, recordsMissing: false },
  { d: 20, volume: 445,  bounceRate: 0.005, complaintRate: 0.0001, recordsMissing: false },
];

// Bands: by reputation range, [inbox, delayed, undelivered]
const BANDS = [
  [80, [0.98, 0.02, 0.00]],
  [60, [0.88, 0.10, 0.02]],
  [40, [0.65, 0.25, 0.10]],
  [20, [0.35, 0.35, 0.30]],
  [0,  [0.10, 0.30, 0.60]],
];
const BLACKLIST_BAND = [0.05, 0.15, 0.80];

const band = (reputation) => BANDS.find(([threshold]) => reputation >= threshold)[1];
const clamp = (x) => Math.max(0, Math.min(100, x));

const MAX_DROP = 35;      // the most points it can fall in one period
const MAX_RECOVERY = 5;   // the most points it can climb in one period

let reputation = STARTING_REPUTATION;
let blacklisted = false;
let cleanPeriods = 0;
let previous = PERIODS[0].volume;
const history = [];

for (const g of PERIODS) {
  const growth = g.volume / previous;
  let target = 100;
  target -= Math.max(0, g.bounceRate - 0.02) * 300;        // bounce rate
  target -= Math.max(0, g.complaintRate - 0.001) * 5000;   // complaint rate
  target -= growth > 3 ? 20 : 0;                            // sudden volume spike
  target -= g.recordsMissing ? 12 : 0;                      // missing verification record
  target = clamp(target);
  reputation = target > reputation
    ? Math.min(target, reputation + MAX_RECOVERY)
    : Math.max(target, reputation - MAX_DROP);

  if (!blacklisted && reputation < BLACKLIST_ENTRY) { blacklisted = true; cleanPeriods = 0; }
  else if (blacklisted) {
    if (reputation >= BLACKLIST_EXIT) cleanPeriods++; else cleanPeriods = 0;
    if (cleanPeriods >= 2) blacklisted = false;
  }

  const [ik, dk, uk] = blacklisted ? BLACKLIST_BAND : band(reputation);
  history.push({
    ...g, target: Math.round(target), reputation: Math.round(reputation), blacklisted,
    inbox: Math.round(g.volume * ik),
    delayed: Math.round(g.volume * dk),
    undelivered: Math.round(g.volume * uk),
  });
  previous = g.volume;
}

console.log('period  volume  bounce%  complaint%  reputation  black  inbox  delayed  undelivered');
for (const s of history) {
  console.log(
    `d${String(s.d).padStart(2, '0')}`.padEnd(8) + String(s.volume).padEnd(8) +
    (s.bounceRate * 100).toFixed(1).padEnd(9) + (s.complaintRate * 100).toFixed(2).padEnd(12) +
    String(s.reputation).padEnd(12) + (s.blacklisted ? 'yes' : '-').padEnd(7) +
    String(s.inbox).padEnd(7) + String(s.delayed).padEnd(9) + String(s.undelivered),
  );
}

const total = (field) => history.reduce((t, s) => t + s[field], 0);
console.log('\ntotal volume:', total('volume'), '| inbox:', total('inbox'),
  '| delayed:', total('delayed'), '| undelivered:', total('undelivered'));
console.log('delivery rate:', (total('inbox') / total('volume') * 100).toFixed(1) + '%',
  '| loss rate:', (total('undelivered') / total('volume') * 100).toFixed(1) + '%');

const after = history.filter((s) => s.d >= 10);
const sum = (arr, field) => arr.reduce((t, s) => t + s[field], 0);
console.log('after the incident (d10-d20) volume:', sum(after, 'volume'),
  '| delayed:', sum(after, 'delayed'), '| undelivered:', sum(after, 'undelivered'));

const declineStart = history.find((s) => s.reputation < 80);
const bottom = history.reduce((a, b) => (b.reputation < a.reputation ? b : a));
const recoveryPoint = history.find((s) => s.d > bottom.d && s.reputation >= 80);
console.log('\nreputation started falling: d' + declineStart.d, '| bottom:', bottom.reputation, '(d' + bottom.d + ')',
  '| periods spent blacklisted:', history.filter((s) => s.blacklisted).length);
console.log('decline: 1 period | return to 80 threshold: d' + recoveryPoint.d,
  '| recovery:', recoveryPoint.d - bottom.d, 'periods');
```

```
period  volume  bounce%  complaint%  reputation  black  inbox  delayed  undelivered
d01     400     1.2      0.04        88          -      392    8        0
d02     410     1.1      0.05        88          -      402    8        0
d03     395     1.3      0.04        88          -      387    8        0
d04     420     1.2      0.06        88          -      412    8        0
d05     405     1.0      0.04        88          -      397    8        0
d06     3200    14.0     0.40        53          -      2080   800      320
d07     3100    13.5     0.52        33          yes    155    465      2480
d08     2900    12.0     0.48        38          yes    145    435      2320
d09     900     4.0      0.20        43          yes    45     135      720
d10     450     1.5      0.08        48          yes    23     68       360
d11     430     1.1      0.05        53          yes    22     65       344
d12     425     0.9      0.04        58          yes    21     64       340
d13     415     0.8      0.03        63          -      365    42       8
d14     420     0.8      0.03        68          -      370    42       8
d15     430     0.7      0.02        73          -      378    43       9
d16     425     0.7      0.02        78          -      374    43       9
d17     440     0.6      0.02        83          -      431    9        0
d18     435     0.6      0.02        88          -      426    9        0
d19     430     0.5      0.01        93          -      421    9        0
d20     445     0.5      0.01        98          -      436    9        0

total volume: 16875 | inbox: 7682 | delayed: 2278 | undelivered: 6918
delivery rate: 45.5% | loss rate: 41.0%
after the incident (d10-d20) volume: 4745 | delayed: 403 | undelivered: 1078

reputation started falling: d6 | bottom: 33 (d7) | periods spent blacklisted: 6
decline: 1 period | return to 80 threshold: d17 | recovery: 10 periods
```

#### Asymmetry

One line carries this entire lesson's weight: **decline takes one period, recovery takes
ten.**

The incident starts at `d06`. A set of addresses left over from years earlier is added to the
field list; volume jumps to eight times its size, the bounce rate spikes to 14%, and
complaints increase. Three factors hit in the same period, and reputation drops from 88 to
53. One period later it falls to 33 and gets **blacklisted**.

What follows from here is no longer about sending behavior. At `d09` the list is cleaned up,
at `d10` volume returns to normal and the rates are good — but reputation can only climb five
points per period. The address stays blacklisted for six periods; after coming off the list,
returning to the 80 threshold takes four more periods. Fixing the mistake took one period;
erasing its consequence took ten. This asymmetry is deliberate and serves the receiving side:
a measure that falls fast stops abuse early, and a measure that climbs slowly requires good
behavior to be sustained for several periods.

The second number shows the price of this. In the eleven periods after the incident — that
is, while sending was entirely normal — **1,078 messages never arrive and 403 messages get
delayed**. These are work-order notifications going to the field team, and there is nothing
wrong with their content; they get dropped purely because of the sender's history. The 41.0%
loss rate in the period total is misleading — most of that loss was already messages that
should not have been sent in the first place. The real cost of this is the normal messages
lost **after** the incident.

#### Counting the Factors Separately

Four factors hit the same number, but their contributions are not equal, and the weights in
the model show this.

Bounce rate is the largest single item: at 14%, it alone takes away 36 points. Complaint rate
carries similar weight with a number that looks smaller — five complaints per thousand means
21 points; this is the factor where a small number of receivers' decisions are weighted very
heavily. A sudden volume spike is a fixed 20-point penalty, independent of content: had the
same messages been sent more slowly, this item would never have arisen.

The fourth behaves differently from the others. The missing verification record produces no
incident at all in the first five periods; the delivery ratio is 98%, nothing is visible. All
it does is hold the reputation ceiling at **88**. What this means becomes clear at `d19` and
`d20`: once the record is completed, reputation climbs to 93, then to 98. In other words,
while the record was missing, the system always sat 12 points lower, and when an incident
arrived, it was 12 points closer to the blacklist threshold. Here, the previous two lessons'
records are measured not as a defense but as a **buffer**.

#### Separating Flows, and the Surface It Brings

All the loss in this run comes from a single number. Reputation is kept per sender, and the
model has exactly one sender: both the bulk announcement and the work-order notification go
out under the same name. The reputation the bulk list damages also drags down the work order
going to the field team — even though the two flows have no connection to each other.

The countermeasure is separating the flows: bulk and opt-in sending is done from a separate
subdomain, and operational notifications stay under their own name. The separation splits
reputation in two as well; what happens on one side does not drag down the other. The model's
structure says this directly, because the bands derive from reputation, and reputation is
tied to a single sender.

The new surface separation brings is the sum of the first three lessons. Every new subdomain
wants its own set of records: an address record, a sender policy record, a signature record,
its own policy declaration, and its own report address. The number of records in the zone
grows, the chance of a dangling record grows, and the time-to-live decision has to be made
once more. The inventory drift counted in the first lesson, the missing records counted in
the second, and the unprepared flow counted in the third — all three are born from scratch
for the new name. The step that reduces the delivery problem doubles the maintenance surface,
and that cost is part of the decision to separate the name.

#### The Part That Cannot Be Measured

**DR23.** The bands convert the reputation range directly into a delivery ratio.
**DR24.** In the model, the sending side sees the count of messages that reach the inbox.

The last assumption does not hold in reality, and it is the part of this topic that cannot be
measured. The sending side knows only two things: whether the message was accepted by the
receiving side, and which messages bounced. Whether an accepted message lands in the inbox or
somewhere else is **invisible**. The "inbox" and "delayed" columns counted separately in the
table above are, in production, a single column: accepted.

Reputation itself is not published either. It can be modeled as a number, but it cannot be
read; every receiving side keeps its own measure, and these measures differ from one another.
Blacklist status can be partly queried, but which threshold the listing decision used, and how
long the exit will take, is unknown from the outside.

Because of this, the only thing that can be measured is **indirect indicators**: bounce rate,
complaint reports, sending to one's own control addresses, and the previous lesson's report
flow. All four show not the outcome itself but the outcome's shadow. When a delivery problem
started is most often learned when a recipient says "the notification is not arriving"; that
is a tip-off, not a measurement.

#### Summary

- Deliverability is not a yes/no; it is the ratio of three outcomes: inbox, delay, never
  arriving.
- Four factors can be counted separately; in the model, bounce rate takes away 36 points,
  complaints 21, sudden volume spike 20, and the missing verification record sets a 12-point
  ceiling.
- Decline takes one period, recovery takes ten: reputation drops from 88 to 53 in one period,
  six periods pass on the blacklist, and returning to the 80 threshold takes ten periods after
  the bottom.
- Even after the incident, while sending is entirely normal, 1,078 messages never arrive and
  403 get delayed; these are work-order notifications with nothing wrong in their content.
- Verification records do not prevent an incident, they raise the ceiling; while missing, the
  system sits 12 points closer to the blacklist threshold at all times.
- The part that cannot be measured is the receiver's decision: the sending side sees
  acceptance and rejection, not whether a message lands in the inbox, and reputation is
  published nowhere.

#### Next Step

This topic has closed the name. The name now knows where it points, who may send messages in
its name is on record, what happens to a noncompliant message is declared, and where delivery
breaks down can be measured. Every item closed across the four lessons was about **the name**.

What remains is what the name points to. Every address record in the zone file leads to a
machine, and that machine has been sitting in a state never questioned throughout this topic:
reachable from everywhere since the day it was set up. Which ports are listening, where
administrative access is open from, and how many credentials with no expiry sit on it were
never counted. The next topic shifts the scale from the name to the machine and asks the same
question there: how many items does the default setup leave open?

## Hardening and Backup

Closing the surface left open by default and proving the way back: a default-deny firewall policy, key-based remote access and the bastion host, a central secret store and the lifetime of a dynamic credential, the patch window and the effect of automation on the exposure window, backup scope, frequency, and retention, and the evidentiary value of the restore drill.

### Firewall Configuration

Source: https://academia.sh/en/courses/servers-and-hardening/firewall-configuration

Open surface is counted item by item at the machine level: how many ports listen after install, how many zones can reach them, how many attempts stop once the switch to default-deny happens, how many legitimate flows close by mistake, and what stays open when the outbound direction is left unrestricted.

The previous topic closed with verifying the name and the sender: the query reaches the right name
server, and the email's origin can be proven. What was verified was the name itself; the machine
the name points to still stands with the settings it was installed with, and those settings left it
reachable from everywhere. The machine behind it can be reached without ever using the name — the
address is enough.

Network policy was measured earlier at the cluster-internal scale: which pod can reach which pod.
The scale here is the machine underneath the cluster. Three things are asked: how many items stay
open when nothing is done, how many rows closing them takes, and how many legitimate jobs stop when
they close.

- **HB1.** The arrangement of six machines and twenty-five listening ports belongs to the fictional
  regional measurement network. No real machine is run; what is measured is the model built with
  `node`.
- **HB2.** Source zones were reduced to four. In reality zone separation is written with address
  ranges; that grows the item count without changing the direction of the result.
- **HB3.** Attempt counts were written by hand; there is no randomness, every run produces the same
  numbers.
- **HB4.** A port is either open or closed; rate limiting and state tracking are not modeled.
- **HB5.** The allow list was not derived from a trace, it was written by hand; the model
  deliberately includes the oversight of the two rare flows marked with `SKIPPED`. In reality the
  number of missed flows depends on the observation period.
- **HB6.** The outbound direction was modeled at the destination-class level. In reality outbound
  rules are written by address and port; the item count grows, but the meaning of an unrestricted
  outbound direction does not change.

#### Rule Set Model

The model holds three things: which port is listening on which machine, how many attempts came to
which port from which zone during the period, and the allow list the operator wrote. Rules are
written in our own small schema: one row names a source zone, a machine, and a port name, and `*`
means no distinction is made on that axis.

```js
// firewall.mjs — machine, listening port, attempt, and rule set model; not a real network
export const ZONE = ['internal-net', 'office-net', 'field-net', 'external'];
const COMMON = [['management-shell', 'management'], ['metric-endpoint', 'auxiliary'], ['log-shipping', 'auxiliary']];
const SPECIFIC = {                          // ports the install leaves open per machine
  'collector-1': [['read-intake', 'work']], 'collector-2': [['read-intake', 'work']],
  'validator': [['validation-api', 'work']], 'billing': [['billing-api', 'work']],
  'data-store': [['data-endpoint', 'work'], ['backup-endpoint', 'work']], 'management': [['management-interface', 'management']],
};
export const MACHINE = Object.keys(SPECIFIC);
export const PORT = MACHINE.flatMap((m) => [...COMMON, ...SPECIFIC[m]].map(([portName, portKind]) => ({ machine: m, name: portName, type: portKind })));
export const key = (z, m, p) => `${z}|${m}|${p}`;

// Thirty-day attempt log: [zone, machine, port, count, legitimate?]
const perMachine = (z, p, f, ms) => MACHINE.map((m, i) => [z, m, p, f(i), ms]);
export const ATTEMPT = [
  ['field-net', 'collector-1', 'read-intake', 41200, true], ['field-net', 'collector-2', 'read-intake', 38900, true],
  ['internal-net', 'validator', 'validation-api', 26400, true], ['internal-net', 'data-store', 'data-endpoint', 91500, true],
  ['internal-net', 'billing', 'billing-api', 8600, true], ['office-net', 'billing', 'billing-api', 1240, true],
  ['office-net', 'management', 'management-interface', 860, true], ['office-net', 'management', 'management-shell', 310, true],
  ['office-net', 'data-store', 'backup-endpoint', 4, true],       // monthly backup verification
  ['office-net', 'validator', 'validation-api', 12, true], // end-of-quarter manual verification
  ...perMachine('internal-net', 'metric-endpoint', () => 8640, true), ...perMachine('internal-net', 'log-shipping', () => 30, true),
  ...perMachine('internal-net', 'management-shell', () => 62, true),     // config distribution
  ...perMachine('external', 'management-shell', (i) => 12000 + i * 1730, false),
  ...perMachine('external', 'metric-endpoint', (i) => 90 + i * 7, false),
  ['external', 'data-store', 'data-endpoint', 2400, false], ['external', 'data-store', 'backup-endpoint', 180, false],
  ['external', 'collector-1', 'read-intake', 640, false], ['external', 'management', 'management-interface', 5100, false],
  ['field-net', 'data-store', 'data-endpoint', 26, false], ['field-net', 'management', 'management-shell', 14, false],
  ['office-net', 'data-store', 'data-endpoint', 310, false],
];
export const LEGITIMATE = ATTEMPT.filter((d) => d[4]).map((d) => key(d[0], d[1], d[2]));
// The operator wrote the allow list by hand; two rare flows were missed.
const SKIPPED = ['office-net|data-store|backup-endpoint', 'office-net|validator|validation-api'];
export const ALLOW = LEGITIMATE.filter((a) => !SKIPPED.includes(a)).map((a) => a.split('|'));
// The same list can be written in three widths; '*' means no distinction on that axis.
export const widths = () => ({
  'port-name': [...new Set(ALLOW.map((r) => r[2]))].map((k) => ['*', '*', k]),
  'zone+port': [...new Set(ALLOW.map((r) => `${r[0]}|${r[2]}`))].map((s) => [s.split('|')[0], '*', s.split('|')[1]]),
  'zone+machine+port': ALLOW,
});
export function opened(rules) {             // rule set -> open item set
  const open = new Set();
  for (const [z, m, p] of rules)
    for (const zz of (z === '*' ? ZONE : [z]))
      for (const port of PORT)
        if ((m === '*' || port.machine === m) && (p === '*' || port.name === p)) open.add(key(zz, port.machine, port.name));
  return open;
}
export const ALL = opened([['*', '*', '*']]);  // the default-allow arrangement equals a single rule
export function count(open) {
  const closed = (d) => !open.has(key(d[0], d[1], d[2]));
  const sum = (f) => ATTEMPT.filter(f).reduce((a, d) => a + d[3], 0);
  return { items: open.size, unmatched: [...open].filter((a) => !LEGITIMATE.includes(a)).length,
    blocked: sum((d) => !d[4] && closed(d)), passed: sum((d) => !d[4] && !closed(d)),
    wronglyBlocked: ATTEMPT.filter((d) => d[4] && closed(d)).length };
}
export const OUTBOUND_LEGITIMATE = [
  ['collector-1', 'internal:queue'], ['collector-2', 'internal:queue'], ['validator', 'internal:data-store'],
  ['billing', 'internal:data-store'], ['billing', 'external:mail-gateway'], ['management', 'external:package-repo'],
  ['data-store', 'internal:backup-area'], ...MACHINE.map((m) => [m, 'external:time-source']),
];
export const OUTBOUND_ALLOW = OUTBOUND_LEGITIMATE.filter(([m, h]) => h !== 'external:time-source' || m === 'management');
```

The `ALL` row is the model's silent claim: the default-allow arrangement is equivalent to a single
`* -> * -> *` rule; nobody wrote that rule, but its effect is as if it had been. `count` takes an
open item set and returns five numbers: how many items are open, how many of them have no match,
how many illegitimate attempts were blocked, how many got through, and how many legitimate flows
were closed by mistake.

#### The Surface Left Open by Default

```js
// run1.mjs — items left open when nothing is done after install are counted
import { ZONE, MACHINE, PORT, ATTEMPT, LEGITIMATE, OUTBOUND_LEGITIMATE, ALL, count } from './firewall.mjs';

const s = count(ALL);
const byType = (t) => PORT.filter((k) => k.type === t).length;
const sum = (f) => ATTEMPT.filter(f).reduce((a, d) => a + d[3], 0);
const portType = (m, p) => PORT.find((x) => x.machine === m && x.name === p).type;
for (const [label, value, note = ''] of [
  ['listening port', PORT.length, `${MACHINE.length} machines: work ${byType('work')}, auxiliary ${byType('auxiliary')}, management ${byType('management')}`],
  ['inbound open item', s.items, `${PORT.length} ports x ${ZONE.length} zones`],
  ['  matching a legitimate flow', LEGITIMATE.length],
  ['  open unmatched', s.unmatched],
  ['management port reachable from external', byType('management')],
  ['outbound restriction', '-', `${MACHINE.length} machines have unrestricted targets, ${OUTBOUND_LEGITIMATE.length} legitimate flows`],
  ['thirty-day illegitimate attempt', sum((d) => !d[4]), `from ${new Set(ATTEMPT.filter((d) => !d[4]).map((d) => d[0])).size} source zones`],
  ['  hitting a management port', sum((d) => !d[4] && portType(d[1], d[2]) === 'management')],
  ['  blocked', s.blocked],
  ['  reached target', s.passed],
]) console.log(`${label.padEnd(41)}${String(value).padStart(7)}  ${note}`.trimEnd());
```

```
listening port                                25  6 machines: work 6, auxiliary 12, management 7
inbound open item                            100  25 ports x 4 zones
  matching a legitimate flow                  28
  open unmatched                              72
management port reachable from external        7
outbound restriction                           -  6 machines have unrestricted targets, 13 legitimate flows
thirty-day illegitimate attempt           107265  from 3 source zones
  hitting a management port               103064
  blocked                                      0
  reached target                          107265
```

Only six of the twenty-five listening ports are open for the work itself. The remaining nineteen
are for metric collection, log shipping, and the management shell; none of them were opened on
purpose, the install opened them. The hundred items come from these ports being reachable from all
four zones at once — twenty-eight of them have a matching legitimate flow, seventy-two do not.

Seven of the management ports are also reachable from the external zone, and one hundred three
thousand of the period's one hundred seven thousand illegitimate attempts land exactly there. The
last two rows are the punch line: zero of these attempts were blocked. A firewall layer whose
default decision is "allow" produces the same number as no layer at all. In the outbound direction
there is not even an item called restriction.

#### Default-Deny and Rule Width

```js
// run2.mjs — switch to default-deny; the same allow list is written at three rule widths
import { ZONE, ALLOW, widths, opened, ALL, count } from './firewall.mjs';

const row = (label, n, open) => {
  const s = count(open);
  console.log(`${label.padEnd(52)} ${String(n).padEnd(6)} ${String(s.items).padEnd(7)} ` +
    `${String(s.unmatched).padEnd(12)} ${String(s.blocked).padEnd(18)} ${String(s.passed).padEnd(17)} ${s.wronglyBlocked}`);
};
console.log(`the same allow list (${ALLOW.length} flows) at three widths:\n` +`${'rule width'.padEnd(52)} rows   items   unmatched    blocked attempts   passed attempts   wrongly blocked`);
row('default allow', 0, ALL);
for (const [label, k] of Object.entries(widths())) row(label, k.length, opened(k));
const narrow = widths()['zone+machine+port'];
const A = ['office-net', '*', '*'], B = ['external', 'management', 'management-shell'], C = ['*', '*', 'metric-endpoint'];
console.log('\nadding a temporary row to the narrowest shape:');
for (const [label, ek] of [['office-net -> * -> *', [A]], ['external -> management -> management-shell', [B]],
  ['all three together (+ * -> * -> metric-endpoint)', [A, B, C]]]) row(`  ${label}`, narrow.length + ek.length, opened([...narrow, ...ek]));
const wide = widths()['port-name'].map(([z, m, p]) => [p, opened([[z, m, p]]).size]).filter(([, n]) => n > 4);
console.log(`\nitems opened by a single rule written for a port name (${ZONE.length} zones x machines listening on that port):`);
console.log('  ' + wide.map(([p, n]) => `${p} ${n}`).join(', '));
```

```
the same allow list (26 flows) at three widths:
rule width                                           rows   items   unmatched    blocked attempts   passed attempts   wrongly blocked
default allow                                        0      100     72           0                  107265            0
port-name                                            8      96      69           180                107085            1
zone+port                                            10     31      5            107265             0                 2
zone+machine+port                                    26     26      0            107265             0                 2

adding a temporary row to the narrowest shape:
  office-net -> * -> *                               27     48      20           106955             310               0
  external -> management -> management-shell         27     27      1            86615              20650             2
  all three together (+ * -> * -> metric-endpoint)   29     61      33           85660              21605             0

items opened by a single rule written for a port name (4 zones x machines listening on that port):
  read-intake 8, management-shell 24, metric-endpoint 24, log-shipping 24
```

Switching to default-deny does not by itself produce a number. The `port-name` row shows this:
eight rules were written, it is default-deny, the allow list is applied — and the open items drop
from a hundred to ninety-six, and only a hundred eighty of the illegitimate attempts are blocked.
The last row gives the reason: `management-shell` written as a single rule opens twenty-four items,
because it spans all six machines and all four zones at once. The allow list had six rows for this
port, and all six were from the internal network; dropping the zone axis opens the external zone
too. Three broad rules produce seventy-two of the ninety-six items.

Adding the zone axis raises the row count from eight to ten: open items drop to thirty-one and all
illegitimate attempts are blocked. Two rows close one hundred seven thousand attempts. Adding the
machine axis too asks for twenty-six rows; unmatched items drop from five to zero, but the blocked
attempt count does not change. The zone axis is the axis that closes things, the machine axis only
cleans up the remainder: sixteen rows of extra work buy five items.

The wrongly-blocked column is two in both narrow shapes — the monthly backup verification and the
end-of-quarter manual verification marked with `SKIPPED` in the model, four and twelve attempts
during the period. In a trace of hundreds of thousands of rows, these sixteen rows are invisible.
Nothing breaks on the day the rule is written; it breaks at month- and quarter-end, and what shows
up is a connection error, not an allow-list row.

The middle block measures the surface closing itself introduces; the rule set is itself an item
that gets written, expanded for a reason, and mostly never rolled back. The wildcard row opened for
`office-net` adds twenty-two items but only lets three hundred ten attempts through; the single
management port opened to the external zone adds one item and lets twenty thousand six hundred
fifty attempts through — which zone an item faces matters more than the count. The wildcard row's
last column shows the real problem: wrongly-blocked drops from two to zero, because the row
accidentally reopens the two skipped legitimate flows. The gap goes unnoticed, and no reason arises
to remove the row. When all three stay together, twenty-nine rows open sixty-one items; the rule
set only grows in one direction, because adding a row has a justification and removing one does not.

#### Outbound Direction

Every item counted so far belonged to the inbound direction. The outbound direction is left
entirely unrestricted in most setups, and because it does not show up in the item count, nobody
notices it is a surface.

```js
// run3.mjs — outbound direction is counted separately; the inbound rule set touches none of these numbers
import { MACHINE, ALLOW, OUTBOUND_LEGITIMATE, OUTBOUND_ALLOW, widths, opened, count } from './firewall.mjs';

const extClasses = (l) => new Set(l.filter(([, h]) => h.startsWith('external:')).map(([, h]) => h)).size;
const items = count(opened(widths()['zone+machine+port'])).items;
const blocked = OUTBOUND_LEGITIMATE.filter(([k, h]) => !OUTBOUND_ALLOW.some(([a, b]) => a === k && b === h));
console.log('machine       default        allowed dest   blocked legitimate flow');
for (const m of MACHINE)
  console.log((`${m.padEnd(13)} ${'unrestricted'.padEnd(14)} ` +
    `${String(OUTBOUND_ALLOW.filter(([k]) => k === m).length).padEnd(14)} ` +
    blocked.filter(([k]) => k === m).map(([, h]) => h).join(', ')).trimEnd());
console.log(`\nlegitimate outbound flow         ${OUTBOUND_LEGITIMATE.length}   (${extClasses(OUTBOUND_LEGITIMATE)} external dest classes)`);
console.log(`outbound rule row                ${OUTBOUND_ALLOW.length}   (0 by default, unrestricted target)`);
console.log(`allowed external dest class      ${extClasses(OUTBOUND_ALLOW)}   (any, by default)`);
console.log(`wrongly blocked legitimate flow  ${blocked.length}`);
console.log(`inbound open item                ${items} -> ${items}   (unchanged)`);
console.log(`total rule row                   ${ALLOW.length} -> ${ALLOW.length + OUTBOUND_ALLOW.length}`);
```

```
machine       default        allowed dest   blocked legitimate flow
collector-1   unrestricted   1              external:time-source
collector-2   unrestricted   1              external:time-source
validator     unrestricted   1              external:time-source
billing       unrestricted   2              external:time-source
data-store    unrestricted   1              external:time-source
management    unrestricted   2

legitimate outbound flow         13   (3 external dest classes)
outbound rule row                8   (0 by default, unrestricted target)
allowed external dest class      3   (any, by default)
wrongly blocked legitimate flow  5
inbound open item                26 -> 26   (unchanged)
total rule row                   26 -> 34
```

Eight outbound rules bring the open target count per machine down to one or two; that column read
"unrestricted" by default. The difference is what a compromised machine can do: even if the inbound
rule set is complete, a process running inside that can connect out can exfiltrate data, take
commands, and establish its own persistence. Closing this path does not change a single inbound
item; open items stay at twenty-six, and the rule count rises to thirty-four.

The wrongly-blocked column is five. Whoever wrote the allow list saw the time-source dependency on
the management node and wrote it there, and did not think of it for the others. Its defining trait
is that it does not show up right away: a clock does not stop at once, it drifts. The drift surfaces
days later in signature verification, log ordering, and metric windows; all of it looks like a
symptom of something else. The two wrongly-blocked items in the inbound direction throw a
connection error and get fixed; the five in the outbound direction can be misdiagnosed for weeks.

This is also where the new surface closing introduces shows up. One of the three allowed external
destination classes is the package repository, and it is open for the management node; the price of
being able to fetch patches is that a path leading outward stays permanently open. The
thirty-four-row rule set leaves one item that cannot be closed: the management node, privileged in
both directions.

#### Summary

- After install, 25 ports listen across six machines; only 6 are for the work itself. Reachability
  from four zones turns this into 100 open items, 72 of which have no matching legitimate flow, and
  all 107,265 of the period's illegitimate attempts reach their target — 103,064 of them at
  management ports.
- Switching to default-deny is not enough by itself: 8 rows written by port name block only 180
  attempts. The zone axis closes all of the attempts with 10 rows; the machine axis asks for 26
  rows and drops the unmatched item count from 5 to 0 without adding anything to the blocked
  attempt count.
- Both narrow shapes wrongly block 2 legitimate flows; because the two together produce only 16
  attempts during the period, they do not show up in the trace and break at month- and quarter-end.
- The rule set itself is a surface: the single row opened to the external zone adds 1 item and lets
  through 20,650 attempts, and the wildcard row opened to the office network adds 22 items and
  hides the incomplete allow list by dropping wrongly-blocked from 2 to 0.
- When the outbound direction is restricted with 8 rows, the open target count per machine drops to
  1–2 and the inbound direction's 26 items do not change; the price is 5 wrongly-blocked flows that
  show up days later as clock drift. The item that cannot be closed is the one machine that faces
  outward so it can fetch patches.

#### Next Step

The narrowest rule set left the management shell open to the internal network on six machines. The
port narrowed, but what is behind it did not change: whoever connects still logs in with a password,
and password attempts can be made freely from the zones the firewall allows. The next lesson's
question is not which zone a port is open to, but what getting through the port depends on: with
password-based access, how many attempts come from how many sources in a period; how much of this
surface closes when switching to key-based access; how many copies a key exists in, where they sit,
and how many machines each is authorized on; how many touches revoking it takes — and reducing
access to a single bastion host lowers the number of directly reachable machines while introducing
what single target in return.

### Remote Access Hardening

Source: https://academia.sh/en/courses/servers-and-hardening/remote-access-hardening

The attempt surface of password-based access is counted over a period, the surface that closes and the surface that opens when switching to key-based access are measured separately, and adding a bastion host compares the number of directly reachable machines against the single target it gives rise to.

The previous lesson left the firewall at default-deny: the management shell is now reachable only
from the internal network, and the management node additionally from the office network. The port
narrowed. What getting through the port depends on did not change — whoever connects still types a
password, and the password can be tried from every source in the allowed zones. This lesson's
measure is not the width of the port, it is the lock behind it.

- **HB7.** Six machines, three shell accounts, and five principals belong to the fictional regional
  measurement network. No real access is set up; what is measured is the model written with `node`.
- **HB8.** The attempt log was produced with a generator whose seed is visible; every run gives the
  same series. In reality the distribution varies by zone and period, but the order of magnitude of
  the numbers does not change.
- **HB9.** The key strings and fingerprints are fictional; no real key was written. The fingerprint
  exists only to show that the same key sits in two places.
- **HB10.** Updating the authorized-keys file on one machine was counted as four minutes. In
  reality this time varies with automation; the total revocation cost stays proportional to the
  radius.
- **HB11.** The key has no lifetime; it is valid until revoked. A time-limited credential is
  outside this lesson.
- **HB12.** The bastion host was modeled as a single machine. A second one would lower the
  availability item and raise the single-target item to two.

#### The Attempt Surface of the Password Arrangement

The model holds three things: which sources the firewall lets reach, the password attempts
observed over a thirty-day period, and the copy set that will appear once the switch to key-based
access happens. The attempt log is built with our own generator instead of being written by hand;
the seed is visible in the code.

```js
// access.mjs — remote access principals, key copies, and the attempt generator; model, not real access
export const MACHINE = ['collector-1', 'collector-2', 'validator', 'billing', 'data-store', 'management'];
export const ACCOUNT = ['ops', 'deploy', 'maintenance'];   // shell accounts open on every machine
export const REACHABLE = { internal: 90, office: 40 };     // sources the firewall lets reach
export const SHARED = { ops: 6, deploy: 6, maintenance: 1 }; // machines carrying the same password
const SEED = 20260401;                                  // generator seed; every run gives the same series
function* generator(t = SEED) {                         // our own linear-congruential generator
  let x = t;
  for (;;) { x = (x * 1103515245 + 12345) % 2147483648; yield x / 2147483648; }
}
export function attemptLog(n = 400) {                    // thirty-day password attempt log
  const g = generator(); const log = [];
  const s = (k) => Math.floor(g.next().value * k);
  for (let i = 0; i < n; i += 1) {
    const zone = g.next().value < 0.7 ? 'internal' : 'office';
    const heavy = g.next().value < 0.04;                // a slice of sources try heavily
    log.push({ source: `${zone}-${s(REACHABLE[zone]) + 1}`, machine: MACHINE[s(6)], account: ACCOUNT[s(3)],
      count: heavy ? 400 + s(900) : 1 + s(6) });
  }
  return log;
}
// Key copies; principal, where the copy sits, and a fictional fingerprint.
export const COPY = [
  ['op-a', 'laptop', 'kz7f-11ac'], ['op-a', 'backup-archive', 'kz7f-11ac'],
  ['op-b', 'laptop', 'kz7f-2be0'], ['op-c', 'laptop', 'kz7f-3d41'],
  ['deploy', 'config-tool', 'kz7f-5f92'], ['deploy', 'integration-runner', 'kz7f-5f92'],
  ['maintenance-ext', 'external-machine', 'kz7f-6a08'],
].map(([principal, location, fingerprint]) => ({ principal, location, fingerprint, name: `${principal}/${location}` }));
// Which principal is authorized on which machine (authorized_keys files).
export const AUTHORIZED = {
  'op-a': MACHINE, 'op-b': MACHINE, 'op-c': ['management', 'billing'],
  'deploy': MACHINE, 'maintenance-ext': ['data-store', 'management'],
};
export const TOUCH_MIN = 4;                              // time to update authorized_keys on one machine
export const radius = (c, a = AUTHORIZED) => a[c.principal].length;
export const item = (a = AUTHORIZED) => COPY.reduce((acc, c) => acc + radius(c, a), 0);
```

```js
// run1.mjs — the attempt surface of password-based access is counted from the thirty-day log
import { MACHINE, ACCOUNT, REACHABLE, SHARED, attemptLog } from './access.mjs';

const log = attemptLog();
const top = (l) => l.reduce((a, x) => a + x.count, 0);
const heavy = log.filter((x) => x.count >= 100);
const reachable = REACHABLE.internal + REACHABLE.office;
for (const [label, v, note = ''] of [
  ['password attempt', top(log)],
  ['distinct source that tried', new Set(log.map((x) => x.source)).size, `${reachable} sources can reach`],
  ['target (machine x account)', MACHINE.length * ACCOUNT.length],
  ['attemptable triple', reachable * MACHINE.length * ACCOUNT.length, 'reachable sources x targets'],
  ['heavy source (>=100 attempts)', new Set(heavy.map((x) => x.source)).size, `produces ${top(heavy)} attempts`],
  ['untried target', MACHINE.length * ACCOUNT.length - new Set(log.map((x) => `${x.machine}|${x.account}`)).size],
]) console.log(`${label.padEnd(31)}${String(v).padStart(6)}  ${note}`.trimEnd());
console.log('\nmachines that fall if a single password leaks:');
for (const h of ACCOUNT) console.log(`  ${h.padEnd(13)} ${SHARED[h]}`);
```

```
password attempt                16505
distinct source that tried        124  130 sources can reach
target (machine x account)         18
attemptable triple               2340  reachable sources x targets
heavy source (>=100 attempts)      17  produces 15159 attempts
untried target                      0

machines that fall if a single password leaks:
  ops           6
  deploy        6
  maintenance   1
```

The firewall closed the external zone, and the attempt surface is nonzero regardless: ninety
sources from the internal network and forty from the office network can reach the management
shell. During the period, one hundred twenty-four of these sources actually tried and produced
sixteen thousand five hundred five attempts. Seventeen heavy sources produce fifteen thousand one
hundred fifty-nine of that on their own; block those seventeen and the remaining hundred seven
sources stay put. The number that needs measuring is not how many tried, it is how many could try: a
hundred thirty sources times eighteen targets, that is 2340 triples. All eighteen targets were
tried at least once during the period.

The bottom block gives the second dimension of the surface. The ops and deploy accounts carry the
same password on six machines, because the machines were built with the same install step. One
password leaking means six machines; the maintenance account is machine-specific, so the number
there is one. Two different radii sit side by side in the same arrangement, and the difference
does not come from a security decision, it comes from install convenience.

#### Key-Based Access

```js
// run2.mjs — switch to key-based access; the closed surface and the newly opened surface are counted separately
import { MACHINE, ACCOUNT, REACHABLE, COPY, TOUCH_MIN, attemptLog, radius, item } from './access.mjs';

const log = attemptLog();
console.log(`closed   ${log.reduce((a, x) => a + x.count, 0)} password attempts, ` +
  `${new Set(log.map((x) => x.source)).size} sources -> 0`);
console.log(`opened   ${COPY.length} copies, ${new Set(COPY.map((k) => k.principal)).size} principals, ` +
  `${new Set(COPY.map((k) => k.fingerprint)).size} distinct keys, no lifetime\n`);
console.log('copy                               location             authorized  revoke min  location carrying same key');
for (const k of COPY) {
  const es = COPY.filter((x) => x.fingerprint === k.fingerprint && x.name !== k.name).map((x) => x.location);
  console.log(`${k.name.padEnd(35)} ${k.location.padEnd(21)} ${String(radius(k)).padEnd(12)} ` +
    `${String(radius(k) * TOUCH_MIN).padEnd(12)} ${es.join(', ') || '-'}`);
}
const triple = (REACHABLE.internal + REACHABLE.office) * MACHINE.length * ACCOUNT.length;
const total = log.reduce((a, x) => a + x.count, 0);
console.log('\narrangement          closed attempts  attemptable triple  key item');
for (const [label, closed, left, ak] of [['password only', 0, triple, 0],
  ['password + key', 0, triple, item()], ['key only', total, 0, item()]])
  console.log(`${label.padEnd(21)} ${String(closed).padEnd(17)} ${String(left).padEnd(20)} ${ak}`);
const loc = [...new Set(COPY.map((k) => k.location))];
console.log(`\n(copy, machine) item               ${item()}`);
console.log(`distinct location a copy sits in   ${loc.length}   ${loc.join(', ')}`);
console.log(`widest copy radius                 ${Math.max(...COPY.map((k) => radius(k)))}`);
console.log(`revoking every copy                ${item() * TOUCH_MIN} min`);
```

```
closed   16505 password attempts, 124 sources -> 0
opened   7 copies, 5 principals, 5 distinct keys, no lifetime

copy                               location             authorized  revoke min  location carrying same key
op-a/laptop                         laptop                6            24           backup-archive
op-a/backup-archive                 backup-archive        6            24           laptop
op-b/laptop                         laptop                6            24           -
op-c/laptop                         laptop                2            8            -
deploy/config-tool                  config-tool           6            24           integration-runner
deploy/integration-runner           integration-runner    6            24           config-tool
maintenance-ext/external-machine    external-machine      2            8            -

arrangement          closed attempts  attemptable triple  key item
password only         0                 2340                 0
password + key        0                 2340                 34
key only              16505             0                    34

(copy, machine) item               34
distinct location a copy sits in   5   laptop, backup-archive, config-tool, integration-runner, external-machine
widest copy radius                 6
revoking every copy                136 min
```

The first row is the surface that closes: sixteen thousand five hundred five attempts drop to zero,
and none of the hundred twenty-four sources can try anymore. What closes is not the password, it is
guessability — there is nothing left to try. The second row counts what opens at the same time:
seven copies, five principals, five distinct keys. The copy count exceeds the principal count
because the same key sits in two separate places.

The table shows where these copies sit and how much each carries. The key sits on a laptop, in a
backup archive, on a config tool, on an integration runner, and on an external machine — five
separate places, each guarded separately. The authorized column is each copy's radius: four copies
are valid on six machines at once. There are thirty-four `(copy, machine)` items in total, and each
one is a door that opens if that copy leaks. The surface counted in the password arrangement did not
disappear, it changed shape: instead of a hundred thirty guessable sources, five places that need
guarding.

The three-row block in the middle measures the transition itself, and it gives this lesson's
easiest number to skip. Turning on key-based access closes nothing by itself: as long as password
entry stays open too, closed attempts are zero and the attemptable triple stays at 2340. Against
that, the new thirty-four-item surface is there the moment it is opened. The in-between state
carries both arrangements' surface at once, and most setups stay in this in-between state for
weeks, because closing password entry before key distribution is finished risks locking someone
out. The step that closes something is not adding the key, it is removing the password; unless the
two are counted separately, the first stands in for the second.

The revoke column gives the cost of the decision. If a copy leaks, the authorized-keys file has to
be touched on every machine that principal is authorized on: the widest copy is six machines,
twenty-four minutes; every copy together, a hundred thirty-six minutes. The last column is
revocation's side effect. The copy in the backup archive is the same key as the copy on the
operator's laptop; revoking one revokes the other. A revocation made without knowing which copy the
leak came from closes the operator's own access too.

The "no lifetime" at the end of the second row is this lesson's item that cannot be closed. A
password attempt was something visible: it was counted, its source was known, its clustering was
noticed. A login made with a leaked key is indistinguishable from a legitimate one, and the key is
valid until revoked. What closes the exposure window is not time running out, it is someone noticing.

#### Bastion Host

```js
// run3.mjs — a bastion host is added; direct access drops, a single target is born in return
import { MACHINE, COPY, AUTHORIZED, TOUCH_MIN, radius, item } from './access.mjs';

const BASTION = Object.fromEntries(Object.keys(AUTHORIZED).map((o) => [o, ['bastion']]));
const INTERNAL_KEY = 1;                                 // key sitting on the bastion, authorized on every machine
const measure = (a) => [item(a), Math.max(...COPY.map((k) => radius(k, a))), item(a) * TOUCH_MIN];
const [aK, aY, aD] = measure(AUTHORIZED), [bK, bY, bD] = measure(BASTION);
console.log('measure                                      direct    with bastion');
for (const [label, x, y] of [
  ['directly reachable machine', MACHINE.length, 1],
  ['(copy, machine) item', aK, bK],
  ['machines that fall if one copy leaks', aY, bY],
  ['revoking every copy (min)', aD, bD],
  ['internal key sitting at a single point', 0, INTERNAL_KEY],
  ['machines the internal key is authorized on', 0, MACHINE.length],
  ['total access item', aK, bK + INTERNAL_KEY * MACHINE.length],
]) console.log(`${label.padEnd(44)} ${String(x).padEnd(9)} ${y}`);
console.log("\nthe bastion host's own surface (item by item):");
for (const [label, n] of [['listening port', 3], ['authorized principal', Object.keys(BASTION).length],
  ['internal key sitting on it', INTERNAL_KEY], ['single point holding session records', 1],
  ['emergency escape route (direct access)', 1], ['machines unreachable if it goes down', MACHINE.length],
  ['machines reached if it is compromised', MACHINE.length]]) console.log(`  ${label.padEnd(40)}${n}`);
```

```
measure                                      direct    with bastion
directly reachable machine                   6         1
(copy, machine) item                         34        7
machines that fall if one copy leaks         6         1
revoking every copy (min)                    136       28
internal key sitting at a single point       0         1
machines the internal key is authorized on   0         6
total access item                            34        13

the bastion host's own surface (item by item):
  listening port                          3
  authorized principal                    5
  internal key sitting on it              1
  single point holding session records    1
  emergency escape route (direct access)  1
  machines unreachable if it goes down    6
  machines reached if it is compromised   6
```

The first four rows make the case for the step: directly reachable machines go from six to one, the
`(copy, machine)` item from thirty-four to seven, machines falling if one copy leaks from six to
one, and the time to revoke every copy drops from a hundred thirty-six minutes to twenty-eight. All
four columns move the same direction. Key copies are now authorized on only a single machine, so a
stolen laptop means one machine, not six.

The next three rows give the price. An internal key authorized on all six machines sits on the
bastion host; without it the bastion host is useless. Total access items drop from thirty-four to
thirteen — not to zero. What actually changes is the distribution: thirty-four items were spread
across five separate places, and six of the thirteen items concentrated on a single machine. The
surface shrank and concentrated at the same time; these two sentences are not substitutes for each
other, both are true at once.

The bottom block counts the bastion host's own surface item by item: three listening ports, five
authorized principals, one internal key sitting on it, a single point holding the session record.
The last two rows read the same machine from two directions. If the bastion host goes down, none of
the six machines keep management access; that is why an emergency escape route is left in place, and
that escape route is exactly the direct access that was meant to be closed. If it is compromised,
the same six machines are reached. A bastion host is not a defense, it is a point of concentration:
its value comes from making it enough to watch one machine instead of six — as long as that one
machine is actually watched.

#### Summary

- Even after the firewall closes the external zone, 130 sources can reach the management shell; 124
  of them tried during the period and produced 16,505 attempts, and 17 heavy sources produce 15,159
  of that on their own. The measured surface is not who tried, it is the 130 × 18 = 2340 attemptable
  triples.
- Sharing the same password on six machines is a second radius item: a single leak in the ops and
  deploy accounts means 6 machines, and in the machine-specific maintenance account, 1 machine.
- Switching to key-based access closes all 16,505 attempts and 124 sources; in return it opens 7
  copies, 5 separate places, and 34 `(copy, machine)` items. The surface does not disappear, it
  changes shape.
- Revocation cost is proportional to radius: the widest copy is 6 machines and 24 minutes, every
  copy together is 136 minutes. The same key sitting in two places gives revocation a side effect —
  revoking one copy closes the other too.
- The key has no lifetime: a login made with a leaked key is indistinguishable from a legitimate
  one, and the window closes only once someone notices. A password attempt could be counted; this
  cannot.
- The bastion host drops directly reachable machines from 6 to 1, items from 34 to 7, and
  revocation time from 136 minutes to 28; in return an internal key authorized on 6 machines
  concentrates on a single machine, total items stay at 13, and going down loses access to 6
  machines while being compromised reaches the same 6 machines.

#### Next Step

This lesson's key is valid until revoked. The same is true of the credentials applications carry:
the data-store password, the queue key, and the mail-gateway token are written once, and nobody
says when they will change. The next lesson's axis is therefore lifetime. In a secret with no
lifetime, the exposure window is infinite; in a credential with a defined lifetime, the window is
as long as the lifetime. Lifetime will be scanned: a short lifetime lowers the window but raises the
renewal load and the dependency on the central store, and how many services become unable to work
and after how many minutes when the store goes down gets counted. The central store itself is also
a surface, and it is measured by how many principals can reach how many secrets.

### Secret Management Systems

Source: https://academia.sh/en/courses/servers-and-hardening/secret-management-systems

A secret's axis is lifetime: in a static secret the exposure window is infinite, in a credential with a defined lifetime the window is as long as the lifetime. Lifetime is scanned, renewal load and resilience to store outages are counted, and the central store's own access surface is worked out item by item.

The previous lesson's key was valid until revoked; that was the one item that could not be measured.
The same is true of the credentials applications carry: the data-store credential, the queue key,
and the mail-gateway token are written on install day, and nowhere does it say when they will change.
How secrets are stored in production was measured in M22/K01, and how the deployment pipeline
reaches secrets was measured in M22/K04; neither is repeated here. This lesson's only axis is
lifetime — how long a credential stays valid, and how wide the window stays open if it leaks.

- **HB13.** Five secrets, six services, and three environments belong to the fictional regional
  measurement network. No real secret was written, the names are fictional; what is measured is the
  model written with `node`.
- **HB14.** The period was counted as thirty days, that is 43,200 minutes.
- **HB15.** On a store outage, instances' remaining lifetime was spread evenly across the lifetime
  interval; there is no randomness. In reality the distribution depends on renewal timing, but the
  direction of the result does not change.
- **HB16.** A credential is either valid or it is not. Partial authorization, retry on renewal
  failure, and cached sessions are not modeled.
- **HB17.** A secret's content, how it is generated, and how it is encrypted are not measured. The
  only things measured are lifetime and access items.
- **HB18.** The central store was modeled as a single place. A replicated store would lower the
  outage item, not the access item.

#### Static Secret Arrangement

The model holds five secrets, the six services that need secrets, three environments, and the
locations a secret sits in under the static arrangement. The lifetime list gives the values to be
scanned; `null` denotes no lifetime, that is, a static secret.

```js
// secret.mjs — secret set, access matrix, and lifetime model; no real secret, names are fictional
export const SECRET = ['data-store-credential', 'queue-key', 'mail-gateway-token',
  'metric-write-key', 'archive-key'];
export const SERVICE = {                                  // service -> secrets it needs
  'read-collector': ['data-store-credential', 'queue-key'],
  'validator': ['data-store-credential', 'queue-key'],
  'billing': ['data-store-credential', 'mail-gateway-token'],
  'work-order': ['data-store-credential', 'queue-key', 'mail-gateway-token'],
  'nightly-batch': ['data-store-credential', 'archive-key'],
  'metric-collector': ['metric-write-key'],
};
export const ENV = ['dev', 'staging', 'prod'];
export const INSTANCE = Object.keys(SERVICE).flatMap((s) => ENV.map((e) => ({ service: s, env: e })));
export const STATIC_LOCATION = ['config-repo', 'env-var', 'backup-image', 'developer-laptop'];
export const PERIOD_MIN = 30 * 24 * 60;                    // thirty-day period
export const LIFETIME = [5, 15, 60, 480, 1440, null];       // minutes; null = static, no lifetime
// Human and automation principals; which secrets each can access.
export const PRINCIPAL = {
  'op-a': SECRET, 'op-b': ['data-store-credential', 'queue-key', 'archive-key'],
  'op-c': ['mail-gateway-token'], 'deploy': SECRET, 'monitoring': ['metric-write-key'],
};
// Not every secret can be generated dynamically; the second field is the reason.
export const DYNAMIC = {
  'data-store-credential': [true, 'the store account can be opened and closed on demand'],
  'queue-key': [true, 'the queue can grant temporary authorization'],
  'metric-write-key': [true, 'write authorization can be granted short-term'],
  'mail-gateway-token': [false, 'issued by an external party, we cannot generate it'],
  'archive-key': [false, 'an encryption key; rotating it makes the old archive unreadable'],
};
export const instanceItem = () => INSTANCE.reduce((a, o) => a + SERVICE[o.service].length, 0);
export const principalItem = () => Object.values(PRINCIPAL).reduce((a, s) => a + s.length, 0);
export const rotationEffect = (s) => INSTANCE.filter((o) => SERVICE[o.service].includes(s)).length;
// Each instance's remaining lifetime on a store outage: interval split into equal steps, no randomness.
export const remainingLifetime = (life, n = INSTANCE.length) =>
  Array.from({ length: n }, (_, i) => (life * (i + 0.5)) / n);
```

```js
// run1.mjs — items and rotation cost in the static secret arrangement are counted
import { SECRET, SERVICE, ENV, INSTANCE, STATIC_LOCATION, instanceItem, rotationEffect } from './secret.mjs';

for (const [label, v, note = ''] of [
  ['secret', SECRET.length],
  ['service x env = instance', INSTANCE.length, `${Object.keys(SERVICE).length} services, ${ENV.length} envs`],
  ['(instance, secret) item', instanceItem()],
  ['distinct location a secret sits in', STATIC_LOCATION.length, STATIC_LOCATION.join(', ')],
  ['secret copy (secret x location)', SECRET.length * STATIC_LOCATION.length],
  ["secret's lifetime", 'none', 'unchanged since install'],
]) console.log(`${label.padEnd(36)}${String(v).padStart(6)}  ${note}`.trimEnd());
console.log('\nrotating a secret restarts how many instances:');
for (const s of SECRET) console.log(`  ${s.padEnd(24)} ${rotationEffect(s)}`);
```

```
secret                                   5
service x env = instance                18  6 services, 3 envs
(instance, secret) item                 36
distinct location a secret sits in       4  config-repo, env-var, backup-image, developer-laptop
secret copy (secret x location)         20
secret's lifetime                     none  unchanged since install

rotating a secret restarts how many instances:
  data-store-credential    15
  queue-key                9
  mail-gateway-token       6
  metric-write-key         3
  archive-key              3
```

Five secrets, spread across eighteen service instances, come to thirty-six `(instance, secret)`
items. On top of that, a secret sits in four separate places: the config repo, the running
process's environment variable, the backup image, and the developer's laptop. Five secrets times
four locations, twenty copies. Two of these locations — the backup image and the laptop — keep no
access record; a secret read from there shows up nowhere as having been read.

The bottom block gives the reason rotation does not happen, in numbers. The data-store credential is
used in fifteen instances; changing it means restarting fifteen instances, and these fifteen
instances are spread across three separate environments. Rotation's cost is exactly the reason it
does not happen. The result is the "lifetime: none" row in the table — the secret has been the same
since install, so if it leaked, it has been valid since the day it leaked. The exposure window here
is not a duration, it is a state: open until somebody notices.

#### Lifetime Scan

```js
// run2.mjs — lifetime is scanned: exposure window, renewal load, and how it survives a store outage
import { INSTANCE, PERIOD_MIN, LIFETIME, remainingLifetime } from './secret.mjs';

const d1 = (x) => (Math.round(x * 10) / 10).toFixed(1);
console.log('lifetime   exposure window    in period   on store outage          on 30 min outage');
console.log('(min)      (avg min)          renewals    first drop / all (min)   instances unable');
for (const life of LIFETIME) {
  if (life === null) {
    console.log(`${'static'.padEnd(10)} ${'infinite'.padEnd(18)} ${String(0).padEnd(11)} ` +
      `${'-'.padEnd(24)} 0`);
    continue;
  }
  const left = remainingLifetime(life);
  console.log(`${String(life).padEnd(10)} ${d1(life / 2).padEnd(18)} ` +
    `${String(INSTANCE.length * Math.round(PERIOD_MIN / life)).padEnd(11)} ` +
    `${`${d1(left[0])} / ${d1(left[left.length - 1])}`.padEnd(24)} ${left.filter((k) => k < 30).length}`);
}
```

```
lifetime   exposure window    in period   on store outage          on 30 min outage
(min)      (avg min)          renewals    first drop / all (min)   instances unable
5          2.5                155520      0.1 / 4.9                18
15         7.5                51840       0.4 / 14.6               18
60         30.0               12960       1.7 / 58.3               9
480        240.0              1620        13.3 / 466.7             1
1440       720.0              540         40.0 / 1400.0            0
static     infinite           0           -                        0
```

The table lays three separate outcomes of one decision side by side. At a five-minute lifetime, the
exposure window drops to an average of two and a half minutes — a leaked credential is almost
useless. The price is in the two columns on the right: one hundred fifty-five thousand five hundred
twenty renewals during the period, and once an outage starts at the store, the first instance
becomes unable to work within six seconds, all of them within five minutes. The static row is the
exact opposite: zero renewals, no effect from an outage at all, and an infinite window.

The rows in between are where the decision actually gets made. At a sixty-minute lifetime the
window drops to half an hour, renewals fall to twelve thousand nine hundred sixty, but on a
thirty-minute store outage, nine of the eighteen instances become unable to work. At an eight-hour
lifetime the same outage drops only one instance, and the window rises to four hours. At a
twenty-four-hour lifetime the outage does not drop a single instance, and the window is twelve
hours.

The second column from the right is this table's real finding: lifetime sets the outage duration
the store can tolerate. As long as the outage stays under the chosen lifetime, most instances stay
up; once the outage exceeds the lifetime, all of them drop. So the decision to shorten the exposure
window is, without anyone noticing, the decision that the store has to run without outages.
When a five-minute lifetime is chosen, the store becomes a dependency that cannot disappear for
longer than five minutes for all eighteen instances at once — and that introduces a new
availability item that no secret used to carry.

#### Secrets Whose Lifetime Cannot Be Shortened

The lifetime scan assumes every secret's lifetime can be chosen. How many of the five secrets
actually have a choosable lifetime is a separate number.

```js
// run4.mjs — not every secret can be generated dynamically; items outside the lifetime axis are counted
import { SECRET, SERVICE, INSTANCE, DYNAMIC, instanceItem } from './secret.mjs';

console.log('secret                   dynamic   reason');
for (const s of SECRET) console.log(`  ${s.padEnd(24)} ${(DYNAMIC[s][0] ? 'yes' : 'no').padEnd(9)} ${DYNAMIC[s][1]}`);
const static_ = (s) => !DYNAMIC[s][0];
const item = INSTANCE.reduce((a, o) => a + SERVICE[o.service].filter(static_).length, 0);
const inst = INSTANCE.filter((o) => SERVICE[o.service].some(static_));
console.log('');
for (const [label, v] of [
  ['secret whose lifetime can be shortened', `${SECRET.filter((s) => !static_(s)).length} / ${SECRET.length}`],
  ['secret that must stay static', `${SECRET.filter(static_).length} / ${SECRET.length}`],
  ['static (instance, secret) item', `${item} / ${instanceItem()}`],
  ['instance carrying at least one static secret', `${inst.length} / ${INSTANCE.length}`],
  ['exposure window in these instances', 'infinite'],
]) console.log(`${label.padEnd(46)}${v}`);
console.log(`\ncarrying services: ${[...new Set(inst.map((o) => o.service))].join(', ')}`);
```

```
secret                   dynamic   reason
  data-store-credential    yes       the store account can be opened and closed on demand
  queue-key                yes       the queue can grant temporary authorization
  mail-gateway-token       no        issued by an external party, we cannot generate it
  metric-write-key         yes       write authorization can be granted short-term
  archive-key              no        an encryption key; rotating it makes the old archive unreadable

secret whose lifetime can be shortened        3 / 5
secret that must stay static                  2 / 5
static (instance, secret) item                9 / 36
instance carrying at least one static secret  9 / 18
exposure window in these instances            infinite

carrying services: billing, work-order, nightly-batch
```

Dynamic generation is possible for three secrets: the data-store account can be opened and closed
on demand, the queue can grant temporary authorization, metric write authorization can be defined
short-term. It is not possible for two secrets, and their reasons are different in kind. The
mail-gateway token is issued by an external party; we are not the party that sets its lifetime, so
there is nothing to do on the lifetime axis. The archive key, on the other hand, is an encryption
key; when it is rotated the old archive becomes unreadable, meaning shortening its lifetime breaks
the secret's function.

The numbers narrow the scan table's scope. Nine of the thirty-six `(instance, secret)` items fall
outside the lifetime axis, and nine of the eighteen instances — billing, work-order, and
nightly-batch — keep carrying at least one static secret. For these nine instances, every column of
the previous section is void: the exposure window is not as long as the lifetime, it is infinite.
Even if a five-minute lifetime is chosen, half the system carries a static secret. The lifetime axis
is a real axis, but it cannot be applied to every secret, and the ones it cannot be applied to are
exactly the two secrets that face outward: the token going to the mail gateway and the key that
opens the archive.

#### The Central Store's Own Surface

```js
// run3.mjs — the central store's own surface: how many principals can access how many secrets
import { SECRET, INSTANCE, STATIC_LOCATION, PRINCIPAL, instanceItem, principalItem } from './secret.mjs';

console.log('principal   secret accessible');
for (const [o, s] of Object.entries(PRINCIPAL))
  console.log(`  ${o.padEnd(12)} ${String(s.length).padEnd(3)} ${s.length === SECRET.length ? '(all)' : s.join(', ')}`);
console.log(`  ${'service'.padEnd(12)} ${instanceItem()}  (${INSTANCE.length} instances, avg ${instanceItem() / INSTANCE.length} secrets)`);
const total = principalItem() + instanceItem();
console.log(`\nmeasure                                             static   central store`);
for (const [label, a, b] of [
  ['distinct location a secret sits in', STATIC_LOCATION.length, 1],
  ['secret copy', SECRET.length * STATIC_LOCATION.length, SECRET.length],
  ['point carrying every secret', STATIC_LOCATION.length, 1],
  ['(principal, secret) item', total, total],
  ['logged and revocable', 0, total],
  ['principal that can access every secret', Object.values(PRINCIPAL).filter((s) => s.length === SECRET.length).length,
    Object.values(PRINCIPAL).filter((s) => s.length === SECRET.length).length],
  ['instance that cannot renew if the store goes down', 0, INSTANCE.length],
]) console.log(`${label.padEnd(51)} ${String(a).padEnd(8)} ${b}`);
```

```
principal   secret accessible
  op-a         5   (all)
  op-b         3   data-store-credential, queue-key, archive-key
  op-c         1   mail-gateway-token
  deploy       5   (all)
  monitoring   1   metric-write-key
  service      36  (18 instances, avg 2 secrets)

measure                                             static   central store
distinct location a secret sits in                  4        1
secret copy                                         20       5
point carrying every secret                         4        1
(principal, secret) item                            51       51
logged and revocable                                0        51
principal that can access every secret              2        2
instance that cannot renew if the store goes down   0        18
```

The top block is the store's own access surface. Two of the five principals can access all five of
the five secrets at once: one operator and the deployment automation. The matrix is written,
genuinely narrowed for three principals, not narrowed at all for two. Service instances add
thirty-six items with an average of two — services are narrow, people are broad. There are
fifty-one `(principal, secret)` items in total, and each one is a secret that opens if that
principal is compromised.

The bottom table lays the two arrangements side by side, and the first three rows are the central
store's gain: the location a secret sits in drops from four to one, copies from twenty to five, the
point carrying every secret from four to one. The fourth row is not a gain: the `(principal, secret)`
item stays at fifty-one. The central store does not reduce the access count. The static arrangement
had the same fifty-one access items; the difference was that none of them were written down
anywhere.

The fifth row is this lesson's real gain. In the static arrangement, logged and revocable access
was zero — a secret read from the backup image had neither a record nor an individually revocable
state. In the central store, fifty-one of the fifty-one accesses are both loggable and revocable.
What the central store gives is not less access, it is access that can be counted and revoked.

The last row repeats the price: when the store goes down, eighteen instances cannot renew, whereas
zero instances would have been affected in the static arrangement. This is also where the item that
cannot be closed sits — the single point carrying every secret. The static arrangement had four of
these points, the central store has one. Going from four to one is a real gain, but it does not
reach zero, and that single point now carries both every secret and every service's ability to keep
working.

#### Summary

- In the static arrangement, 5 secrets spread across 18 instances come to 36 `(instance, secret)`
  items and sit as 20 copies total across 4 separate locations; two of these locations keep no
  access record. Rotation cost is the reason rotation does not happen: changing the most-shared
  secret requires restarting 15 instances, so the secret has been the same since install and the
  exposure window is a state, not a duration — open until somebody notices.
- The lifetime scan gives three outcomes together: at a 5-minute lifetime the window is 2.5
  minutes, but the period has 155,520 renewals and an outage drops all 18 instances within 5
  minutes; in the static row the window is infinite, renewals are 0, and an outage has no effect at
  all.
- Lifetime sets the outage duration the store can tolerate: at a 60-minute lifetime a 30-minute
  outage drops 9 of 18 instances, at 480 minutes it drops 1, at 1440 minutes it drops none.
- The lifetime axis cannot be applied to every secret: 2 of the 5 secrets must stay static (one is
  issued by an external party, the other is an encryption key), and 9 of the 36 items, 9 of the 18
  instances, fall outside the lifetime axis and keep an infinite window.
- The central store drops the location a secret sits in from 4 to 1 and copies from 20 to 5, but
  the `(principal, secret)` item stays at 51; 2 of the 5 principals access every secret. The gain
  is not less access, it is loggable and revocable access going from 0 to 51.
- The item that cannot be closed is the single point carrying every secret: there were 4 in the
  static arrangement, 1 in the central store — but that single point now also carries the working
  ability of 18 instances.

#### Next Step

These three lessons left three separate points of concentration: the management node that faces
outward, the bastion host that all access passes through, and the store that carries every secret.
All three share one trait — all three are software, and all three run software that ages. A
hardening step holds only as long as the code it runs has no known vulnerability. The next lesson's
question is therefore patches: between the moment a fix is published and the moment it is applied to
machines, how many machines stay open for how long; how many restarts and how many minutes of
outage does narrowing the patch window cost; and what does the difference between an automatically
applied patch and an approved-then-applied patch come to, in days and in machines.

### Patch Management

Source: https://academia.sh/en/courses/servers-and-hardening/patch-management

Splitting the exposure window between a vulnerability announcement and the patch being applied into four parts: comparing the average exposure window, outage count, and rollback count across five patch regimes; reading the day automatic patching buys in the same table as the untested-patch risk it produces; and naming the item that cannot be patched.

The previous lesson narrowed access and shortened secrets' lifetime. But the software using that
credential — the kernel, the system library, the language runtime, the reverse proxy — ages right
where it stands. The moment a vulnerability is announced, **every machine** that has that component
installed turns into an open-surface item, and the item stays open until the patch is applied. What
the default leaves here is not a setting, it is a **gap**: nobody knows which version is running on
which machine, and the patch gets applied when something breaks. The measure is the **exposure
window** — the time between announcement and application; how many items stay open for how long is
**open item-days**.

#### The Exposure Window Has Four Parts

The exposure window is not a single number, it is the sum of four independent parts; on a buggy patch the
retry time is added on top of these four, which is why the table's total column is larger than the sum of
the four. **Notice** is the time between the announcement and seeing that the component is installed on
the fleet, and it depends on the inventory scan interval. **Test** is trying the patch in a non-production
environment. **Window wait** is the ready patch waiting for the next maintenance window. **Apply** is the
install and restart.

The setup below is a **model**. The fictional regional measurement network's machines and
components are written as a data structure; no real fleet is scanned.

- **HB19** — The fleet is fictional: twelve machines, eight components, and sixty (machine,
  component) items.
- **HB20** — Fourteen announcements happen over the ninety-day window; their days and components
  are written in the model.
- **HB21** — The inventory scan interval sets notice time: an announcement is seen at the next
  scan.
- **HB22** — The maintenance window is fixed-interval and fixed-hour; `window: 0` is a wait-free
  regime.
- **HB23** — Whether a patch is buggy is decided by our own generator, the seed is visible. If
  testing catches the bug, the fixed patch arrives three days later; if it misses it, the patch
  goes to production, is seen half a day later, and is rolled back.
- **HB24** — Three items cannot be patched in any regime; their reasons are written in the model.

```js
// patch/model.mjs — the patch flow is a MODEL: twelve fictional machines, eight components, fourteen announcements.
// No real fleet is scanned, no real patch is downloaded; every number derives from this file.
const GROUP = { collector: 3, validator: 2, billing: 1, "work-order": 1, data: 2, gateway: 2, queue: 1 };
export const MACHINES = Object.entries(GROUP).flatMap(([a, n]) =>
  Array.from({ length: n }, (_, i) => `${a}-${i + 1}`));

export const COMPONENT = {                          // test and apply durations in days
  kernel:                { test: 2.0, apply: 0.20, group: "all" },
  "system-library":      { test: 1.0, apply: 0.15, group: "all" },
  "monitoring-agent":    { test: 0.25, apply: 0.05, group: "all" },
  scheduler:             { test: 0.25, apply: 0.05, group: "all" },
  "language-runtime":    { test: 1.5, apply: 0.10, group: ["collector", "validator", "billing", "work-order"] },
  "database-server":     { test: 2.0, apply: 0.25, group: ["data"] },
  "reverse-proxy":       { test: 0.5, apply: 0.05, group: ["gateway"] },
  "message-queue":       { test: 1.0, apply: 0.10, group: ["queue"] },
};
export const machinesFor = (b) => (COMPONENT[b].group === "all" ? MACHINES
  : MACHINES.filter((m) => COMPONENT[b].group.includes(m.slice(0, m.lastIndexOf("-")))));
export const ITEM = Object.keys(COMPONENT).reduce((a, b) => a + machinesFor(b).length, 0);

export const UNPATCHABLE = [                      // item = (machine, component); reason given by name
  ["billing-1", "language-runtime", "application compatibility"],
  ["data-1", "database-server", "service that must run continuously"],
  ["queue-1", "message-queue", "single-copy queue that cannot be drained"],
];
export const patchable = (b) =>
  machinesFor(b).filter((m) => !UNPATCHABLE.some((k) => k[0] === m && k[1] === b));

export const ANNOUNCEMENT = [                           // fourteen vulnerability announcements over a 90-day window
  [3.2, "system-library"], [8.6, "language-runtime"], [12.1, "kernel"], [19.4, "monitoring-agent"],
  [23.0, "reverse-proxy"], [27.8, "system-library"], [34.5, "database-server"], [39.9, "scheduler"],
  [45.3, "kernel"], [51.7, "message-queue"], [58.2, "language-runtime"], [63.5, "system-library"],
  [71.1, "monitoring-agent"], [78.4, "kernel"],
];
export const HORIZON = 90, FIX_DELAY = 3.0, NOTICE_DELAY = 0.5, SEED = 4127;

export function generator(seed) {                   // we write the generator ourselves, the seed is visible
  let x = seed >>> 0;
  return () => { x = (Math.imul(x, 1103515245) + 12345) >>> 0; return x / 4294967296; };
}
const r = generator(SEED);                          // independent of the draw order: all regimes see the same patches
export const BUGGY = ANNOUNCEMENT.map(() => r() < 0.20);
export const CAUGHT = ANNOUNCEMENT.map(() => r() < 0.75);

export const REGIME = {                            // scan = inventory interval, window = maintenance interval
  default: { scan: 30, test: false, window: 30, offset: 5.0 },
  weekly: { scan: 1, test: true, window: 7, offset: 2.0 },
  nightly: { scan: 1, test: true, window: 1, offset: 2 / 24 },
  continuous: { scan: 0.25, test: true, window: 0, offset: 0 },
  automatic: { scan: 0.25, test: false, window: 0, offset: 0 },
};
const firstWindow = (reg, ready) => (reg.window === 0 ? ready
  : Math.ceil((ready - reg.offset) / reg.window) * reg.window + reg.offset);

// One announcement's flow: notice, test, window wait, apply.
export function flow(i, name, buggy = BUGGY[i]) {
  const reg = REGIME[name], [day, b] = ANNOUNCEMENT[i], comp = COMPONENT[b], events = [];
  const notice = Math.ceil(day / reg.scan) * reg.scan;
  const test = reg.test ? comp.test : 0;
  let ready = notice + test, rollback = 0;
  if (buggy && reg.test && CAUGHT[i]) ready += FIX_DELAY;      // test caught it: the patch is delayed
  else if (buggy) {                                            // untested or missed: it went to production
    const first = firstWindow(reg, ready);
    events.push(first, first + NOTICE_DELAY);                  // apply, then rollback
    rollback = 1;
    ready = first + NOTICE_DELAY + FIX_DELAY;
  }
  const applyAt = firstWindow(reg, ready);
  events.push(applyAt);
  return { notice: notice - day, test, wait: applyAt - ready, applyDur: comp.apply,
    close: applyAt + comp.apply, window: applyAt + comp.apply - day, rollback, events };
}

export function scan(name, buggyOverride = null) {
  const outage = new Set();
  let avg = 0, rollback = 0, itemDays = 0, open = 0;
  for (let i = 0; i < ANNOUNCEMENT.length; i += 1) {
    const f = flow(i, name, buggyOverride === null ? BUGGY[i] : buggyOverride);
    const p = patchable(ANNOUNCEMENT[i][1]);
    avg += f.window / ANNOUNCEMENT.length;
    rollback += f.rollback;
    itemDays += f.window * p.length;
    if (f.close > HORIZON) open += p.length;
    for (const t of f.events) for (const m of p) outage.add(`${m}@${t.toFixed(4)}`);
  }
  return { avg, rollback, itemDays, outage: outage.size,
    open: open + UNPATCHABLE.length };
}
```

#### Five Regimes Scanned

```js
// patch/window.mjs — the exposure window's four parts and their costs are scanned across five regimes
import { ANNOUNCEMENT, REGIME, MACHINES, COMPONENT, ITEM, BUGGY, SEED, flow, scan } from "./model.mjs";

const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
console.log(`${MACHINES.length} machines, ${Object.keys(COMPONENT).length} components, ${ITEM} items; ` +
  `90 days, ${ANNOUNCEMENT.length} announcements, ${BUGGY.filter(Boolean).length} buggy patches (seed ${SEED}).`);
console.log("\nregime     |   scan |   test |  window |  avg open |  open item |  outage | rollback");
console.log("-----------|--------|--------|---------|-----------|------------|---------|----------");
for (const [name, d] of Object.entries(REGIME)) {
  const t = scan(name);
  console.log(`${name.padEnd(10)} | ${g(d.scan, 6)} | ${(d.test ? "yes" : "no").padStart(6)}` +
    ` | ${(d.window ? d.window + " days" : "no").padStart(7)} | ${g(t.avg, 9)}` +
    ` | ${String(t.open).padStart(10)} | ${String(t.outage).padStart(7)} | ${String(t.rollback).padStart(8)}`);
}

console.log("\nThe exposure window's four parts (days), averaged over fourteen announcements:");
console.log("regime     |    notice |   test |     window wait |    apply |  total");
console.log("-----------|-----------|--------|-----------------|----------|-------");
for (const name of Object.keys(REGIME)) {
  const p = { notice: 0, test: 0, wait: 0, applyDur: 0, window: 0 };
  for (let i = 0; i < ANNOUNCEMENT.length; i += 1)
    for (const k of Object.keys(p)) p[k] += flow(i, name)[k] / ANNOUNCEMENT.length;
  console.log(`${name.padEnd(10)} | ${g(p.notice, 9)} | ${g(p.test, 6)} | ${g(p.wait, 15)}` +
    ` | ${g(p.applyDur, 8)} | ${g(p.window, 6)}`);
}
```

```
12 machines, 8 components, 60 items; 90 days, 14 announcements, 4 buggy patches (seed 4127).

regime     |   scan |   test |  window |  avg open |  open item |  outage | rollback
-----------|--------|--------|---------|-----------|------------|---------|----------
default    |  30.00 |     no | 30 days |     28.94 |         45 |      54 |        4
weekly     |   1.00 |    yes |  7 days |      6.79 |          3 |     126 |        1
nightly    |   1.00 |    yes |  1 days |      3.09 |          3 |     147 |        1
continuous |   0.25 |    yes |      no |      2.28 |          3 |     147 |        1
automatic  |   0.25 |     no |      no |      1.22 |          3 |     205 |        4

The exposure window's four parts (days), averaged over fourteen announcements:
regime     |    notice |   test |     window wait |    apply |  total
-----------|-----------|--------|-----------------|----------|-------
default    |     15.24 |   0.00 |           11.14 |     0.13 |  28.94
weekly     |      0.52 |   1.16 |            3.89 |     0.13 |   6.79
nightly    |      0.52 |   1.16 |            0.33 |     0.13 |   3.09
continuous |      0.09 |   1.16 |            0.00 |     0.13 |   2.28
automatic  |      0.09 |   0.00 |            0.00 |     0.13 |   1.22
```

The second table shows exactly where the default loses: 15.24 of the 28.94 days is **notice
alone**. This time has nothing to do with the patch itself or fear of outages; its only cause is
that there is no record of which version is running where. Once the inventory scan moves to
nightly, this part drops to 0.52 days — 14.72 days in a single step, and without restarting a
single machine.

Test is the one item that works in the opposite direction: it rises from zero to 1.16 days, meaning
it **lengthens** the window. Its payoff is in the rollback column: four of the fourteen patches are
buggy, and in the three regimes that test, three of these were caught before they reached
production, bringing rollback to 1 instead of 4.

Outage, on the other hand, works opposite to the window interval. The default regime is the
quietest with 54 outages, because patches that accumulate over a monthly window get bundled into a
single restart — and that same bundling is what pushes the window out to 28.94 days. In the nightly
regime, outage rises from 126 to 147, and wait drops from 3.89 days to 0.33. **Outage count and
exposure window are the two ends of the same lever.** The day-ninety count is the outcome: 45 items
open in the default regime, 3 in the other four.

#### The Day Automation Buys and the Risk It Produces

```js
// patch/automation.mjs — the day automation buys, the risk it produces, and the new surface it introduces
import { ANNOUNCEMENT, REGIME, MACHINES, UNPATCHABLE, ITEM, HORIZON, BUGGY, scan } from "./model.mjs";

const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
const base = scan("weekly");
console.log("regime     |   avg open |   gain vs weekly |    applied |   rollback |   rollback added |    extra");
console.log("           |     (days) |           (days) |   untested |    in prod |           (days) |   outage");
console.log("-----------|------------|------------------|------------|------------|------------------|---------");
for (const name of Object.keys(REGIME)) {
  const t = scan(name), s = scan(name, false);          // s: if no patch is buggy
  console.log(`${name.padEnd(10)} | ${g(t.avg, 10)} | ${g(base.avg - t.avg, 16)}` +
    ` | ${String(REGIME[name].test ? 0 : ANNOUNCEMENT.length).padStart(10)} | ${String(t.rollback).padStart(10)}` +
    ` | ${g(t.avg - s.avg, 16)} | ${String(t.outage - s.outage).padStart(8)}`);
}

console.log("\nUnpatchable items (announcement arrives, application never happens in any regime):");
let unpatchableDays = 0;
for (const [m, b, gerekce] of UNPATCHABLE) {
  const first = ANNOUNCEMENT.find(([, x]) => x === b)[0];
  unpatchableDays += HORIZON - first;
  console.log(`${(m + " / " + b).padEnd(34)} | ${gerekce.padEnd(42)} | announced day ${g(first, 4, 1)}` +
    ` | ${g(HORIZON - first, 5, 1)} days open`);
}
const itemDays = scan("nightly").itemDays;
console.log(`patchable in the nightly regime ${ITEM - UNPATCHABLE.length} items : ${g(itemDays, 7)} item-days`);
console.log(`unpatchable ${UNPATCHABLE.length} items${" ".repeat(21)} : ${g(unpatchableDays, 7)} item-days ` +
  `(${g(100 * unpatchableDays / (itemDays + unpatchableDays), 5)}% of total)`);

// the new surface closing introduces: one patch agent per machine, one package source, one inventory record
const added = MACHINES.length + 3;
console.log(`\nnew surface: ${MACHINES.length} patch agents + package source + inventory record + maintenance window`);
console.log(`${ITEM} items -> ${ITEM + added} items (+${added}); exposure window dropped from ` +
  `${g(scan("default").avg, 5)} days to ${g(scan("automatic").avg, 5)} days, ` +
  `rollback ${scan("automatic").rollback} instead of ${base.rollback}, ` +
  `outage ${scan("automatic").outage} instead of ${base.outage}.`);
console.log(`buggy patch ${BUGGY.filter(Boolean).length}/${ANNOUNCEMENT.length}; ` +
  `testing caught ${BUGGY.filter(Boolean).length - base.rollback} of these before production.`);
```

```
regime     |   avg open |   gain vs weekly |    applied |   rollback |   rollback added |    extra
           |     (days) |           (days) |   untested |    in prod |           (days) |   outage
-----------|------------|------------------|------------|------------|------------------|---------
default    |      28.94 |           -22.14 |         14 |          4 |             8.57 |       18
weekly     |       6.79 |             0.00 |          0 |          1 |             1.50 |       17
nightly    |       3.09 |             3.70 |          0 |          1 |             0.93 |       24
continuous |       2.28 |             4.52 |          0 |          1 |             0.89 |       24
automatic  |       1.22 |             5.57 |         14 |          4 |             1.00 |       82

Unpatchable items (announcement arrives, application never happens in any regime):
billing-1 / language-runtime       | application compatibility                  | announced day  8.6 |  81.4 days open
data-1 / database-server           | service that must run continuously         | announced day 34.5 |  55.5 days open
queue-1 / message-queue            | single-copy queue that cannot be drained   | announced day 51.7 |  38.3 days open
patchable in the nightly regime 57 items :  407.50 item-days
unpatchable 3 items                      :  175.20 item-days (30.07% of total)

new surface: 12 patch agents + package source + inventory record + maintenance window
60 items -> 75 items (+15); exposure window dropped from 28.94 days to  1.22 days, rollback 4 instead of 1, outage 205 instead of 126.
buggy patch 4/14; testing caught 3 of these before production.
```

The automatic regime buys 5.57 days over weekly and has the shortest window. The same row has three
more numbers: all fourteen of the fourteen patches were applied untested, four were rolled back in
production, and rollback produced 82 extra outages. The continuous regime buys 4.52 days — 81
percent of the gain — but zero patches are applied untested and extra outage is 24. The gap between
them is 1.06 days. **The argument is not automation itself, it is whether testing gets skipped.** The
rollback-added-days column shows that a buggy patch lengthens its own window: 1.00 day in the
automatic regime, 8.57 days in the default — because the fixed version waits for the next monthly
window.

The second table names the item that cannot be closed. The language runtime on the billing machine
cannot be upgraded because the end-of-day batch job depends on the old version; the database server
on the primary data machine cannot be restarted because there is no room for a write outage in the
nightly batch window; the single-copy queue cannot be stopped because it cannot be drained. These
three items carry 175.20 item-days — 5 percent of the sixty items, **30.07 percent** of the open
item-days.

Closing this in itself brings surface. Patch automation puts a process running code with elevated
privilege on each of the twelve machines; the package source is the single point of trust for an
incoming binary; the inventory record is a complete vulnerability map of the fleet for anyone who
can read it; the maintenance window is a known interval of disruption repeated at the same hour
every night.

#### Summary

- The exposure window has four parts: notice, test, window wait, apply. In the default regime,
  15.24 of the 28.94 days is notice alone, and the inventory scan drops it to 0.52 days.
- Test lengthens the window by 1.16 days, and in return drops rollback in production from 4 to 1.
- Outage count and exposure window work in opposite directions: 54 outages and 28.94 days in a
  monthly window, 147 outages and 3.09 days in a nightly one. Open items at day ninety drop from
  45 to 3.
- Automatic patching buys 5.57 days but applies all fourteen patches untested, raises rollback to
  4, and produces 82 extra outages; the continuous regime gives 81 percent of the gain without
  dropping testing.
- The three unpatchable items — application compatibility, a service that must run continuously, a
  single-copy queue — carry 30.07 percent of the open item-days despite being 5 percent of the
  sixty items.
- The new surface closing introduces is fifteen items: the patch agent, the package source, the
  inventory record, and the maintenance window; the total rises from 60 to 75.

#### Next Step

This lesson never tested one assumption: that a **prior state** exists to return to when a patch is
rolled back. The fix for the unpatchable items leaned on the same assumption. Recovery objectives
were measured in the Resilience and Reliability course and are taken here as input; the next lesson
does not measure objectives, it measures **the backup itself**: how many of production's items are
actually backed up, which item is left out by name, and where frequency meets the data-loss window
and retention meets the point you can go back to. The backup itself is also a surface: how many
copies, where, and who can read them.

### Backup Strategy

Source: https://academia.sh/en/courses/servers-and-hardening/backup-strategy

Measuring the backup itself: counting the coverage gap item by item and naming the item that is not backed up, scanning frequency against the data-loss window and retention against the point you can go back to alongside storage cost, counting how many identities can read the backup copies, and showing the new single point the encryption key introduces.

The previous lesson ended with an assumption: that a prior state exists to return to when a patch
is rolled back. The same assumption sits under every recovery plan. Recovery time and data-loss
objectives were measured in the Resilience and Reliability course; they are taken here as input.
What is measured is **the backup itself**.

The item the default leaves is a gap here too: after install, the only backup that comes to mind is
the nightly dump of the database. The rest — config, secrets, object store, data in an external
service — stays out of the plan in a way nobody decided. The measure is the **coverage gap**: the
number of items that exist in production but are not in the backup set.

#### Coverage Gap Item by Item

- **HB25** — The production inventory is fictional: sixteen items; their type, size, and daily
  change are written in the model.
- **HB26** — The item set each of the four services needs is written in the model; if one item is
  missing, that service cannot be restored.
- **HB27** — Three items cannot be backed up; the reason is the item's type.

- **HB28** — The compression ratio is 0.42; an incremental backup is the interval's share of the
  daily change, and every point has 0.02 GiB of overhead.
- **HB29** — The data-loss window is the interval's half on average, and the interval itself in
  the worst case.
- **HB30** — The encryption section really runs; time is measured but not printed, it is passed
  through a band check.

```js
// backup/model.mjs — production inventory is a MODEL: sixteen items, four services. No real backup is taken.
const K = (name, type, gib, change, capturable = true) => ({ name, type, gib, change, capturable });
export const INVENTORY = [
  K("read-database", "database", 420, 6.0), K("rate-database", "database", 4, 0.02),
  K("work-order-database", "database", 18, 0.6), K("invoice-document-store", "object store", 260, 1.4),
  K("field-photo-store", "object store", 610, 3.1), K("service-config", "config", 0.02, 0.001),
  K("firewall-rules", "config", 0.001, 0.0002),
  K("zone-file", "config", 0.0005, 0.0001),
  K("deployment-config", "config", 0.05, 0.01),
  K("secret-store-content", "secret", 0.002, 0.0005), K("certificate-private-keys", "secret", 0.0004, 0.00005),
  K("monitoring-metric-series", "metric", 95, 4.2), K("audit-log", "log", 140, 2.8),
  K("message-queue-state", "queue", 2.5, 2.5, false),
  K("external-billing-service-data", "external service", 12, 0.3, false),
  K("external-notification-lists", "external service", 0.8, 0.05, false),
];
// A service can only be restored if every item it needs is in the backup.
export const SERVICE = {
  collector: ["read-database", "service-config", "secret-store-content", "deployment-config"],
  validator: ["read-database", "service-config", "secret-store-content", "message-queue-state"],
  billing: ["read-database", "rate-database", "invoice-document-store", "service-config",
    "secret-store-content", "external-billing-service-data"],
  "work-order": ["work-order-database", "field-photo-store", "service-config",
    "certificate-private-keys", "zone-file"],
};
export const PLAN = {                             // which types go into the backup set
  default: ["database"],
  "data-and-object": ["database", "object store"],
  "config-added": ["database", "object store", "config"],
  "secret-added": ["database", "object store", "config", "secret"],
  full: ["database", "object store", "config", "secret", "metric", "log", "queue", "external service"],
};
export const covered = (p) =>
  INVENTORY.filter((k) => PLAN[p].includes(k.type) && k.capturable);
```

```js
// backup/scope.mjs — coverage gap is counted item by item and restorable service is measured
import { INVENTORY, SERVICE, PLAN, covered } from "./model.mjs";

const TOTAL_GIB = INVENTORY.reduce((a, k) => a + k.gib, 0);
const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
console.log(`${INVENTORY.length} production items, ${g(TOTAL_GIB, 0, 1)} GiB, ${Object.keys(SERVICE).length} services; ` +
  `${INVENTORY.filter((k) => !k.capturable).length} items cannot be backed up.`);
console.log("\nplan                 | covered   | gap    | backed up GiB   | byte share    | restorable");
console.log("---------------------|-----------|--------|-----------------|---------------|------------------");
for (const p of Object.keys(PLAN)) {
  const inside = covered(p), names = new Set(inside.map((k) => k.name));
  const gib = inside.reduce((a, k) => a + k.gib, 0);
  const service = Object.entries(SERVICE).filter(([, needed]) => needed.every((x) => names.has(x)));
  console.log(`${p.padEnd(20)} | ${String(inside.length).padStart(9)} | ` +
    `${String(INVENTORY.length - inside.length).padStart(6)} | ${g(gib, 15, 1)} | ` +
    `${g(100 * gib / TOTAL_GIB, 12)}% | ${(service.length + "/" + Object.keys(SERVICE).length).padStart(18)}`);
}

console.log("\nCoverage gap item by item (left out of the config-added plan):");
const inside = new Set(covered("config-added").map((k) => k.name));
for (const k of INVENTORY.filter((x) => !inside.has(x.name)))
  console.log(`${k.name.padEnd(31)} | ${k.type.padEnd(18)} | ${g(k.gib, 7, 3)} GiB | ` +
    `${k.capturable ? "out of plan" : "UNCAPTURABLE"}`);

const blocking = INVENTORY.map((k) => ({ k, e: Object.keys(SERVICE).filter((s) => SERVICE[s].includes(k.name)) }))
  .filter((x) => x.e.length > 0).sort((a, b) => a.k.gib - b.k.gib).slice(0, 4);
const small = blocking.reduce((a, x) => a + x.k.gib, 0);
console.log(`\nfour smallest items blocking a service: ${blocking.map((x) => x.k.name).join(", ")}`);
console.log(`total ${g(small * 1024, 6)} MiB = ${g(100 * small / TOTAL_GIB, 6, 4)}% of production bytes; ` +
  `together blocking ${new Set(blocking.flatMap((x) => x.e)).size}/${Object.keys(SERVICE).length} services.`);
```

```
16 production items, 1562.4 GiB, 4 services; 3 items cannot be backed up.

plan                 | covered   | gap    | backed up GiB   | byte share    | restorable
---------------------|-----------|--------|-----------------|---------------|------------------
default              |         3 |     13 |           442.0 |        28.29% |                0/4
data-and-object      |         5 |     11 |          1312.0 |        83.97% |                0/4
config-added         |         9 |      7 |          1312.1 |        83.98% |                0/4
secret-added         |        11 |      5 |          1312.1 |        83.98% |                2/4
full                 |        13 |      3 |          1547.1 |        99.02% |                2/4

Coverage gap item by item (left out of the config-added plan):
secret-store-content            | secret             |   0.002 GiB | out of plan
certificate-private-keys        | secret             |   0.000 GiB | out of plan
monitoring-metric-series        | metric             |  95.000 GiB | out of plan
audit-log                       | log                | 140.000 GiB | out of plan
message-queue-state             | queue              |   2.500 GiB | UNCAPTURABLE
external-billing-service-data   | external service   |  12.000 GiB | UNCAPTURABLE
external-notification-lists     | external service   |   0.800 GiB | UNCAPTURABLE

four smallest items blocking a service: certificate-private-keys, zone-file, secret-store-content, service-config
total  23.45 MiB = 0.0015% of production bytes; together blocking 4/4 services.
```

The default plan covers three of the sixteen items, a coverage gap of 13. Once the object store is
added, byte share jumps from 28.29 percent to 83.97 percent and the plan looks complete — but
restorable services are still 0/4. Once config is added, byte share rises by 0.01 points, services
are still 0/4. Once secrets are added, byte share **does not change at all**, and services go from 0
to 2. **Byte share does not measure coverage.** The last two rows give the reason: the four smallest
blocking items add up to 23.45 MiB, 0.0015 percent of production bytes, but they block all four
services.

Even in the full plan, services stay at 2/4: a consistent snapshot of the message queue state cannot
be captured, and the data in the external billing service and the external notification lists have
no export interface. The validator depends on the first, billing on the second; neither can be
restored. There is no backup decision to make for these items — the decision is moving the item to a
place that can be backed up.

#### Frequency, Retention, and Storage

```js
// backup/frequency.mjs — frequency, retention, point count, and storage are scanned together (MODEL)
import { INVENTORY, covered } from "./model.mjs";

const COMPRESSION = 0.42, OVERHEAD = 0.02;         // overhead per point, GiB
const items = covered("full");
const FULL_GIB = items.reduce((a, k) => a + k.gib, 0) * COMPRESSION;
const DAILY_GIB = items.reduce((a, k) => a + k.change, 0) * COMPRESSION;
const PRODUCTION_GIB = INVENTORY.reduce((a, k) => a + k.gib, 0);
const SCHEDULE = {                                   // full: days, inc: hours (null = no incremental)
  "weekly-full": { full: 7, inc: null }, "nightly-full": { full: 1, inc: null },
  "nightly+6h": { full: 1, inc: 6 }, "weekly+1h": { full: 7, inc: 1 },
  "weekly+5min": { full: 7, inc: 1 / 12 },
};
const points = (d, R) => Math.floor(R / d.full) + 1 + (d.inc ? Math.floor(R * 24 / d.inc) : 0);
const stored = (d, R) => (Math.floor(R / d.full) + 1) * FULL_GIB + (d.inc ? R * DAILY_GIB : 0)
  + points(d, R) * OVERHEAD;
const g = (x, n, d = 2) => x.toFixed(d).padStart(n);

console.log(`backing up ${items.length} items: full backup ${g(FULL_GIB, 0, 1)} GiB (compression ${COMPRESSION}), ` +
  `daily change ${g(DAILY_GIB, 0, 2)} GiB, overhead per point ${OVERHEAD} GiB.`);
console.log("\nschedule     |      full |   incr. |   avg data loss (min) |   worst (min) | daily written GiB");
console.log("-------------|-----------|---------|-----------------------|---------------|--------------------");
for (const [name, d] of Object.entries(SCHEDULE)) {
  const min = (d.inc ?? d.full * 24) * 60;
  console.log(`${name.padEnd(12)} | ${(d.full + " days").padStart(9)} | ` +
    `${(d.inc ? (d.inc < 1 ? d.inc * 60 + " min" : d.inc + " h") : "none").padStart(7)} | ` +
    `${g(min / 2, 21, 1)} | ${g(min, 13, 1)} | ${g(FULL_GIB / d.full + (d.inc ? DAILY_GIB : 0), 19)}`);
}

const RETENTION = [7, 30, 90, 365];        // retention = oldest point you can go back to
console.log("\nstored GiB (by retention days) and points recoverable at 30 days:");
console.log(`schedule     |${RETENTION.map((R) => (R + " days").padStart(11)).join(" |")} | 30d point`);
console.log("-------------|------------|------------|------------|------------|----------");
for (const [name, d] of Object.entries(SCHEDULE))
  console.log(`${name.padEnd(12)} |${RETENTION.map((R) => g(stored(d, R), 11, 1)).join(" |")} | ` +
    `${String(points(d, 30)).padStart(9)}`);
console.log(`production data ${g(PRODUCTION_GIB, 0, 1)} GiB; stored in the weekly+1h schedule, at 7 days retention ` +
  `${g(stored(SCHEDULE["weekly+1h"], 7) / PRODUCTION_GIB, 0)}x, at 365 days ${g(stored(SCHEDULE["weekly+1h"], 365) / PRODUCTION_GIB, 0)}x.`);
```

```
backing up 13 items: full backup 649.8 GiB (compression 0.42), daily change 7.62 GiB, overhead per point 0.02 GiB.

schedule     |      full |   incr. |   avg data loss (min) |   worst (min) | daily written GiB
-------------|-----------|---------|-----------------------|---------------|--------------------
weekly-full  |    7 days |    none |                5040.0 |       10080.0 |               92.82
nightly-full |    1 days |    none |                 720.0 |        1440.0 |              649.77
nightly+6h   |    1 days |     6 h |                 180.0 |         360.0 |              657.39
weekly+1h    |    7 days |     1 h |                  30.0 |          60.0 |              100.44
weekly+5min  |    7 days |   5 min |                   2.5 |           5.0 |              100.44

stored GiB (by retention days) and points recoverable at 30 days:
schedule     |     7 days |    30 days |    90 days |   365 days | 30d point
-------------|------------|------------|------------|------------|----------
weekly-full  |     1299.6 |     3249.0 |     8447.3 |    34438.9 |         5
nightly-full |     5198.3 |    20143.5 |    59131.0 |   237823.5 |        31
nightly+6h   |     5252.2 |    20374.4 |    59823.6 |   240632.3 |       151
weekly+1h    |     1356.2 |     3491.8 |     9175.9 |    37393.7 |       725
weekly+5min  |     1393.2 |     3650.2 |     9651.1 |    39320.9 |      8645
production data 1562.4 GiB; stored in the weekly+1h schedule, at 7 days retention 0.87x, at 365 days 23.93x.
```

The nightly full backup is the most common default habit, and it loses on both ends: it writes
649.77 GiB a day, stores 20,143.5 GiB over thirty days, has a 720-minute window, and leaves 31
points. The weekly-full-plus-hourly-incremental schedule writes 100.44 GiB a day, stores 3,491.8
GiB over thirty days, drops the window to 30 minutes, and leaves 725 points — six times less
writing, five times less storage, a window twenty-four times smaller.

Raising frequency has almost no effect on storage: once the incremental interval drops to five
minutes, thirty-day storage rises from 3,491.8 GiB to 3,650.2 GiB — 4.5 percent — and the loss
window drops to 2.5 minutes. What rises is not the data stored, it is the point count and the
overhead. **What is expensive is not frequency, it is retention:** once retention goes from 7 days to
365 days, storage goes from 1,356.2 GiB to 37,393.7 GiB, from 0.87 times production data to 23.93
times.

Retention is also the oldest point you can go back to, and it is not a number until it is compared
against how long silent corruption takes to notice: at seven-day retention, corruption noticed on
the eighth day has no clean point to return to — all 725 points belong to after the corruption.

#### The Backup Itself Is a Surface

Copy placement is also a model. Copies lost in a single event are the copy-per-zone ratio rounded
up; **readable surface** is the copy count times the number of reading identities.

| location | copies | zones | reading identities | can delete | encrypted | lost in one event | readable surface |
|---|---|---|---|---|---|---|---|
| default | 1 | 1 | 9 | 9 | no | 1 | 9 |
| two-copy | 2 | 1 | 9 | 9 | no | 2 | 18 |
| separate-zone | 3 | 2 | 5 | 5 | yes | 2 | 15 |
| narrowed | 3 | 2 | 2 | 2 | yes | 2 | 6 |
| immutable | 4 | 2 | 2 | 0 | yes | 2 | 8 |

```js
// backup/key.mjs — the encryption section REALLY runs: recovery is tried with the correct and the wrong
// key. Time is measured but the RAW MEASUREMENT IS NOT PRINTED, only the band check is written.
import { createCipheriv, createDecipheriv, scryptSync } from "node:crypto";

const SEED = 90211, MIB = 24;           // data comes from our own generator, seed is visible: fictional
let x = SEED >>> 0;
const next = () => { x = (Math.imul(x, 1103515245) + 12345) >>> 0; return x; };
const plain = Buffer.allocUnsafe(MIB * 1024 * 1024);
for (let i = 0; i < plain.length; i += 4) plain.writeUInt32LE(next(), i);
const iv = Buffer.allocUnsafe(12);       // the init vector comes from the same generator: no randomness
for (let i = 0; i < 12; i += 4) iv.writeUInt32LE(next(), i);
const salt = Buffer.from("mock-salt-example-invalid");   // no real secret or key is written
const key = scryptSync("mock-password-example-invalid", salt, 32);
const other = scryptSync("mock-other-password-example-invalid", salt, 32);

const t0 = process.hrtime.bigint();
const s = createCipheriv("aes-256-gcm", key, iv);
const cipher = Buffer.concat([s.update(plain), s.final()]), tag = s.getAuthTag();
const decrypt = (a) => {                 // the integrity tag will not hold with the wrong key
  try {
    const d = createDecipheriv("aes-256-gcm", a, iv).setAuthTag(tag);
    return Buffer.concat([d.update(cipher), d.final()]);
  } catch { return null; }
};
const back = decrypt(key), wrong = decrypt(other);
const sec = Number(process.hrtime.bigint() - t0) / 1e9;

const [LOW, HIGH] = [5, 40000];           // MiB/s band; not the measured value, the check result is printed
const print = (e, d) => console.log(`${e.padEnd(44)}: ${d}`);
console.log(`\nmock backup set ${MIB} MiB (seed ${SEED}), encrypted size ${cipher.length} bytes`);
print("returned identical to input with the correct key", back && back.equals(plain) ? "yes" : "no");
print("could be recovered with a different key", wrong ? "yes" : "no");
print(`speed within ${LOW}-${HIGH} MiB/s band`, MIB / sec >= LOW && MIB / sec <= HIGH ? "yes" : "no");
```

```

mock backup set 24 MiB (seed 90211), encrypted size 25165824 bytes
returned identical to input with the correct key: yes
could be recovered with a different key     : no
speed within 5-40000 MiB/s band             : yes
```

The default location holds a full copy of production data in the same place as production,
readable by the same nine identities, and unencrypted: readable surface is 9, and a single event
takes out the whole copy. The second copy raises durability but pushes the surface to 18 — **taking
a backup is exposing the data a second time**. Once access is narrowed, the surface drops to 6 with
three copies: the copy count triples while the surface drops to a third. The fourth, immutable copy
zeroes out who can delete.

The item encryption introduces shows up in the run: recovery with the correct key returns exactly
the input, recovery with a different key does not. This is also the definition of the new single
point — if the key is lost, the backup is lost too. The key is a secret, and it sits in the secret
store; and the secret store's content is itself one of the items that needs backing up: **the
backup's key cannot be stored inside the backup.**

#### Summary

- Coverage gap is not measured by byte share: once the object store is added, byte share rises to
  83.97 percent while restorable services stay at 0/4; once secrets are added, byte share does not
  change at all but services rise to 2/4.
- Four small items — certificate private keys, the zone file, secret store content, service config
  — total 23.45 MiB, 0.0015 percent of production bytes, but block all four services.
- The three items that cannot be backed up are named: the instantaneous queue state, the data in the
  external billing service, the external notification lists. Even in the full plan, services stay
  at 2/4.
- Frequency is cheap, retention is expensive: dropping the incremental interval to five minutes
  raises thirty-day storage by 4.5 percent, and raising retention to 365 days carries it from
  1,356.2 GiB to 37,393.7 GiB.
- The backup itself is a surface: readable surface is 9 by default, 18 with two copies, 6 with
  three access-narrowed copies. The encryption key is a new single point and cannot be stored inside
  the backup.

#### Next Step

Every number in this lesson came from a backup **set**; none of them came from actually performing
a restore. The coverage table says an item is in the backup set, not that it is readable; the point
count says you can return to a point, not how long the return takes. The next lesson tests this
with a controlled drill: real files are backed up and restored, and what comes back is verified to
be **byte-for-byte identical** to what went in. Three things get measured — the real time the drill
takes, the item count the coverage table counted correctly, and which assumptions a first restore,
in a setup without a drill, breaks all at once.

### Restore Drill

Source: https://academia.sh/en/courses/servers-and-hardening/restore-drill

Proving that the backup actually works: real files are backed up, deleted, and restored; being byte-for-byte identical is verified with a digest; five defects seen only through the drill are counted; restore duration is split into five components and compared against the recovery target; and drill frequency is scanned alongside the uncertainty that accumulates between drills.

The previous lesson tied backup coverage, frequency, and retention to numbers; those numbers were
read from a backup **set**. The item the default leaves is the quietest one: **the backup step
returns successful.** The archive is produced, its size is written, a record is kept — and that
record says nothing about whether the backup can actually be restored. The measure is **proof** of
reversibility: a digest comparing the byte in production with the byte that comes back.

#### The Drill Really Runs

- **HB31** — The fictional regional measurement network's production tree is really written to
  disk: five items, their contents from our own generator, seed 51703; the temp directory is
  deleted at the end of the run.
- **HB32** — The archive container is our own format; the password and salt are fictional.
- **HB33** — Three of the five defects are really injected (an out-of-scope item, a flipped byte,
  a different key); wrong order and missing permission are modeled.
- **HB34** — The recovery time objective is input from the Resilience and Reliability course: 240
  minutes. Download time derives from the previous lesson's 649.8 GiB full backup and a fictional
  80 MiB/s line; the other four components are modeled.
- **HB35** — The production set gets three config changes a month and one item a quarter; one
  drill is six hours of one person's time, and runs in a separate restore environment.
- **HB36** — Time is measured but not printed; it is only passed through a band check.

```js
// drill/container.mjs — the archive container is our own format: manifest + sequential contents.
// The password and salt are fictional, no real secret is written.
import { readFileSync, writeFileSync, readdirSync, mkdirSync } from "node:fs";
import { join } from "node:path";
import { gzipSync, gunzipSync } from "node:zlib";
import { createCipheriv, createDecipheriv, createHash, scryptSync } from "node:crypto";

export const SEED = 51703;              // we write the generator ourselves, the seed is visible
let x = SEED >>> 0;
export const generate = (n) => {
  const b = Buffer.allocUnsafe(n);
  for (let i = 0; i < n; i += 4) { x = (Math.imul(x, 1103515245) + 12345) >>> 0; b.writeUInt32LE(x, i); }
  return b;
};
export const digest = (b) => createHash("sha256").update(b).digest("hex");
const salt = Buffer.from("mock-salt-example-invalid");
export const KEY = scryptSync("mock-backup-password-example-invalid", salt, 32);
export const OTHER = scryptSync("mock-other-password-example-invalid", salt, 32);
const IV = Buffer.from("mock-iv-12b");  // 12 bytes, fixed and fictional

export function pack(dir, skip = []) {       // skip: item left out of the plan
  const names = readdirSync(dir).sort().filter((a) => !skip.includes(a));
  const body = names.map((a) => readFileSync(join(dir, a)));
  const manifest = Buffer.from(JSON.stringify(names.map((a, i) => [a, body[i].length])));
  const head = Buffer.allocUnsafe(4);
  head.writeUInt32LE(manifest.length, 0);
  return Buffer.concat([head, manifest, ...body]);
}
export function unpack(archive, dir) {
  const n = archive.readUInt32LE(0);
  let p = 4 + n;
  mkdirSync(dir, { recursive: true });
  for (const [a, size] of JSON.parse(archive.subarray(4, 4 + n).toString())) {
    writeFileSync(join(dir, a), archive.subarray(p, p + size));
    p += size;
  }
}
export function encrypt(archive) {
  const s = createCipheriv("aes-256-gcm", KEY, IV);
  return { body: Buffer.concat([s.update(gzipSync(archive)), s.final()]), tag: s.getAuthTag() };
}
export function decrypt({ body, tag }, key = KEY) {
  try {                                  // a corrupt archive or the wrong key falls through
    const d = createDecipheriv("aes-256-gcm", key, IV).setAuthTag(tag);
    return gunzipSync(Buffer.concat([d.update(body), d.final()]));
  } catch { return null; }
}
```

```js
// drill/drill.mjs — REALLY runs: a fictional production tree is written to disk, archived,
// DELETED, and restored; being byte-for-byte identical is verified with a digest. Time is not printed.
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, readdirSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { SEED, generate, digest, pack, unpack, encrypt, decrypt, OTHER } from "./container.mjs";

const ITEM = [["metric.data", 262144], ["work-order.data", 131072], ["rate.data", 65536],
  ["service.config", 2048], ["zone.record", 1024]];   // fictional measurement network tree
const CONTENT = ITEM.map(([name, n]) => [name, generate(n)]);
const root = mkdtempSync(join(tmpdir(), "drill-"));
const production = join(root, "production");
const state = () => readdirSync(production).sort().map((a) => [a, digest(readFileSync(join(production, a)))]);

function run({ skip = [], corrupt = false, key = undefined }) {
  rmSync(production, { recursive: true, force: true });
  mkdirSync(production, { recursive: true });
  for (const [name, b] of CONTENT) writeFileSync(join(production, name), b);
  const before = state();
  let arc = encrypt(pack(production, skip));
  if (corrupt) arc = { ...arc, body: Buffer.concat([arc.body.subarray(0, 64),         // one byte is flipped
    Buffer.from([arc.body[64] ^ 1]), arc.body.subarray(65)]) };
  rmSync(production, { recursive: true });                  // production is deleted: return comes from the archive
  const container = decrypt(arc, key);
  if (container) unpack(container, production);
  const after = new Map(container ? state() : []);
  return { before, opened: !!container, archiveBytes: arc.body.length,
    same: before.filter(([name, o]) => after.get(name) === o).length };
}

const t0 = process.hrtime.bigint();
const healthy = run({});
const sec = Number(process.hrtime.bigint() - t0) / 1e9;
const bytes = CONTENT.reduce((a, [, b]) => a + b.length, 0);
const back = new Map(state());
console.log(`${ITEM.length} items, ${bytes} bytes (seed ${SEED}); archive ${healthy.archiveBytes} bytes`);
console.log("item                | prod digest (first 16)   | same on return");
console.log("--------------------|---------------------------|-----------------");
for (const [name, o] of healthy.before)
  console.log(`${name.padEnd(19)} | ${o.slice(0, 16).padStart(25)} | ` +
    `${(back.get(name) === o ? "yes" : "no").padStart(16)}`);

// MODEL: if order breaks, how many steps come up with empty data
const DEPENDS = { secret: [], data: ["secret"], config: [], service: ["data", "config"] };
const WRONG_ORDER = ["service", "data", "config", "secret"];
const empty = WRONG_ORDER.filter((a, i) => DEPENDS[a].some((b) => WRONG_ORDER.indexOf(b) > i)).length;
// MODEL: required/granted permissions for the restore identity
const REQUIRED = { "backup-store": "read", "object-store": "write", "secret-store": "read", "service-manager": "start" };
const GRANTED = { "backup-store": "read", "secret-store": "read" };
const missing = Object.keys(REQUIRED).filter((k) => GRANTED[k] !== REQUIRED[k]);

console.log("\ntrial                         |   opened |   match |   missing | backup step");
console.log("------------------------------|----------|---------|-----------|------------------");
let found = 2;                                        // two MODEL defects to start
for (const [name, option] of [["healthy drill", {}],
  ["missing item (config)", { skip: ["service.config"] }],
  ["corrupt archive (one byte)", { corrupt: true }], ["undecryptable cipher", { key: OTHER }]]) {
  const r = run(option), n = r.before.length;
  if (r.same < n) found += 1;
  console.log(`${name.padEnd(29)} | ${(r.opened ? "yes" : "no").padStart(8)} | ` +
    `${(r.same + "/" + n).padStart(7)} | ${String(n - r.same).padStart(9)} | ` +
    `${(r.same < n ? "looked successful" : "successful").padStart(16)}`);
}
console.log(`wrong order (MODEL)           | ${empty}/${WRONG_ORDER.length} steps open with empty data` +
  `${" ".repeat(2)}| looked successful`);
console.log(`missing permission (MODEL)    | ${missing.length} items: ${missing.join(", ")}  | looked successful`);
console.log(`\ndrill found ${found} defects; seen in the backup step 0`);
const mib = bytes / 1024 / 1024, [LOW, HIGH] = [0.5, 20000];   // MiB/s band; the raw measurement is not printed
console.log(`backup+delete+restore speed within ${LOW}-${HIGH} MiB/s band: ` +
  `${mib / sec >= LOW && mib / sec <= HIGH ? "yes" : "no"}`);
rmSync(root, { recursive: true, force: true });               // the temp tree is deleted
```

```
5 items, 461824 bytes (seed 51703); archive 462107 bytes
item                | prod digest (first 16)   | same on return
--------------------|---------------------------|-----------------
metric.data         |          3edf42c01d4f883f |              yes
rate.data           |          434443d05cecee54 |              yes
service.config      |          4ead0d5566a44d75 |              yes
work-order.data     |          5c4512d9a8016d51 |              yes
zone.record         |          b7c0e693cd032524 |              yes

trial                         |   opened |   match |   missing | backup step
------------------------------|----------|---------|-----------|------------------
healthy drill                 |      yes |     5/5 |         0 |       successful
missing item (config)         |      yes |     4/5 |         1 | looked successful
corrupt archive (one byte)    |       no |     0/5 |         5 | looked successful
undecryptable cipher          |       no |     0/5 |         5 | looked successful
wrong order (MODEL)           | 2/4 steps open with empty data  | looked successful
missing permission (MODEL)    | 2 items: object-store, service-manager  | looked successful

drill found 5 defects; seen in the backup step 0
backup+delete+restore speed within 0.5-20000 MiB/s band: yes
```

The first table is this lesson's proof: the production tree was written, archived, **deleted**, and
brought back from the archive alone; digests came out identical for all five of the five items. Not
a sentence, five comparisons.

The last column of the second table is the finding. The config left out of scope does not come
back, but the archive still opens; the archive with one flipped byte and the archive encrypted with
a different key do not open at all, and all five items go missing at once. In the wrong order, two
of the four steps open with empty data, and the identity is missing two permissions. **All five of
the five defects were invisible in the backup step.**

#### Duration Splits Into Components

```js
// drill/duration.mjs — restore duration splits into components, is compared against the target, and
// drill frequency is scanned. MODEL; download time derives from 05's full backup size.
const TARGET = 240;                       // minutes; input from the previous course
const FULL_GIB = 649.8, LINE_MIBS = 80;    // full backup size; line speed is fictional
const DOWNLOAD = (FULL_GIB * 1024) / LINE_MIBS / 60;
const COMPONENT = {                        // [undrilled, drilled] minutes
  find: [95, 12], download: [DOWNLOAD, DOWNLOAD], extract: [65, 38],
  verify: [40, 15], "start service": [85, 22],
};
const g = (x, n, d = 1) => x.toFixed(d).padStart(n);
const sum = (i) => Object.values(COMPONENT).reduce((a, b) => a + b[i], 0);
const SHARE = TARGET / Object.keys(COMPONENT).length;         // equal share

console.log(`target ${TARGET} min; full backup ${FULL_GIB} GiB, line ${LINE_MIBS} MiB/s, equal share ${SHARE} min.`);
console.log("\ncomponent       |   undrilled |    drilled |    percent of target");
console.log("----------------|-------------|------------|---------------------");
for (const [name, [y, t]] of Object.entries(COMPONENT))
  console.log(`${name.padEnd(15)} | ${g(y, 11)} | ${g(t, 10)} | ${(g((100 * t) / TARGET, 6) + "%").padStart(20)}`);
console.log(`${"total".padEnd(15)} | ${g(sum(0), 11)} | ${g(sum(1), 10)} | ` +
  `${(g((100 * sum(1)) / TARGET, 6) + "%").padStart(20)}`);
const over = Object.entries(COMPONENT).filter(([, [, t]]) => t > SHARE).map(([a]) => a);
console.log(`${over.length} components exceed the equal share in drilled time: ${over.join(", ")}`);

const CHANGE_RATE = 3 / 30 + 1 / 90;         // daily change: 3 configs a month, 1 item a quarter
const ITEM = 16, HOURS = 6;              // inventory; one drill is six hours
console.log("\nfrequency        |   yearly |   yearly hours |        unverified change |      restore |     target");
console.log("-----------------|----------|----------------|--------------------------|--------------|-----------");
for (const days of [90, 180, 365, null]) {
  const n = days ? Math.floor(365 / days) : 0;
  const uncertain = days ? days * CHANGE_RATE : 365 * CHANGE_RATE + ITEM;
  const dur = days ? sum(1) : sum(0);
  console.log(`${(days ? days + " days" : "never").padEnd(16)} | ${String(n).padStart(8)} | ` +
    `${String(n * HOURS).padStart(14)} | ${(g(uncertain, 4) + (days ? " items" : " items+set")).padStart(25)}` +
    ` | ${g(dur, 12)} | ${(dur > TARGET ? "exceeded" : "held").padStart(10)}`);
}
```

```
target 240 min; full backup 649.8 GiB, line 80 MiB/s, equal share 48 min.

component       |   undrilled |    drilled |    percent of target
----------------|-------------|------------|---------------------
find            |        95.0 |       12.0 |                 5.0%
download        |       138.6 |      138.6 |                57.8%
extract         |        65.0 |       38.0 |                15.8%
verify          |        40.0 |       15.0 |                 6.3%
start service   |        85.0 |       22.0 |                 9.2%
total           |       423.6 |      225.6 |                94.0%
1 components exceed the equal share in drilled time: download

frequency        |   yearly |   yearly hours |        unverified change |      restore |     target
-----------------|----------|----------------|--------------------------|--------------|-----------
90 days          |        4 |             24 |                10.0 items |        225.6 |       held
180 days         |        2 |             12 |                20.0 items |        225.6 |       held
365 days         |        1 |              6 |                40.6 items |        225.6 |       held
never            |        0 |              0 |            56.6 items+set |        423.6 |   exceeded
```

In the undrilled arrangement, restore takes 423.6 minutes, missing the target by 183.6 minutes; in
the drilled arrangement it drops to 225.6 minutes and holds the target with a 14.4-minute margin.
What the drill shortens is knowledge: finding the right point drops from 95 minutes to 12, starting
services from 85 to 22 — rehearsed steps.

The download column does not change at all: 138.6 minutes, 57.8 percent of the target, and the only
component that exceeds the equal share. This time does not come from a lack of knowledge, it comes
from pushing 649.8 GiB through an 80 MiB/s line; the drill measures it but does not shorten it. **The
way to hit the target is not more drilling, it is moving the backup closer to where it will be
restored.** The frequency scan puts the cost and the alternative side by side: a drill every
ninety days is twenty-four hours a year and 10 items of uncertainty, a drill once a year is six
hours and 40.6 items. Never drilling costs zero, the uncertainty is all sixteen items, and duration
misses the target.

#### Summary

- The backup step returned successful for all five of the five defects; all of them were seen only
  in the drill: an out-of-scope item, a corrupt archive, an undecryptable cipher, wrong order,
  missing permission.
- Proof is a digest comparison: five items were written, archived, deleted, restored, and digests
  came out identical.
- Restore duration is 423.6 minutes undrilled, 225.6 minutes drilled; the target holds at 94
  percent full. The drill shortens four components and does not touch download: 138.6 minutes is
  57.8 percent of the target and the only component exceeding the equal share.
- Drill frequency is a cost-uncertainty trade-off: a drill every ninety days is 24 hours and 10
  items a year, never drilling is 0 hours and all sixteen items.

#### Course Wrap-Up

| Lesson | What the default leaves open | Closed / wrongly closed | New surface closing introduces |
|---|---|---|---|
| Web Server Roles | 18 items | 13 closed / 4 legitimate jobs | 3 new items |
| Reverse Proxy Configuration | 4 pieces of missing info | 4 resolved / 2 items hidden | spoofed header, 4 requests |
| TLS Termination | 19 weak items | 19 closed / 65 devices | certificate at 6 points |
| Certificate Automation | 1,700 hours window | 0 hours / 12 unnecessary renewals | 4 new items |
| Cache Server | 179 leaked requests | 0 leaks / 2x load | escaping broadcast, silent window |
| Service Mesh | 40 touches for 40 services | 508 work items / expensive at 3 services | 162 sidecar items |
| DNS Management | 4 dead, 2 takeover-able records | to 5 minutes / 288x queries | identity writing to the zone |
| Sender Verification Records | unowned sending is free | 5/6 spoofed / 2 legitimate dropped | 254 excess authorized addresses |
| Domain Policy Record | 61 spoofed reach | 49 blocked / 43 legitimate separated | 12 compliant impersonations |
| Mail Delivery Issues | 12-point ceiling | ceiling lifted / 1,078 messages lost | recipient decision unmeasurable |
| Firewall Configuration | 100 items, 107,265 attempts | 10 rows all / 2 flows | external row, 20,650 attempts |
| Remote Access Hardening | 2,340 triples, 16,505 attempts | 16,505 closed / 0 legitimate | 34 key copies |
| Secret Management Systems | 36 items, infinite window | 2.5 minutes / 9 items lifetime-less | single point, 18 instances |
| Patch Management | 60 items, 28.94 days | to 1.22 days / 4 rollbacks, 82 outages | 15 items, patch agent |
| Backup Strategy | 13 items out of scope | to 3 items / services still 2/4 | readable surface doubled |
| Restore Drill | 5 defects invisible, 423.6 min | 5 found / 225.6 min | 24 hours, 2 missing permissions |

No row in the second column shows a trace of a decision: **the default is not a decision, it is the
decision nobody made.** In the third column, wrong closing fills fourteen of the sixteen rows: no
step is free. In the fourth column, three types of surface repeat: **a new identity or write
authorization** in five lessons, **a single point carrying the whole load** in four lessons, **a
new layer that itself needs configuring** in four lessons. Every defense brings its own surface — a
closed item does not disappear, it changes shape.

Seven courses in, every layer has been built and hardened separately. Every row in the table is a
sequence of decisions: which port, which threshold, which item. But does every team have to rebuild
all of this from scratch? Platform Engineering starts with this question.