---
title: 'Observability and Reliability'
source: 'https://academia.sh/en/courses/observability'
language: en
updated: '2026-08-23T07:00:27+00:00'
license: 'CC BY-SA 4.0'
lessons: 19
outcomes:
    - 'Seats the log, metric, and trace triad on a consistent telemetry model'
    - 'Defines service level objectives with measurable indicators'
    - 'Manages partial failure with resilience patterns'
---

# Observability and Reliability

Making a system that runs across several processes visible, and making it withstand partial failure: seating the log, metric, and trace triad on a consistent telemetry model, structured logging with a correlation id, the distinction between counter, gauge, and histogram, propagation context and trace sampling, vendor-neutral instrumentation, joining the three signals in a single event, the distinction between an indicator and an objective, the golden signals, the error budget, symptom-based alerting, the distinction between liveness and readiness, and where resilience patterns sit inside the code.

## Telemetry

Producing the three signals in application code: which question the log, the metric, and the trace each answer, how the field-based log and the correlation id make a request traceable end to end, how the counter, gauge, and histogram correspond to different questions, how the propagation context crosses the process boundary and what trace sampling costs in lost information, a vendor-neutral collection interface, and combining the three signals into a single event.

### The Three Signals of Observability

Source: https://academia.sh/en/courses/observability/three-signals-of-observability

Actually producing the log, metric, and trace signals in a split loan system: how many steps each signal takes to answer the same three questions, which question each signal leaves unanswered, and how the three signals grow with request count.

The previous course split the system into five processes and measured how every boundary gave
birth to a new failure mode. The question left standing at its close was this: no tool was ever
built for seeing where a request slows down or gets lost. This lesson builds the three parts of
that tool and counts what each part can and cannot answer.

The three signals are these. A **log** is the text record of individual events. A **metric** is
events reduced to a pre-chosen aggregate. A **trace** is the record that holds a job's steps
together with their durations, and each step is a **span**. All three measure, but each answers
a different question; this lesson shows that not as a claim but as a step count.

Monitoring categories and alert thresholds were established in the Performance Anti-Patterns and
Monitoring course; the category list is not repeated here. This course's subject is the
production side: how a signal is born in application code, which field it carries, and which it
does not.

#### The Service Producing Three Signals

The setup continues as the previous course left it: catalog, membership, loan, notification, and
fee are separate processes. The loan service calls the membership, catalog, and notification
services; the membership service calls the fee service. The single source file below takes on
all of their roles and produces all three signals on every request.

**TL1 — all three signals are produced inside the process handling the request, with no extra
collector process involved.** Rationale: what gets measured is not how the signal travels, but
which field it carries at the point where it is born. The question of wiring to a collector is
handled separately in the fifth lesson.

```js
// service.mjs — <name> <port> [downstream...]; every request produces all three signals; long-lived process
import { createServer } from "node:http";
import { appendFileSync, writeFileSync } from "node:fs";
const [name, port, ...downstream] = process.argv.slice(2);
writeFileSync(`log-${name}.txt`, ""); writeFileSync(`trace-${name}.txt`, "");
const metrics = { requests: 0, errors: 0, duration: 0 };  // metrics: totals only, no id
let localSeq = 0;                                  // the trace number is meaningful only in this process
const wait = (ms) => new Promise((c) => setTimeout(c, ms));

createServer(async (req, res) => {
  const u = new URL(req.url, "http://y");
  if (u.pathname === "/metrics") { res.end(JSON.stringify({ name, ...metrics, duration: Math.round(metrics.duration) })); return; }
  const member = u.searchParams.get("member"), copy = Number(u.searchParams.get("copy"));
  const t0 = performance.now(), no = ++localSeq;
  metrics.requests += 1;
  await wait(name === "catalog" && copy % 5 === 0 ? 100 : 5);  // every fifth copy is on a slow shelf
  let status = name === "fee" && member === "4023" ? 402 : 200;   // unpaid fee
  for (const p of downstream) {
    const y = await fetch(`http://127.0.0.1:${p}/work?member=${member}&copy=${copy}`);
    if (y.status !== 200) status = y.status;
  }
  const duration = performance.now() - t0;
  metrics.duration += duration; if (status !== 200) metrics.errors += 1;
  appendFileSync(`log-${name}.txt`,
    `[${new Date().toISOString()}] ${name}: member ${member} copy ${copy} request took ` +
    `${Math.round(duration)} ms, result ${status}\n`);
  appendFileSync(`trace-${name}.txt`, `${no} ${name} ${Math.round(duration)}\n`);
  res.statusCode = status; res.end(JSON.stringify({ name, status }));
}).listen(Number(port));
```

This first version of the three signals is deliberately plain. The log line is a flat sentence.
The metric is three totals. The span carries only a number that increases within its own
process. The next four lessons close these three gaps one at a time.

The source of slowness and errors is fixed: the catalog service spends a hundred milliseconds on
every copy whose number is a multiple of five, every request from member 4023 comes back from
the fee service with a 402, and that status climbs through membership up to the loan service.
The client sends requests to the loan endpoint in sequence, picking members from a four-member
rotation.

```js
// run.mjs — <request count>; sends requests to the loan endpoint in sequence
const n = Number(process.argv[2]);
const MEMBER = ["4021", "4022", "4023", "4024"];      // 4023 is the member with an unpaid fee
for (let i = 1; i <= n; i += 1) {
  await fetch(`http://127.0.0.1:9101/work?member=${MEMBER[(i - 1) % 4]}&copy=${i}`);
}
console.log(`${n} requests sent`);
```

#### Same Question, Three Signals

The query tool tries to answer three questions with three signals. The questions are real ones:
where did this request slow down, how many requests failed, and which user was affected.

A **step** is one pass the query makes over a record store: reading a file or an endpoint is a
step, parsing plain text is a step, filtering is a step, trying to join two stores is a step. The
counter lives in the code, it is not a guess. Steps are counted even for a question that goes
unanswered — the work spent before giving up is also a cost.

```js
// query.mjs — tries to answer three questions with three signals; every store pass is one step
import { readFileSync, statSync } from "node:fs";
const SERVICES = ["loan", "membership", "catalog", "fee", "notification"];
const PORT = { loan: 9101, membership: 9102, catalog: 9103, fee: 9104, notification: 9105 };
const RE = /member (\d+) copy (\d+) request took ([\d.]+) ms, result (\d+)/;
let step = 0; const count = () => { step += 1; };
const lines = (prefix, s) => { count(); return readFileSync(`${prefix}-${s}.txt`, "utf8").split("\n").filter(Boolean); };
const parseLog = () => {                       // plain text: one parsing pass per service
  const o = {};
  for (const s of SERVICES) o[s] = lines("log", s).map((l) => { const m = l.match(RE); return m && { member: m[1], duration: +m[3], status: +m[4] }; });
  for (const s of SERVICES) count();
  return o;
};
const readTrace = () => { const o = {}; for (const s of SERVICES) o[s] = lines("trace", s).map((l) => { const [no, , duration] = l.split(" "); return { no: +no, duration: +duration }; }); return o; };
const readMetrics = async () => { const o = {}; for (const s of SERVICES) { count(); o[s] = await (await fetch(`http://127.0.0.1:${PORT[s]}/metrics`)).json(); } return o; };
const print = (question, signal, answer) => console.log(`${question.padEnd(12)}${signal.padEnd(9)}${String(step).padStart(5)}  ${answer}`);

console.log(`${"question".padEnd(12)}${"signal".padEnd(9)}${"step".padStart(5)}  answer`);
const metrics = await readMetrics(); const baseStep = step;

// S1 — where did this request slow down (target: the slowest request at the loan endpoint)
const THRESHOLD = 60;                                // above 60 ms counts as a slow request
step = 0; const g = parseLog(); count();
const slow = g.loan.filter((r) => r.duration > THRESHOLD).length; count();
print("S1 where", "log", `unanswered (${slow} loan records above ${THRESHOLD} ms; no shared id with downstream records)`);
step = baseStep; count();
print("S1 where", "metric", `unanswered (no per-request breakdown; only the service average)`);
step = 0; const trace = readTrace(); count();
const slowSpans = trace.loan.filter((a) => a.duration > THRESHOLD).length; count();
print("S1 where", "trace", `partial (${slowSpans} spans above ${THRESHOLD} ms; which downstream leg is unclear)`);

// S2 — how many requests failed
step = 0; const g2 = parseLog(); count();
print("S2 errors", "log", `${g2.loan.filter((r) => r.status !== 200).length} requests`);
step = baseStep; count(); print("S2 errors", "metric", `${metrics.loan.errors} requests`);
step = 0; readTrace(); count(); print("S2 errors", "trace", `unanswered (no status field in the trace record)`);

// S3 — which user was affected
step = 0; const g3 = parseLog(); count();
print("S3 who", "log", `member ${[...new Set(g3.loan.filter((r) => r.status !== 200).map((r) => r.member))].join(",")}`);
step = baseStep; count(); print("S3 who", "metric", `unanswered (total counter, carries no id)`);
step = 0; readTrace(); count(); print("S3 who", "trace", `unanswered (no id field in the span)`);

const bytes = (prefix) => SERVICES.reduce((t, s) => t + statSync(`${prefix}-${s}.txt`).size, 0);
const records = (prefix) => SERVICES.reduce((t, s) => t + lines(prefix, s).length, 0);
const avg = (prefix) => Math.round(bytes(prefix) / records(prefix));   // bytes per record: robust to digit-count noise
const metricBytes = SERVICES.reduce((t, s) => t + JSON.stringify(metrics[s]).length, 0);
console.log(`\nvolume: log ${records("log")} records x avg. ${avg("log")} bytes, ` +
  `trace ${records("trace")} spans x avg. ${avg("trace")} bytes, ` +
  `metric ${SERVICES.length * 4} fields (independent of request count) / ${metricBytes} bytes`);
```

The block below brings up the five processes, sends twenty requests and asks the three
questions, then sends a hundred and eighty more requests and reprints only the volume line.

```bash
node service.mjs fee 9104 &
node service.mjs catalog 9103 &
node service.mjs notification 9105 &
node service.mjs membership 9102 9104 &
node service.mjs loan 9101 9102 9103 9105 &
sleep 1

node run.mjs 20
node query.mjs
node run.mjs 180
node query.mjs | tail -1
pkill -f "node service.mjs"
```

```
20 requests sent
question    signal    step  answer
S1 where    log         12  unanswered (4 loan records above 60 ms; no shared id with downstream records)
S1 where    metric       6  unanswered (no per-request breakdown; only the service average)
S1 where    trace        7  partial (4 spans above 60 ms; which downstream leg is unclear)
S2 errors   log         11  5 requests
S2 errors   metric       6  5 requests
S2 errors   trace        6  unanswered (no status field in the trace record)
S3 who      log         11  member 4023
S3 who      metric       6  unanswered (total counter, carries no id)
S3 who      trace        6  unanswered (no id field in the span)

volume: log 100 records x avg. 86 bytes, trace 100 spans x avg. 13 bytes, metric 20 fields (independent of request count) / 292 bytes
180 requests sent
volume: log 1000 records x avg. 87 bytes, trace 1000 spans x avg. 14 bytes, metric 20 fields (independent of request count) / 305 bytes
```

#### Five of Nine Attempts Came Back Empty

The table has nine rows and five are unanswered. The ones that are answered: the metric gives
the error count in six steps, the log in eleven; only the log gives the affected member, also in
eleven steps. No signal fully answers the first question.

The difference is not signal quality but which field it carries. The metric gives the cheapest
answer because the question it can answer was already chosen when the metric was written: the
`errors` counter is the "how many requests failed" question, frozen into code. For the same
reason it cannot answer "which member" — there is no id inside the total, and none can be
recovered afterward. The log is the opposite: every event has its own line, so it answers the id
question, but because it is a flat sentence, every query needs five separate parsing passes and
the regular expression depends on the line format.

#### Volume

The last two lines are the third measure. Twenty external requests produced a hundred log
records and a hundred spans across five processes; two hundred external requests produced a
thousand records and a thousand spans. The ratio is fixed: five records per external request,
because the call amplification is five and every process writes its own record. The average size
per record is eighty-six to eighty-seven bytes for the log, thirteen to fourteen bytes for the
trace; the roughly sixfold gap comes from the log carrying a fresh timestamp and sentence pattern
on every line.

The metric sits outside this growth. Five services, four fields per service: twenty fields, no
matter how many requests come in. The total payload grew from 292 to 305 bytes; the entire
increase comes from the digit count of the accumulated duration total, not from the record
count. The metric's cost grows with the **field count**, not the request count; the log's and
the trace's cost grow with the request count. This is also why the three signals are kept
together: they answer the same question at different costs and different resolutions.

#### Summary

- The log records individual events, the metric records pre-chosen totals, the trace records a
  job's steps as spans; all three were born from the same request in the same process.
- Five of nine attempts stayed unanswered: the metric could not answer the id question, the
  trace could not answer the status or id question, and no signal fully answered "where did this
  request slow down."
- The metric spent the fewest steps (6), the log the most (11–12); the gap comes from plain text
  needing one parsing pass per service.
- The metric chooses its question when it is written: the `errors` counter makes one question
  cheap and closes off the id question permanently.
- Volume follows two different laws: 20 requests produced 100 records, 200 requests produced
  1000; the metric stayed at 20 fields in both cases and grew only by its digit count.

#### Next Step

Three of the five unanswered questions share a single cause: there is no field in the records
that identifies the request. Nothing connects the line the loan service writes to the line the
catalog service writes, because the lines are written as sentences and a sentence carries only
what that one process knows. The next lesson takes the log out of being a flat sentence, splits
it into fields, and attaches to the request an id that crosses the process boundary. What gets
measured is clear: how many of the same question's filtering and parsing steps drop away, how
many services a request becomes traceable across, and how many call sites are left without the
id carried through them.

### Structured Logging

Source: https://academia.sh/en/courses/observability/structured-logging

Turning the flat-sentence log into a field-based record and attaching to the request a correlation id that crosses the process boundary: how many parsing and filtering steps the same question takes to answer, how many services a request becomes traceable across, and counting the call sites left without the id carried through them.

The previous lesson left five questions unanswered, and three of them shared a single cause: no
field in the records identifies the request. The log line was a sentence, the sentence carried
only what that one process knew, and nothing joined one service's line to another's. This lesson
makes two changes: the record stops being a sentence and splits into fields, and the request
gets a **correlation id** that crosses the process boundary. Both changes have a cost, and both
are measured.

**Structured logging** means the record is made of named fields instead of free text. A
**correlation id** is a value given to an external request and carried through every internal
request it produces. The second is useless without the first: the id is ultimately a field
itself, and it needs to be a value that can be filtered by name, not a string that has to be
hunted for inside a free-text sentence.

#### Records Split into Fields

The logger is hand-written and has a single job: merge the fixed fields with the call-time
fields and append them as a single line.

```js
// logger.mjs — field-based logger; every record is a single line, every field stands under its own name
import { appendFileSync } from "node:fs";
export const logger = (file, fixed) => (fields) =>
  appendFileSync(file, JSON.stringify({ ...fixed, ...fields }) + "\n");
```

The service writes **two** records on every request: the previous lesson's flat sentence and its
field-based counterpart. Both describe the same fact, from the same run, with the same values;
the only difference is format, so the step difference this lesson measures is purely a
difference in format.

**TL2 — the correlation id is generated by the service that first receives the external
request; if a header already carries an id, it is not regenerated, it is carried through as
is.** Rationale: the id must have a single point of generation, otherwise the same request picks
up a new id at every service and joining still cannot be done. Here the generation rule is tied
to the copy number, so the ids in the measurement stay independent of the run.

```js
// service2.mjs — <name> <port> [downstream...]; writes two log formats for the same request
import { createServer } from "node:http";
import { writeFileSync, appendFileSync } from "node:fs";
import { logger } from "./logger.mjs";
const [name, port, ...downstream] = process.argv.slice(2);
const NOTIFICATION = "9205";
writeFileSync(`plain-${name}.txt`, ""); writeFileSync(`fields-${name}.txt`, "");
const writeFields = logger(`fields-${name}.txt`, { service: name });
let counter = 0;
const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const url = (p, u) => `http://127.0.0.1:${p}/work?member=${u.get("member")}&copy=${u.get("copy")}`;
const call = (p, u, correlationId) => fetch(url(p, u), { headers: { "x-correlation": correlationId } });
const notify = (p, u) => fetch(url(p, u));      // old helper: never took a header parameter

createServer(async (req, res) => {
  const u = new URL(req.url, "http://y").searchParams;
  const member = u.get("member"), copy = Number(u.get("copy"));
  const correlationId = req.headers["x-correlation"] ?? `${name}-${copy}`;   // generated when no id is present
  const t0 = performance.now(); counter += 1;
  await wait(name === "catalog" && copy % 5 === 0 ? 45 : 5);
  let status = name === "fee" && member === "4023" ? 402 : 200;
  for (const p of downstream) {
    const y = p === NOTIFICATION ? await notify(p, u) : await call(p, u, correlationId);
    if (y.status !== 200) status = y.status;
  }
  const duration = Math.round(performance.now() - t0);
  appendFileSync(`plain-${name}.txt`, `${name}: member ${member} copy ${copy} request took ` +
    `${duration} ms, result ${status}\n`);
  writeFields({ correlationId, member, copy, duration, status });
  res.statusCode = status; res.end("done");
}).listen(Number(port));
```

The id crosses the boundary in a single line: `call` adds the header. Next to it, `notify` is an
old helper that never took a header parameter, and the notification call is made through it.
This is not a staged mistake, it is the typical outcome of adding an id: the id gets added to a
client helper, but a call path that does not use that helper survives in the system. The
measurement's job is to count that path.

#### Measurement

The query tool asks two questions in two formats. The step definition is the same as the
previous lesson: reading a file, parsing, filtering, or trying to join two stores are each one
step. In plain text every service has its own line pattern, so parsing takes one pass per
service; in the field-based record the format is uniform, so parsing takes a single pass.

```js
// query2.mjs — the same questions in two log formats; a step is one store pass
import { readFileSync } from "node:fs";
const SERVICES = ["loan", "membership", "catalog", "fee", "notification"];
const CALLS = [["loan", "membership"], ["loan", "catalog"], ["loan", "notification"], ["membership", "fee"]];
const RE = /member (\d+) copy (\d+) request took (\d+) ms, result (\d+)/;
let step = 0; const count = () => { step += 1; };
const line = (prefix, s) => { count(); return readFileSync(`${prefix}-${s}.txt`, "utf8").split("\n").filter(Boolean); };
const readPlain = () => {                       // every service has its own line pattern: one parsing pass per service
  const o = {};
  for (const s of SERVICES) o[s] = line("plain", s).map((l) => { const m = l.match(RE); return m && { member: m[1], copy: +m[2], duration: +m[3], status: +m[4] }; });
  for (const s of SERVICES) count();
  return o;
};
const readFields = () => {                      // field-based: uniform format, a single parsing pass
  const o = {}; for (const s of SERVICES) o[s] = line("fields", s);
  count(); for (const s of SERVICES) o[s] = o[s].map((l) => JSON.parse(l));
  return o;
};
const print = (question, format, answer) => console.log(`${question.padEnd(11)}${format.padEnd(13)}${String(step).padStart(5)}  ${answer}`);
console.log(`${"question".padEnd(11)}${"format".padEnd(13)}${"step".padStart(5)}  answer`);

// S3 — which member was affected (both formats can answer)
step = 0; const plain = readPlain(); count();
print("S3 who", "plain text", `member ${[...new Set(plain.loan.filter((r) => r.status !== 200).map((r) => r.member))].join(",")}`);
step = 0; const fields = readFields(); count();
print("S3 who", "field-based", `member ${[...new Set(fields.loan.filter((r) => r.status !== 200).map((r) => r.member))].join(",")}`);

// S1 — where did this request slow down; target: the request for copy number 5
const TARGET = "loan-5";
step = 0; readPlain(); count(); count();
print("S1 where", "plain text", "unanswered (no id field in the line, services cannot be joined)");
step = 0; const fields2 = readFields(); count();
const path = SERVICES.filter((s) => fields2[s].some((r) => r.correlationId === TARGET));
const longest = path.filter((s) => s !== "loan").sort((x, y) =>
  fields2[y].find((r) => r.correlationId === TARGET).duration - fields2[x].find((r) => r.correlationId === TARGET).duration)[0];
count(); print("S1 where", "field-based", `${TARGET}: ${path.join(" ")} (${path.length}/${SERVICES.length} services), longest downstream ${longest}`);

console.log(`\n${"call site".padEnd(26)}${"id".padEnd(8)}records with id`);
let carrying = 0;
for (const [upper, lower] of CALLS) {
  const set = new Set(fields[upper].map((r) => r.correlationId));
  const shared = fields[lower].filter((r) => set.has(r.correlationId)).length;
  if (shared > 0) carrying += 1;
  console.log(`${`${upper} -> ${lower}`.padEnd(26)}${(shared > 0 ? "yes" : "no").padEnd(8)}${shared}/${fields[lower].length}`);
}
console.log(`call sites carrying id ${carrying}/${CALLS.length}, not carrying ${CALLS.length - carrying}; ` +
  `a request is traceable across ${path.length}/${SERVICES.length} services`);
const bytes = (prefix) => Math.round(SERVICES.reduce((t, s) => t + readFileSync(`${prefix}-${s}.txt`, "utf8").length, 0) / 100);
console.log(`avg. bytes per record: plain text ${bytes("plain")} bytes, field-based ${bytes("fields")} bytes (100 records)`);
```

```bash
cat > run.mjs <<'EOF'
const MEMBER = ["4021", "4022", "4023", "4024"];      // 4023 is the member with an unpaid fee
for (let i = 1; i <= 20; i += 1)
  await fetch(`http://127.0.0.1:9201/work?member=${MEMBER[(i - 1) % 4]}&copy=${i}`);
EOF
node service2.mjs fee 9204 &
node service2.mjs catalog 9203 &
node service2.mjs notification 9205 &
node service2.mjs membership 9202 9204 &
node service2.mjs loan 9201 9202 9203 9205 &
sleep 1

node run.mjs
node query2.mjs
pkill -f "node service2.mjs"
```

```
question   format        step  answer
S3 who     plain text      11  member 4023
S3 who     field-based      7  member 4023
S1 where   plain text      12  unanswered (no id field in the line, services cannot be joined)
S1 where   field-based      8  loan-5: loan membership catalog fee (4/5 services), longest downstream catalog

call site                 id      records with id
loan -> membership        yes     20/20
loan -> catalog           yes     20/20
loan -> notification      no      0/20
membership -> fee         yes     20/20
call sites carrying id 3/4, not carrying 1; a request is traceable across 4/5 services
avg. bytes per record: plain text 59 bytes, field-based 101 bytes (100 records)
```

#### Four Steps and One Blind Spot

The first comparison is an ordinary win. The same question took eleven steps in plain text,
seven in the field-based record. All four steps of the difference come from parsing: because
every service has its own line pattern in plain text, the regular expression has to run once per
service, while the field-based record's uniform format needs only a single pass. This gap grows
with the number of services — with twenty services instead of five, plain text would spend
twenty-five steps and the field-based record twenty-two, and the gap would widen to nineteen.
The fragility of parsing is here too: when a line pattern changes, the regular expression
silently stops matching, while when a field name changes, the record openly loses that field.

The second comparison is this lesson's real point. The question no signal in the previous lesson
could answer got answered in eight steps by the field-based record: the `loan-5` id shows up in
four services, and the longest of the downstream legs is catalog. Plain text is still unanswered
on the same question — the problem was never a lack of durations, it was a lack of the field
that would tie those durations together.

The third table shows the limit of the win. Three of the four call sites carry the id, one does
not; none of the notification service's twenty records carries the loan request's id. The
measured result is that a request is traceable across **four** of five services. Adding an id
costs a single line, but its coverage is measured in call paths, and a single skipped call site
leaves an entire service off the map. Notification was not picked at random here: "why did this
notification not go out" is the most frequently asked question in this system, and exactly that
service is left in the blind spot.

The last line gives the cost. The field-based record is 101 bytes per record, plain text 59:
because the field names get rewritten on every line, the volume grows by roughly seventy
percent. In exchange, query steps went down and a question that could not be answered became
answerable. That is the measured trade-off — bytes on the write side, steps on the read side.

#### Summary

- The field-based record answered the same question in 7 steps instead of 11; the entire gap
  comes from parsing and grows with the number of services.
- The correlation id turned the previous lesson's unanswerable question into one answered in 8
  steps: the `loan-5` id was traced across four services, and the longest downstream leg turned
  out to be catalog.
- 3 of 4 call sites carry the id; the one that does not leaves the notification service entirely
  off the map, and a request is traceable across 4 of 5 services.
- A skipped call site is the natural residue of adding an id: the id gets added to a client
  helper, and an old call path that does not use that helper silently stays without it.
- The cost sits on the write side: the rise from 59 to 101 bytes per record comes from the field
  names repeating on every line.

#### Next Step

The log is now split into fields, and a request's path is traceable across four services. But
every query still starts by reading and filtering every record: a hundred records were read for
twenty requests, a thousand will be read for two hundred. The previous lesson's metric signal
sat outside this cost, because the question it could answer was already chosen when the metric
was written. The next lesson looks at that choice itself: the same fact is measured with three
separate types — counter, gauge, and histogram — the question each type can and cannot answer is
counted, and the histogram's memory cost and percentile error are compared against its bucket
count.

### Metric Types

Source: https://academia.sh/en/courses/observability/metric-types

Measuring the same fact with a counter, a gauge, and a histogram: the question each type answers and forecloses, how many observations the gauge loses between reads, and the histogram's memory cost and percentile error as a function of its bucket count.

The previous two lessons matured the log: the record split into fields, and a request became
traceable across four services. But every query still starts by reading every record. In the
first lesson the metric sat outside that cost, because the question it could answer was already
chosen when the metric was written. This lesson looks at that choice: the same fact is measured
with three metric types, and what question each type opens up and what question it forecloses
is counted.

The fact being measured is a single one: **the count of open loans**. Every request opens a
loan, and in between there are batch returns and occasional batch-loan days. The same number is
measured with three types. A **counter** only ever increases. A **gauge** holds the current
value. A **histogram** distributes values into buckets. All three feed off the same fact, and
all three answer a different question.

Instrumentation's own cost — the overhead producing a metric adds to a request — was measured in
the Performance Anti-Patterns and Monitoring course and is not repeated here. The cost here is
memory and accuracy: how many fields a type holds, and how far off its answer really is.

#### Three Types

```js
// metrics.mjs — three metric types; all three are hand-written, nothing comes from outside
export class Counter {                     // only increases; the past is folded into a single number
  constructor() { this.value = 0; }
  add(n = 1) { this.value += n; }
  read() { return this.value; }
}
export class Gauge {                       // holds the current value; the previous value is lost
  constructor() { this.value = 0; }
  set(v) { this.value = v; }
  read() { return this.value; }
}
export class Histogram {                   // bucket counts; the distribution is preserved approximately
  constructor(bounds) { this.bounds = bounds; this.bucket = new Array(bounds.length + 1).fill(0); this.count = 0; }
  observe(v) { let i = 0; while (i < this.bounds.length && v > this.bounds[i]) i += 1; this.bucket[i] += 1; this.count += 1; }
  size() { return this.bucket.length; }
  percentile(p) {                          // linear interpolation inside the bucket that crosses the target
    const target = p * this.count; let accumulated = 0;
    for (let i = 0; i < this.bucket.length; i += 1) {
      if (accumulated + this.bucket[i] >= target) {
        const lower = i === 0 ? 0 : this.bounds[i - 1];
        const upper = i < this.bounds.length ? this.bounds[i] : lower * 2;
        return lower + ((target - accumulated) / this.bucket[i]) * (upper - lower);
      }
      accumulated += this.bucket[i];
    }
    return this.bounds[this.bounds.length - 1];
  }
}
export const equalBounds = (k, max) =>     // k buckets: the [0, max] range is split evenly
  Array.from({ length: k - 1 }, (_, i) => Math.round(((i + 1) * max) / (k - 1)));
```

The three classes' total state is this: the counter is one number, the gauge is one number, the
histogram is as many numbers as it has buckets. The types differ not in their behavior but in
**what they forget**. The counter folds every observation into the total and forgets the
observation. The gauge holds the latest observation and forgets the one before it. The histogram
holds which bucket an observation fell into and forgets the value itself.

**TL3 — the pattern by which the open-loan count changes is independent of the run: return and
batch-loan decisions come from a generator with a visible seed.** Rationale: what is being
compared is the behavior of the metric types, not the speed of the environment; if the same
observation sequence is not produced on every run, the error margins cannot be compared.

#### Same Fact, Three Types

```js
// service3.mjs — loan service; measures the same fact (open loan count) with three types; long-lived process
import { createServer } from "node:http";
import { Counter, Gauge, Histogram, equalBounds } from "./metrics.mjs";
const opened = new Counter(), open = new Gauge(), distribution = new Histogram(equalBounds(8, 400));
let seed = 20260731;                       // visible seed: the return pattern is independent of the run
const next = () => (seed = (seed * 1103515245 + 12345) % 2147483648);
let value = 0, peak = 0;                   // peak: kept to show what the gauge misses
const sample = [];                         // full sample: kept only for comparison

createServer((req, res) => {
  if (req.url === "/metrics") {
    res.end(JSON.stringify({ opened: opened.read(), open: open.read(),
      bucket: distribution.bucket, bounds: distribution.bounds, peak }));
    return;
  }
  if (req.url === "/samples") { res.end(JSON.stringify(sample)); return; }
  value += 1; opened.add();                          // every request opens a loan
  if (next() % 60 === 0) value = Math.max(0, value - Math.ceil(value / 2));  // batch return
  if (next() % 211 === 0) value += 180;               // batch-loan day
  peak = Math.max(peak, value);
  open.set(value); distribution.observe(value); sample.push(value);
  res.end("done");
}).listen(9301);
```

The service also keeps two things purely for comparison: the true peak value and the full
sample. Neither would be kept in a real system — precisely because of the cost this lesson
measures. They stay here so that the answer each of the three types gives can be compared
against the truth.

#### Measurement

The query tool does two things. First it lines up the three types side by side and prints the
question each one answers and cannot answer. Then it re-summarizes the same observation sequence
with histograms of four, eight, sixteen, thirty-two, and sixty-four buckets, and works out each
one's memory cost and p95 error.

```js
// query3.mjs — the question the three types answer, and the cost of the histogram's bucket count
import { Histogram, equalBounds } from "./metrics.mjs";
const m = await (await fetch("http://127.0.0.1:9301/metrics")).json();
const sample = await (await fetch("http://127.0.0.1:9301/samples")).json();
const sorted = [...sample].sort((a, b) => a - b);
const actual = (p) => sorted[Math.floor(p * sorted.length)];
const h = new Histogram(m.bounds); h.bucket = m.bucket; h.count = sample.length;

console.log(`${"type".padEnd(11)}${"size".padStart(5)}  ${"question answered".padEnd(36)}question left unanswered`);
console.log(`${"counter".padEnd(11)}${"1".padStart(5)}  ${`loans opened: ${m.opened}`.padEnd(36)}loans open right now`);
console.log(`${"gauge".padEnd(11)}${"1".padStart(5)}  ${`loans open right now: ${m.open}`.padEnd(36)}what the past peak was`);
console.log(`${"histogram".padEnd(11)}${String(m.bucket.length).padStart(5)}  ` +
  `${`95% of observations under ${Math.round(h.percentile(0.95))}`.padEnd(36)}which observation was the peak`);
console.log(`actual p95 ${actual(0.95)}, actual peak ${m.peak}; ` +
  `full sample ${sample.length} values x 8 bytes = ${sample.length * 8} bytes`);

console.log(`\n${"buckets".padStart(8)}${"memory".padStart(10)}${"p95 estimate".padStart(14)}${"error".padStart(7)}${"relative".padStart(10)}`);
for (const k of [4, 8, 16, 32, 64]) {
  const hh = new Histogram(equalBounds(k, 400));
  for (const v of sample) hh.observe(v);
  const t = hh.percentile(0.95), error = Math.abs(t - actual(0.95));
  console.log(`${String(k).padStart(8)}${`${hh.size() * 8} bytes`.padStart(10)}${t.toFixed(1).padStart(14)}` +
    `${error.toFixed(1).padStart(7)}${`%${((100 * error) / actual(0.95)).toFixed(1)}`.padStart(10)}`);
}
```

```bash
cat > run.mjs <<'EOF'
const READ_EVERY = 100;                            // the gauge is read once every 100 requests
let highest = 0, reads = 0;
for (let i = 1; i <= 2000; i += 1) {
  await fetch("http://127.0.0.1:9301/work");
  if (i % READ_EVERY !== 0) continue;
  const m = await (await fetch("http://127.0.0.1:9301/metrics")).json();
  highest = Math.max(highest, m.open); reads += 1;
}
const m = await (await fetch("http://127.0.0.1:9301/metrics")).json();
console.log(`gauge: 2000 observations, ${reads} reads, highest seen ${highest}, ` +
  `actual peak ${m.peak}; observations missed between reads ${2000 - reads}`);
EOF
node service3.mjs &
sleep 1

node run.mjs
echo
node query3.mjs
pkill -f "node service3.mjs"
```

```
gauge: 2000 observations, 20 reads, highest seen 199, actual peak 293; observations missed between reads 1980

type        size  question answered                   question left unanswered
counter        1  loans opened: 2000                  loans open right now
gauge          1  loans open right now: 26            what the past peak was
histogram      8  95% of observations under 159       which observation was the peak
actual p95 159, actual peak 293; full sample 2000 values x 8 bytes = 16000 bytes

 buckets    memory  p95 estimate  error  relative
       4  32 bytes         209.1   50.1     %31.5
       8  64 bytes         158.5    0.5      %0.3
      16 128 bytes         157.1    1.9      %1.2
      32 256 bytes         161.5    2.5      %1.6
      64 512 bytes         158.5    0.5      %0.3
```

#### Each Type Forecloses One Question

The top table gives the division of labor between the types. The counter says in a single field
that two thousand loans were opened, but because it does not know how many closed, it cannot
answer "how many loans are open right now" — an increase that cannot be taken back is the
counter's definition. The gauge answers that question in a single field: twenty-six. The
histogram answers what neither of the other two can: ninety-five percent of observations stayed
under 159.

The gauge's cost sits in the first line, and it is this lesson's sharpest number. Two thousand
observations were produced, the gauge was read twenty times, and the highest value read was 199
— the actual peak was 293. The nineteen hundred and eighty observations that fell between reads
were never seen at all, and the peak was reported thirty-two percent short. Because what a gauge
holds is "the value right now," whatever movement happens between reads is lost no matter how
often it is read. When a gauge is asked whether a threshold was crossed, a "no" answer can mean
the threshold was never crossed, or it can mean the read never landed on the moment it was
crossed.

The bottom table ties the histogram's bucket count to its cost. Four buckets take thirty-two
bytes and give a p95 of 209: a deviation of roughly thirty-one and a half percent from the true
159. At eight buckets the error drops to half a unit, 0.3 percent. What comes after this is
surprising, and it is the lesson's second finding: **error does not shrink evenly with the
bucket count**. It is 1.9 at sixteen buckets, 2.5 at thirty-two, and back down to 0.5 at
sixty-four. The reason is plain — the error depends on where the bucket boundaries happen to
fall relative to the distribution. The ninety-fifth percentile sits around 159; if a boundary
happens to fall close to it, the interpolation covers a short distance and the error shrinks, and
if the boundary is far off, the inside of the bucket is assumed linear and the error grows.
Raising the bucket count lowers the expected error, but it gives no guarantee for any single
percentile.

The comparison line is the last one: the full sample takes 16,000 bytes and gives the p95
exactly. The eight-bucket histogram takes 64 bytes and gives the same answer with half a unit of
error — two hundred fifty times less memory. This is the metric's real job: to hold an
approximate answer at a fixed cost. A fixed cost is the opposite of the previous lesson's log
volume, which grows with the request count.

#### Summary

- The same fact was measured with three types: the counter gave 2000 opens, the gauge gave 26
  open loans, the histogram gave 95% of observations under 159; none answered the others'
  question.
- The types differ in what they forget: the counter forgets the observation, the gauge forgets
  the previous value, the histogram forgets the value itself.
- The gauge saw 20 of 2000 observations; the peak read was 199, the actual peak was 293 — the
  1980 observations between reads were lost, and the peak was reported short.
- The histogram's bucket count grows memory linearly (4 buckets, 32 bytes; 64 buckets, 512
  bytes), but the error does not shrink evenly: 31.5% → 0.3% → 1.2% → 1.6% → 0.3%.
- An 8-bucket histogram gave the same p95 as the 16,000-byte full sample using 64 bytes and half
  a unit of error; a metric's cost grows with its field count, not its observation count.

#### Next Step

Two of the three signals are now produced in code: the field-based log traces a request across
four services, and the metrics give a fixed-cost summary. What remains is the original question
the first lesson left unanswered. The correlation id says which services a request passed
through, but it does not say how the durations nest inside one another: did the catalog call
start inside the loan request, did it finish before the membership call, where did the waiting
happen. The next lesson hand-writes a trace context and carries it across the process boundary.
What gets measured is clear: how many traces break where the context is not carried through, how
the sampling rate affects the percentage of problem requests caught, and the volume of the trace
record.

### Distributed Tracing

Source: https://academia.sh/en/courses/observability/distributed-tracing

Hand-writing the trace context and carrying it across the process boundary, and measuring the sampling rate: how many traces break at the call site that does not carry the context, how the sampling rate affects the percentage of problem requests caught, and the trace record's volume per span.

The correlation id says which services a request passed through, and the metrics give a
fixed-cost summary. Neither says one thing: how the durations nest inside one another. Where
inside the loan request did the catalog call start, did it run at the same time as the
membership branch, in which call did the waiting happen. This lesson writes the signal that
builds that structure.

A **trace** is the shared record of all the work an external request gives birth to; a **span**
is a single piece of that record — the time between one piece of work's start and finish, and
its link to its parent span. The span here is entirely separate from the query range covered in
the data access lessons: there, a range was the lower and upper bound of a key set; here it is a
piece of work's start and finish point in time. **Propagation** is carrying the context across
the process boundary; where it is missing, the trace breaks.

#### The Trace Context

The context is made of three values: the trace's id, the parent span's id, and whether it was
sampled. All three travel in a single header, because the only thing that can cross to the other
side of the boundary is the request itself.

```js
// trace.mjs — trace context and span; the context travels in a header, the span is written to a file
import { appendFileSync } from "node:fs";
export const HEADER = "x-trace";
export const parseContext = (value) => {         // "<traceId>|<parent span>|<sampled>"
  if (!value) return null;
  const [traceId, parent, s] = value.split("|");
  return { traceId, parent, sampled: s === "1" };
};
export const buildContext = (c) => `${c.traceId}|${c.parent}|${c.sampled ? 1 : 0}`;
export const hash = (s) => {                     // hash that spreads ids evenly; the sampling decision rests on it
  let h = 2166136261;
  for (const c of s) { h ^= c.charCodeAt(0); h = Math.imul(h, 16777619) >>> 0; }
  return h;
};
export const samplingDecision = (traceId, rate) => hash(traceId) % 1000 < rate * 1000;

export class Trace {
  constructor(file, service) { this.file = file; this.service = service; this.no = 0; }
  start(name, context) {
    return { traceId: context.traceId, span: `${this.service}${(this.no += 1)}`, parent: context.parent,
      name, t0: performance.now(), sampled: context.sampled };
  }
  finish(a) {
    if (!a.sampled) return;                      // an unsampled span is never written
    appendFileSync(this.file, JSON.stringify({ trace: a.traceId, span: a.span, parent: a.parent,
      service: this.service, name: a.name, duration: Math.round(performance.now() - a.t0) }) + "\n");
  }
}
```

**TL4 — the sampling decision is made only at the root and carried with the context.**
Rationale: if the decision were made separately at every service, some spans of the same trace
would be recorded and others would not, and half-traces would be indistinguishable from complete
ones. Making the decision at the root ensures every recorded trace is whole; its cost is what
the next section measures.

#### Carrying the Context Across the Boundary

The service does three things: parses the context from the incoming header, becomes the root and
produces the trace id and the sampling decision if no context arrived, and updates the context
with its own span id before putting it in the header when it calls downstream. The notification
call site from the previous lessons still does not carry the context here either.

```js
// service4.mjs — <name> <port> [downstream...]; the trace context crosses the process boundary; long-lived process
import { createServer } from "node:http";
import { writeFileSync } from "node:fs";
import { Trace, HEADER, parseContext, buildContext, samplingDecision } from "./trace.mjs";
const [name, port, ...downstream] = process.argv.slice(2);
const NOTIFICATION = "9405";
const file = `trace-${name}.txt`; writeFileSync(file, "");
const trace = new Trace(file, name);
let rate = 1;
const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const url = (p, u) => `http://127.0.0.1:${p}/work?member=${u.get("member")}&copy=${u.get("copy")}`;

createServer(async (req, res) => {
  const u = new URL(req.url, "http://y").searchParams;
  if (req.url.startsWith("/rate")) {             // set the rate and empty the trace file
    rate = Number(u.get("value")); writeFileSync(file, ""); res.end("done"); return;
  }
  const member = u.get("member"), copy = Number(u.get("copy"));
  let context = parseContext(req.headers[HEADER]);
  if (!context) {                                // root: the trace id and sampling decision are given here
    const traceId = `${name}-${copy}`;
    context = { traceId, parent: "", sampled: samplingDecision(traceId, rate) };
  }
  const a = trace.start(`${name}/work`, context);
  await wait(name === "catalog" && copy % 5 === 0 ? 40 : 3);
  let status = name === "fee" && member === "4023" ? 402 : 200;
  for (const p of downstream) {
    const headers = p === NOTIFICATION ? {}       // the notification call site does not carry the context
      : { [HEADER]: buildContext({ ...context, parent: a.span }) };
    const y = await fetch(url(p, u), { headers });
    if (y.status !== 200) status = y.status;
  }
  trace.finish(a);
  res.statusCode = status; res.end("done");
}).listen(Number(port));
```

When the notification service receives a request with no context, it takes itself for the root
and opens its own trace. The result is not a missing span, it is an **extra trace**: the loan
request's trace ends at four spans, and the notification work stands as a separate trace that
connects to nothing.

#### Measurement

The query tool reads the trace files and groups spans by trace id. Traces that begin with the
loan id are real traces; ones that begin with the notification id are broken traces. The
definition of a problem request lives in the code: copies whose number is a multiple of five
come from the slow shelf, and one of the four members gets an error for an unpaid fee.

```js
// query4.mjs — <rate>; trace integrity and the sampling rate's effect on problem requests
import { readFileSync, statSync } from "node:fs";
const SERVICES = ["loan", "membership", "catalog", "fee", "notification"];
const N = 100;
const problematic = (n) => n % 5 === 0 || (n - 1) % 4 === 2;   // slow shelf or unpaid fee
const rate = process.argv[2];
const spans = SERVICES.flatMap((s) =>
  readFileSync(`trace-${s}.txt`, "utf8").split("\n").filter(Boolean).map((l) => JSON.parse(l)));
const traces = new Map();
for (const a of spans) { if (!traces.has(a.trace)) traces.set(a.trace, []); traces.get(a.trace).push(a); }
const roots = [...traces.keys()].filter((k) => k.startsWith("loan"));
const broken = [...traces.keys()].filter((k) => k.startsWith("notification"));
const complete = roots.filter((k) => traces.get(k).length === 4).length;
const caught = roots.filter((k) => problematic(Number(k.split("-")[1]))).length;
const total = Array.from({ length: N }, (_, i) => i + 1).filter(problematic).length;
const bytes = SERVICES.reduce((t, s) => t + statSync(`trace-${s}.txt`).size, 0);
console.log(`${rate.padStart(5)}${String(spans.length).padStart(9)}${String(roots.length).padStart(5)}` +
  `${String(complete).padStart(8)}${String(broken.length).padStart(10)}${`${caught}/${total}`.padStart(12)}` +
  `${`%${Math.round((100 * caught) / total)}`.padStart(7)}` +
  `${String(spans.length ? Math.round(bytes / spans.length) : 0).padStart(12)}`);
```

```bash
cat > run.mjs <<'EOF'
const MEMBER = ["4021", "4022", "4023", "4024"];      // 4023 is the member with an unpaid fee
const PORT = [9401, 9402, 9403, 9404, 9405];
for (const rate of ["1.00", "0.50", "0.10", "0.02"]) {
  for (const p of PORT) await fetch(`http://127.0.0.1:${p}/rate?value=${rate}`);
  for (let i = 1; i <= 100; i += 1)
    await fetch(`http://127.0.0.1:9401/work?member=${MEMBER[(i - 1) % 4]}&copy=${i}`);
  await new Promise((c) => setTimeout(c, 100));
  const { execSync } = await import("node:child_process");
  process.stdout.write(execSync(`node query4.mjs ${rate}`).toString());
}
EOF
node service4.mjs fee 9404 &
node service4.mjs catalog 9403 &
node service4.mjs notification 9405 &
node service4.mjs membership 9402 9404 &
node service4.mjs loan 9401 9402 9403 9405 &
sleep 1

printf '%5s%9s%5s%8s%10s%12s%7s%12s\n' rate spans root whole broken caught pct "bytes/span"
node run.mjs
pkill -f "node service4.mjs"
```

```
 rate    spans root   whole    broken      caught    pct  bytes/span
 1.00      500  100     100       100       40/40   %100         113
 0.50      233   46      46        49       18/40    %45         115
 0.10       35    7       7         7        2/40     %5         115
 0.02        8    2       2         0        0/40     %0         111
```

#### The Broken Trace and the Cost of Sampling

The first row shows that the context truly crosses the boundary: a hundred external requests, a
hundred traces, and every one of them **whole** — four spans. The loan span sits at the top, the
membership and catalog spans sit below it, and the fee span sits below membership. The "where did
the waiting happen" question the previous lessons could not answer now lives in the record's own
structure.

Right beside it there are a hundred **broken traces**. Because the notification call site does
not carry the context, every piece of work in that service opens its own root: a hundred
separate, single-span traces for a hundred requests. This is why the measure matters — a broken
trace does not look like missing data, it looks like **extra data**. Two hundred traces sit in
the records, and nothing says that a hundred of them are really part of the other hundred.
Someone trying to follow a request's path would conclude that notification was never called at
all.

The bottom three rows give the cost of the sampling rate. When the rate is halved, the spans
written drop from five hundred to two hundred thirty-three; at a tenth, to thirty-five; at one in
fifty, to eight. Volume falls roughly in step with the rate, and the size per span stays close to
a fixed range of 111–115 bytes: sampling changes the count of the record, not its size.

What gets lost in exchange is in the columns on the right. All forty problem requests were
caught only at full sampling. At the fifty-percent rate, eighteen of the forty problem requests
were caught; at a tenth, two; at one in fifty, **none**. The numbers come out in the same
neighborhood as the rate itself, and that is exactly this lesson's finding: **a sampling decision
made at the root does not favor the problem request.** The decision is made at the start of the
request, before it is known whether it will slow down or fail, so problem requests get thinned
out by the rate just like every other request. Sampling at two percent does not mean "I am
keeping two percent of the traces," it means "I will never see ninety-eight percent of the
problems."

Making the sampling decision at the root did buy one thing: **every** recorded trace is whole. At
the fifty-percent rate, all forty-six of the forty-six traces written have four spans. Had the
decision been made separately at each service, the same volume would have produced far more
traces, but half of them would be half-traces — and a half-trace, like a broken trace, makes the
data misleading.

#### Summary

- The trace context is made of three values (trace id, parent span, sampling decision) and
  crosses the process boundary in a single header; at full sampling, 100 requests turned into
  100 whole traces and 400 linked spans.
- The one call site that does not carry the context produced 100 broken traces; a broken trace
  does not show up as missing data, it shows up as extra, unlinked traces.
- The sampling rate lowers volume roughly in step with the rate: 500, 233, 35, and 8 spans; the
  size per span stayed close to a fixed range of 111–115 bytes.
- A sampling decision made at the root does not favor the problem request: of 40 problem
  requests, the number caught was 40, 18, 2, and 0 in turn.
- Making the decision at the root ensured every recorded trace was whole: at every rate, all the
  traces written had four spans.

#### Next Step

All three signals are now produced in code, but each still follows its own path. The logger
writes to its own file, the metrics sit behind an endpoint, the spans fall into another file;
where each one gets written is buried inside the service code. This ties the application code to
a single way of collecting telemetry: if the write destination changes, all five services have to
be searched. The next lesson cuts that tie and designs an interface. What gets measured is a
single thing: how many files get touched when the collector implementation changes.

### Instrumentation Standards

Source: https://academia.sh/en/courses/observability/instrumentation-standards

Separating telemetry collection from application code: meeting the same interface with two collector implementations, measuring how many files get touched when the implementation changes in a bound layout versus an interfaced one, and counting how many application files the collector's name appears in.

All three signals are produced in code, but where each one gets written still lives inside the
service code: the logger opens its own file, the metrics are served from an endpoint, the spans
fall into another file. This tie looks harmless at first. Its harm only becomes visible when the
collection method changes, and the moment it does, it turns into a measurable number: how many
files had to be touched.

**Vendor-neutral instrumentation** means the application code does not know where or how
telemetry gets sent. What this lesson builds is not a name, it is an **interface**: the
application code only says "this event happened"; whether the event gets written to a file,
counted, or sent somewhere else is someone else's job. The measure comes down to a single
question — how many files change when the implementation changes.

#### Two Implementations of the Same Interface

The interface is made of two operations: `collect` takes an event, `summary` returns the state
of what has been collected. The two implementations are direct opposites in behavior. The first
writes every event to disk as its own line; the second never stores the individual event at all,
it only counts by type.

```js
// collectors.mjs — two implementations of the same interface; both implement collect and summary
import { appendFileSync, writeFileSync } from "node:fs";
export const fileCollector = (file) => {        // writes every event as its own line
  writeFileSync(file, "");
  return { collect: (o) => appendFileSync(file, JSON.stringify(o) + "\n"),
    summary: () => `events in ${file}` };
};
export const summaryCollector = () => {         // counts events by type, never writes a single event
  const count = new Map();
  return { collect: (o) => count.set(o.type, (count.get(o.type) ?? 0) + 1),
    summary: () => [...count].map(([t, n]) => `${t}=${n}`).join(" ") };
};
```

The face the application code sees is different. The service code never calls `collect`; it
calls the three operations that correspond to the three signals, and it does not know where the
event goes. The binding happens in a single file, in a single line.

```js
// measurement.mjs — the single face the application code sees; it does not know which collector is bound
import { collector } from "./binding.mjs";
export const log = (fields) => collector.collect({ type: "log", ...fields });
export const metric = (name, value) => collector.collect({ type: "metric", name, value });
export const span = (name, duration) => collector.collect({ type: "trace", name, duration });
export const summary = () => collector.summary();
```

```js
// binding.mjs — the single binding point; the implementation is chosen only in this line
import { fileCollector } from "./collectors.mjs";
export const collector = fileCollector("event-interfaced.txt");        // BINDING
```

**TL5 — the five services are separate deployment units, and therefore separate source files.**
Rationale: what is being measured is a change's blast radius; in a setup derived from a single
source file with a role argument, every change lands in one file by definition and the measure
loses its meaning. All five files are generated from the same template, and the only difference
between them is the role name and the port.

#### Two Layouts

Both layouts do the same job. In the bound layout, the service file calls the collector by name
and formats the events itself. In the interfaced layout, no collector name appears in the
service file at all.

```js
// template-bound.mjs — five service files are generated by substituting ROLE and PORT
import { createServer } from "node:http";
import { fileCollector } from "./collectors.mjs";
const collector = fileCollector("event-bound.txt");                  // BINDING
let request = 0;
createServer((req, res) => {
  if (req.url === "/summary") { res.end(collector.summary()); return; }
  const t0 = performance.now(); request += 1;
  collector.collect({ type: "log", service: "ROLE", event: "loan request processed" });
  collector.collect({ type: "metric", name: "ROLE_request", value: request });
  collector.collect({ type: "trace", name: "ROLE/work", duration: Math.round(performance.now() - t0) });
  res.end("done");
}).listen(PORT);
```

```js
// template-interfaced.mjs — five service files are generated by substituting ROLE and PORT
import { createServer } from "node:http";
import { log, metric, span, summary } from "./measurement.mjs";
let request = 0;
createServer((req, res) => {
  if (req.url === "/summary") { res.end(summary()); return; }
  const t0 = performance.now(); request += 1;
  log({ service: "ROLE", event: "loan request processed" });
  metric("ROLE_request", request);
  span("ROLE/work", Math.round(performance.now() - t0));
  res.end("done");
}).listen(PORT);
```

#### Measurement

The measurement actually performs the switch. First both layouts run with the file collector,
then the `switchOver` function changes the implementation in the relevant files, then both run
again. The number of files touched is not estimated: a directory copy taken before the switch is
compared against the state after.

```bash
SERVICES="loan membership catalog fee notification"
rm -rf bound interfaced bound-before interfaced-before; mkdir -p bound interfaced
cp collectors.mjs bound/ ; cp collectors.mjs measurement.mjs binding.mjs interfaced/
i=0
for s in $SERVICES; do
  sed -e "s/ROLE/$s/g" -e "s/PORT/$((9501 + i))/" template-bound.mjs > bound/$s.mjs
  sed -e "s/ROLE/$s/g" -e "s/PORT/$((9511 + i))/" template-interfaced.mjs > interfaced/$s.mjs
  i=$((i + 1))
done
switchOver() {   # replaces the file collector with the summary collector
  perl -pi -e 's/fileCollector/summaryCollector/g; s/\("event-[a-z]+\.txt"\)/()/' "$1"
}
runTrial() {   # brings up the loan service in $1, sends 5 requests, prints the summary
  (cd "$1" && node loan.mjs) & PID=$!
  sleep 0.7
  for n in 1 2 3 4 5; do curl -s -o /dev/null "http://127.0.0.1:$2/work"; done
  printf '%-10s %s\n' "$1" "$(curl -s http://127.0.0.1:$2/summary)"
  kill $PID 2>/dev/null; wait $PID 2>/dev/null
}

echo "before the switch:"
runTrial bound 9501
runTrial interfaced 9511
echo "first event record: $(head -1 bound/event-bound.txt)"

cp -r bound bound-before; cp -r interfaced interfaced-before
for s in $SERVICES; do switchOver bound/$s.mjs; done
switchOver interfaced/binding.mjs

echo
echo "after the switch:"
runTrial bound 9501
runTrial interfaced 9511

echo
printf '%-10s%34s%28s\n' layout "app files naming collector" "files touched by switch"
for y in bound interfaced; do
  named=$(grep -l "Collector" $y/*.mjs | grep -v collectors | grep -v binding | wc -l | tr -d ' ')
  touched=$(diff -rq $y-before $y | wc -l | tr -d ' ')
  printf '%-10s%34s%28s\n' "$y" "$named" "$touched"
done
echo "files touched (interfaced): $(diff -rq interfaced-before interfaced | awk '{print $4}' | paste -sd' ' -)"
```

```
before the switch:
bound      events in event-bound.txt
interfaced events in event-interfaced.txt
first event record: {"type":"log","service":"loan","event":"loan request processed"}

after the switch:
bound      log=5 metric=5 trace=5
interfaced log=5 metric=5 trace=5

layout            app files naming collector     files touched by switch
bound                                      5                           5
interfaced                                 0                           1
files touched (interfaced): interfaced/binding.mjs
```

#### Five Files versus One

Both layouts give the same summary after the switch: `log=5 metric=5 trace=5`. The behavior is
equal, the cost is not. Five files changed in the bound layout, one in the interfaced layout. The
name of that one changed file is printed too: `binding.mjs`. None of the five service files was
touched — their state before and after the switch is byte for byte the same.

The middle column gives the reason for the gap. In the bound layout, the collector's name
appears in five application files; in the interfaced layout, in zero. The number of files
touched is the direct consequence of that: wherever a name is written, changing that name
requires touching just as many places. What the interface does is confine the name to a single
place; it does nothing else.

How the number grows is also part of the measure. Five services today means five files; as the
system grows, this count grows with the number of services, and every increase makes the same
switch a notch more expensive. In the interfaced layout the count is independent of the service
count: with fifty services, the binding point is still one. This is the same shape as the
previous lessons' metric field count being independent of the request count — the cost is tied
to something fixed, not to something that grows.

The interface's cost is just as plain, and should not be underestimated. Two extra files were
written (`measurement.mjs` and `binding.mjs`), a layer of indirection now sits between the
application code and the collector, and the interface itself became a decision: `collect` only
takes an object with a `type` field, so if a collector needs a piece of information that does not
fit this shape, the interface has to change. And an interface change lands right back in the
application files — the independence gained is only as good as the interface's design.

There is one more boundary. The switch was made here by changing a single line, because what was
measured was the change's **blast radius**. Choosing the collector at runtime, writing to more
than one collector at once, or what happens to events while a collector is down are outside this
lesson's measure; none of them change the file count.

#### Summary

- The interface is just two operations: `collect` takes an event, `summary` returns the state;
  the two implementations met the same events by one writing to disk, the other only counting.
- After the switch, both layouts gave the same summary (`log=5 metric=5 trace=5`); the difference
  was not in behavior, it was in the cost of the change.
- The collector's name appears in 5 application files in the bound layout, and in 0 in the
  interfaced layout.
- The same switch touched 5 files in the bound layout and 1 in the interfaced layout; the five
  service files stayed byte for byte the same before and after.
- In the bound layout the cost grows with the number of services; in the interfaced layout it is
  fixed; in exchange there is a two-file indirection and the interface itself becoming a
  decision.

#### Next Step

The three signals are now produced in code, none stands in for another, and none is bound to a
collector. But they still live in three separate places: when a question is asked, the log is
read separately, the metric separately, the trace separately, and the link between them is
rebuilt by hand every time. The second lesson produced the correlation id, the fourth lesson
carried the trace context across the boundary; both name the same request, but no single record
holds all three together. The next lesson does that, and closes the topic: the three signals are
combined into a single event, and the same question is compared for how many steps and how many
signals it takes to answer, combined and uncombined.

### Correlation

Source: https://academia.sh/en/courses/observability/correlation

Combining the three signals into a single event by correlation id: how many steps and how many signals the same question takes to answer, combined and uncombined, the cost of combining that gets paid once, and the sampling rate's effect on how many events carry all three signals.

All three signals are produced in code, none is bound to a collector, and all of them carry the
same correlation id. But they still live in three separate places. When a question is asked, the
log is read separately, the metric separately, the trace separately, and the link between them
is rebuilt every time. This lesson builds that link once, stores it, and compares the cost of
the two paths.

The question being asked has three parts, and each part falls to a different signal: **why** did
this request fail (log), **where** did it slow down (trace), **is** this slowness normal
(metric). The target is a request that both comes off the slow shelf and belongs to a member
with an unpaid fee; all three questions apply to it at once.

#### Three Signals, One Id

The service writes all three signals, and every one of them carries the same correlation id. The
second lesson's uncovered call site is closed here: the notification call now carries the
context too, so no broken trace remains.

**TL6 — the quantity the histogram buckets is the request's reported work, not its wall-clock
duration.** Rationale: wall-clock duration varies from run to run, and requests near a bucket
boundary shift from one run to the next; reported work is the same on every run, so the bucket
shares stay comparable. Wall-clock duration still lives in the span, and that is what answers
the "where did it slow down" question.

```js
// service6.mjs — <name> <port> [downstream...]; all three signals carry the same correlation id
import { createServer } from "node:http";
import { appendFileSync, writeFileSync } from "node:fs";
const [name, port, ...downstream] = process.argv.slice(2);
const BOUNDS = [10, 45, 150, 400];                  // reported-work buckets (ms)
const bucket = new Array(BOUNDS.length + 1).fill(0);
const counter = { requests: 0, errors: 0 };
let rate = 1, no = 0;
const hash = (s) => { let h = 2166136261; for (const c of s) { h ^= c.charCodeAt(0); h = Math.imul(h, 16777619) >>> 0; } return h; };
const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const url = (p, u) => `http://127.0.0.1:${p}/work?member=${u.get("member")}&copy=${u.get("copy")}`;
const reset = () => { writeFileSync(`log-${name}.txt`, ""); writeFileSync(`trace-${name}.txt`, ""); };
reset();

createServer(async (req, res) => {
  const u = new URL(req.url, "http://y").searchParams;
  if (req.url.startsWith("/rate")) {
    rate = Number(u.get("value")); counter.requests = 0; counter.errors = 0; bucket.fill(0); no = 0;
    reset(); res.end("done"); return;
  }
  if (req.url === "/metrics") { res.end(JSON.stringify({ service: name, ...counter, bounds: BOUNDS, bucket })); return; }
  const member = u.get("member"), copy = Number(u.get("copy"));
  const incoming = req.headers["x-trace"];
  const [traceId, parent, s] = incoming ? incoming.split("|") : [`loan-${copy}`, "", ""];
  const sampled = incoming ? s === "1" : hash(traceId) % 1000 < rate * 1000;
  const span = `${name}${(no += 1)}`, t0 = performance.now();
  counter.requests += 1;
  const ownWork = name === "catalog" && copy % 5 === 0 ? 80 : 3;   // reported work: independent of the run
  await wait(ownWork);
  let status = name === "fee" && member === "4023" ? 402 : 200, downstreamWork = 0;
  for (const p of downstream) {
    const y = await fetch(url(p, u), { headers: { "x-trace": `${traceId}|${span}|${sampled ? 1 : 0}` } });
    const body = await y.json(); downstreamWork += body.work;
    if (y.status !== 200) status = y.status;
  }
  const work = ownWork + downstreamWork, duration = Math.round(performance.now() - t0);
  if (status !== 200) counter.errors += 1;
  let k = 0; while (k < BOUNDS.length && work > BOUNDS[k]) k += 1; bucket[k] += 1;
  appendFileSync(`log-${name}.txt`, JSON.stringify({ correlationId: traceId, service: name, member, copy, status }) + "\n");
  if (sampled) appendFileSync(`trace-${name}.txt`,
    JSON.stringify({ trace: traceId, span, parent, service: name, duration, work, bucket: k }) + "\n");
  res.statusCode = status; res.end(JSON.stringify({ work, status }));
}).listen(Number(port));
```

#### Two Paths

The query tool answers the same question twice. In the first path, the three stores are read
separately, each is filtered by the target id, and the three results are aligned by hand. In the
second path, every record is first gathered into a single event by correlation id, and then the
question is asked of that one store. The step definition is the same as it has been throughout
the topic: reading a file or an endpoint, parsing, filtering, or aligning are each one step.

```js
// query6.mjs — <rate> [mode]; the same question answered first from three stores, then from the combined event
import { readFileSync } from "node:fs";
const SERVICES = ["loan", "membership", "catalog", "fee", "notification"];
const PORT = { loan: 9601, membership: 9602, catalog: 9603, fee: 9604, notification: 9605 };
const TARGET = "loan-15";                          // both a slow shelf copy and an unpaid fee
const [rate, mode] = process.argv.slice(2);
let step = 0; const count = () => { step += 1; };
const read = (prefix, field) => {                  // 5 file reads + 1 parsing pass
  const k = SERVICES.flatMap((s) => { count(); return readFileSync(`${prefix}-${s}.txt`, "utf8").split("\n").filter(Boolean); });
  count(); return k.map((l) => JSON.parse(l)).map((r) => ({ ...r, key: r[field] }));
};
const readMetrics = async () => { const o = {}; for (const s of SERVICES) { count(); o[s] = await (await fetch(`http://127.0.0.1:${PORT[s]}/metrics`)).json(); } return o; };
const label = (m, k) => `[${k === 0 ? 0 : m.bounds[k - 1]},${k < m.bounds.length ? m.bounds[k] : "+"})`;

// uncombined: each signal is read separately, results are aligned by hand
step = 0;
const g = read("log", "correlationId"), trace = read("trace", "trace"), m = await readMetrics();
const gh = g.filter((r) => r.key === TARGET); count();
const th = trace.filter((r) => r.key === TARGET); count();
const source = gh.find((r) => r.status !== 200 && r.service !== "loan" && r.service !== "membership");
const longest = th.filter((r) => r.service !== "loan").sort((a, b) => b.duration - a.duration)[0];
const rootSpan = th.find((r) => r.service === "loan");
const share = Math.round((100 * m.loan.bucket[rootSpan.bucket]) / m.loan.requests);
count();
const answer = `member ${gh[0].member}, status ${gh[0].status} (source ${source.service}); longest downstream ` +
  `${longest.service}; work in the ${label(m.loan, rootSpan.bucket)} bucket, %${share} of requests`;
if (mode === "question") {
  console.log(`question: why did the ${TARGET} request fail, where did it slow down, is this work normal`);
  console.log(`${"method".padEnd(17)}${"step".padStart(5)}${"signal".padStart(8)}  answer`);
  console.log(`${"uncombined".padEnd(17)}${String(step).padStart(5)}${"3".padStart(8)}  ${answer}`);
}

// combined: the three signals are gathered into a single event by correlation id
step = 0;
const g2 = read("log", "correlationId"), trace2 = read("trace", "trace"), m2 = await readMetrics();
const event = new Map();
for (const r of g2) {
  if (!event.has(r.key)) event.set(r.key, { correlationId: r.key, signals: 2, spans: 0 });
  const e = event.get(r.key);
  if (r.service === "loan") { e.member = r.member; e.status = r.status; }
  if (r.status !== 200 && r.service !== "loan" && r.service !== "membership") e.source = r.service;
}
for (const r of trace2) {
  const e = event.get(r.key); e.spans += 1;
  if (e.signals === 2) e.signals = 3;
  if (r.service === "loan") { e.bucket = label(m2.loan, r.bucket); e.share = Math.round((100 * m2.loan.bucket[r.bucket]) / m2.loan.requests); }
  else if (!e.longest || r.duration > e.longestDuration) { e.longest = r.service; e.longestDuration = r.duration; }
}
count();
const combining = step;
step = 0; count(); const e = event.get(TARGET); count();
if (mode === "question") {
  console.log(`${"combined".padEnd(17)}${String(step).padStart(5)}${"1".padStart(8)}  ` +
    `member ${e.member}, status ${e.status} (source ${e.source}); longest downstream ${e.longest}; ` +
    `work in the ${e.bucket} bucket, %${e.share} of requests`);
  console.log(`combining cost: ${combining} steps, ${g2.length + trace2.length} records -> ` +
    `${event.size} events (paid once, not on every query)`);
  console.log(`\n${"rate".padStart(5)}${"count".padStart(6)}${"3 signals".padStart(12)}${"2 signals".padStart(12)}  missing signal`);
}
const three = [...event.values()].filter((x) => x.signals === 3).length;
console.log(`${rate.padStart(5)}${String(event.size).padStart(6)}${String(three).padStart(12)}` +
  `${String(event.size - three).padStart(12)}  ${event.size - three === 0 ? "-" : "trace (unsampled request)"}`);
```

```bash
cat > run.mjs <<'EOF'
const MEMBER = ["4021", "4022", "4023", "4024"];      // 4023 is the member with an unpaid fee
const PORT = [9601, 9602, 9603, 9604, 9605];
const rate = process.argv[2];
for (const p of PORT) await fetch(`http://127.0.0.1:${p}/rate?value=${rate}`);
for (let i = 1; i <= 100; i += 1)
  await fetch(`http://127.0.0.1:9601/work?member=${MEMBER[(i - 1) % 4]}&copy=${i}`);
EOF
node service6.mjs fee 9604 &
node service6.mjs catalog 9603 &
node service6.mjs notification 9605 &
node service6.mjs membership 9602 9604 &
node service6.mjs loan 9601 9602 9603 9605 &
sleep 1

node run.mjs 1.00 && node query6.mjs 1.00 question
node run.mjs 0.50 && node query6.mjs 0.50
pkill -f "node service6.mjs"
```

```
question: why did the loan-15 request fail, where did it slow down, is this work normal
method            step  signal  answer
uncombined          20       3  member 4023, status 402 (source fee); longest downstream catalog; work in the [45,150) bucket, %20 of requests
combined             2       1  member 4023, status 402 (source fee); longest downstream catalog; work in the [45,150) bucket, %20 of requests
combining cost: 18 steps, 1000 records -> 100 events (paid once, not on every query)

 rate count   3 signals   2 signals  missing signal
 1.00   100         100           0  -
 0.50   100          46          54  trace (unsampled request)
```

#### Twenty Steps versus Two

The two answers are word for word the same. The difference is in the cost: the uncombined path
spent twenty steps and went to three signals separately, the combined path spent two steps and
looked at a single event. The breakdown of the twenty steps is instructive too — fifteen are
reads alone (five log files, five trace files, five metric endpoints), two are parsing, three are
filtering and alignment. Because the question has three parts, all three stores had to be
visited.

Combining does not eliminate this work, it does it **once**. The eighteen-step cost is printed
separately, marked with the line: paid once, not on every query. A thousand records were read
and reduced to a hundred events. When a second question is asked, the uncombined path will again
spend twenty steps, the combined path again two. Correlation's payoff does not grow within a
single query, it grows with the number of queries.

The one thing that makes combining possible is that all three signals carry the same id. That id
entered the log in the second lesson, entered the trace context in the fourth; the metric does
not carry an id and cannot — the third lesson measured exactly this, the id getting lost inside
the total. That is why the metric joins the event differently: a single request cannot be tied
to the metric, but **which bucket** the request fell into, and that bucket's share of the
population, can be. That is the combined event's last field, and it is the only thing that
answers "is this slowness normal": request 15's work falls in the bucket that holds twenty
percent of requests. The trace alone says "eighty milliseconds were spent in catalog," the
metric alone says "twenty percent of requests are in this bucket"; combined, they say "this
request is one of the normal twenty percent that runs slow."

The bottom table gives the limit of combining. At full sampling, all hundred of the hundred
events carry three signals. When the sampling rate is halved, the event count does not change —
the log is written for every request, so a hundred events still stand — but fifty-four of them
carry only two signals: their traces were never written at all. On these events "why" and "is
this normal" can still be asked, "where" cannot. The fourth lesson's sampling measure shows up
here a second time: what gets thinned out is the trace record's volume, what gets lost is the
combined event's resolution.

#### Summary

- Once all three signals carried the same correlation id, they could be combined into a single
  event: 1000 records reduced to 100 events.
- The same question spent 20 steps and 3 signals uncombined, 2 steps and 1 signal combined; the
  two answers came out word for word the same.
- Combining's 18-step cost is paid once; the payoff accumulates not within a single query but
  across the number of queries.
- Because the metric carries no id, it joined the event not as an individual value but as the
  bucket the request fell into and that bucket's share — the one field that answers "is this
  slowness normal."
- Sampling limits combining's resolution: when the rate dropped to 0.50, 54 of 100 events were
  left without the trace signal, and the "where" question was closed off on those events.

#### Next Step

The topic closes here. A request's path is now visible: who asked, where it slowed down, which
service gave birth to the error, where this request falls in the population — all from a single
record, in two steps. There is no such thing anymore as a request lost among five processes.

What remains is this: none of these records says whether the request was **good or bad**. The
measurement says request 15's work falls in the bucket that holds twenty percent of requests; it
does not say whether that twenty percent is acceptable. It says the fee service returned a 402;
it does not say whether twenty-five errors in a hundred requests is normal or a disaster.
Telemetry produces **observation**, not **judgment**; what a judgment would rest on has not been
built yet. The next lesson closes that gap: an indicator is defined from the measured signal, an
objective is attached to the indicator, and the objective is tied to the promise made to the
outside world.

## Service Level Management

The path from telemetry to a decision: distinguishing the concepts of indicator, objective, and agreement, computing the golden signals — latency, traffic, errors, and saturation — in application code, the error budget balancing rate of change against stability, symptom-based alert design cutting down noise, and separating the liveness and readiness endpoints.

### Indicator, Objective, and Agreement

Source: https://academia.sh/en/courses/observability/indicator-objective-and-agreement

Turning telemetry into a decision: showing that indicator, objective, and agreement are separate sentences, computing three indicator definitions — request-based, minute-based, and member-based — from the loan system's thirty-day request log, and counting how the same period can be described by twelve different numbers.

The previous topic joined the three signals in a single event: which process a request passed
through, where it waited, where it was lost, all now readable from a single correlation id. The
one thing missing is the sentence that says whether that record is **good or bad**. Telemetry
produces observation, not judgment. Knowing that the loan service answered a request in 380
milliseconds does not say whether that request counts as successful.

This lesson lays the first stone of the decision layer. Three concepts that are often confused
with one another are defined separately, then three separate indicator definitions are derived
from the same telemetry, and the lesson counts how many different numbers the same thirty days
can be described with.

#### Three Concepts, Three Separate Sentences

A **service level indicator** is a measured ratio. It is computed from telemetry and carries no
commitment by itself: "what percentage of the loan system's requests are good" is an indicator.

A **service level objective** is the internal commitment placed on top of that indicator, and it
has three parts: a definition, a window, a number. "Over a thirty-day window, the request-based
indicator does not drop below 99.5%" is an objective. No one is paid anything in return, but what
the team does when the objective is missed is written down in advance.

A **service level agreement** is the commitment given to the outside world, and it carries a
penalty: if it is violated, a refund, a membership extension, or some other cost follows. An
agreement is always looser than the objective, because there has to be time left to fix things
after the objective is missed.

The question that separates the three is this: the indicator answers "what is being measured",
the objective answers "when does someone intervene", and the agreement answers "what is paid if
it is violated". All three draw on the same telemetry. That is why, when the indicator's
definition changes, the meaning of the other two changes with it — and that is exactly what this
lesson measures.

#### The Thirty-Day Log

The measurement comes from the split loan system's request log. The generator below writes
thirty days of requests across five services using seeded integer arithmetic; four degradation
windows and one scheduled maintenance window are placed on known schedules.

```js
// slo/telemetry.mjs — the split loan system's 30-day request log. Generated with seeded
// integer arithmetic. Record format: [minute, service, status, duration_ms, member, type].
export const MIN = 30 * 1440, MEMBER = 4000, SEED = 20260731;
export const SERVICE = ["catalog", "membership", "loan", "notification", "fee"];
const BASE = [40, 25, 60, 30, 45];
// SD1 (assumption): requests per minute follow the daily rhythm (40 between 08-22, 10 outside it).
// SD2 (assumption): four incidents [day, hour, duration min, service (-1 all), mode, severity (per mille)].
export const EVENT = [
  [4, 8, 90, 2, "error", 60], [11, 14, 240, 0, "slow", 1000],
  [19, 2, 30, -1, "error", 600], [26, 20, 120, 3, "error", 250],
];
// SD3 (assumption): the scheduled maintenance window is day 15's 03:00-04:00 slot.
export const MAINTENANCE = [15 * 1440 + 180, 60];

export function records() {
  let s = SEED;
  const rand = (n) => Math.floor(((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
  const out = [];
  for (let t = 0; t < MIN; t += 1) {
    const hour = Math.floor(t / 60) % 24;
    const maintenance = t >= MAINTENANCE[0] && t < MAINTENANCE[0] + MAINTENANCE[1];
    for (let i = 0, n = hour >= 8 && hour < 22 ? 40 : 10; i < n; i += 1) {
      const sv = rand(5);
      let status = 200, duration = BASE[sv] + rand(60);
      if (rand(1000) < 8) status = 404;
      if (rand(1000) < 1) status = 500;
      if (rand(1000) === 0) duration += 400 + rand(600);
      for (const [d, h, dr, sd, mode, sev] of EVENT) {
        const start = d * 1440 + h * 60;
        if (t < start || t >= start + dr || (sd >= 0 && sd !== sv)) continue;
        if (mode === "error" && rand(1000) < sev) status = 500;
        if (mode === "slow") duration += 300 + rand(sev);
      }
      if (maintenance) status = 503;
      out.push([t, SERVICE[sv], status, duration, rand(MEMBER), rand(50) === 0 ? "crawler" : "human"]);
    }
  }
  return out;
}

if (import.meta.filename === process.argv[1]) {
  const R = records();
  const count = (f) => R.filter(f).length;
  console.log(`seed ${SEED}, ${MIN} minutes, ${R.length} requests, ${MEMBER} members`);
  console.log(`status 5xx ${count((r) => r[2] >= 500)}, status 4xx ${count((r) => r[2] >= 400 && r[2] < 500)}, ` +
    `duration > 400 ms ${count((r) => r[3] > 400)}, crawler ${count((r) => r[5] === "crawler")}`);
}
```

```
seed 20260731, 43200 minutes, 1188000 requests, 4000 members
status 5xx 2214, status 4xx 9457, duration > 400 ms 3087, crawler 23868
```

#### Three Indicators from the Same Telemetry

The definition of a bad request is shared by all three indicators: a request is bad if its status
code is 400 or above, or its duration exceeds 400 milliseconds. What changes is **what plays the
numerator and what plays the denominator**.

- **Request-based**: good request count divided by total request count.
- **Minute-based**: a minute is good if its bad ratio is under 5%; the indicator is good-minute
  count divided by populated-minute count.
- **Member-based**: a member-day is good if at most 1% of that member's requests that day are
  bad; the indicator is the thirty-day average of the daily good-member ratio.

Four filters are layered on top of this. A filter states what the indicator **does not count**:
crawler requests, requests with a client error, and the scheduled maintenance window are excluded
in turn.

```js
// slo/indicator.mjs — three indicator definitions and four filters from the same telemetry.
// Bad request: status 400 and above, or duration above 400 ms. The three definitions' results
// are compared against the 99.5% objective.
import { records, MIN, MEMBER, MAINTENANCE } from "./telemetry.mjs";
const ALL = records(), OBJECTIVE = 0.995, pct = (x) => `%${(x * 100).toFixed(3)}`;
const bad = (r) => r[2] >= 400 || r[3] > 400;

const requestBased = (rows) => 1 - rows.filter(bad).length / rows.length;

function minuteBased(rows) {                  // one minute: good if the bad ratio is under 5%
  const total = new Int32Array(MIN), badC = new Int32Array(MIN);
  for (const r of rows) { total[r[0]] += 1; if (bad(r)) badC[r[0]] += 1; }
  let good = 0, filled = 0;
  for (let t = 0; t < MIN; t += 1) if (total[t] > 0) { filled += 1; if (badC[t] <= total[t] * 0.05) good += 1; }
  return good / filled;
}

function memberBased(rows) {                  // one member-day: good if the bad ratio is under 1%
  let sum = 0, reqSum = 0, dayCount = 0;
  for (let d = 0; d < 30; d += 1) {
    const total = new Int32Array(MEMBER), badC = new Int32Array(MEMBER);
    for (const r of rows) {
      if (Math.floor(r[0] / 1440) !== d) continue;
      total[r[4]] += 1; if (bad(r)) badC[r[4]] += 1;
    }
    let active = 0, good = 0, n = 0;
    for (let u = 0; u < MEMBER; u += 1) if (total[u] > 0) {
      active += 1; n += total[u]; if (badC[u] <= total[u] * 0.01) good += 1;
    }
    sum += good / active; reqSum += n / active; dayCount += 1;
  }
  return [sum / dayCount, reqSum / dayCount];
}

const FILTER = [
  ["raw", () => true],
  ["crawler excluded", (r) => r[5] !== "crawler"],
  ["+ client error excluded", (r) => r[5] !== "crawler" && (r[2] < 400 || r[2] >= 500)],
  ["+ maintenance excluded", (r) => r[5] !== "crawler" && (r[2] < 400 || r[2] >= 500)
    && (r[0] < MAINTENANCE[0] || r[0] >= MAINTENANCE[0] + MAINTENANCE[1])],
];

console.log(`${"filter".padEnd(24)}${"requests".padStart(9)}${"excluded".padStart(9)}` +
  `${"request-based".padStart(15)}${"minute-based".padStart(16)}${"member-based".padStart(13)}` +
  `${"over objective".padStart(15)}`);
let memberReq = 0;
for (const [name, f] of FILTER) {
  const rows = ALL.filter(f), [mb, n] = memberBased(rows);
  const u = [requestBased(rows), minuteBased(rows), mb];
  memberReq = n;
  console.log(`${name.padEnd(24)}${String(rows.length).padStart(9)}` +
    `${String(ALL.length - rows.length).padStart(9)}` +
    `${u.map((v, i) => pct(v).padStart([15, 16, 13][i])).join("")}` +
    `${`${u.filter((v) => v >= OBJECTIVE).length}/3`.padStart(15)}`);
}
console.log(`objective ${pct(OBJECTIVE)}; same 30 days, same ${ALL.length} records, 12 numbers`);
console.log(`average requests per member per day ${memberReq.toFixed(1)}: ` +
  `the 1% member threshold allows ${Math.floor(memberReq * 0.01)} bad requests`);
```

```
filter                   requests excluded  request-based    minute-based member-based over objective
raw                       1188000        0        %98.760         %94.655      %88.763            0/3
crawler excluded          1164132    23868        %98.761         %93.074      %88.980            0/3
+ client error excluded   1154873    33127        %99.553         %98.123      %96.089            1/3
+ maintenance excluded    1154280    33720        %99.604         %98.259      %96.540            1/3
objective %99.500; same 30 days, same 1188000 records, 12 numbers
average requests per member per day 9.6: the 1% member threshold allows 0 bad requests
```

#### What the Definition Counts

Without a single record changing, the same thirty days produced twelve numbers, spread between
88.763% and 99.604% — a range of 10.8 points. Only two of the twelve numbers clear the 99.5%
objective. The sentence "we hit the objective" cannot be verified until the definition is written
down.

**There is no relationship between how many records are excluded and how much the indicator
moves.** Crawler requests excluded 23,868 records and moved the request-based indicator from
98.760% to 98.761%; client errors excluded 9,259 records and jumped the same indicator by 0.79
points. What decides the outcome is not how many records are excluded, but whether they were bad.
The maintenance window's 593 records added 0.05 points — a small number, but once the distance to
the objective narrows to 0.05 points, whether these records are excluded decides the result.

Excluding crawler requests **lowered** the minute-based indicator (from 94.655% to 93.074%).
Since the excluded requests were not bad, this looks backwards; the reason lies in the
denominator. In a minute with forty requests, two bad requests make exactly 5%, and the minute
counts as good. Once one crawler request is excluded, the denominator becomes 39, the same two
requests make 5.1%, and the minute turns bad. **Excluding good requests distorts a ratio-based
indicator**: a filter raises the indicator when it excludes bad requests, and lowers it when it
excludes good ones.

The member-based indicator is the strictest of the three: where the request-based one says
99.553%, it says 96.089%. The source of the strictness is not the threshold but the denominator.
An average of 9.6 requests falls to each member per day, and a 1% threshold applied to that
denominator allows zero bad requests; in practice, the definition becomes "the member sees no bad
request that day at all". The three definitions measure three separate faces of the same event:
the request-based one measures its **volume**, the minute-based one its **duration**, the
member-based one its **spread**.

#### The Gap Between Objective and Agreement

An objective cannot be measured unless it is written together with its definition. "99.5%" by
itself does not say which of the twelve numbers above is being looked at, and depending on which
number is looked at, the same month can be both hit and missed. The objective's three written
parts — definition, window, number — must all be present at once.

An agreement demands one more thing: **the other party must be able to compute the same number.**
An agreement cannot be written on a member-based indicator, because a member does not see their
own member-days. This is why an agreement uses the coarsest and most verifiable definition:
request-based, crawler and client error excluded, maintenance window excluded. If the loan system
sets the agreement at 99.0% and the objective at 99.5%, the 0.5-point gap between them is the
intervention zone; when the objective is missed, no penalty has come due yet, and there is still
room left in the month to fix it.

The counterpart of this distinction in code is the `FILTER` array. Every exclusion sentence in
the agreement's text is a row in that array: four filter levels, three exclusion rules. If what
the text says and what the array holds drift apart, the committed number and the measured number
diverge — if the maintenance window's 593 records are not excluded in the text, the agreement
looks violated even though the measurement is working correctly.

#### Summary

- An indicator is the measured ratio, an objective is the internal commitment placed on that
  ratio, and an agreement is the external commitment that carries a penalty; an agreement is
  always looser than the objective.
- The same thirty-day log of 1,188,000 records produced twelve numbers from three definitions and
  four filters: between 88.763% and 99.604%, and only two of them clear the 99.5% objective.
- The effect of an exclusion is measured not by how many records it excludes but by whether those
  records were bad: 23,868 crawler records moved the indicator by 0.001 points, 9,259 client
  errors moved it by 0.79 points.
- Excluding good requests lowers the minute-based indicator, because it shrinks the denominator:
  from 94.655% to 93.074%.
- With 9.6 requests falling to each member per day, the member-based indicator turns the 1%
  threshold into zero tolerance; the three definitions measure the same event's volume, duration,
  and spread.
- The agreement's exclusion list turns into code; if the filter array and the agreement text
  drift apart, the measured number and the committed number drift apart with them.

#### Next Step

All three definitions here were built on a single "bad request" measure: status code or duration.
But in the loan system, that is not the only way a request can go bad. The catalog service can
slow down without ever returning an error, the notification service can fill its queue while
running error-free, and the fee service receiving no requests at all can be a failure signal on
its own. All of this is written in the same telemetry, but an indicator that looks at a single
ratio cannot see part of it. The next lesson computes four separate signals from the same records
and counts what each signal **alone fails to catch**.

### Golden Signals

Source: https://academia.sh/en/courses/observability/golden-signals

Computing latency, traffic, errors, and saturation from the loan system's seven-day records: counting the events each signal alone misses across six event classes, measuring how many minutes earlier saturation triggers than the error signal, and showing the silent loss even the four signals together cannot see.

The previous lesson built all three indicator definitions on a single bad-request measure:
status code or duration. In the loan system, that is not the only way a request can go bad. The
catalog service can slow down without ever returning an error, the notification queue can fill up
while running error-free, and the fee service receiving no requests at all can be a failure
signal on its own. All of this is written in the same telemetry, but an indicator that looks at a
single ratio cannot see part of it.

This lesson computes four signals from the same records, runs them across six separate event
classes, and counts what each signal **alone fails to catch**.

#### Four Signals, Four Separate Questions

- **Latency**: how long a request took. Its question is "is it slow".
- **Traffic**: how many requests arrive per unit time. Its question is "what is the load".
- **Errors**: how many requests came back failed. Its question is "is it broken".
- **Saturation**: how much of the system's scarcest resource is full. Its question is "how much
  room is left".

The four are chosen together because the first three describe **what has already happened**, and
the fourth describes **what is about to happen**. This signal is the same concept as saturation
in the Introduction to System Design course.

The first three come directly from the request log. Saturation does not: what the scarce resource
is depends on the application, and it **must be measured explicitly in code**. In the split loan
system, that resource is the notification queue; every time a notification request is written to
the queue, its own record also captures the queue's depth at that moment. Queue depth is a gauge,
not a counter: it rises and falls.

#### The Seven-Day Log and Six Event Classes

```js
// signal/record.mjs — the loan system's 7-day request log and notification queue gauge.
// Record: [minute, service, status, duration_ms, queue_depth, delivery(-1: not a notification)].
export const MIN = 7 * 1440, SEED = 20260731, LIMIT = 500, DRAIN = 25;
export const SERVICE = ["catalog", "membership", "loan", "notification", "fee"];
const BASE = [40, 25, 60, 30, 45];
// SD1 (assumption): 100 requests/minute between 08-22, 30 outside it; day 0 is event-free (base day).
// SD2 (assumption): six event classes [day, hour, duration min, name].
export const EVENT = [
  [1, 10, 60, "error burst"], [2, 14, 90, "silent slowdown"],
  [3, 3, 120, "traffic collapse"], [4, 15, 120, "queue saturation"],
  [5, 12, 30, "traffic spike"], [6, 9, 180, "silent delivery loss"],
];
export const eventWindow = (i) => [EVENT[i][0] * 1440 + EVENT[i][1] * 60, EVENT[i][2]];

export function records() {
  let s = SEED;
  const rand = (n) => Math.floor(((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
  const out = [];
  let queue = 0;
  for (let t = 0; t < MIN; t += 1) {
    const hour = Math.floor(t / 60) % 24;
    const within = (i) => t >= eventWindow(i)[0] && t < eventWindow(i)[0] + EVENT[i][2];
    let n = hour >= 8 && hour < 22 ? 100 : 30;
    if (within(2)) n = Math.round(n * 0.1);
    if (within(4)) n = Math.round(n * 2.5);
    queue = Math.max(0, queue - (within(3) ? 14 : DRAIN));
    for (let i = 0; i < n; i += 1) {
      const sv = rand(5);
      let status = 200, duration = BASE[sv] + rand(60), delivery = -1;
      if (rand(1000) < 1) status = 500;
      if (within(0) && rand(1000) < 80) status = 500;
      if (within(1) && sv === 0) duration += 700 + rand(400);
      if (sv === 3 && status === 200) {
        if (queue >= LIMIT) status = 503;
        else { queue += 1; delivery = rand(1000) < (within(5) ? 970 : 4) ? 0 : 1; }
      }
      out.push([t, SERVICE[sv], status, duration, queue, delivery]);
    }
  }
  return out;
}

if (import.meta.filename === process.argv[1]) {
  const R = records();
  console.log(`seed ${SEED}, ${MIN} minutes, ${R.length} requests, queue limit ${LIMIT}`);
  console.log(`event classes: ${EVENT.map((o) => o[3]).join(", ")}`);
}
```

```
seed 20260731, 10080 minutes, 715260 requests, queue limit 500
event classes: error burst, silent slowdown, traffic collapse, queue saturation, traffic spike, silent delivery loss
```

Six event classes were chosen separately: one moves only the status code, one moves only the
duration, one moves only the request count, one moves only the queue. In the last one,
notification requests return 200, their durations do not change, the queue drains normally — but
the notification sent does not reach the member.

#### Computing the Four

Each signal is turned into a decision with a threshold. The latency and traffic thresholds are
derived from the event-free day 0; the error and saturation thresholds are fixed. For a signal to
count as having caught an event, it must trigger for **three consecutive minutes** within the
window; a single-minute trigger does not count.

```js
// signal/golden.mjs — the four golden signals computed minute by minute from the same
// records, and a count of which event class each signal alone fails to catch. Catch
// criterion: trigger for 3 consecutive minutes within the window.
import { records, MIN, EVENT, eventWindow, LIMIT } from "./record.mjs";
const R = records();
const count = new Int32Array(MIN), errC = new Int32Array(MIN), queueD = new Int32Array(MIN);
const ms = Array.from({ length: MIN }, () => []);
for (const [t, , status, duration, queue] of R) {
  count[t] += 1; ms[t].push(duration); queueD[t] = Math.max(queueD[t], queue);
  if (status >= 500) errC[t] += 1;
}
const p95 = (a) => [...a].sort((x, z) => x - z)[Math.floor(a.length * 0.95)];
const lat = new Int32Array(MIN);
for (let t = 0; t < MIN; t += 1) lat[t] = count[t] ? p95(ms[t]) : 0;

// SD3 (assumption): the latency and traffic thresholds are derived from day 0, the error and
// saturation thresholds are fixed.
const P95 = p95(R.filter((r) => r[0] < 1440).map((r) => r[3])), hourlyBase = new Float64Array(24);
for (let t = 0; t < 1440; t += 1) hourlyBase[Math.floor(t / 60)] += count[t] / 60;
const CHECK = {
  latency: (t) => lat[t] > 2 * P95,
  traffic: (t) => count[t] < 0.5 * hourlyBase[Math.floor(t / 60) % 24] || count[t] > 2 * hourlyBase[Math.floor(t / 60) % 24],
  error: (t) => count[t] > 0 && errC[t] / count[t] > 0.01,
  saturation: (t) => queueD[t] / LIMIT > 0.85,
};
const NAMES = Object.keys(CHECK);
const firstThree = (name, start, dur) => {
  for (let t = start, run = 0; t < start + dur; t += 1)
    if ((run = CHECK[name](t) ? run + 1 : 0) === 3) return t - start - 2;
  return -1;
};
console.log(`base p95 ${P95} ms, latency threshold ${2 * P95} ms, error threshold %1, ` +
  `saturation threshold ${0.85 * LIMIT}/${LIMIT}\n`);
console.log(`${"event class".padEnd(22)}${"duration".padStart(9)}` +
  `${NAMES.map((a) => a.padStart(14)).join("")}${"caught".padStart(11)}`);
const caught = NAMES.map(() => new Set());
EVENT.forEach(([, , dr, name], i) => {
  const [start] = eventWindow(i);
  const s = NAMES.map((a) => {
    let n = 0;
    for (let t = start; t < start + dr; t += 1) if (CHECK[a](t)) n += 1;
    return [n, firstThree(a, start, dr)];
  });
  s.forEach(([, k], j) => { if (k >= 0) caught[j].add(i); });
  console.log(`${name.padEnd(22)}${(dr + " min").padStart(9)}` +
    `${s.map(([n, k]) => (k >= 0 ? `${n} min @ ${k}` : "-").padStart(14)).join("")}` +
    `${String(s.filter(([, k]) => k >= 0).length).padStart(11)}`);
});
const duringAnyEvent = (t) => EVENT.some((o, i) => t >= eventWindow(i)[0] && t < eventWindow(i)[0] + o[2]);
console.log(`\n${"signal".padEnd(12)}${"caught".padStart(11)}${"off-event trigger".padStart(18)}` +
  `  missed event classes`);
NAMES.forEach((a, j) => {
  let d = 0;
  for (let t = 0; t < MIN; t += 1) if (CHECK[a](t) && !duringAnyEvent(t)) d += 1;
  console.log(`${a.padEnd(12)}${`${caught[j].size}/${EVENT.length}`.padStart(11)}${(d + " min").padStart(18)}` +
    `  ${EVENT.filter((_, i) => !caught[j].has(i)).map((o) => o[3]).join(", ")}`);
});
const all = new Set(caught.flatMap((s) => [...s]));
console.log(`\nall four together: ${all.size}/${EVENT.length} event classes; ` +
  `missed: ${EVENT.filter((_, i) => !all.has(i)).map((o) => o[3]).join(", ")}`);
const deliveryRate = (a, b) => {
  const rows = R.filter((r) => r[5] >= 0 && r[0] >= a && r[0] < b);
  return `%${((100 * rows.filter((r) => r[5] === 1).length) / rows.length).toFixed(1)}`;
};
const [b5, d5] = [eventWindow(5)[0], EVENT[5][2]];
console.log(`delivery rate: before event ${deliveryRate(b5 - d5, b5)}, during event ${deliveryRate(b5, b5 + d5)}`);
let low = 0, off = 0, firstQueueTrig = -1;
for (let t = 0; t < MIN; t += 1) if (!duringAnyEvent(t)) {
  if (CHECK.error(t)) { off += 1; if (count[t] <= 40) low += 1; }
  if (CHECK.saturation(t) && firstQueueTrig < 0) firstQueueTrig = t;
}
const [eb, ed] = [eventWindow(3)[0], EVENT[3][2]];
console.log(`of the error signal's ${off} off-event triggers, ${low} fell in minutes under 40 ` +
  `requests/min; saturation's first off-event trigger was minute ${firstQueueTrig}, the event ended ` +
  `at minute ${eb + ed}`);
console.log(`during queue saturation: saturation caught it at minute ${firstThree("saturation", eb, ed)}, ` +
  `error caught it at minute ${firstThree("error", eb, ed)}`);
```

```
base p95 105 ms, latency threshold 210 ms, error threshold %1, saturation threshold 425/500

event class            duration       latency       traffic         error    saturation     caught
error burst              60 min             -             -    60 min @ 0             -          1
silent slowdown          90 min    90 min @ 0             -             -             -          1
traffic collapse        120 min             -   120 min @ 0             -             -          1
queue saturation        120 min             -             -   43 min @ 72   59 min @ 61          2
traffic spike            30 min             -    30 min @ 0   12 min @ 18   15 min @ 15          3
silent delivery loss    180 min             -             -             -             -          0

signal           caught off-event trigger  missed event classes
latency             1/6             0 min  error burst, traffic collapse, queue saturation, traffic spike, silent delivery loss
traffic             2/6             0 min  error burst, silent slowdown, queue saturation, silent delivery loss
error               3/6           162 min  silent slowdown, traffic collapse, silent delivery loss
saturation          2/6            23 min  error burst, silent slowdown, traffic collapse, silent delivery loss

all four together: 5/6 event classes; missed: silent delivery loss
delivery rate: before event %99.7, during event %3.0
of the error signal's 162 off-event triggers, 136 fell in minutes under 40 requests/min; saturation's first off-event trigger was minute 6780, the event ended at minute 6780
during queue saturation: saturation caught it at minute 61, error caught it at minute 72
```

#### Each Signal's Blind Spot

No signal sees more than four of the six events. Latency catches one in six, traffic and
saturation catch two in six, error catches three in six. These numbers have nothing to do with
the quality of the signals: each one misses a different event class because it answers **a
different question**.

The most instructive row is the traffic collapse. For two hours, the request count drops to a
tenth of normal; the error rate is zero throughout, latency sits at its base value, the queue is
empty. An indicator that looks only at the error rate sees **one hundred percent success** in
those two hours, because every one of the few requests that did arrive came back successful. This
is exactly the picture a failed upstream producer or a broken route produces: the ratio is clean,
because the denominator has collapsed.

The queue saturation row shows the signal's **time** value. The notification consumer slows down,
and the queue grows by six notifications a minute. Saturation clears its three-minute threshold
at minute 61; the error signal also triggers once the queue hits its limit and notification
requests start getting rejected, but that happens at minute 72. The **11 minutes** in between is
the window in which the incident can be addressed before it reaches the member. If saturation is
not measured, those eleven minutes are lost and the incident is first seen through dropped
requests.

The off-event triggers show two separate things. Of the error signal's 162 triggers, 136 fall in
minutes receiving fewer than 40 requests: with a denominator of 30 in night traffic, a single 5xx
makes 3.3% and clears the 1% threshold. A ratio-based signal gets noisier as its denominator
shrinks — the previous lesson's denominator discussion comes back here as alert noise. Saturation's
23 triggers, on the other hand, are not wrong: the first one starts at exactly minute 6780, right
where the queue saturation event ends, and measures how long the queue takes to drain. **A
signal's window does not have to match the event's window.**

#### What the Four Together Cannot See

The four signals together cover five of the six event classes. The sixth one that slips through
is silent delivery loss: notification requests return 200, their durations do not change, the
queue drains normally, no threshold moves. Yet the delivery field inside those same records says
99.7% before the event and 3.0% during it.

The distinction here is a lasting one: the four golden signals measure **transport**, not
**correctness**. A request being accepted, answered quickly, and not consuming resources does not
mean the work got done. In the loan system, the work being done means "the overdue notification
reached the member", and only a business-level counter can say that. The four signals are placed
on every service the same way; a fifth counter has to be thought through separately for each
service, and for that reason it is often never written at all.

#### Summary

- The four golden signals ask four separate questions: latency asks "is it slow", traffic asks
  "what is the load", errors ask "is it broken", saturation asks "how much room is left";
  saturation is the same concept as in the Introduction to System Design course.
- The first three come directly from the request log; saturation does not, and the scarce
  resource must be measured in code — here, every notification request writes the queue depth
  into its own record.
- Across the six event classes, no signal alone caught more than three: latency 1/6, traffic 2/6,
  saturation 2/6, error 3/6.
- In the traffic collapse, the error rate is zero and the indicator looks flawless; a ratio-based
  signal goes blind when its denominator collapses and gets noisy when its denominator shrinks
  (136 of 162 off-event triggers fall in minutes under 40 requests/min).
- Saturation triggered at minute 61 while the queue was filling; error only triggered at minute
  72, once requests started getting rejected — an 11-minute early warning.
- The four together cover five of the six classes; the one that slips through is silent delivery
  loss (the delivery rate dropped from 99.7% to 3.0%), because the golden signals measure
  transport, not correctness.

#### Next Step

The four signals are now computed minute by minute and can be compared against an objective. What
happens once it drops below that objective is still not written down. The loan system might get
two releases a week, or ten; every release leaves a measurable degradation in the records, and
every minute spent below the objective is paid for from somewhere. The next lesson computes that
payment from telemetry: it derives a budget from the same request records, counts how fast two
different release rates consume that budget, and shows what changes in the code once the budget
runs out.

### Error Budget

Source: https://academia.sh/en/courses/observability/error-budget

Turning the objective's complement into a spendable number: computing the budget from the loan system's real request records as a count of bad requests, counting how fast four release rates consume that budget, and measuring how many releases the release gate blocks and how much it shrinks the overage once the budget runs out.

The previous lesson computed four signals at minute resolution; it is now possible to tell when
something drops below an objective. What **happens** once it drops below is still not written
down. The loan system might get two releases a week, or two a day; every release leaves a
measurable trace in the records, and every minute spent below the objective is paid for from
somewhere. This lesson keeps the ledger for that payment.

#### The Budget Comes from Telemetry

If the objective is 99.9%, one in a thousand requests is allowed to be bad. Once that ratio is
converted into a **count**, it becomes a spendable quantity: whatever number of requests arrived
over thirty days, one-thousandth of that is the month's **error budget**. A budget is a number
whose remainder can be read, whose daily consumption can be plotted, and whose depletion day is
known.

The Introduction to System Design course set up the same quantity as monthly downtime minutes;
the conversion between percentage and minutes was done there and is not repeated here. The
difference here is where the denominator comes from: when the budget is computed from the
request log, **it moves with traffic.** If traffic doubles, the allowed number of bad requests
doubles too, because the same ratio is applied to a larger base. A budget written in minutes has
no such give.

#### How Release Rate Eats the Budget

The run below steps through thirty days minute by minute. Releases happen at fixed intervals,
each release leaves a degradation behind with some probability, and the degradation's severity
and duration depend on the **release's size**. The measure of release size is the number of days
between two releases: the longer the interval, the more change a release carries.

```js
// budget/run.mjs — the error budget derived from telemetry: if the objective is 99.9%, the
// allowed bad-request count is 1 per thousand of total requests. Rates are in per-ten-thousand.
export const MIN = 30 * 1440, SEED = 20260731, OBJECTIVE = 0.999, BASE = 2;
const makeRand = (t) => (n) => Math.floor(((t = (t * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
const hour = (t) => Math.floor(t / 60) % 24;
export const reqPerMin = (t) => (hour(t) >= 8 && hour(t) < 22 ? 100 : 30);
export const TOTAL = [...Array(MIN)].reduce((a, _, t) => a + reqPerMin(t), 0);
export const BUDGET = Math.round(TOTAL * (1 - OBJECTIVE));
export const gate = (remaining) => remaining > 0;        // release gate: no release once the budget is out

// SD1 (assumption): as the interval between releases grows, the probability, severity, and
// recovery time of a degradation grow with it: probability 15% + 15%*days, severity 200 + 400*days
// per ten-thousand, recovery 10 + 20*days min.
export function run(intervalDays, gateOpen) {
  const rRel = makeRand(SEED), rReq = makeRand(SEED), period = Math.round(intervalDays * 1440);
  // intervalDays 0: no release is ever made, what is measured is the base error eating the budget alone
  const p = Math.min(900, 150 + 150 * intervalDays), sev = Math.min(2500, 200 + 400 * intervalDays);
  const dur = Math.min(120, Math.round(10 + 20 * intervalDays));
  let remaining = BUDGET, releases = 0, degraded = 0, blocked = 0, depletedDay = 0, degradedUntil = -1;
  const daily = [];
  for (let t = 0; t < MIN; t += 1) {
    if (period > 0 && t >= 600 && (t - 600) % period === 0) {
      if (gateOpen && !gate(remaining)) blocked += 1;
      else { releases += 1; if (rRel(1000) < p) { degraded += 1; degradedUntil = t + dur; } }
    }
    const rate = BASE + (t < degradedUntil ? sev : 0);
    for (let i = 0, n = reqPerMin(t); i < n; i += 1) if (rReq(10000) < rate) remaining -= 1;
    if (remaining <= 0 && depletedDay === 0) depletedDay = Math.floor(t / 1440) + 1;
    if ((t + 1) % 1440 === 0) daily.push(remaining / BUDGET);
  }
  return { releases, degraded, blocked, depletedDay, spent: BUDGET - remaining, daily };
}

export const RATE = [["no releases", 0], ["2/day", 0.5], ["1/day", 1],
  ["every 3 days", 3], ["every 5 days", 5]];

if (import.meta.filename === process.argv[1]) {
  console.log(`${TOTAL} requests in 30 days, objective %99.9 -> budget ${BUDGET} bad requests; base error ` +
    `at ${BASE} per ten-thousand alone eats ${Math.round((TOTAL * BASE) / 10000)} requests`);
  console.log(`\n${"release interval".padEnd(18)}${"releases".padStart(9)}${"degraded".padStart(9)}` +
    `${"bad minutes".padStart(14)}${"spent".padStart(10)}${"of budget".padStart(10)}` +
    `${"ran out".padStart(11)}`);
  for (const [name, g] of RATE) {
    const r = run(g, false), min = r.degraded * Math.min(120, Math.round(10 + 20 * g));
    console.log(`${name.padEnd(18)}${String(r.releases).padStart(9)}${String(r.degraded).padStart(9)}` +
      `${(min + " min").padStart(14)}${String(r.spent).padStart(10)}` +
      `${`%${(100 * r.spent / BUDGET).toFixed(1)}`.padStart(10)}` +
      `${(r.depletedDay ? `${r.depletedDay}.` : "-").padStart(11)}`);
  }
}
```

```
3060000 requests in 30 days, objective %99.9 -> budget 3060 bad requests; base error at 2 per ten-thousand alone eats 612 requests

release interval   releases degraded   bad minutes     spent of budget    ran out
no releases               0        0         0 min       631     %20.6          -
2/day                    60       13       260 min      1486     %48.6          -
1/day                    30        6       180 min      1717     %56.1          -
every 3 days             10        5       350 min      5523    %180.5        16.
every 5 days              6        4       440 min     10248    %334.9         6.
```

#### Not a Count, a Magnitude

The first row says that part of the budget has nothing to do with change at all: even with no
releases at all, the base error alone eats 20.6% of the budget over thirty days. Even a system
left untouched for the whole month has a budget, and not all of it can be set aside for change.

The real contrast is between the second and fourth rows. The team releasing twice a day shipped
**60 releases** for the month and spent 48.6% of the budget; the team releasing once every three
days shipped **10 releases** and spent 180.5% of the budget, and the budget ran out on day 16.
Six times as many releases, roughly a third of the consumption.

The difference shows up in the cost per release. Subtracting the base consumption (631), the
twice-a-day rate has 855 bad requests attributable to releases: 14 per release. The every-three-
days rate has 4,892 bad requests attributable to releases: 489 per release. Thirty-five times as
much. The bad-minutes column shows why: the fast rate had 13 degradations but they lasted 260
minutes total; the slow rate had 5 degradations and they lasted 350 minutes. Degradation
**count** is higher on the fast team, degradation **duration and severity** are higher on the
slow team, and it is the latter that eats the budget.

This result is baked into SD1 and should be argued there: the model assumes that a degradation's
probability, severity, and recovery time all grow with release size. If the assumption changes,
the table changes with it. The way to turn the assumption into a measurement is written in the
telemetry: every release is timestamped, the degradation window following it is extracted from
the records, and the three parameters — probability, severity, duration — are plotted against
release size. Once those three numbers are measured, the release-rate debate stops being a matter
of preference.

#### Once the Budget Runs Out

A budget is a permission slip: while the remainder is positive, change is free to happen; once it
is spent, priority shifts back to stability. The counterpart of this in code is a one-line gate —
when `gate(remaining)` returns false, no release happens. The run below executes the same four
rates twice, once with the gate closed and once with it open.

```js
// budget/gate.mjs — what changes when the release gate is open: once the budget runs out,
// releases are blocked. The same four release intervals are run twice, gate closed and open.
import { run, RATE, BUDGET } from "./run.mjs";
const pct = (x) => `%${(100 * x).toFixed(1)}`;

console.log(`budget ${BUDGET} bad requests\n`);
console.log(`${"release interval".padEnd(18)}${"gate".padStart(8)}${"releases".padStart(9)}` +
  `${"blocked".padStart(12)}${"spent".padStart(10)}${"of budget".padStart(10)}` +
  `${"overage".padStart(9)}`);
for (const [name, g] of RATE)
  for (const open of [false, true]) {
    const r = run(g, open);
    console.log(`${name.padEnd(18)}${(open ? "open" : "closed").padStart(8)}` +
      `${String(r.releases).padStart(9)}${String(r.blocked).padStart(12)}` +
      `${String(r.spent).padStart(10)}${pct(r.spent / BUDGET).padStart(10)}` +
      `${(r.spent > BUDGET ? String(r.spent - BUDGET) : "-").padStart(9)}`);
  }

const DAY = [3, 6, 9, 12, 15, 18, 21, 24, 27, 30];
console.log(`\nremaining budget (gate closed)\n${"day".padEnd(18)}${DAY.map((g) => String(g).padStart(8)).join("")}`);
for (const [name, g] of RATE) {
  const r = run(g, false);
  console.log(`${name.padEnd(18)}${DAY.map((d) => pct(r.daily[d - 1]).padStart(8)).join("")}`);
}
```

```
budget 3060 bad requests

release interval      gate releases     blocked     spent of budget  overage
no releases         closed        0           0       631     %20.6        -
no releases           open        0           0       631     %20.6        -
2/day               closed       60           0      1486     %48.6        -
2/day                 open       60           0      1486     %48.6        -
1/day               closed       30           0      1717     %56.1        -
1/day                 open       30           0      1717     %56.1        -
every 3 days        closed       10           0      5523    %180.5     2463
every 3 days          open        6           4      3531    %115.4      471
every 5 days        closed        6           0     10248    %334.9     7188
every 5 days          open        2           4      5416    %177.0     2356

remaining budget (gate closed)
day                      3       6       9      12      15      18      21      24      27      30
no releases          %98.0   %96.2   %93.5   %91.2   %89.2   %87.2   %85.3   %82.9   %81.2   %79.4
2/day                %98.0   %92.0   %86.7   %80.7   %76.2   %68.8   %63.9   %59.2   %56.8   %51.4
1/day                %98.0   %96.2   %93.5   %80.4   %78.3   %70.2   %62.1   %53.8   %45.7   %43.9
every 3 days         %67.0   %65.2   %62.5   %27.8   %25.8   %-7.6   %-9.5  %-43.9  %-45.7  %-80.5
every 5 days         %19.9  %-60.2  %-62.8  %-65.2  %-67.2 %-148.2 %-150.1 %-152.5 %-233.1 %-234.9
```

The gate says three things at once. **It never stops the fast team:** in the twice-a-day and
once-a-day rows, the number of blocked releases is zero, because the budget never runs out. The
gate is not a release-rate limit, it is a budget limit; a team releasing fast never sees it.

**On the slow team, four releases are blocked** and the overage drops from 2,463 to 471: 81% of
the consumption beyond the budget is prevented. Six of the ten releases go out, four are left for
the end of the month.

**The gate is late.** Even with the gate open, there is still an overage of 471 requests, because
the gate engages after the budget runs out, not before. The remaining-budget table shows this day
by day: for the team releasing every three days, the remainder had dropped to 27.8% by day 12 and
crossed into negative by day 18. Six days pass between the day the remainder reads 27.8% and the
day the overage begins, and two more releases went out in those days. This is why the gate alone
is not enough: a gate that closes **after** a threshold is crossed does not prevent the overage
itself; what is needed is something that gives warning **before** the budget runs out.

#### Summary

- An error budget is the objective's complement applied to the request log: 3,060,000 requests
  over thirty days and a 99.9% objective give a budget of 3,060 bad requests, and this budget
  flexes with traffic.
- Part of the budget has nothing to do with change: with no releases at all, the base error alone
  ate 20.6% of the budget.
- Six times as many releases does not mean six times the consumption: 60 releases spent 48.6% of
  the budget, 10 releases spent 180.5%, and in the latter case the budget ran out on day 16.
- Cost per release: 14 bad requests versus 489 bad requests; degradation count is higher on the
  fast team (13 to 5), degradation duration is higher on the slow team (260 minutes to 350), and
  it is duration that eats the budget.
- The release gate is a one-line condition and never stops the fast team; on the slow team, it
  blocked four releases and brought the overage down from 2,463 to 471.
- Because the gate closes after the budget runs out, it cannot prevent the entire overage; a
  mechanism is needed that gives warning before the budget is depleted.

#### Next Step

Every row in the remaining-budget table can be read at the end of the month, yet the decision
needed to be made on day 12. If no one is looking at the table that day, the number has no value
— someone has to be paged. Which rule the page comes from is still an open question: does the
rule watch the notification queue's depth, the loan service's memory usage, or the error rate the
member sees directly? That choice determines how many alerts fire, how many times the same rule
fires for the same incident, and how many incidents never fire an alert at all. The next lesson
runs two separate rule sets across the same incidents and compares them on these three numbers.

### Alert Design

Source: https://academia.sh/en/courses/observability/alert-design

Showing that an alert rule's source determines the outcome: running twelve cause-based rules and four symptom-based rules across the same seven days, and counting the alerts they produce, how many rules fire together for the same event, and how many real events produce no alert at all.

In the previous lesson's remaining-budget table, the decision needed to be made on day 12. If no
one is looking at the table that day, the number has no value; someone has to be paged. Which
rule the page comes from is still an open question. Does the rule watch the notification queue's
depth, the loan service's memory usage, or the error rate the member sees directly?

This lesson does not choose thresholds. The threshold-and-window search was done in the
Performance Anti-Patterns and Monitoring course; the question here is different: the rule's
**source**. Two separate rule sets are run across the same seven days, and three numbers are
compared — alerts produced, rules firing together for the same event, events that produce no
alert at all.

#### The Rule's Source

A **cause-based rule** reads a component's internal metric: memory usage, disk occupancy, cache
hit rate, connection pool utilization, garbage collection duration. These metrics come from
inside the processes, and their count grows with the number of services.

A **symptom-based rule** reads what the member sees: error rate, latency, traffic deviation,
notification delivery rate. These metrics look the same from outside the system, and their count
is independent of the number of services.

The distinction is not arbitrary: an alert pages a person, and the person paged has to have work
to do. The loan service's memory sitting at 91% does not, by itself, define any work; 4% of loan
requests coming back failed does. This is what the distinction between a dashboard and an alert
answering different questions looks like once it lands in how a rule is written.

#### Sixteen Rules and Six Events

The generator below writes seven days as per-minute samples. Twelve component metrics and four
user metrics are generated at the same minutes; each metric's base, noise, and threshold sit on
their own row. Six events move some of these metrics toward a target value.

```js
// alert/metric.mjs — a 7-day per-minute metric sample. Twelve component metrics (cause) and
// four user metrics (symptom) are generated at the same minutes. Rule format: [name, base,
// noise, direction, threshold]; direction "over" means above the threshold is bad, "under"
// means below it is bad.
export const MIN = 7 * 1440, SEED = 20260731;
export const CAUSE = [
  ["catalog.cache_hit_rate", 92, 2, "under", 70], ["catalog.cpu", 38, 6, "over", 85],
  ["loan.memory", 58, 3, "over", 90], ["loan.gc_ms", 25, 6, "over", 120],
  ["loan.restarts", 0, 0, "over", 0], ["membership.pool_usage", 44, 6, "over", 90],
  ["membership.dependency_ms", 40, 8, "over", 300], ["notification.queue", 12, 8, "over", 400],
  ["notification.active_workers", 8, 1, "under", 3], ["fee.disk", 70, 1, "over", 95],
  ["fee.file_handles", 900, 60, "over", 1800], ["general.retries", 3, 2, "over", 30],
];
export const SYMPTOM = [
  ["error_rate", 0.2, 0.15, "over", 1], ["p95_latency_ms", 180, 25, "over", 400],
  ["traffic_deviation", 0, 6, "over", 45], ["delivery_rate", 99.5, 0.3, "under", 97],
];
// SD1 (assumption): six events [day, hour, duration min, name, [[metric, value during event], ...]]
export const EVENT = [
  [1, 10, 90, "cache collapse", [["catalog.cache_hit_rate", 35], ["catalog.cpu", 91],
    ["p95_latency_ms", 520]]],
  [2, 8, 240, "memory leak", [["loan.memory", 96], ["loan.gc_ms", 190],
    ["loan.restarts", 2]]],
  [3, 0, 300, "disk fill-up", [["fee.disk", 98], ["fee.file_handles", 2100]]],
  [4, 14, 120, "dependency slowdown", [["membership.dependency_ms", 680],
    ["membership.pool_usage", 96], ["general.retries", 48], ["p95_latency_ms", 690],
    ["error_rate", 1.8]]],
  [5, 9, 180, "silent delivery loss", [["delivery_rate", 3]]],
  [6, 11, 60, "post-release error increase", [["error_rate", 4.6]]],
];
export const eventWindow = (i) => [EVENT[i][0] * 1440 + EVENT[i][1] * 60, EVENT[i][2]];

// SD2 (assumption): a component metric is read from a single process, so it spikes for
// 3-7 minutes every 1500 minutes; a user metric is averaged over thousands of requests,
// so it spikes every 6000 minutes.
export function series() {
  let s = SEED;
  const rand = (n) => Math.floor(((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
  const out = {};
  for (const [name, base, noise, dir, threshold] of [...CAUSE, ...SYMPTOM]) {
    const isCause = CAUSE.some((k) => k[0] === name);
    const v = new Float64Array(MIN);
    let spike = 0;
    for (let t = 0; t < MIN; t += 1) {
      let val = base + (noise ? (rand(2001) - 1000) / 1000 * noise : 0);
      if (spike > 0) { spike -= 1; val = dir === "over" ? threshold * 1.15 + 1 : threshold * 0.85; }
      else if (rand(isCause ? 1500 : 6000) === 0) spike = 3 + rand(5);
      for (let i = 0; i < EVENT.length; i += 1) {
        const [start, dur] = eventWindow(i), target = EVENT[i][4].find((h) => h[0] === name);
        if (!target || t < start || t >= start + dur) continue;
        const progress = Math.min(1, (t - start + 1) / Math.max(1, dur * 0.2));
        val = base + (target[1] - base) * progress;
      }
      v[t] = val;
    }
    out[name] = v;
  }
  return out;
}

if (import.meta.filename === process.argv[1]) {
  const S = series();
  console.log(`seed ${SEED}, ${MIN} minutes, ${CAUSE.length} component metrics, ` +
    `${SYMPTOM.length} user metrics, ${EVENT.length} events`);
  const [b, dur] = eventWindow(0);
  console.log(`sample — cache collapse window (minute ${b}, ${dur} min): ` +
    `cache hit ${S["catalog.cache_hit_rate"][b + 30].toFixed(1)}, ` +
    `cpu ${S["catalog.cpu"][b + 30].toFixed(1)}, ` +
    `p95 ${S["p95_latency_ms"][b + 30].toFixed(0)} ms, ` +
    `error rate ${S["error_rate"][b + 30].toFixed(2)}`);
}
```

```
seed 20260731, 10080 minutes, 12 component metrics, 4 user metrics, 6 events
sample — cache collapse window (minute 2040, 90 min): cache hit 35.0, cpu 91.0, p95 520 ms, error rate 0.07
```

#### Running Two Rule Sets Through the Same Week

A rule fires an alert once its condition holds for three consecutive minutes, and does not repeat
until the condition clears. The two sets run independently over the same metrics.

```js
// alert/rule.mjs — the same seven days are run through two rule sets: twelve cause-based
// rules and four symptom-based rules. A rule fires an alert once its condition holds for 3
// consecutive minutes, and does not fire again until the condition clears.
import { series, CAUSE, SYMPTOM, EVENT, MIN, eventWindow } from "./metric.mjs";
const S = series();
const crosses = (k, v) => (k[3] === "over" ? v > k[4] : v < k[4]);

function alerts(set) {
  const m = new Map();
  for (const k of set) {
    const v = S[k[0]], mins = [];
    let run = 0, active = false;
    for (let t = 0; t < MIN; t += 1) {
      if (crosses(k, v[t])) { run += 1; if (run >= 3 && !active) { active = true; mins.push(t - 2); } }
      else { run = 0; active = false; }
    }
    m.set(k[0], mins);
  }
  return m;
}
const SET = [["cause-based", CAUSE, alerts(CAUSE)], ["symptom-based", SYMPTOM, alerts(SYMPTOM)]];

console.log(`${"event".padEnd(29)}${"duration".padStart(9)}` +
  `${SET.map(([name]) => `${name} rules`.padStart(24) + "first alert".padStart(13)).join("")}`);
const caught = SET.map(() => new Set());
EVENT.forEach(([, , dr, name], i) => {
  const [start] = eventWindow(i);
  const s = SET.map(([, , m], j) => {
    const c = [...m.values()].map((mins) => mins.filter((t) => t >= start && t < start + dr));
    const rules = c.filter((x) => x.length > 0).length;
    const first = Math.min(...c.flat().map((t) => t - start), Infinity);
    if (rules > 0) caught[j].add(i);
    return [rules, first];
  });
  console.log(`${name.padEnd(29)}${(dr + " min").padStart(9)}` +
    `${s.map(([rules, first]) => String(rules).padStart(24) +
      (first === Infinity ? "-" : `${first} min`).padStart(13)).join("")}`);
});

const duringAnyEvent = (t) => EVENT.some((o, i) => t >= eventWindow(i)[0] && t < eventWindow(i)[0] + o[2]);
console.log(`\n${"set".padEnd(16)}${"rules".padStart(7)}${"alerts".padStart(7)}` +
  `${"off-event".padStart(11)}${"caught".padStart(11)}${"per event".padStart(13)}  missed events`);
SET.forEach(([name, set, m], j) => {
  const all = [...m.values()].flat();
  const off = all.filter((t) => !duringAnyEvent(t)).length;
  const on = all.length - off;
  console.log(`${name.padEnd(16)}${String(set.length).padStart(7)}${String(all.length).padStart(7)}` +
    `${String(off).padStart(11)}${`${caught[j].size}/${EVENT.length}`.padStart(11)}` +
    `${(on / caught[j].size).toFixed(1).padStart(13)}  ` +
    `${EVENT.filter((_, i) => !caught[j].has(i)).map((o) => o[3]).join(", ") || "-"}`);
});

const together = new Set([...caught[0], ...caught[1]]);
console.log(`\nboth together: ${CAUSE.length + SYMPTOM.length} rules, ` +
  `${together.size}/${EVENT.length} events`);
const noisiest = EVENT.map((o, i) => {
  const [start] = eventWindow(i);
  return [o[3], [...SET[0][2].values()].filter((mins) =>
    mins.some((t) => t >= start && t < start + o[2])).length];
}).sort((a, b) => b[1] - a[1])[0];
console.log(`noisiest event: ${noisiest[0]} — ${noisiest[1]} cause-based alerts at once in one event`);
const busiest = [...SET[0][2]].sort((a, b) => b[1].length - a[1].length)[0];
console.log(`busiest single rule: ${busiest[0]} — ${busiest[1].length} alerts in 7 days`);
```

```
event                         duration       cause-based rules  first alert     symptom-based rules  first alert
cache collapse                  90 min                       2        6 min                       1       11 min
memory leak                    240 min                       5        0 min                       0            -
disk fill-up                   300 min                       3       45 min                       0            -
dependency slowdown            120 min                       3        9 min                       2       10 min
silent delivery loss           180 min                       0            -                       1        0 min
post-release error increase     60 min                       0            -                       1        2 min

set               rules alerts  off-event     caught    per event  missed events
cause-based          12     81         67        4/6          3.5  silent delivery loss, post-release error increase
symptom-based         4      9          4        4/6          1.3  memory leak, disk fill-up

both together: 16 rules, 6/6 events
noisiest event: memory leak — 5 cause-based alerts at once in one event
busiest single rule: notification.queue — 10 alerts in 7 days
```

#### Three Numbers

**Alerts produced.** The twelve cause-based rules produced 81 alerts over seven days, the four
symptom-based rules produced 9. Three times the rules, nine times the alerts. 67 of those 81 do
not line up with any event: 83% of the cause-based alerts turn out empty. In the symptom-based
set, the same ratio is 4 of 9. The source of the difference is written in SD2: a component metric
is read from a single process and produces short spikes, a user metric is an average of thousands
of requests and stays smooth. One single rule — the notification queue — produced 10 alerts over
seven days.

**Rules firing together for the same event.** The cause-based set averages 3.5 alerts per event,
the symptom-based set 1.3. During the memory leak, five rules fired at once: three were the event
itself (memory, garbage collection, restarts), two were unrelated spikes that happened to land
inside that four-hour window. When the person paged opens five alerts, they cannot tell which one
is the root cause; the count of alerts carries no information, because all of them came from the
same event.

**Events that produce no alert at all.** Both sets caught four of the six events, but **missed a
different two.** The cause-based rules could not see the silent delivery loss or the post-release
error increase: in both, the component metrics were normal and what was broken was the work
itself. The symptom-based rules could not see the memory leak or the disk fill-up: in both, the
member had not noticed anything yet, because the resource had not run out yet.

#### The Rule That Wakes You, the Metric That Diagnoses

The timing column separates the two sets' roles. In the cache collapse, the cause-based rule
fired at minute 6, the symptom-based rule at minute 11; in the dependency slowdown, the gap
narrows to one minute. Cause-based metrics fire a little earlier, but the cost of those five
minutes is 67 empty alerts.

The pattern that emerges is this: **the twelve component metrics are not deleted, they stop being
alert rules.** The rules that wake someone up are written from symptoms — four rules, the four
metrics the member sees. The component metrics stay on the dashboard, and the person paged looks
there after receiving the alert. An alert says "something must be done"; a dashboard says "where
to look". The union of the two sets covers all six of the six events, but it does that not with
16 alert rules, but with 4 alert rules and 12 dashboard panels.

The two events the symptom-based set misses are this pattern's real gap, and changing a threshold
does not close it. With the disk at 98% full, the member still sees nothing; the symptom only
appears once the disk actually fills up, and by then reversing course is expensive. For resources
that run out, the rule is written not on a threshold but on **time to exhaustion**: the sentence
"at this fill rate, the disk fills up within four hours" is a symptom, because it defines work for
a person to do. The same sentence can be written for the memory leak; both are read from a
cause-based metric but answer a symptom-based question.

#### Summary

- An alert rule's source is one of two kinds: a component's internal metric (cause) or a metric
  the member sees (symptom); the test for the distinction is whether the person paged has work to
  do.
- The twelve cause-based rules produced 81 alerts over seven days and 67 lined up with no event;
  the four symptom-based rules produced 9 alerts and 4 turned out empty.
- The cause-based set averages 3.5 alerts per event: during the memory leak, five rules fired at
  once, three were the event itself, two were unrelated spikes.
- Both sets caught four of six events but missed a different two: the cause-based rules missed
  events where the work itself was broken, the symptom-based rules missed exhaustions that had
  not yet reached the member.
- The cause-based rules fired five minutes earlier in the cache collapse; the cost of those five
  minutes is 67 empty alerts.
- The fix is not to delete the component metrics but to stop them from being alert rules: four
  alert rules and twelve dashboard panels cover all six of the six events.

#### Next Step

Every rule set up in this lesson pages a person, and the person responds within minutes. Some
decisions are never put to a person at all. Whether one of the loan system's processes is alive
and whether it should keep receiving traffic is decided by a machine, within seconds and
thousands of times a day. That decision looks at two separate questions — is the process dead,
and can the process take work right now — and when the two questions are wired to a single
endpoint, the answer given is not just wrong, it builds a self-feeding loop. The next lesson
separates the two endpoints in code and measures the cost of not separating them in restart count
and dropped request count.

### Health Endpoints

Source: https://academia.sh/en/courses/observability/health-endpoints

Keeping liveness and readiness separate in code: the question each endpoint answers, the restart loop produced when a dependency check lands in the liveness endpoint, and running the same scenario in two modes to count the dropped requests and recovery time each one produces.

Every rule set up in the previous lesson pages a person, and the person responds within minutes.
Some decisions are never put to a person at all. Whether one of the loan system's processes is
alive and whether it should keep receiving traffic is decided by a machine, within seconds and
thousands of times a day. That decision looks at two separate questions; when the two are wired
to a single endpoint, the answer given is not just wrong, it builds a self-feeding loop.

The Resilience and Reliability course measured the health endpoint's content and check interval;
that survey is not repeated here. The question here is single: are the two endpoints kept
separate in code, and what happens when they are not.

#### Two Questions, Two Actions

The **liveness** endpoint's question is this: is this process dead, and does restarting it fix
the situation. If the answer is no, the controller takes exactly one action — it kills the
process and restarts it.

The **readiness** endpoint's question is different: can this process take work right now. If the
answer is no, the action is different too — traffic is cut, but the process keeps living and is
expected to recover on its own.

The rule for the distinction follows from these two actions: **for a check to belong in the
liveness endpoint, restarting has to be the fix when that check fails.** A process's own lock,
its own memory, its own event loop meet this test; they get cleaned up when the process is
reborn. A dependency does not. Restarting the loan process while the membership service is
unreachable does not bring the membership service back; the check fails in the new process too,
the controller kills that one as well, and the loop continues until the dependency comes back.

#### The Node: The Distinction Is Two Lines

```js
// health/node.mjs — the loan node: two health endpoints (/liveness, /readiness) and two
// work endpoints. In "separate" mode, liveness only reads the process's own state; in
// "combined" mode, liveness also checks the dependency on the membership service. The
// entire distinction is these two lines.
import { createServer } from "node:http";
import { readFileSync } from "node:fs";
// SD1 (assumption): a restarted process rebuilds its local directory over 24 requests.
export const WARMUP = 24;

export function start(mode) {
  let processed = 0, locked = false;
  const dependencyOk = () => readFileSync("dependency.status", "utf8").trim() === "up";
  const s = createServer((req, res) => {
    let code = 200;
    if (req.url === "/liveness") code = locked || (mode === "combined" && !dependencyOk()) ? 500 : 200;
    else if (req.url === "/readiness") code = dependencyOk() && processed >= WARMUP ? 200 : 503;
    else {
      processed += 1;
      if (processed <= WARMUP) code = 503;                          // local directory not yet rebuilt
      else if (req.url === "/loan" && !dependencyOk()) code = 503;   // needs the membership service
    }
    res.writeHead(code).end();
  });
  s.listen(0, "127.0.0.1", () => process.send({ port: s.address().port }));
}

if (process.argv[2]) start(process.argv[2]);
else console.log(`endpoints: /liveness /readiness /search /loan; warmup ${WARMUP} requests; ` +
  `/search does not need the dependency, /loan does`);
```

```
endpoints: /liveness /readiness /search /loan; warmup 24 requests; /search does not need the dependency, /loan does
```

The entire difference between the two endpoints is which line calls `dependencyOk()`. It belongs
in the readiness endpoint, because the process cannot take work without the dependency. It should
not appear in the liveness endpoint; the `combined` mode makes exactly this mistake, adding a
single condition term. The process serves two work endpoints: `/search` answers from the local
directory and does not need the membership service, `/loan` does. A process that just restarted
loses its local directory and cannot answer either one for twenty-four requests.

#### The Same Scenario, Two Modes

```js
// health/controller.mjs — the same scenario is run in two modes: the membership service goes
// down at tick 15, comes back up at tick 40. The controller polls the liveness and readiness
// endpoints every 2 ticks, and restarts the process after 3 consecutive liveness failures.
// Every tick sends 2 search requests and 2 loan requests.
import { fork } from "node:child_process";
import { writeFileSync } from "node:fs";
import { request } from "node:http";
// SD2 (assumption): poll interval 2 ticks, restart threshold 3 consecutive liveness failures.
const TICKS = 60, DOWN = [15, 40], POLL = 2, THRESHOLD = 3, WARMUP = 24;

const call = (port, path) => new Promise((c) => {
  const req = request({ host: "127.0.0.1", port, path }, (res) => { res.resume(); c(res.statusCode); });
  req.on("error", () => c(0));
  req.end();
});
const spawn = (mode) => new Promise((c) => {
  const p = fork(new URL("node.mjs", import.meta.url), [mode],
    { stdio: ["ignore", "ignore", "ignore", "ipc"] });
  p.once("message", (m) => c([p, m.port]));
});

async function run(mode) {
  writeFileSync("dependency.status", "up");
  let [p, port] = await spawn(mode);
  for (let i = 0; i < WARMUP; i += 1) await call(port, "/search");   // the first warmup does not count
  const s = { restarts: 0, search: 0, loan: 0, notReady: 0, recovery: -1 };
  let consecutive = 0;
  for (let t = 0; t < TICKS; t += 1) {
    writeFileSync("dependency.status", t >= DOWN[0] && t < DOWN[1] ? "down" : "up");
    if (t % POLL === 0) {
      consecutive = (await call(port, "/liveness")) === 200 ? 0 : consecutive + 1;
      if (consecutive >= THRESHOLD) {
        consecutive = 0; s.restarts += 1;
        p.kill(); await new Promise((c) => p.once("exit", c));
        [p, port] = await spawn(mode);
      }
      if ((await call(port, "/readiness")) !== 200) s.notReady += 1;
    }
    let good = 0;
    for (const path of ["/search", "/search", "/loan", "/loan"]) {
      if ((await call(port, path)) === 200) good += 1;
      else if (path === "/search") s.search += 1; else s.loan += 1;
    }
    if (t >= DOWN[1] && good === 4 && s.recovery < 0) s.recovery = t - DOWN[1];
  }
  p.kill();
  return s;
}

const results = [];
for (const mode of ["separate", "combined"]) results.push([mode, await run(mode)]);
console.log(`${TICKS} ticks, 4 requests per tick; membership service down between ticks ${DOWN[0]}-` +
  `${DOWN[1]} (${DOWN[1] - DOWN[0]} ticks), warmup ${WARMUP} requests`);
console.log(`\n${"mode".padEnd(10)}${"restarts".padStart(18)}${"dropped /search".padStart(17)}` +
  `${"dropped /loan".padStart(15)}${"not ready".padStart(13)}${"recovery".padStart(12)}`);
for (const [mode, s] of results)
  console.log(`${mode.padEnd(10)}${String(s.restarts).padStart(18)}${String(s.search).padStart(17)}` +
    `${String(s.loan).padStart(15)}${(s.notReady + " polls").padStart(13)}` +
    `${(s.recovery + " ticks").padStart(12)}`);
const [a, b] = results.map(([, s]) => s);
console.log(`\nin combined mode, ${b.search} dropped search requests never needed the membership ` +
  `service; in separate mode that count is ${a.search}`);
console.log(`total dropped requests: separate ${a.search + a.loan}, combined ${b.search + b.loan} ` +
  `(${b.search + b.loan - a.search - a.loan} requests from misclassification alone)`);
```

```
60 ticks, 4 requests per tick; membership service down between ticks 15-40 (25 ticks), warmup 24 requests

mode                restarts  dropped /search  dropped /loan    not ready    recovery
separate                   0                0             50     12 polls     0 ticks
combined                   4               48             58     14 polls     4 ticks

in combined mode, 48 dropped search requests never needed the membership service; in separate mode that count is 0
total dropped requests: separate 50, combined 106 (56 requests from misclassification alone)
```

#### The Cost of Misclassification

In `separate` mode, there are no restarts. While the membership service was down for twenty-five
ticks, the readiness endpoint answered negatively on twelve polls — meaning the traffic router
could have taken this node out of rotation — but the process stayed alive. The 50 dropped
requests are all `/loan` requests: work that could not have been done without the membership
service anyway. None of the search requests dropped. When the dependency came back at tick 40,
recovery happened **on that same tick**, because the process had never lost its in-memory
directory.

In `combined` mode, once the liveness endpoint looked at the dependency, a chain formed. After
three consecutive negative polls, the controller killed the process; the new process could not
find the same dependency either, so it got killed too: **four restarts** across the twenty-five
ticks the dependency was down. Every restart reset the local directory, and `/search` could not
answer for twenty-four requests either. The result: 48 search requests dropped, even though those
requests never needed the membership service at all. The `/loan` drop count also rose from 50 to
58, because the process was still warming up after the dependency came back.

Two numbers are this lesson's measure. **Total dropped requests rose from 50 to 106; 56 requests
dropped for misclassification alone.** And recovery time rose from 0 ticks to 4: even after the
dependency came back, the system could not answer until it had paid off the last restart's
warm-up debt. That second number shows the loop's real character — restarting extends the
failure, it does not shorten it.

#### Keeping the Distinction in Code

Reducing the two endpoints to a single helper function is the most common mistake: a function
named `healthStatus()` is written, it runs all the checks, and both endpoints call it. In this
shape, the distinction is lost the next day, because whoever wants to add a new dependency check
adds it to a single place, and that place feeds both endpoints at once. In the node above, the
two endpoints are separate lines, and the only thing shared between them is the `dependencyOk()`
reader; which endpoint calls it is spelled out explicitly every time.

A three-item test is enough for the decision. The process's own state — lock, memory, event loop
— belongs in liveness, because restarting fixes them. Dependencies and warm-up belong in
readiness, because waiting fixes them, killing does not. Resources that run out — disk,
connection count — belong in neither; they belong to the previous lesson's alert rule, because
neither restarting nor cutting traffic fixes them.

#### Summary

- Liveness answers "is this process dead" and triggers a restart; readiness answers "can it take
  work right now" and only cuts traffic.
- A check belongs in the liveness endpoint only if restarting is the fix when it fails; a
  dependency check does not meet that test.
- Over the same twenty-five-tick dependency outage, keeping the endpoints separate produced zero
  restarts; combining them produced four.
- Misclassification dropped 48 search requests that never needed the dependency at all; total
  dropped requests rose from 50 to 106, and 56 of those came from this mistake alone.
- Restarting extended the failure: recovery took 4 ticks instead of 0 after the dependency
  returned, because every restart reset the local directory.
- The distinction in code is a single condition term, and reducing the two endpoints to a shared
  `healthStatus()` function loses it the first time someone adds to it.

#### Next Step

This topic turned telemetry into a decision: an indicator was defined, four signals were
computed, the objective's complement turned into a spendable budget, an alert rule was written
from symptoms, and the two health endpoints were separated. A breach of a threshold can now be
**stated.** Nothing changes just from stating it, though. While the membership service was down,
the loan node kept sending the same request the same way, waiting the same duration, and getting
the same error; the readiness endpoint reported this, but no code stopped the request, shortened
it, or routed it another way. Measuring changes observation, not behavior. The next topic places
the patterns that change behavior into the code itself, and counts which layer each one sits in,
how many call sites it wraps, and which call site it skips.

## Resilience

Where patterns that withstand partial failure sit in application code: the duration limit on every outbound call, keeping retry and backoff from turning into a storm, isolating a broken dependency with a circuit breaker, separating resource pools with bulkheads, graceful degradation that keeps the core service running, rate limiting and throttling for fair sharing, classifying the types of partial failure, and post-incident analysis with a blameless root-cause exercise.

### Timeouts

Source: https://academia.sh/en/courses/observability/timeouts

The duration limit of every outbound call: measuring in source whether the limit sits at the call site or in the client wrapper, counting wrapped and uncovered call sites, the orphaned completion produced when the inner limit exceeds the outer one, and the testability difference between the two placements.

The previous topic built indicators and left this behind: a crossed limit is now visible, but the
system still behaves the same way — measuring changes nothing. What changes it is a limit placed
in the code, and this lesson's job is to measure **where** it is placed.

The effect of a timeout was measured in **M19/K05 Resilience and Reliability** and is not repeated
here. What is measured here differs: how many lines and configuration values appear when the same
limit sits at two layers, how many call sites it covers, and **how many it skips**.

**RS1 — the unit of measure is the call site:** every source expression making an outbound call is
a call site. The loan service reaches four dependencies from seven call sites.

#### Mechanism

The dependencies are local processes: catalog and membership respond fast, billing takes 500 ms,
and notification keeps the connection open and **never writes a response**. The loan service also
runs; in the nested-limits section it is the middle layer.

```js
// services.mjs — the split loan system's dependencies and the loan service itself.
// RS2: notification keeps the connection open, never responds. 8103/counter returns all counters.
import { createServer, request } from "node:http";
const s = { open: { notification: 0 }, total: { notification: 0 }, orphaned: 0 };  // orphaned: work that finishes after the caller gave up
for (const [port, name, delay] of [[8101, "catalog", 5], [8102, "membership", 5], [8104, "billing", 500]]) {
  s.open[name] = 0; s.total[name] = 0;
  createServer((req, res) => { s.open[name] += 1; s.total[name] += 1;
    setTimeout(() => { s.open[name] -= 1; res.end("{}"); }, delay);
  }).listen(port);
}
createServer((req, res) => {                        // 8103 notification + counter endpoint
  if (req.url === "/counter") return res.end(JSON.stringify(s));
  if (req.url === "/reset") { s.orphaned = 0; for (const k in s.total) s.total[k] = 0; return res.end("{}"); }
  s.open.notification += 1; s.total.notification += 1;
  req.on("close", () => { s.open.notification -= 1; });   // response never written: call stays hanging
}).listen(8103);
createServer((req, res) => {                        // 8100 loan: applies its own inner limit
  let gaveUp = false;
  req.on("close", () => { gaveUp = !res.writableEnded; });
  const r = request({ port: 8104, path: "/fee", agent: false }, (c) => {
    c.resume(); c.on("end", () => { if (gaveUp) s.orphaned += 1; else res.end('{"fee":1}'); });
  });
  r.setTimeout(Number(new URL(req.url, "http://x").searchParams.get("inner")), () => r.destroy(new Error("inner")));
  r.on("error", () => { if (!gaveUp) res.end('{"error":"inner"}'); });
  r.end();
}).listen(8100);
```

The calling side is written three times, in three directories. `none/` is the unlimited case; `a/`
places the limit **at the call site**, so every request carries its own `setTimeout` line and
value; `b/` places it **in the client wrapper**. All three come from the same seven call sites, so
the difference comes only from placement.

```js
// setup.mjs — generates the same seven call sites in three placements: unlimited, at the site, in the wrapper.
import { mkdirSync, writeFileSync } from "node:fs";
const UNCOVERED = "repeatNotification";              // RS4: call site forgotten when moving to the wrapper
const C = [["loan", "searchBook", 8101, "/search", 200], ["loan", "bookDetail", 8101, "/book", 200],
  ["loan", "verifyMember", 8102, "/member", 200], ["loan", "memberPenalty", 8102, "/penalty", 200],
  ["loan", "lateFee", 8104, "/fee", 800],
  ["notification", "sendNotification", 8103, "/send", 800],
  ["notification", UNCOVERED, 8103, "/repeat", 800]];
const raw = ([, name, port, path, limit], place) => `export function ${name}() {
  return new Promise((resolve, reject) => {
    const r = request({ port: ${port}, path: "${path}" }, (y) => { y.resume(); y.on("end", resolve); });${
  place ? `\n    r.setTimeout(${limit}, () => r.destroy(new Error("timeout")));` : ""}
    r.on("error", reject); r.end();
  });
}`;
const wrapped = ([, name, port, path]) => `export const ${name} = () => sendRequest(${port}, "${path}", "${path.slice(1)}");`;
for (const [dir, place, wrap] of [["none", false, false], ["a", true, false], ["b", false, true]]) {
  mkdirSync(dir, { recursive: true });
  for (const m of ["loan", "notification"]) {
    const g = C.filter((x) => x[0] === m);
    const s = wrap ? g.filter((x) => x[1] !== UNCOVERED) : [], h = g.filter((x) => !s.includes(x));
    writeFileSync(`${dir}/${m}.mjs`, `// ${dir}/${m}.mjs\n` +
      (h.length ? 'import { request } from "node:http";\n' : "") +
      (s.length ? 'import { sendRequest } from "../client.mjs";\n' : "") +
      [...s.map(wrapped), ...h.map((x) => raw(x, place))].join("\n") + "\n");
  }
}
```

`UNCOVERED` is this lesson's core: the repeat path in notification never moved into the wrapper.
Once a limit sits somewhere, a call site that never passes through it shows no gap in the source.

```js
// client.mjs — client wrapper: every outbound call goes through here, the duration limit applies here.
import { request } from "node:http";
export const DEFAULT = { duration: 800 };            // RS3: one default, can be passed at the call site
export function sendRequest(port, path, name, options = {}) {
  const duration = options.duration ?? DEFAULT.duration;
  return new Promise((resolve, reject) => {
    const r = request({ port, path }, (y) => { y.resume(); y.on("end", () => resolve(name)); });
    r.setTimeout(duration, () => r.destroy(new Error(`timeout ${duration} ms`)));
    r.on("error", reject); r.end();
  });
}
```

#### The Cost of Placement in Source and at Runtime

The measurement script scans the three directories, runs all three versions against the live
services and **reads how many requests stay open on the dependency from its own counter**, then
tries the outer and inner limit with four pairs.

```js
// measure.mjs — covered/uncovered call sites, hanging calls, and the conflict between nested limits.
import { readdirSync, readFileSync } from "node:fs";
import { request } from "node:http";
const ROUND = 4, WINDOW = 1500, N = 10, DEPENDENCY = 500;   // billing's fixed delay is 500 ms
const V = { none: "no limit", a: "at call site", b: "in client wrapper" };
const count = (m, d) => (m.match(d) || []).length;
const print = (g, a, ...r) => console.log(String(a).padEnd(26) + r.map((x) => String(x).padStart(g)).join(""));
const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const counter = (path = "/counter") => new Promise((c) => request({ port: 8103, path }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(g === "{}" ? null : JSON.parse(g)));
}).end());
const scan = (d) => readdirSync(d).map((a) => readFileSync(`${d}/${a}`, "utf8"))   // [file, line, raw, wrapped, limit]
  .reduce((a, m) => [a[0] + 1, a[1] + m.split("\n").filter((x) => x.trim()).length,
    a[2] + count(m, /\brequest\(\{/g), a[3] + count(m, /\bsendRequest\(/g), a[4] + count(m, /\.setTimeout\(/g)], [0, 0, 0, 0, 0]);

async function run(d) {                              // runs the seven call sites against the live services
  const m = { ...(await import(`./${d}/loan.mjs`)), ...(await import(`./${d}/notification.mjs`)) };
  const c = { done: 0, failed: 0 }, name = Object.keys(m), once = (await counter()).open.notification;
  for (let t = 0; t < ROUND; t += 1) for (const a of name) m[a]().then(() => { c.done += 1; }, () => { c.failed += 1; });
  await wait(WINDOW);
  return { n: name.length * ROUND, ...c, open: (await counter()).open.notification - once };
}
const gateway = (outer, inner) => new Promise((resolve) => {   // gateway -> loan:8100 -> billing:8104
  const t0 = process.hrtime.bigint();
  const done = (s) => resolve({ s, ms: Number(process.hrtime.bigint() - t0) / 1e6 });
  const r = request({ port: 8100, path: `/fee?inner=${inner}`, agent: false }, (y) => {
    let g = ""; y.on("data", (p) => (g += p)); y.on("end", () => done(g.includes("error") ? "inner error" : "response"));
  });
  r.setTimeout(outer, () => r.destroy(new Error("outer")));
  r.on("error", () => done("outer timeout")); r.end();
});

const wrapperLines = readFileSync("client.mjs", "utf8").split("\n").filter((x) => x.trim()).length;
print(13, "placement", "file", "line", "call site", "wrapped", "uncovered", "limit line");
for (const [d, name] of Object.entries(V)) {
  const [file, line, raw, wrp, lim] = scan(d), b = d === "b";
  print(13, name, file + b, line + (b ? wrapperLines : 0), raw + wrp, d === "a" ? lim : wrp, d === "a" ? 0 : raw, lim + b);
}
const values = ["a/loan.mjs", "a/notification.mjs"].flatMap((f) => readFileSync(f, "utf8").match(/setTimeout\((\d+)/g));
console.log(`configuration surface: ${values.length} numbers at the call site, ${new Set(values).size} distinct values ` +
  `(${[...new Set(values)].map((x) => x.slice(11)).join(" / ")} ms); 1 default in the wrapper, 0 overrides`);

console.log(`\n${ROUND} rounds x 7 call sites; observation window ${WINDOW} ms`);
print(13, "placement", "calls", "response", "timeout", "hanging", "open request");
for (const [d, name] of Object.entries(V)) {
  const r = await run(d);
  print(13, name, r.n, r.done, r.failed, r.n - r.done - r.failed, r.open);
}

console.log(`\nnested limits: dependency's fixed delay ${DEPENDENCY} ms, ${N} requests per pair;` +
  `\n  the measured ms is this run's average, rounded to 100`);
print(22, "outer limit / inner limit", "result", "measured ms", "calls to dependency", "orphaned completion");
for (const [outer, inner] of [[300, 800], [800, 300], [800, 800], [800, 100]]) {
  await counter("/reset");
  const r = await Promise.all(Array.from({ length: N }, () => gateway(outer, inner)));
  await wait(DEPENDENCY + 300);                     // waits for the dependency to finish its work
  const s = await counter();
  print(22, `${outer} / ${inner}`, [...new Set(r.map((x) => x.s))].join("+"),
    Math.round(r.reduce((a, x) => a + x.ms, 0) / N / 100) * 100, s.total.billing, s.orphaned);
}
process.exit(0);
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1; node measure.mjs; kill $SP
```

```
placement                          file         line    call site      wrapped    uncovered   limit line
no limit                              2           46            7            0            7            0
at call site                          2           53            7            7            0            7
in client wrapper                     3           28            7            6            1            1
configuration surface: 7 numbers at the call site, 2 distinct values (200 / 800 ms); 1 default in the wrapper, 0 overrides

4 rounds x 7 call sites; observation window 1500 ms
placement                         calls     response      timeout      hanging open request
no limit                             28           20            0            8            8
at call site                         28           20            8            0            0
in client wrapper                    28           20            4            4            4

nested limits: dependency's fixed delay 500 ms, 10 requests per pair;
  the measured ms is this run's average, rounded to 100
outer limit / inner limit                 result           measured ms   calls to dependency   orphaned completion
300 / 800                          outer timeout                   300                    10                    10
800 / 300                            inner error                   300                    10                     0
800 / 800                               response                   500                    10                     0
800 / 100                            inner error                   100                    10                     0
```

The unlimited row says the client **has no default timeout**: 20 of 28 calls got a response, 8 got
neither response nor error, and 8 stayed open on the dependency when the window closed. This 8 is
independent of run duration — it would still be 8 at a 15-minute window, since nothing exists to
finish those calls.

The limit at the call site takes the source from 46 lines to 53 — one extra line per call site —
and **spreads the configuration surface across seven numbers**; those seven hold only two distinct
values, so five are duplicates. Coverage, in return, is complete: 7 wrapped, 0 uncovered, 0
hanging. With the limit at the call site, **no call site can be forgotten** — a forgotten limit
would be a line never written.

The wrapper version runs the other way. Its total is **shorter** — two modules and the wrapper come
to 28 lines, below even the unlimited case — because it absorbs the limit and the repetition of
building the request. What is lost is per-dependency tuning: catalog was 200 ms at the call site,
800 ms in the wrapper.

#### Uncovered Call Site

The last two columns of the wrapper row are this lesson's real measure: **6 wrapped, 1 uncovered
call site**. The cost shows at runtime: 4 of 28 calls stay hanging — exactly half the unlimited
case.

The difference is not only in number but in information state: in the unlimited case every call
site is suspect and the author knows it; with the wrapper, six are safe and one lies. The rule
that follows: **wrapper placement is incomplete without a coverage check.** What `scan` does —
searching for a raw call constructor — must keep running.

#### The Conflict of Nested Limits

The third table runs the same work through three layers, and each layer's limit knows nothing of
the others.

The first row names the misconfiguration: **the inner limit is larger than the outer one.** The
gateway gives up at 300 ms, the loan service keeps waiting, the dependency finishes at 500 ms, and
the response **goes to no one**: 10 orphaned completions. No indicator counts this as an error —
from its own view, the loan service made a successful call.

In the second row the inner limit is smaller: the loan service gives up first, a named error goes
back to the gateway, and orphaned completions drop to zero. **Limits growing from the inside out
is an ordering rule, not a preference.** The fourth row is the other end of it: once the inner
limit drops below the dependency's delay (100 ms against 500 ms), every call fails though nothing
is wrong with the dependency.

The column common to all three rows gives the real lesson: **calls reaching the dependency stay at
10 for every setting.** A timeout does not reduce load, it only frees the one waiting.

#### Testability

Where the limit sits decides how its behavior gets triggered in a test.

```js
// test/limit.test.mjs — how the same behavior is triggered under two placements.
import { test } from "node:test";
import assert from "node:assert/strict";
import { DEFAULT } from "../client.mjs";
import * as A from "../a/notification.mjs";
import * as B from "../b/notification.mjs";
const bucket = (g) => (g < 100 ? "under 100 ms" : `~${Math.round(g / 100) * 100} ms`);
const duration = async (f) => { const t = Date.now(); await assert.rejects(f()); return Date.now() - t; };

test("the limit at the call site is a constant: the test waits that long", async () => {
  const g = await duration(A.sendNotification);
  assert.ok(g >= 700, `limit 800 ms; measured ${g}`);
  console.log(`  at call site: rejected ${bucket(g)}`);
});
test("the limit in the wrapper is lowered from one place", async () => {
  DEFAULT.duration = 20;
  const g = await duration(B.sendNotification);
  assert.ok(g < 200, `limit 20 ms; measured ${g}`);
  console.log(`  in wrapper: rejected ${bucket(g)}`);
});
test("the uncovered call site is unaffected by the wrapper's setting", async () => {
  assert.equal(await Promise.race([B.repeatNotification().then(() => "response", () => "rejected"),
    new Promise((c) => setTimeout(() => c("still hanging"), 300))]), "still hanging");
});
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1
node --test --test-force-exit --test-reporter=tap test/limit.test.mjs |
  grep -E "^(ok|not ok|#   |# (tests|pass|fail) )"
kill $SP
```

```
#   at call site: rejected ~800 ms
ok 1 - the limit at the call site is a constant: the test waits that long
#   in wrapper: rejected under 100 ms
ok 2 - the limit in the wrapper is lowered from one place
ok 3 - the uncovered call site is unaffected by the wrapper's setting
# tests 3
# pass 3
# fail 0
```

The limit at the call site is a constant; the test has no handle to change it and must wait out
the real duration. The limit in the wrapper is a variable; the test lowers the default to 20 ms
and verifies the same branch in under 100 ms. The run-independent quantity is the limits' ratio:
forty times.

The third test turns this lesson's measure into a test: even with the default at 20 ms, the
uncovered call site is still hanging after 300 ms. **If coverage can be measured as a number, it
can be verified in a test.** No such test can be written for the call-site placement — there is no
coverage concept left to verify.

#### Summary

- The effect of a timeout was measured in M19/K05 Resilience and Reliability; measured here is
  where the limit sits in the code. The client has no default limit: 8 of 28 calls never resolved,
  and 8 requests stayed open on the dependency.
- Call-site placement takes 46 lines to 53 and spreads the configuration surface across seven
  numbers (two distinct values, five duplicates), but coverage is complete: 7 wrapped, 0 uncovered.
- Wrapper placement is shorter (28 lines) with a single configuration surface, but does not
  guarantee coverage: 6 wrapped, 1 uncovered, 4 hanging calls. The gap shows nowhere in the
  source; only scanning finds it.
- When the inner limit exceeds the outer one, the outer layer gives up while the inner keeps
  working: 10 of 10 requests produce an orphaned completion — an ordering rule, not a preference.
- Calls reaching the dependency stay at 10 across all four settings; a timeout frees the one
  waiting, it does not reduce the load. Whether the limit is a variable decides testability: the
  uncovered call site can be tested directly.

#### Next Step

The limit is placed and the waiting side is freed, but it is left holding an error, and nothing
says what to do with it. The closest answer is retrying. Where that decision sits is also a
placement question, and more dangerous than the timeout: a timeout does not change the
dependency's load, while a retry multiplies it directly. The next lesson measures **which layer
the retry sits in** — the same rule built into three layers, then all three opened at once,
counting how many calls one user request turns into on the dependency.

### Retry and Backoff

Source: https://academia.sh/en/courses/observability/retry-and-backoff

Measuring which layer a retry sits in: building the same rule separately into the client wrapper, the middle layer, and the gateway, whether the healthy call site gets retried too, the multiplier produced when three placements overlap, and the repeated side effect counted in deliveries.

The previous lesson placed the duration limit and freed the caller, but left it holding an error.
The closest answer is retrying: if the dependency dropped only briefly, a second call may succeed.
This decision also has a place, more dangerous than the timeout — a timeout did not change the
dependency's load, a retry multiplies it directly.

Exponential backoff, jitter, the cap, and retry-budget arithmetic were built in **M14/K06
Application Architecture: Routing, State and Data**, **M16/K05**, and **M19/K05 Resilience and
Reliability**; the pattern's effect on the system was measured in M19/K05. None of it repeats
here. This lesson has one question: **which layer the retry sits in**, and what happens to calls
reaching the dependency when that layer changes.

The same rule can be placed in three spots. The **client wrapper** retries a single outbound call.
The **middle layer** reruns the request's whole body. The **gateway** recalls the entire service
and has not one line in the service's code. All three make the same promise; each repeats
something different.

#### Mechanism

The notification dependency carries this lesson's measure. **RS5 — notification actually
delivers; what's lost is the acknowledgment:** the delivery counter goes up on every call, but the
first `failure` calls return 503. So what the retry repeats is not just the call but the **side
effect**. Membership is a healthy dependency and always responds.

```js
// services.mjs — 8202 notification (failing, counts deliveries), 8203 membership (healthy),
// 8201 loan: carries both retry placements. Counters live at the 8202/counter endpoint.
import { createServer, request } from "node:http";
const s = { loan: 0, membership: 0, notification: 0, delivered: 0, remaining: 0 };  // delivered: what notification actually sends
createServer((req, res) => {                        // 8202 notification
  const q = new URL(req.url, "http://x").searchParams;
  if (req.url === "/counter") return res.end(JSON.stringify(s));
  if (req.url.startsWith("/reset")) {
    for (const k of ["loan", "membership", "notification", "delivered"]) s[k] = 0;
    s.remaining = Number(q.get("failure")); return res.end("{}");
  }
  s.notification += 1; s.delivered += 1;            // delivery happens; the acknowledgment is lost for the first `remaining` calls
  if (s.remaining > 0) { s.remaining -= 1; res.statusCode = 503; }
  res.end("{}");
}).listen(8202);
createServer((req, res) => { s.membership += 1; res.end("{}"); }).listen(8203);   // healthy dependency

const call = (port, path) => new Promise((resolve) => {
  const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); });
  r.on("error", () => resolve(0)); r.end();
});
// >>> placement: client wrapper — retries only the call site it wraps
const callWithRetry = async (port, path, tries) => {
  for (let n = 1; ; n += 1) {
    const k = await call(port, path);
    if (k < 500 || n >= tries) return k;
  }
};
// <<<
const handle = async (inner) => {                   // body of the loan request: two outbound call sites
  await callWithRetry(8203, "/member", 1);          // healthy dependency, not retried
  return callWithRetry(8202, "/send", inner);
};
createServer(async (req, res) => {                  // 8201 loan
  const q = new URL(req.url, "http://x").searchParams;
  s.loan += 1;
  // >>> placement: middle layer — reruns the request's whole body
  let k = 0;
  for (let n = 1; n <= Number(q.get("middle")); n += 1) { k = await handle(Number(q.get("inner"))); if (k < 500) break; }
  // <<<
  res.statusCode = k; res.end("{}");
}).listen(8201);
```

The third placement lives in the measurement script, because the gateway is outside the service.
**RS6 — every layer's retry count is 3**; the comparison only means something while that number
stays fixed. The `>>> placement:` markers exist to count code cost; the script reads them straight
from the source.

```js
// measure.mjs — which layer the retry sits in, the multiplier of overlapping retries, code cost.
import { readFileSync } from "node:fs";
import { request } from "node:http";
const call = (port, path) => new Promise((resolve) => {
  const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); });
  r.on("error", () => resolve(0)); r.end();
});
const read = (path) => new Promise((c) => request({ port: 8202, path }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(g === "{}" ? null : JSON.parse(g)));
}).end());
const print = (g, a, ...r) => console.log(String(a).padEnd(32) + r.map((x) => String(x).padStart(g)).join(""));
// >>> placement: gateway — not one line lives in the service's code
const gateway = async (outer, middle, inner) => {
  for (let n = 1; ; n += 1) {
    const k = await call(8201, `/loan?middle=${middle}&inner=${inner}`);
    if (k < 500 || n >= outer) return k;
  }
};
// <<<
const attempt = async (failure, outer, middle, inner) => {
  await read(`/reset?failure=${failure}`);
  const k = await gateway(outer, middle, inner);
  return { k, ...(await read("/counter")) };
};

console.log("transient failure: notification delivers on the first 2 calls but loses the acknowledgment; single user request");
print(20, "placement (outer/middle/inner)", "result", "loan calls", "membership calls", "notification calls", "delivered");
for (const [name, o, m, i] of [["none (1/1/1)", 1, 1, 1], ["wrapper (1/1/3)", 1, 1, 3],
  ["middle layer (1/3/1)", 1, 3, 1], ["gateway (3/1/1)", 3, 1, 1]]) {
  const r = await attempt(2, o, m, i);
  print(20, name, r.k, r.loan, r.membership, r.notification, r.delivered);
}

console.log("\npersistent failure: overlapping retries");
print(20, "outer / middle / inner", "multiplier", "notification calls", "delivered", "membership calls");
for (const [o, m, i] of [[1, 1, 1], [3, 1, 1], [3, 3, 1], [3, 3, 3], [5, 3, 3]]) {
  const r = await attempt(999, o, m, i);
  print(20, `${o} / ${m} / ${i}`, o * m * i, r.notification, r.delivered, r.membership);
}

const code = (file) => [...readFileSync(file, "utf8").matchAll(/\/\/ >>> placement: ([^\n—]+)—[^\n]*\n([\s\S]*?)\/\/ <<</g)]
  .map(([, name, g]) => [name.trim(), file, g.trim().split("\n").length]);
const sites = [...readFileSync("services.mjs", "utf8").matchAll(/callWithRetry\(\d+, "[^"]+", ([^)]+)\)/g)].map((m) => m[1]);
console.log("\ncode cost and coverage");
print(15, "placement", "file", "line");
for (const [name, d, n] of [...code("services.mjs"), ...code("measure.mjs")]) print(15, name, d, n);
console.log(`the loan body has ${sites.length} outbound call sites; the wrapper retries ${sites.filter((x) => x !== "1").length} ` +
  `of them, the middle layer and the gateway retry all ${sites.length}.`);
console.log("configuration surface: 3 retry counts, in 3 separate places; none sees the others' value.");
process.exit(0);
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1; node measure.mjs; kill $SP
```

```
transient failure: notification delivers on the first 2 calls but loses the acknowledgment; single user request
placement (outer/middle/inner)                result          loan calls    membership calls  notification calls           delivered
none (1/1/1)                                     503                   1                   1                   1                   1
wrapper (1/1/3)                                  200                   1                   1                   3                   3
middle layer (1/3/1)                             200                   1                   3                   3                   3
gateway (3/1/1)                                  200                   3                   3                   3                   3

persistent failure: overlapping retries
outer / middle / inner                    multiplier  notification calls           delivered    membership calls
1 / 1 / 1                                          1                   1                   1                   1
3 / 1 / 1                                          3                   3                   3                   3
3 / 3 / 1                                          9                   9                   9                   9
3 / 3 / 3                                         27                  27                  27                   9
5 / 3 / 3                                         45                  45                  45                  15

code cost and coverage
placement                                  file           line
client wrapper                     services.mjs              6
middle layer                       services.mjs              2
gateway                             measure.mjs              6
the loan body has 2 outbound call sites; the wrapper retries 1 of them, the middle layer and the gateway retry all 2.
configuration surface: 3 retry counts, in 3 separate places; none sees the others' value.
```

#### How Three Placements Close the Same Failure

The first table meets a transient failure with three placements, and the result column is the
same in all three: a 503 turns into 200. The retry keeps its promise. The difference is in the
other columns.

Calls reaching the dependency stay at 3 in all three. This is not surprising; the failure lasts
two calls and clears on the third. The retry's load on the dependency comes from the retry count,
not from the placement.

Calls reaching the healthy dependency, though, are 1, 3, and 3. This is where the split lies. The
client wrapper retries only the call site it wraps; the membership call happens once, and that
dependency — which has nothing to do with the failure — is never strained. The middle layer
reruns the request body from the start, repeating the failure-free call site too. The gateway does
the same and, on top of it, calls the loan service three times — shown in the loan-calls column as
1, 1, 3.

The rule that follows is a selection criterion: **the outer placement cannot discriminate.** The
gateway does not know which call failed, so it repeats the whole request; the further out you go,
the more repeated work grows. The inner placement discriminates, but it only covers the call site
that passes through it — the previous lesson's uncovered-call-site problem applies here exactly
the same.

The delivered column is 3 in all three placements. For a single user request, notification was
sent three times. Changing the placement does not fix this; the side effect comes from the retry
itself. Whether a call site is safe to repeat is a property of the call site, not the placement,
and the fix belongs there too: that call site must keep its retry count at 1.

#### Overlapping Retries

The second table opens all three placements at once. Under a persistent failure, every layer
spends its share to the end, and calls reaching the dependency track the multiplier column
exactly: 1, 3, 9, 27, 45. A single user request turns into 27 calls on the dependency, and a
single notification gets delivered 27 times.

The real information in this table is not the multiplier itself, but that **no layer sees it.**
Whoever wrote the gateway wrote 3 retries, whoever wrote the middle layer wrote 3, whoever wrote
the wrapper wrote 3; none chose a wrong number. The configuration surface sits in three separate
places and none can see the others' value. The last row shows how fast this grows: the outer layer
going from three to five takes the total from 27 to 45 — **one number changing in one place adds
eighteen calls to the dependency.**

The membership-calls column is 9 on those same rows, not 27. The healthy dependency is only called
as many times as the product of the two outer layers, because the innermost placement does not
repeat it. Which factors of the multiplier land on which call site is a direct result of
placement.

#### Code Cost and Coverage

The third table gives the size of each placement: client wrapper 6 lines, middle layer 2, gateway
6 — and the gateway's six lines are not in the service's code at all. This is what makes the outer
placement attractive: every call site is covered without the service changing, even ones not yet
written. It is also exactly the cost; coverage having no selectivity is not a flaw, it is that
placement's definition.

The inner placement is the opposite: of the two outbound call sites in the loan body, it retries
one and deliberately leaves the other alone. Selectivity is only achieved by deciding call site by
call site, and that decision is not made once — it is made again every time a new call site is
added.

#### Testability

Placement also decides where the behavior can be pinned down in a test.

```js
// test/retry.test.mjs — testability of placement: where the assertion is pinned.
import { test } from "node:test";
import assert from "node:assert/strict";
import { request } from "node:http";
const call = (port, path) => new Promise((resolve) => {
  const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); });
  r.on("error", () => resolve(0)); r.end();
});
const read = (path) => new Promise((c) => request({ port: 8202, path }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(g === "{}" ? null : JSON.parse(g)));
}).end());
const run = async (failure, outer, middle, inner) => {
  await read(`/reset?failure=${failure}`);
  for (let n = 1; n <= outer; n += 1) if (await call(8201, `/loan?middle=${middle}&inner=${inner}`) < 500) break;
  return read("/counter");
};

test("wrapper placement: the assertion pins to a single call site", async () => {
  assert.equal((await run(2, 1, 1, 3)).notification, 3);
  assert.equal((await run(2, 1, 1, 3)).membership, 1);   // healthy call site is not retried
});
test("gateway placement: the same assertion changes with the inner setting", async () => {
  assert.equal((await run(999, 3, 1, 1)).notification, 3);
  assert.equal((await run(999, 3, 1, 3)).notification, 9);   // gateway unchanged, result triples
});
test("retry multiplies delivery", async () => {
  const r = await run(2, 1, 1, 3);
  assert.equal(r.delivered, 3);                     // single user request, three deliveries
});
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1
node --test --test-force-exit --test-reporter=tap test/retry.test.mjs |
  grep -E "^(ok|not ok|# (tests|pass|fail) )"
kill $SP
```

```
ok 1 - wrapper placement: the assertion pins to a single call site
ok 2 - gateway placement: the same assertion changes with the inner setting
ok 3 - retry multiplies delivery
# tests 3
# pass 3
# fail 0
```

The first test shows the wrapper placement's testability: the assertion pins to a single call
site, the number is local, and no other setting can change it. The second proves the gateway
placement's problem with the test itself: the gateway's own setting is 3 in both runs, but the
expected call count goes from 3 to 9, because a different number changed further in. An assertion
about the outer placement's behavior depends on a setting it cannot see, and breaks when that
setting changes. The third pins the side effect directly: one user request, three deliveries.

#### Summary

- Retry arithmetic and its system effect were measured in M14/K06, M16/K05, and M19/K05 Resilience
  and Reliability; measured here is which layer it sits in.
- Under a transient failure all three placements save the request, and calls reaching the
  dependency stay at 3 in all three; the difference is in calls reaching the healthy dependency: 1
  wrapper, 3 middle layer, 3 gateway. The gateway also calls the loan service 3 times instead of 1.
- The outer placement cannot discriminate: not knowing which call failed, it repeats the whole
  request. The inner placement discriminates, but covers only the call site passing through it.
- With all three placements open at once, calls reaching the dependency track the multiplier
  exactly: 1, 3, 9, 27, 45. No layer sees another's number; the outer one going from three to five
  adds eighteen calls at the dependency. The healthy dependency gets 9 calls, not 27.
- The delivery counter is 3 in all three placements: the repeated side effect is a call-site
  problem, not a placement one, and the fix is keeping that call site's retry count at 1.
- Code cost is 6 / 2 / 6 lines, and the gateway's six are not in the service's code. In tests the
  wrapper's assertion is local; the gateway's depends on a setting it cannot see and breaks when
  that setting changes.

#### Next Step

A retry closes a transient failure but worsens a persistent one: all 27 calls failed, none saved
anything, and a dependency already down took twenty-seven times the load on top of it. What's
missing is not a number but a **decision** — recognizing the dependency is down and not making
the call at all. The next lesson places the circuit breaker in the code and measures: whether the
breaker's state is kept per dependency or per call site, when the two placements open under the
same failure, how many parameters make up the configuration surface, and what behavior a wrong
threshold produces.

### Circuit Breaker

Source: https://academia.sh/en/courses/observability/circuit-breaker

Measuring where the breaker sits in the code: whether its state is kept per dependency or per call site, the calls reaching the dependency before each placement opens, the same threshold carrying two separate meanings under the two placements, and a rarely called path whose breaker never opens.

The previous lesson measured how a retry worsens a persistent failure: all 27 of the 27 calls
failed, and a dependency that was already down took twenty-seven times the load. What's missing is
not a number but a decision — recognizing that the dependency is down and not making the call at
all.

What a circuit breaker is, and its closed, open, and half-open states, were built in the
**M14/K06 Application Architecture: Routing, State and Data** course; the pattern's effect on the
system, the cost of a failure-free day, and the price of a false trip were measured in **M19/K05
Resilience and Reliability**. Neither repeats here. This lesson's question is: **where is the
breaker's state kept.** A breaker is nothing more than a number and a timestamp; where that number
is placed decides what the pattern does from the start.

There are two placements, and the difference between them is a single symbol in the code: is
state stored under the dependency's name, or under the call site's name. The loan service reaches
the same notification dependency from three separate call sites; in the first placement, all
three share one state, and in the second, each keeps its own.

#### Mechanism

The notification dependency is permanently faulty and returns 503 on every call. Catalog is
healthy and, although it passes through the breaker too, it never opens — it is there to show
where the breaker's scope ends. **RS7 — the open duration outlasts the run**, so once a breaker
opens it does not close, and the half-open transition does not interfere with the measurement.

```js
// services.mjs — 8302 notification (permanently faulty), 8303 catalog (healthy), 8301 loan.
// The breaker is the same code under both placements; the only change is which key holds the state.
import { createServer, request } from "node:http";
const s = { catalog: 0, send: 0, remind: 0, cancel: 0 };
createServer((req, res) => {                        // 8302 notification: 503 on every call
  if (req.url === "/counter") return res.end(JSON.stringify(s));
  if (req.url === "/reset") { for (const k in s) s[k] = 0; return res.end("{}"); }
  s[req.url.slice(1)] += 1; res.statusCode = 503; res.end("{}");
}).listen(8302);
createServer((req, res) => { s.catalog += 1; res.end("{}"); }).listen(8303);

const call = (port, path) => new Promise((resolve) => {
  const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); });
  r.on("error", () => resolve(0)); r.end();
});
let THRESHOLD = 5, state = new Map();               // RS7: open duration outlasts the run; half-open transition built in M14/K06
const OPEN_DURATION = 3_600_000;
const breakerFor = (a) => { if (!state.has(a)) state.set(a, { consecutive: 0, openUntil: 0 }); return state.get(a); };
const guardedCall = async (port, path, key) => {    // breaker lives here; the key decides placement
  const d = breakerFor(key);
  if (Date.now() < d.openUntil) return 0;           // open: the call is never made
  const k = await call(port, path);
  if (k >= 500) { d.consecutive += 1; if (d.consecutive >= THRESHOLD) d.openUntil = Date.now() + OPEN_DURATION; } else d.consecutive = 0;
  return k;
};
createServer(async (req, res) => {                  // 8301 loan: three call sites reach the same dependency
  const q = new URL(req.url, "http://x").searchParams;
  if (req.url.startsWith("/configure")) { THRESHOLD = Number(q.get("threshold")); state = new Map(); return res.end("{}"); }
  if (req.url.startsWith("/state"))
    return res.end(JSON.stringify([...state].map(([a, d]) => [a, Date.now() < d.openUntil])));
  const n = Number(q.get("n")), placement = q.get("by");
  const keyFor = (dependency, callSite) => (placement === "dependency" ? dependency : callSite);
  await guardedCall(8303, "/book", keyFor("catalog", "book"));
  await guardedCall(8302, "/send", keyFor("notification", "send"));
  await guardedCall(8302, "/remind", keyFor("notification", "remind"));
  if (n % 20 === 0) await guardedCall(8302, "/cancel", keyFor("notification", "cancel"));
  res.end("{}");
}).listen(8301);
```

The `keyFor` function is this entire lesson. The breaker's code does not change, the threshold
does not change, the call sites do not change; the only thing that changes is whether the key
passed into `guardedCall` names the dependency or the call site. Two of the three call sites run
on every request; the `cancel` path is called only every twentieth request — in a real codebase,
frequent and rare paths always coexist.

```js
// measure.mjs — where the breaker's state is kept: per dependency or per call site.
import { request } from "node:http";
const N = 60;                                       // 60 user requests; the cancel path is called every 20th
const call = (port, path) => new Promise((resolve) => {
  const r = request({ port, path, agent: false }, (y) => {
    let g = ""; y.on("data", (p) => (g += p)); y.on("end", () => resolve(g));
  });
  r.on("error", () => resolve("")); r.end();
});
const print = (a, ...r) => console.log(String(a).padEnd(24) + r.map((x) => String(x).padStart(20)).join(""));

const run = async (threshold, by) => {
  await call(8302, "/reset");
  await call(8301, `/configure?threshold=${threshold}`);
  for (let n = 1; n <= N; n += 1) await call(8301, `/request?n=${n}&by=${by}`);
  const s = JSON.parse(await call(8302, "/counter"));
  const d = JSON.parse(await call(8301, "/state"));
  return { ...s, objects: d.length, open: d.filter(([, a]) => a).length };
};

console.log(`${N} user requests; send and remind on every request, cancel every 20th request`);
console.log("notification dependency is permanently faulty: 503 on every call");
print("threshold / placement", "state objects", "open objects", "notification calls", "send", "remind", "cancel", "catalog");
const t = {};
for (const threshold of [3, 5, 10])
  for (const [by, name] of [["dependency", "dependency"], ["callSite", "call site"]]) {
    const r = await run(threshold, by);
    t[`${threshold}${by}`] = r;
    print(`${threshold} / ${name}`, r.objects, r.open, r.send + r.remind + r.cancel, r.send, r.remind, r.cancel, r.catalog);
  }
const total = (r) => r.send + r.remind + r.cancel;
console.log(`\nconfiguration surface: 2 numbers (threshold, open duration) and 1 decision (the state's key).`);
console.log(`same threshold, two separate meanings under two placements: for threshold 5, calls reaching the dependency go ` +
  `${total(t["5dependency"])} -> ${total(t["5callSite"])}; for threshold 10, ${total(t["10dependency"])} -> ${total(t["10callSite"])}.`);
console.log(`under per-call-site placement the effective threshold at the dependency level = threshold x openable call sites; ` +
  `at threshold 10 the cancel path never reached the threshold with ${t["10callSite"].cancel} calls and its breaker stayed closed.`);
process.exit(0);
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1; node measure.mjs; kill $SP
```

```
60 user requests; send and remind on every request, cancel every 20th request
notification dependency is permanently faulty: 503 on every call
threshold / placement          state objects        open objects  notification calls                send              remind              cancel             catalog
3 / dependency                             2                   1                   3                   2                   1                   0                  60
3 / call site                              4                   3                   9                   3                   3                   3                  60
5 / dependency                             2                   1                   5                   3                   2                   0                  60
5 / call site                              4                   2                  13                   5                   5                   3                  60
10 / dependency                            2                   1                  10                   5                   5                   0                  60
10 / call site                             4                   2                  23                  10                  10                   3                  60

configuration surface: 2 numbers (threshold, open duration) and 1 decision (the state's key).
same threshold, two separate meanings under two placements: for threshold 5, calls reaching the dependency go 5 -> 13; for threshold 10, 10 -> 23.
under per-call-site placement the effective threshold at the dependency level = threshold x openable call sites; at threshold 10 the cancel path never reached the threshold with 3 calls and its breaker stayed closed.
```

#### The State-Object Column

The leftmost numeric column is the placement's direct definition: 2 state objects under
per-dependency placement, 4 under per-call-site placement. Both reach the same two dependencies;
the difference is whether state multiplies by dependency count or by call-site count. This number
grows with the codebase: adding a call site changes nothing under per-dependency placement, but
under per-call-site placement one more state object is born, and it starts learning from zero.

Whether the learning is shared shows up in the notification-calls column. At threshold 5,
per-dependency placement makes 5 calls to the broken dependency: `send` fails three times,
`remind` fails twice, the shared breaker opens on the fifth failure, and the `cancel` path never
reaches the dependency at all — 0 in the column. What one call site learns, the other already
knows.

Under per-call-site placement, the same threshold sends 13 calls: `send` 5, `remind` 5, `cancel`
3. Every call site learns the same fact from scratch. While the dependency is already down, the
third call site behaves as if it knows nothing.

#### The Same Threshold, Two Meanings

Reading the threshold rows together reveals the real problem with the configuration surface. For
thresholds 3, 5, and 10, calls reaching the dependency are 3, 5, and 10 under per-dependency
placement, and 9, 13, and 23 under per-call-site placement. Whoever writes `threshold 5` has said
"at most five failed calls to the dependency" under one placement, and "five per call site, up to
fifteen in total" under the other. The number is the same, its meaning is not, and the only thing
in the code that says which meaning applies is the name of a key.

Under per-call-site placement, the effective threshold at the dependency level is the threshold
multiplied by the number of call sites that can open. This multiplier appears nowhere in the
configuration; it is the call-site count in the source, and it grows silently with every new call
site.

#### The Breaker That Never Opens

The last row is this lesson's harshest measure. At threshold 10, under per-call-site placement
only 2 of the 4 state objects are open — `send` and `remind` opened, `cancel` did not. The reason
is arithmetic: the `cancel` path is called only 3 times across 60 requests, and 3 failures never
reach a threshold of 10.

The result is this: even though the dependency is known to be broken, that call site still goes
out every time it is called. The breaker is there, its code is correct, its setting matches the
others, and it will never do anything useful. A rarely called call site can never fill a threshold
on its own — **what stays unprotected is exactly the least exercised path.** Under
per-dependency placement, that same path makes zero calls, because it inherits what its frequently
called neighbors already learned.

The catalog column is 60 on every row. The breaker never touches the healthy dependency and
behaves the same under both placements; sharing state is not a question between dependencies, it
is a question between the call sites of the same dependency.

#### Testability

```js
// test/breaker.test.mjs — the assertion pins wherever the breaker's state is kept.
import { test } from "node:test";
import assert from "node:assert/strict";
import { request } from "node:http";
const call = (port, path) => new Promise((resolve) => {
  const r = request({ port, path, agent: false }, (y) => {
    let g = ""; y.on("data", (p) => (g += p)); y.on("end", () => resolve(g));
  });
  r.on("error", () => resolve("")); r.end();
});
const run = async (threshold, by, n = 60) => {
  await call(8302, "/reset");
  await call(8301, `/configure?threshold=${threshold}`);
  for (let i = 1; i <= n; i += 1) await call(8301, `/request?n=${i}&by=${by}`);
  return { s: JSON.parse(await call(8302, "/counter")), d: JSON.parse(await call(8301, "/state")) };
};

test("per dependency: what one call site learns, the other already knows", async () => {
  const { s, d } = await run(5, "dependency");
  assert.equal(s.cancel, 0);                        // the cancel path never reached the dependency
  assert.equal(d.length, 2);                        // one assertion: two dependencies, two state objects
});
test("per call site: the rare call site never reaches the threshold", async () => {
  const { s, d } = await run(10, "callSite");
  assert.equal(s.cancel, 3);                        // called three more times while the dependency was broken
  assert.deepEqual(d.find(([a]) => a === "cancel"), ["cancel", false]);
});
test("the breaker's scope is bound to the dependency", async () => {
  for (const by of ["dependency", "callSite"]) assert.equal((await run(5, by)).s.catalog, 60);
});
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1
node --test --test-force-exit --test-reporter=tap test/breaker.test.mjs |
  grep -E "^(ok|not ok|# (tests|pass|fail) )"
kill $SP
```

```
ok 1 - per dependency: what one call site learns, the other already knows
ok 2 - per call site: the rare call site never reaches the threshold
ok 3 - the breaker's scope is bound to the dependency
# tests 3
# pass 3
# fail 0
```

Triggering the breaker in a test is easy — keep the dependency broken and make as many calls as
the threshold. What is hard is deciding **which state is under test.** Under per-dependency
placement there is a single object, and one assertion covers every call site; that is what the
first test's `d.length` check rests on. Under per-call-site placement, each call site's state must
be tested separately, and as the second test shows, an untested call site stays silently
unprotected. Placement decides not only runtime behavior but how many assertions a test must
write: one per dependency, or one per call site.

#### Summary

- The breaker's states were built in M14/K06 Application Architecture: Routing, State and Data,
  its effect on the system in M19/K05 Resilience and Reliability; what is measured here is where
  its state is kept in the code. The difference between the two placements is the name of a
  single key.
- State-object count is the direct result of placement: 2 per dependency, 4 per call site. Adding
  a call site under the latter births one more state object that starts learning from zero.
- At the same threshold, calls reaching the broken dependency are 5 against 13; under
  per-dependency placement, one call site's learning carries over to the others, and the cancel
  path never reaches the dependency.
- The same threshold carries two separate meanings under the two placements: thresholds of 3, 5,
  and 10 correspond to 3, 5, and 10 calls at the dependency, but to 9, 13, and 23 calls under
  per-call-site placement. The effective threshold at the dependency level is the threshold times
  the call-site count, and that multiplier appears in no configuration.
- A rarely called path cannot fill its own threshold: at threshold 10, the cancel path's breaker
  stayed closed after 3 calls and kept going out on every call even though the dependency was
  known to be broken.
- The breaker never touches the healthy dependency (catalog is 60 on every row); the sharing
  question sits between the call sites of one dependency, not between dependencies. In tests,
  per-dependency placement closes with one assertion, per-call-site placement demands one per call
  site.

#### Next Step

The breaker drew a boundary, but the boundary's material is information: protection only begins
after enough failures accumulate, and on rare paths it never begins at all. Beyond that, the
breaker understands that a dependency is broken, not that the work waiting on it is consuming the
caller's own resources — all three call sites run in the same process, from the same pool, and
when one slows down, what is left for the others shrinks. So one more question remains: can the
contagion be cut off from the start, without ever depending on detecting it. The next lesson tries
this by splitting the resource pool, and measures where the pool is defined in the code, how many
call sites moving a dependency into its own pool touches, which paths bypass the pool's limit, and
what behavior a wrong pool size produces.

### Bulkhead Pattern

Source: https://academia.sh/en/courses/observability/bulkhead-pattern

Measuring where the resource pool sits in the code: comparing a single shared pool against a pool per dependency at the same total slot count, a call site that never uses the pool exceeding the limit, and the behavior wrong sizing produces in concurrency and duration.

The previous lesson placed the breaker's state in the code and showed that protection only starts
after enough failures accumulate, and never starts on rarely called paths. The breaker understands
that a dependency is broken; it does not understand that the work waiting on it consumes the
caller's own resources. All three call sites run in the same process, from the same slot pool.

The bulkhead pattern and its effect on the system were built and measured in **M19/K05 Resilience
and Reliability**. None of it repeats here. This lesson's question is: **where the pool is defined
in the code**, how many call sites using it changes, and which paths stay outside the limit.

#### Mechanism

Two dependencies: notification takes 300 ms, catalog takes 5 ms. Forty concurrent calls are made
to each. Every dependency records its own peak concurrent request count — this is what the pool
limit looks like from the dependency's side. **RS8 — the total slot count is equal under both
placements:** eight in the single pool, four plus four in the pool-per-dependency case. The
comparison measures placement only while the total stays fixed.

```js
// services.mjs — 8402 notification (slow), 8403 catalog (fast), 8401 loan: pools are defined here.
import { createServer, request, Agent } from "node:http";
const N = 40;                                       // 40 concurrent calls per stream
const s = { notification: { open: 0, peak: 0 }, catalog: { open: 0, peak: 0 } };
const service = (name, delay, port) => createServer((req, res) => {
  const d = s[name]; d.open += 1; d.peak = Math.max(d.peak, d.open);
  setTimeout(() => { d.open -= 1; res.end("{}"); }, delay);
}).listen(port);
service("notification", 300, 8402);
service("catalog", 5, 8403);

// >>> pool definition: two placements, same total slots
const pools = (placement, notif, cat) => {
  if (placement === "single") { const a = new Agent({ maxTotalSockets: notif + cat }); return () => a; }
  const t = { 8402: new Agent({ maxTotalSockets: notif }), 8403: new Agent({ maxTotalSockets: cat }) };
  return (port) => t[port];
};
// <<<
const call = (port, path, pool) => new Promise((resolve) => {
  const r = request({ port, path, agent: pool }, (y) => { y.resume(); y.on("end", resolve); });
  r.on("error", resolve); r.end();
});

createServer(async (req, res) => {                  // 8401 loan: runs one round start to finish
  const q = new URL(req.url, "http://x").searchParams;
  const poolFor = pools(q.get("placement"), Number(q.get("notif")), Number(q.get("cat")));
  const skip = Number(q.get("skip"));               // call sites that never use the pool at all
  for (const d of Object.values(s)) { d.open = 0; d.peak = 0; }
  const t0 = Date.now();
  let done = 0, catalogBefore = null;
  const slow = Array.from({ length: N }, (_, j) => call(8402, "/send", j < skip ? false : poolFor(8402)));
  const fast = Array.from({ length: N }, () => call(8403, "/book", poolFor(8403)).then(() => { done += 1; }));
  Promise.race(slow).then(() => { catalogBefore = done; });
  await Promise.all([...slow, ...fast]);
  res.end(JSON.stringify({ n: N, catalogBefore, notification: s.notification.peak, catalog: s.catalog.peak,
    duration: Math.round((Date.now() - t0) / 500) * 500 }));
}).listen(8401);
```

`catalogBefore` is this lesson's main measure: how many catalog calls finished by the time the
first notification call completes. Since catalog answers in five milliseconds, a stream with its
own resources should finish all forty of its calls before notification gives even its first
response. If it cannot, that means it found no slot.

The `skip` parameter produces a call site that never uses the pool at all: a request made with
`agent: false` opens its own socket and never looks at the pool table. In the source this looks
like no shortcoming at all; it is just another option.

```js
// measure.mjs — where the pool is defined, the skipped path, and wrong sizing.
import { readFileSync } from "node:fs";
import { request } from "node:http";
const call = (path) => new Promise((c) => request({ port: 8401, path, agent: false }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(JSON.parse(g)));
}).end());
const print = (a, ...r) => console.log(String(a).padEnd(34) + r.map((x) => String(x).padStart(20)).join(""));

console.log("40 concurrent calls per stream: notification 300 ms, catalog 5 ms; total slots 8 under every placement");
console.log("the duration column is this run's measurement, rounded to 500 ms; the expected value is ceil(40/slots) x 300 ms");
print("placement", "catalog done before", "notification peak", "catalog peak", "duration ms");
for (const [name, q] of [["single shared pool", "placement=single&notif=4&cat=4&skip=0"],
  ["pool per dependency", "placement=separate&notif=4&cat=4&skip=0"],
  ["pool per dependency + 8 skipped", "placement=separate&notif=4&cat=4&skip=8"]]) {
  const r = await call(`/run?${q}`);
  print(name, `${r.catalogBefore} / ${r.n}`, r.notification, r.catalog, r.duration);
}

console.log("\nwrong sizing: slot counts under pool-per-dependency placement");
print("notification slots / catalog slots", "notification peak", "catalog peak", "duration ms");
for (const [notif, cat] of [[1, 4], [4, 4], [40, 4], [4, 1]]) {
  const r = await call(`/run?placement=separate&notif=${notif}&cat=${cat}&skip=0`);
  print(`${notif} / ${cat}`, r.notification, r.catalog, r.duration);
}

const source = readFileSync("services.mjs", "utf8");
const block = source.match(/\/\/ >>> pool definition[^\n]*\n([\s\S]*?)\/\/ <<</)[1].trim().split("\n").length;
const sites = (source.match(/call\(\d+, "/g) || []).length;
console.log(`\npool definition is ${block} lines, in one place; outbound call sites ${sites} and none of them changed — ` +
  `what moves the pool to the call site is ${(source.match(/poolFor\(\d+\)/g) || []).length} poolFor(port) expressions.`);
process.exit(0);
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1; node measure.mjs; kill $SP
```

```
40 concurrent calls per stream: notification 300 ms, catalog 5 ms; total slots 8 under every placement
the duration column is this run's measurement, rounded to 500 ms; the expected value is ceil(40/slots) x 300 ms
placement                          catalog done before   notification peak        catalog peak         duration ms
single shared pool                              0 / 40                   8                   1                2000
pool per dependency                            40 / 40                   4                   4                3000
pool per dependency + 8 skipped                40 / 40                  12                   4                2500

wrong sizing: slot counts under pool-per-dependency placement
notification slots / catalog slots   notification peak        catalog peak         duration ms
1 / 4                                                1                   4               12000
4 / 4                                                4                   4                3000
40 / 4                                              40                   4                 500
4 / 1                                                4                   1                3000

pool definition is 5 lines, in one place; outbound call sites 2 and none of them changed — what moves the pool to the call site is 2 poolFor(port) expressions.
```

#### Where the Pool Sits in the Code

The last line gives the placement's code cost: five lines, in one place. Neither outbound call
site changed at all; what moves the pool to the call site is two expressions of the form
`poolFor(port)`. This is where the bulkhead parts ways with the previous three patterns. A timeout
adds a behavior, a retry adds a loop, a breaker holds state — all change what the call site does.
A pool only says which resource the call uses; the call site's code stays the same.

The result: pool placement is nearly invisible in the source. Whether a call site uses the right
pool cannot be told by looking at the call site itself; you must check which expression was passed
in. Writing `false` on one line instead of `poolFor(8402)` is syntactically flawless.

#### Two Placements, the Same Eight Slots

The first two rows of the first table split the same eight slots two different ways.

Under the single shared pool, `catalog done before` is 0 / 40 — not one catalog call finished
before the first notification response arrived. The reason is in the notification-peak column:
the slow dependency holds all eight slots, and catalog can find at most one. Nothing is wrong with
catalog, it answers in five milliseconds; even so, all forty of its requests wait. **When the pool
is shared, one dependency's slowness becomes latency for a stream that has nothing to do with
it.**

Under pool per dependency, the same column is 40 / 40. Catalog finds its own four slots and
finishes all forty calls before notification gives even its first response. Notification's
concurrency drops from eight to four, and duration rises from 2000 ms to 3000 ms — the bulkhead's
cost, shown directly in the table. The pattern cuts off the contagion without ever depending on
detecting it: no threshold, no counter, no learning period. **The limit was set from the start.**

#### The Skipped Path

The third row is this lesson's most valuable measure. The pool configuration did not change — it
still declares four slots for notification. Eight of the forty calls do not use the pool at
all; they open their own sockets. The result: notification concurrency is 12 instead of 4.

The pool's limit says four, the dependency sees twelve. The excess is exactly the skipped-call
count, appearing nowhere in the configuration: someone reading the slot count sees four, someone
watching the dependency sees twelve. The previous two lessons' coverage problem is at its
sneakiest here, because pool placement does not change the call site's code, so it does not change
the gap either — a skipped call site looks nearly identical to a wrapped one.

Notably, the neighboring stream still comes out fine: `catalog done before` is still 40 / 40. The
skipped path breaches the notification pool's limit but not the catalog pool. **The bulkhead's
protection is that the leak stays inside its own bulkhead** — the leak still voids that bulkhead's
promise regardless.

#### Wrong Sizing

The second table shows both ends of pool sizing, and both leave the pattern useless, but in
different directions.

**One slot** queues the notification stream: concurrency 1, duration 12000 ms. Forty calls run
back to back while nothing is wrong with the dependency. Here the bulkhead produces not protection
but a bottleneck; the neighboring stream comes out fine, but the protected stream itself becomes
the slowest part of the system.

**Forty slots** is the opposite end: concurrency 40, duration 500 ms. The pool exists in the code,
is defined, is named, and binds nothing. This is the most dangerous configuration of all — it is
indistinguishable from a system with no pattern at all, yet anyone reading the source sees the
bulkhead set up.

The last row says the limit applies just as much on the neighboring side: when the catalog pool
drops to one, catalog concurrency becomes 1 too. **Every pool is a ceiling on its own stream** —
the bulkhead limits not only the slow dependency but the fast one too, and wrong sizing pays its
cost in both directions the same way.

#### Testability

```js
// test/pool.test.mjs — is the pool limit actually binding.
import { test } from "node:test";
import assert from "node:assert/strict";
import { request } from "node:http";
const run = (path) => new Promise((c) => request({ port: 8401, path, agent: false }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(JSON.parse(g)));
}).end());

test("pool per dependency saves the neighboring stream", async () => {
  assert.equal((await run("/run?placement=single&notif=4&cat=4&skip=0")).catalogBefore, 0);
  assert.equal((await run("/run?placement=separate&notif=4&cat=4&skip=0")).catalogBefore, 40);
});
test("a skipped call site exceeds the pool limit", async () => {
  assert.equal((await run("/run?placement=separate&notif=4&cat=4&skip=0")).notification, 4);
  assert.equal((await run("/run?placement=separate&notif=4&cat=4&skip=8")).notification, 12);   // 4 + 8
});
test("the pool limit binds only its own dependency", async () => {
  for (const notif of [1, 4, 40]) assert.equal((await run(`/run?placement=separate&notif=${notif}&cat=4&skip=0`)).catalog, 4);
});
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1
node --test --test-force-exit --test-reporter=tap test/pool.test.mjs |
  grep -E "^(ok|not ok|# (tests|pass|fail) )"
kill $SP
```

```
ok 1 - pool per dependency saves the neighboring stream
ok 2 - a skipped call site exceeds the pool limit
ok 3 - the pool limit binds only its own dependency
# tests 3
# pass 3
# fail 0
```

The bulkhead's testability differs from the previous three patterns: its behavior shows up not in
the caller's code but in the concurrency the dependency observes. A timeout or breaker test can be
written from the error the caller receives; a pool test can only be written by counting how many
connections the other side sees. The second test uses this directly: the pool says four, the
assertion is built on twelve, and the skipped call site becomes a number this way. The third pins
the limit's scope — whatever the notification pool is, catalog stays at four.

#### Summary

- The bulkhead's effect was measured in M19/K05 Resilience and Reliability; what is measured here
  is where the pool is defined. The definition is five lines, one place; neither call site
  changes, and what moves the pool is the `poolFor(port)` expression passed into the call.
- When the same eight slots are pooled together, the slow dependency holds all of them and catalog
  calls finished before the first notification response are 0 / 40; catalog concurrency drops to 1.
- Under pool per dependency, the same number is 40 / 40. The cost: notification concurrency from
  8 to 4, duration from 2000 ms to 3000 ms. Protection does not depend on a threshold or learning
  — it was set from the start.
- The eight of forty calls that bypass the pool deliver twelve concurrent connections while the
  configuration says four slots. The excess is exactly the skipped-call count, invisible in
  the configuration.
- Both ends of pool sizing leave the pattern useless: one slot drops concurrency to 1 and pushes
  duration to 12000 ms, forty slots binds nothing (concurrency 40, duration 500 ms) and looks like
  a bulkhead was set up.
- Every pool is a ceiling on its own stream: the catalog pool at 1 makes catalog concurrency 1
  too. A pool test can only be written by counting the concurrency the dependency observes.

#### Next Step

All four patterns did the same thing: placed a limit and refused the request beyond it. The
timeout left the waiter waiting, the retry multiplied and gave up, the breaker never made the
call, the bulkhead handed out no slot. What was left behind every time was the same thing — an
unmet request. Where that rejected request goes was never a question for any of these four
patterns; each contained the failure, none met it. The next lesson fills this gap: a request that
hits the limit gets an incomplete but usable response instead of a full one, so the system returns
something even while degrading.

### Graceful Degradation

Source: https://academia.sh/en/courses/observability/graceful-degradation

Placing graceful degradation in the code: gathering the core/optional field split into a schema object, comparing the shared wrapper at the call site as 7 and 12 lines, 6 of 14 call sites wrapped and 2 uncovered, and a fallback value that makes itself indistinguishable from a genuine zero.

The bulkhead pattern shut a broken dependency inside its own pool: once the pool fills, the call
fails fast, and other dependencies' pools stay unaffected. What gets saved is the downstream side.
Upstream, nothing has changed — the call site still gets an error object, the endpoint still
throws it upward, the client still comes back empty. Even with four of the five fields in the
panel response on hand, all five get thrown away because the fifth never arrived.

Graceful degradation closes this gap. Its effect was measured in the **Resilience and
Reliability** course: which policy saves how many requests, the cost of a failure-free day, and
what the pattern can do when a core dependency goes down were all counted there. What is measured
here is something else: **where this pattern sits in the code**, how many call sites it covers,
and which call sites it skips.

#### The Schema Object and Call Sites

The mechanism is five services of the split loan system. Each is a local `node:http` server, and
refuses the connection when its flag is turned off — for the caller, this means an error object
it has no control over.

```js
// degradation/services.mjs — upstream services of the split loan system.
// Each is a local node:http process; when the RUNNING flag turns off, the connection is refused.
import http from "node:http";

export const PORT = {};
export const RUNNING = { membership: 1, loan: 1, billing: 1, notification: 1, catalog: 1 };
export const BODY = {
  membership: { name: "N.A.", status: "active" },
  loan: { open: 3, overdue: 1 },
  billing: { balance: 12.5 },
  notification: { unread: 4 },
  catalog: { suggestions: ["K-1180"] },
};
const SERVERS = [];

export async function start() {
  for (const name of Object.keys(BODY)) {
    const s = http.createServer((req, res) => {
      if (RUNNING[name] === 0) { res.socket.destroy(); return; }
      res.writeHead(200, { "content-type": "application/json" });
      res.end(JSON.stringify(BODY[name]));
    });
    await new Promise((r) => s.listen(0, "127.0.0.1", r));
    PORT[name] = s.address().port;
    SERVERS.push(s);
  }
}
export function stop() { for (const s of SERVERS) s.close(); }

// Plain call site: throws an error when the service is down. Every unwrapped call behaves this way.
export async function call(name) {
  const y = await fetch(`http://127.0.0.1:${PORT[name]}/`);
  if (y.ok === false) throw new Error(`${name}: HTTP ${y.status}`);
  return y.json();
}
```

**RS10 — the panel's field split.** The member panel has five fields. `member` and `loan` are
core: without these two the page shows nothing. `balance`, `notification`, and `suggestions` are
optional, and each has a fallback value to stand in for it. **RS11 — the receipt and summary
endpoints were written before the panel** and make their own calls; this is typical of the
situation when the degradation pattern enters a system after the fact.

The split does not scatter through the code; it lives in a single schema object. Both placements
read the same schema.

```js
// degradation/panel.mjs — the panel response's schema and the pattern's two placements.
import { call } from "./services.mjs";

// RS10: the response's core/optional split lives in a schema object, not scattered through the code.
export const SCHEMA = {
  member: { service: "membership", core: 1 },
  loan: { service: "loan", core: 1 },
  balance: { service: "billing", core: 0, fallback: { balance: 0 } },
  notification: { service: "notification", core: 0, fallback: { unread: 0 } },
  suggestions: { service: "catalog", core: 0, fallback: { suggestions: [] } },
};

// Placement A — at the call site: every optional call carries its own try/catch and its own fallback.
export async function panelA() {
  const y = { member: await call("membership"), loan: await call("loan") };
  try { y.balance = await call("billing"); } catch { y.balance = { balance: 0 }; }
  try { y.notification = await call("notification"); } catch { y.notification = { unread: 0 }; }
  try { y.suggestions = await call("catalog"); } catch { y.suggestions = { suggestions: [] }; }
  return y;
}

// Placement B — shared wrapper: the fallback is read from the schema, the dropped field is marked in the response.
export async function optional(field, missing) {
  const t = SCHEMA[field];
  try { return await call(t.service); } catch { missing.push(field); return t.fallback; }
}
export async function panelB() {
  const missing = [];
  const y = { member: await call("membership"), loan: await call("loan") };
  y.balance = await optional("balance", missing);
  y.notification = await optional("notification", missing);
  y.suggestions = await optional("suggestions", missing);
  return { ...y, missing };
}

// The other two endpoints that touch the same services. Both were written before the panel.
export async function receipt() {
  const u = await call("membership");
  const b = await call("billing");
  return { member: u.name, balance: b.balance };
}
export async function summary() {
  const o = await call("loan");
  const b = await call("notification");
  return { open: o.open, unread: b.unread };
}
```

#### Source Scan and Run

The measurement has two parts. First the source text is scanned and every call site is
classified: does it reach a core service, an optional service that is wrapped, or an optional
service that is not wrapped. The third case is the uncovered call site. Then the same code runs
under failure scenarios, and the scan is compared against the run.

```js
// degradation/measurement.mjs — first the source is counted, then the same code runs under failure scenarios.
import { readFileSync } from "node:fs";
import { start, stop, RUNNING, BODY } from "./services.mjs";
import { SCHEMA, panelA, panelB, receipt, summary } from "./panel.mjs";

const OPTIONAL = new Set(Object.values(SCHEMA).filter((t) => !t.core).map((t) => t.service));
const sites = [], lines = {};            // every call site: which function, which service, wrapped or not
let current = null;
for (const s of readFileSync(new URL("./panel.mjs", import.meta.url), "utf8").split("\n")) {
  const head = s.match(/^export async function (\w+)/);
  if (head) current = head[1];
  if (current === null) continue;
  if (s.trim() !== "") lines[current] = (lines[current] ?? 0) + 1;
  for (const m of s.matchAll(/call\("(\w+)"\)/g))              // plain call site with the name written out
    sites.push({ fn: current, service: m[1], wrapped: s.includes("try {") });
  for (const m of s.matchAll(/optional\("(\w+)"/g))            // call site handed to the wrapper
    sites.push({ fn: current, service: SCHEMA[m[1]].service, wrapped: true });
  if (s.startsWith("}")) current = null;
}
const classOf = (y) => (OPTIONAL.has(y.service) ? (y.wrapped ? "wrapped" : "uncovered") : "core");

console.log("-- source scan --");
console.log(`${"function".padEnd(13)}${"lines".padStart(6)}${"call sites".padStart(11)}${"core".padStart(9)}${"wrapped".padStart(11)}${"uncovered".padStart(10)}`);
for (const i of [...new Set(sites.map((y) => y.fn))]) {
  const t = sites.filter((y) => y.fn === i);
  const n = (k) => String(t.filter((y) => classOf(y) === k).length);
  console.log(`${i.padEnd(13)}${String(lines[i]).padStart(6)}${String(t.length).padStart(11)}${n("core").padStart(9)}${n("wrapped").padStart(11)}${n("uncovered").padStart(10)}`);
}
console.log(`total ${sites.length} call sites; ${sites.filter((y) => classOf(y) === "wrapped").length} wrapped, ${sites.filter((y) => classOf(y) === "uncovered").length} uncovered`);
console.log(`placement A ${lines.panelA} lines, placement B ${lines.panelB}+${lines.optional}=${lines.panelB + lines.optional} lines\n`);

await start();
const ENDPOINT = { panelA, panelB, receipt, summary };
const SCENARIO = [["all healthy", []], ["billing down", ["billing"]],
  ["billing+notification+catalog", ["billing", "notification", "catalog"]], ["membership down", ["membership"]]];
console.log("-- run: fate of the call sites reaching the down service --");
console.log(`${"scenario".padEnd(31)}${"touched".padStart(10)}${"fell back".padStart(12)}${"threw".padStart(14)}${"endpoints answering".padStart(20)}`);
const body = {};
for (const [name, down] of SCENARIO) {
  for (const s of Object.keys(RUNNING)) RUNNING[s] = down.includes(s) ? 0 : 1;
  let answered = 0;
  for (const [ep, f] of Object.entries(ENDPOINT)) {
    try { body[`${name}|${ep}`] = JSON.stringify(await f()); answered += 1; } catch { /* endpoint fell */ }
  }
  const d = sites.filter((y) => down.includes(y.service));
  const fallenBack = d.filter((y) => classOf(y) === "wrapped").length;
  console.log(`${name.padEnd(31)}${String(d.length).padStart(10)}${String(fallenBack).padStart(12)}${String(d.length - fallenBack).padStart(14)}${`${answered}/4`.padStart(20)}`);
}

for (const s of Object.keys(RUNNING)) RUNNING[s] = 1;
BODY.billing.balance = 0;                            // a member whose balance is genuinely zero
const zeroA = JSON.stringify(await panelA()), zeroB = JSON.stringify(await panelB());
const downA = body["billing down|panelA"], downB = body["billing down|panelB"];
console.log("\n-- the fallback's silent falsehood: service down / balance genuinely 0 --");
console.log(`A down: ${downA}`);
console.log(`A zero: ${zeroA}`);
console.log(`same body: ${downA === zeroA}`);
console.log(`B down: ${downB.slice(downB.indexOf('"balance"'))}`);
console.log(`B zero: ${zeroB.slice(zeroB.indexOf('"balance"'))}`);
console.log(`same body: ${downB === zeroB}; the marker costs ${downB.length - downA.length} bytes, ${(((downB.length - downA.length) / downA.length) * 100).toFixed(1)}% of the response`);
stop();
```

```
-- source scan --
function      lines call sites     core    wrapped uncovered
panelA            7          5        2          3         0
panelB            8          5        2          3         0
receipt           5          2        1          0         1
summary           5          2        1          0         1
total 14 call sites; 6 wrapped, 2 uncovered
placement A 7 lines, placement B 8+4=12 lines

-- run: fate of the call sites reaching the down service --
scenario                          touched   fell back         threw endpoints answering
all healthy                             0           0             0                 4/4
billing down                            3           2             1                 3/4
billing+notification+catalog            8           6             2                 2/4
membership down                         3           0             3                 1/4

-- the fallback's silent falsehood: service down / balance genuinely 0 --
A down: {"member":{"name":"N.A.","status":"active"},"loan":{"open":3,"overdue":1},"balance":{"balance":0},"notification":{"unread":4},"suggestions":{"suggestions":["K-1180"]}}
A zero: {"member":{"name":"N.A.","status":"active"},"loan":{"open":3,"overdue":1},"balance":{"balance":0},"notification":{"unread":4},"suggestions":{"suggestions":["K-1180"]}}
same body: true
B down: "balance":{"balance":0},"notification":{"unread":4},"suggestions":{"suggestions":["K-1180"]},"missing":["balance"]}
B zero: "balance":{"balance":0},"notification":{"unread":4},"suggestions":{"suggestions":["K-1180"]},"missing":[]}
same body: false; the marker costs 22 bytes, 13.2% of the response
```

#### Two Placements and the Uncovered Call Site

Both placements do the same job and produce the same result at the panel endpoint: each wraps the
same three optional calls, and each produces a response when billing goes down. The line cost
differs — the call-site placement is 7 lines, the shared-wrapper placement is 8 plus 4, 12 lines
total. At first glance, the call-site placement looks cheaper.

What makes the difference is not the line count but **where the fallback is written.** In the
call-site placement, the expression `{ balance: 0 }` is buried inside a `catch` block, with no
link at all to the `fallback` field in the schema. The two can be edited separately and drift
apart silently. In the shared wrapper, the fallback is read from a single place, the schema; the
decision on whether a field is core lives there too.

The real measure is in the second table. When the billing service goes down, three call sites
reach it: the two placements in the panel and the call inside the receipt endpoint. Two of them
return a fallback, one still throws — the receipt endpoint cannot answer, three of the four
endpoints stay up. When three services go down at once, the touched call sites rise to eight, six
return a fallback, two throw, and the summary endpoint falls too.

The degradation pattern settled into the panel, not into the system. The schema object declares
the `balance` field optional, but that declaration only binds the code path that reads it. The
receipt endpoint calls the same service, gets the same error, and keeps throwing it upward. **The
pattern's coverage is not where it was set up, it is the set of places where it is called.** The
source scan counts this set as six wrapped and two uncovered call sites; the two uncovered call
sites turn into two fallen endpoints at run time.

The last row shows another side of the split. When the membership service goes down, all three
call sites reaching it are classed as core, and none is wrapped — this is not a shortcoming, it is
the design itself. Yet the summary endpoint stays up, because membership is not a dependency for
it. **Coreness is a property of the endpoint, not of the field:** the same service is core at one
endpoint and absent entirely at another.

#### The Fallback's Silent Falsehood

The last section measures a flaw the pattern produces itself. The body the call-site placement
produces while billing is down is compared against the body of a member whose balance is
genuinely zero on a healthy day: **byte for byte identical.** The value `balance: 0` means two
different things, and the client has no way to tell them apart. The member's screen says you have
no balance due, while in reality there is a twelve-fifty balance, and no one sees an error.

This is the one point where the degradation pattern can be **worse** than throwing code. A thrown
error is visible: the endpoint falls, it gets logged, it shows up in a counter. A fallback value
is invisible, because it sits inside a successful response. The wrapped placement fixes this: the
`missing` list carries which field was filled with a fallback right in the response itself, and
the two bodies diverge. The cost is 22 bytes, 13.2 percent of this response.

These 22 bytes are not just one extra field, they are the pattern's **observable form.** A
degradation carrying no marker is invisible to every tool built in this course's first two topics:
no log is produced, no counter moves, no indicator shifts. Without a marker on the fallback value,
degradation cannot be measured; and degradation that cannot be measured is, before long, mistaken
for normal.

#### Summary

- Graceful degradation settles into the code as a schema object: which field is core, which is
  optional, and what its fallback is all live in one place. The pattern's effect was measured in
  the Resilience and Reliability course; the measure here is the placement itself.
- Two placements were compared: try/catch at the call site is 7 lines, the shared wrapper is 12.
  What the line difference buys is the fallback being read from the schema instead of buried in a
  catch block.
- The source scan found 14 call sites: 6 wrapped, 2 uncovered, the rest core calls. The two
  uncovered call sites sit in the receipt and summary endpoints, both written before the panel.
- The run confirmed the scan: when billing went down, 2 of 3 call sites returned a fallback, 1
  threw, and one of four endpoints fell. When three services went down at once, 6 of 8 call sites
  returned a fallback and two endpoints fell.
- An unmarked fallback silently produces false data: the body produced while the service is down
  is byte for byte identical to a member's body whose balance is genuinely zero. The `missing`
  list separates the two and adds 22 bytes, 13.2 percent, to the response.
- Coreness is a property of the endpoint, not the field: the membership service is core for the
  panel and the receipt, and absent entirely for the summary endpoint.

#### Next Step

The five patterns so far all look in the same direction: what this system does when a downstream
dependency breaks. The timeout cuts the waiting short, the retry asks again, the breaker stops
asking, the bulkhead separates resources, degradation shrinks the response. All of them have the
same subject: **the caller.**

The other side of the same system has never been addressed. These services do not only call, they
also get called, and there is no limit on how many requests the calling side can send. One
client's badly written loop can send thousands of requests a minute to a single endpoint, and none
of the patterns here can stop it — all of them regulate what happens to a request that has already
arrived. The next lesson places the limit on the **accepting side**, and measures: which layer of
the code the limiter sits in, whom the limit applies to, which paths stay outside it, and what
happens when the wrong key is chosen.

### Rate Limiting and Throttling

Source: https://academia.sh/en/courses/observability/rate-limiting-and-throttling

The rate limiter's placement and key in the code: the same limiter seeing 2 of 8 path-access pairs at the gateway and 6 in the service's middle layer, 2 pairs falling outside every placement, and a single parameter of 20 turning into a cap ranging from 20 to 100,000 depending on the key.

The subject of the previous five patterns was always the caller: what happens when this system
calls another service was what got regulated. The panel's optional fields were filled with
fallbacks, call sites that throw were counted, and two endpoints outside the pattern's coverage
were found. All of them looked downstream.

The upstream side was never regulated. The loan service does not only call, it also gets called,
with no limit on how many calls it will accept. When an overnight batch job falls into a badly
written loop, the timeout, retry, and breaker in front of that service do nothing useful — all of
them regulate what happens to a request already accepted. The limit is placed on the accepting
side and its name is **rate limiting**; the form that delays instead of rejecting is called
**throttling**.

The limiter's effect and a comparison of algorithms — fixed window, sliding window, token bucket —
were measured in **Resilience and Reliability** and in a case study on rate-limiter design. Here
the algorithm is held fixed and two other questions are asked: **which layer of the code the
limiter sits in**, and **whom the limit applies to**.

#### Three Layers, One Limiter

The limiter is a single function; it can be placed in three spots. At the gateway, it sits in one
place in front of every service. In a service's middle layer, it protects only that service, but
every one of its routes. At the call site, it regulates only the call it wraps — this third case
is throttling on the way out, and its coverage is counted the same way as the wrapped-call-site
measure from the previous lesson. The first two are built here.

```js
// limit/layer.mjs — the limiter and the two layers that can host it: gateway and middle layer.
import http from "node:http";

// Fixed window. RS12: the window was chosen longer than the run; algorithm choice is not measured here.
export function limiter(key, limit = 20, window = 60_000) {
  const bucket = new Map();
  let start = Date.now();
  const s = { passed: 0, rejected: 0, member: new Map(), seen: 0 };
  return { s, bucket, allow(i) {
    if (Date.now() - start >= window) { bucket.clear(); start = Date.now(); }
    s.seen += 1;
    const n = (bucket.get(key(i)) ?? 0) + 1;
    bucket.set(key(i), n);
    const t = s.member.get(i.member) ?? { passed: 0, rejected: 0 };
    if (n <= limit) { t.passed += 1; s.passed += 1; } else { t.rejected += 1; s.rejected += 1; }
    s.member.set(i.member, t);
    return n <= limit;
  } };
}

const parseRequest = (req) => ({ path: req.url, client: req.headers["x-client"], member: req.headers["x-member"] });
const respond = (res, code, g) => { res.writeHead(code, { "content-type": "application/json" }); res.end(JSON.stringify(g)); };

// Service: covers every one of its routes if given a middle layer, none if not.
export function service(routes, middleLayer) {
  return http.createServer((req, res) => {
    const i = parseRequest(req);
    if (routes.includes(i.path) === false) return respond(res, 404, { path: i.path });
    if (middleLayer && middleLayer.allow(i) === false) return respond(res, 429, { limit: 1 });
    respond(res, 200, { path: i.path });
  });
}

// Gateway: the limit applies only if the written prefix matches. Other prefixes pass through.
export function gateway(limit, prefix, target) {
  return http.createServer(async (req, res) => {
    const i = parseRequest(req);
    if (i.path.startsWith(prefix) && limit.allow(i) === false) return respond(res, 429, { limit: 1 });
    const y = await fetch(`http://127.0.0.1:${target(i.path)}${i.path}`,
      { headers: { "x-client": i.client ?? "", "x-member": i.member ?? "" } });
    respond(res, y.status, await y.json());
  });
}

export async function listen(s) { await new Promise((r) => s.listen(0, "127.0.0.1", r)); return s.address().port; }
export async function sendRequest(port, path, client, member) {
  const y = await fetch(`http://127.0.0.1:${port}${path}`, { headers: { "x-client": client, "x-member": member } });
  await y.json();
  return y.status;
}
```

The measurement has two parts. First, coverage: four paths, two access routes, and two placements
are set up; for every request, which limiter sees it is counted. Then, the key: the placement with
wider coverage is held fixed, and four separate key choices run on the same request stream.

```js
// limit/measurement.mjs — first coverage (which path is counted at which layer), then key selection.
import { limiter, service, gateway, listen, sendRequest } from "./layer.mjs";

const LOAN = ["/loan/borrow", "/loan/list", "/loan-bulk/list"];
const CATALOG = ["/catalog/search"];
const PREFIX = "/loan/";                     // the single prefix written into the gateway

// Two separate setups: in A the limit is only at the gateway, in B only in the service's middle layer.
const gwA = limiter(() => "global", 1e9);
const midB = limiter(() => "global", 1e9);
const setup = async (mid) => {
  const o = await listen(service(LOAN, mid)), c = await listen(service(CATALOG, null));
  return { o, c, target: (u) => (u.startsWith("/catalog") ? c : o) };
};
const A = await setup(null), B = await setup(midB);
const gwPortA = await listen(gateway(gwA, PREFIX, A.target));
const gwPortB = await listen(gateway(limiter(() => "global", 1e9), "/none/", B.target));

console.log("-- coverage: which placement counts a request --");
console.log(`${"path".padEnd(20)}${"access".padEnd(11)}${"gateway".padStart(10)}${"middle layer".padStart(14)}`);
const coverage = { gateway: 0, middle: 0, none: 0, total: 0 };
for (const path of [...LOAN, ...CATALOG]) {
  for (const access of ["gateway", "direct"]) {
    const g0 = gwA.s.seen, a0 = midB.s.seen;
    await sendRequest(access === "direct" ? (path.startsWith("/catalog") ? A.c : A.o) : gwPortA, path, "measure", "u0");
    await sendRequest(access === "direct" ? (path.startsWith("/catalog") ? B.c : B.o) : gwPortB, path, "measure", "u0");
    const g = gwA.s.seen > g0, a = midB.s.seen > a0;
    coverage.total += 1; if (g) coverage.gateway += 1; if (a) coverage.middle += 1; if (!g && !a) coverage.none += 1;
    console.log(`${path.padEnd(20)}${access.padEnd(11)}${(g ? "counted" : "skipped").padStart(10)}${(a ? "counted" : "skipped").padStart(14)}`);
  }
}
console.log(`of ${coverage.total} path-access pairs, the gateway sees ${coverage.gateway}, the middle layer sees ${coverage.middle}; ` +
  `${coverage.none} pairs are outside every placement\n`);

// RS13: the overnight batch job runs uninterrupted first, the mobile client cycles through its members, web arrives last.
const STREAM = [["batch", "u5", 40], ["mobile", "u1", 12], ["mobile", "u2", 8], ["mobile", "u3", 4], ["web", "u4", 6]];
const PLAN = [];
for (const [client, member, n] of STREAM)
  for (let i = 0; i < n; i += 1) PLAN.push({ client, member, path: client === "mobile" ? LOAN[i % 3] : LOAN[1] });

const KEY = [["fixed", () => "global"], ["client", (i) => i.client],
  ["member", (i) => i.member], ["client+path", (i) => `${i.client} ${i.path}`]];
const MEMBER = ["u1", "u2", "u3", "u4", "u5"], LIMIT = 20, ACTIVE_MEMBERS = 5000;
console.log(`-- key selection: ${PLAN.length} requests, limit per bucket ${LIMIT}, placement middle layer --`);
console.log(`${"key".padEnd(12)}${"buckets".padStart(8)}${"cap".padStart(7)}${"passed".padStart(8)}${"rejected".padStart(9)}` +
  MEMBER.map((u) => `${u} p/r`.padStart(11)).join(""));
for (const [name, f] of KEY) {
  const lim = limiter(f, LIMIT);
  const port = await listen(service(LOAN, lim));
  for (const p of PLAN) await sendRequest(port, p.path, p.client, p.member);
  console.log(`${name.padEnd(12)}${String(lim.bucket.size).padStart(8)}${String(lim.bucket.size * LIMIT).padStart(7)}` +
    `${String(lim.s.passed).padStart(8)}${String(lim.s.rejected).padStart(9)}` +
    MEMBER.map((u) => `${lim.s.member.get(u).passed}/${lim.s.member.get(u).rejected}`.padStart(11)).join(""));
}
console.log(`\nthe one written parameter is ${LIMIT}; the real cap is multiplied by the bucket count. If the key is member, the ` +
  `bucket count is the member count: RS14's ${ACTIVE_MEMBERS} active members mean the same configuration is ${ACTIVE_MEMBERS * LIMIT} requests/window.`);
process.exit(0);
```

```
-- coverage: which placement counts a request --
path                access        gateway  middle layer
/loan/borrow        gateway       counted       counted
/loan/borrow        direct        skipped       counted
/loan/list          gateway       counted       counted
/loan/list          direct        skipped       counted
/loan-bulk/list     gateway       skipped       counted
/loan-bulk/list     direct        skipped       counted
/catalog/search     gateway       skipped       skipped
/catalog/search     direct        skipped       skipped
of 8 path-access pairs, the gateway sees 2, the middle layer sees 6; 2 pairs are outside every placement

-- key selection: 70 requests, limit per bucket 20, placement middle layer --
key          buckets    cap  passed rejected     u1 p/r     u2 p/r     u3 p/r     u4 p/r     u5 p/r
fixed              1     20      20       50       0/12        0/8        0/4        0/6      20/20
client             3     60      46       24       12/0        8/0        0/4        6/0      20/20
member             5    100      50       20       12/0        8/0        4/0        6/0      20/20
client+path        5    100      50       20       12/0        8/0        4/0        6/0      20/20

the one written parameter is 20; the real cap is multiplied by the bucket count. If the key is member, the bucket count is the member count: RS14's 5000 active members mean the same configuration is 100000 requests/window.
```

#### The Path Outside the Limit

The gateway placement sees two of the eight path-access pairs. This has nothing to do with the
location itself; it comes down to a single line at the gateway: the `/loan/` prefix. The loan
service's third path is named `/loan-bulk/list`, which does not match the prefix. The path was
added later, the prefix list was never updated, and the limiter never saw it. Nothing in the code
raises an error; the rule stays silently incomplete.

The second, heavier gap is in the access route. The gateway can only see a request that passes
through it. Every request reaching the service's own listener directly — another internal
service's call, a maintenance script, a health check — never touches the limiter. All four
`direct` rows show `skipped` in the gateway column. **A gateway is not a boundary, it is a gate**;
walk around the gate and there is no boundary.

The middle layer sees six pairs, because it wraps the entire handler table: whatever the path's
name, wherever the request comes from, every request reaching that service gets counted. Its cost
is narrow coverage — only that one process. Catalog is a separate process with no middle layer of
its own; `/catalog/search` never enters any limiter's view under either access route. Even with
both layers built together, these two pairs would stay open, because the gap is not in the layer,
it is in the **process the limiter was never built into.**

The throttling form does not change this table. Even if the limiter queues and delays the request
instead of returning 429, the requests it can queue are still only the ones it sees. The choice
between rejecting and delaying is about the request's fate; the coverage question is independent
of it, and comes first.

#### Whom the Key Limits

The second table runs the same placement, the same limit, and the same seventy-request stream
through four separate keys. The stream is **RS13**: the overnight batch job starts with forty
requests, the mobile client cycles through its three members in turn, and the web client arrives
last.

**A fixed key** — the key function returns the same value for every request — pools every client
into one bucket. The batch job exhausts it with its first twenty requests, and the remaining fifty
are rejected: all four other members end up with zero passed requests. Not one of the mobile
client's twelve requests or the web client's six reaches the system. This is worse than no limiter
at all: without it, the batch job's load would only have slowed the service down; with it, the
batch job shuts everyone else out entirely. **The wrong key turns a protection tool into a tool of
contagion.**

**The client key** splits the bucket into three and confines the batch job to its own quota;
passed requests rise from twenty to forty-six. But behind the mobile client sit three separate
members sharing one bucket. The first two exhaust the mobile quota, and all four of the third
member's requests get rejected — a member who sent four requests and had none pass, for behavior
that belonged to someone else. **The client key feeds users behind a shared client on each
other's quota.**

**The member key** fixes this: all three mobile members pass every one of their requests, and only
the batch job gets limited. In the measured stream, **client+path** gives the same result, because
the mobile client's twenty-four requests spread across three paths and no bucket reaches the
limit.

#### The Real Cap

The last two columns show the sneakiest side of the configuration surface. The one number written
in the configuration is 20. The upper bound the system actually accepts is **20 × bucket count**:
20 under the fixed key, 60 under the client key, 100 under the member key.

The bucket count is not a configuration value; it comes from the data. Under the member key, the
bucket count equals the active-member count. **RS14** assumes five thousand active members here;
the same value of 20 then means one hundred thousand requests per window. Someone reading the
configuration assumes the service is limited to twenty requests. **Key choice is more decisive
than the limit value itself**, and since the two sit in separate places, their product is written
nowhere.

This also explains why the limiter is not enough alone. A per-member limit throttles a single
member; it does not protect the service's total capacity. The limit that protects total capacity
carries a fixed key, with the cost already seen in the first table. Both keys need to be built in
together — two bucket sets, two limit values, two reasons for rejection.

#### Summary

- The limiter's effect and algorithm choice were measured elsewhere; what is measured here is
  placement and key. The same limiter function was built into the gateway and the middle layer.
- The gateway sees 2 of 8 path-access pairs, the middle layer sees 6. Two causes: the `/loan/`
  prefix does not catch the later-added `/loan-bulk/list` path, and no direct request passes
  through the gateway.
- 2 pairs sit outside every placement: catalog is a separate process the limiter was never built
  into. The gap is not in the choice of layer, it is in the missing process.
- A fixed key pools every client into one bucket: the batch job exhausts it with twenty requests,
  and all thirty of the other members' requests are rejected — worse than no limiter at all.
- The client key feeds members behind a shared client on each other's quota: a member who sent
  four requests was rejected on all four. Under the member key, that member passed all four.
- The one written parameter is 20, the real cap is 20 × bucket count: 20 fixed, 60 client, 100
  member, 100,000 at RS14's five thousand members. Bucket count comes from data, not config.

#### Next Step

Six patterns were built in this topic, each written against a specific way things break: no
response arrives, the response is delayed, a dependency stays down, resources run out, a field is
missing, too many calls come in. Every pattern has a place in the code, a coverage, and call sites
outside that coverage.

Which failure mode each pattern corresponds to, though, was never spelled out. When a dependency
refuses a connection, answers slowly, returns an empty body, or returns a wrong value that looks
valid, application code sees each differently — one as an error object, one as a timeout, one with
no symptom at all. The next lesson produces each of these appearances one by one, writes down how
each looks at the call site, matches which pattern catches which, and counts what is left over:
the failure modes no pattern meets.

### Failure Modes

Source: https://academia.sh/en/courses/observability/failure-modes

What failure modes correspond to in application code: the same endpoint breaking eight separate ways, producing five different error identities at the call site, the pattern stack seeing five of them and missing three entirely, and the same zero value reaching the call site under six modes.

Six patterns were built in this topic. Each was written against a way of breaking, each has a
place in the code, and each had its uncovered call sites counted. What the patterns do is clear;
which failure they correspond to was never spelled out one by one.

The failure-mode vocabulary — how many ways a dependency can break, and each one's effect on the
system — was built in **Resilience and Reliability**. The question here is different: **what does
each mode appear as in application code.** Does the call site get an error object, a timeout, an
empty body, or a valid response showing no symptom at all. This is the topic's closing lesson, and
it counts the patterns' total coverage.

#### Producing Eight Modes in Code

**RS15 — eight modes.** A single billing endpoint breaks eight separate ways: connection refusal,
a delayed response, a server error, an empty body, a body cut off partway through, a field with
the right shape but wrong type, a valid and correctly typed but wrong value, and slowness just
under the timeout. The last three all return HTTP 200 and valid JSON. The actual balance is 12.5,
and the late fee adds five to it.

**RS16 — the stack's settings.** Timeout 120 ms, retry count 2, breaker threshold 3 consecutive
failures, 5 calls per mode. These values were measured in earlier lessons; here they are held
fixed.

```js
// mode/mode.mjs — a dependency's eight failure modes and the patterns stacked on top.
import http from "node:http";

export const ACTUAL = 12.5, TIMEOUT = 120, RETRIES = 2, THRESHOLD = 3, CALLS = 5;
const send = (res, g) => { res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify(g)); };

// Every mode breaks the same endpoint a different way. The last two return HTTP 200 and valid JSON.
export const MODE = {
  "connection-refused": null,                                 // no listener
  "timeout": (res) => setTimeout(() => send(res, { balance: ACTUAL }), 400),
  "server-error": (res) => { res.writeHead(500, { "content-type": "application/json" }); res.end("{}"); },
  "empty-body": (res) => { res.writeHead(200, { "content-type": "application/json" }); res.end(""); },
  "truncated-body": (res) => {
    res.writeHead(200, { "content-type": "application/json", "content-length": "20" });
    res.write('{"balance":'); res.socket.destroy();
  },
  "wrong-type": (res) => send(res, { balance: String(ACTUAL.toFixed(2)) }),
  "silent-wrong": (res) => send(res, { balance: 0 }),
  "slow-under-limit": (res) => setTimeout(() => send(res, { balance: ACTUAL }), 90),
};

export function breaker() {
  let consecutive = 0;
  return { isOpen: () => consecutive >= THRESHOLD, success() { consecutive = 0; }, failure() { consecutive += 1; } };
}

// Timeout + retry + circuit breaker in a single wrapper. Which one triggers is written to the trail.
export async function guarded(port, k, trail) {
  if (k.isOpen()) { trail.add("breaker"); throw new Error("circuit open"); }
  let last;
  for (let d = 1; d <= RETRIES; d += 1) {
    try {
      const y = await fetch(`http://127.0.0.1:${port}/`, { signal: AbortSignal.timeout(TIMEOUT) });
      if (y.ok === false) { const e = new Error(`HTTP ${y.status}`); e.name = `HTTP ${y.status}`; throw e; }
      const g = await y.json();
      k.success();
      return g;
    } catch (e) {
      last = e;
      trail.add(e.name === "TimeoutError" ? "expired" : "error");
      if (d < RETRIES) trail.add("retry");
    }
  }
  k.failure();
  throw last;
}

export const symptom = (e) => (e.cause?.code ? `${e.name}/${e.cause.code}` : e.name);

// Call site: the degradation fallback lives here. The fallback value is the same as what silent-wrong returns.
export async function callSite(port, k, trail) {
  try { return { value: await guarded(port, k, trail), fallback: 0, label: "HTTP 200" }; }
  catch (e) { trail.add("fallback"); return { value: { balance: 0 }, fallback: 1, label: symptom(e) }; }
}
```

#### The Pattern Stack and the Measurement

The measurement produces each mode in turn and makes five calls. Four things get recorded: the
error identity that shows at the call site, the patterns that trigger, how many calls pass without
falling back, and the balance value reaching the call site along with the late fee calculated from
it.

```js
// mode/measurement.mjs — produces each mode one at a time, writes what shows at the call site and which pattern triggers.
import http from "node:http";
import { MODE, ACTUAL, TIMEOUT, RETRIES, THRESHOLD, CALLS, breaker, callSite } from "./mode.mjs";

let current = "silent-wrong";
const s = http.createServer((req, res) => MODE[current](res));
await new Promise((r) => s.listen(0, "127.0.0.1", r));
const PORT = s.address().port;
const dead = http.createServer(() => {});           // a port whose listener has been closed
await new Promise((r) => dead.listen(0, "127.0.0.1", r));
const DEAD_PORT = dead.address().port;
await new Promise((r) => dead.close(r));

console.log(`run-independent: timeout ${TIMEOUT} ms, retries ${RETRIES}, breaker threshold ${THRESHOLD}, ${CALLS} calls per mode`);
console.log(`actual balance ${ACTUAL}; the late-fee calculation should be balance + 5 = ${ACTUAL + 5}\n`);
console.log(`${"mode".padEnd(20)}${"seen at call site".padEnd(27)}${"pattern triggered".padEnd(33)}${"passed".padStart(6)}${"balance".padStart(9)}${"balance+5".padStart(11)}`);

const rows = [];
for (const mode of Object.keys(MODE)) {
  current = mode;
  const k = breaker(), trail = new Set();
  let passed = 0, first = null;
  for (let c = 0; c < CALLS; c += 1) {
    const y = await callSite(mode === "connection-refused" ? DEAD_PORT : PORT, k, trail);
    if (y.fallback === 0) passed += 1;
    first ??= y;                                   // the symptom is read from the first call
  }
  const seen = trail.size > 0, correct = first.value.balance === ACTUAL;
  rows.push({ mode, seen, correct, trail });
  console.log(`${mode.padEnd(20)}${first.label.padEnd(27)}${([...trail].join("+") || "none").padEnd(33)}` +
    `${`${passed}/${CALLS}`.padStart(6)}${String(first.value.balance).padStart(9)}${String(first.value.balance + 5).padStart(11)}`);
}
s.close();

const unnoticed = rows.filter((x) => x.seen === false);
console.log(`\n${rows.length - unnoticed.length} of ${rows.length} modes triggered at least one pattern, ` +
  `${unnoticed.length} triggered none: ${unnoticed.map((x) => x.mode).join(", ")}`);
const wrong = rows.filter((x) => x.correct === false);
console.log(`the actual value never reached the call site in ${wrong.length}/${rows.length} modes: a pattern reported it in ${wrong.filter((x) => x.seen).length} modes, ` +
  `${wrong.filter((x) => !x.seen).length} modes carried no report at all`);
console.log("\ncoverage per pattern: " + ["expired", "error", "retry", "breaker", "fallback"]
  .map((kl) => `${kl} ${rows.filter((x) => x.trail.has(kl)).length}/${rows.length}`).join(", "));
console.log(`run-independent upper bound: a failed call waits at most ${RETRIES * TIMEOUT} ms, 0 ms once the circuit is open`);
```

```
run-independent: timeout 120 ms, retries 2, breaker threshold 3, 5 calls per mode
actual balance 12.5; the late-fee calculation should be balance + 5 = 17.5

mode                seen at call site          pattern triggered                passed  balance  balance+5
connection-refused  TypeError/ECONNREFUSED     error+retry+fallback+breaker        0/5        0          5
timeout             TimeoutError               expired+retry+fallback+breaker      0/5        0          5
server-error        HTTP 500                   error+retry+fallback+breaker        0/5        0          5
empty-body          SyntaxError                error+retry+fallback+breaker        0/5        0          5
truncated-body      TypeError/UND_ERR_SOCKET   error+retry+fallback+breaker        0/5        0          5
wrong-type          HTTP 200                   none                                5/5    12.50     12.505
silent-wrong        HTTP 200                   none                                5/5        0          5
slow-under-limit    HTTP 200                   none                                5/5     12.5       17.5

5 of 8 modes triggered at least one pattern, 3 triggered none: wrong-type, silent-wrong, slow-under-limit
the actual value never reached the call site in 7/8 modes: a pattern reported it in 5 modes, 2 modes carried no report at all

coverage per pattern: expired 1/8, error 4/8, retry 5/8, breaker 5/8, fallback 5/8
run-independent upper bound: a failed call waits at most 240 ms, 0 ms once the circuit is open
```

#### Five Identities at the Call Site, Two Paths

The second column gives the eight modes' identities on the application side, and they do not
resemble each other. Connection refusal produces a `TypeError`, its real cause only showing up in
`cause.code`. Timeout is a separate class, `TimeoutError`. A server error throws nothing at all —
a resolved response comes back, and checking the status code is left to the caller; skip that
check, and a 500 counts as success. An empty body is a `SyntaxError`, a **parsing** error: nothing
is wrong at the network layer, the body itself is invalid. A truncated body is again a `TypeError`,
with a different `cause.code`.

Five separate identities, read from five separate places: the error class, the error cause, the
status code, the parse result. **Recognizing a failure mode in code requires looking at all four
of these sources at once.**

The behavior in the last columns, by contrast, is uniform. The same four patterns trigger on all
five of those modes, and all five calls fall back. The pattern stack does not distinguish between
modes; it reduces every one to a single category: **the call failed.** This is not a flaw in the
patterns, it is the limit of their scope — but the difference between connection refusal, which
says the service is down, and an empty body, which says it produced a broken response, disappears
once it passes through the stack. Only a log entry recording that error identity can carry the
difference forward.

The coverage-per-pattern row puts a number on this. Timeout triggers on only one of eight modes:
the delayed response. The remaining four failing modes already throw an error on their own and
never need it. Retry, breaker, and the degradation fallback each trigger on five modes. The most
expensive pattern — the breaker — is also the most commonly triggered; timeout, narrowest in
coverage, is the only one whose absence leaves a call hanging indefinitely. **Coverage width and
importance are not the same thing.**

#### The Modes No Pattern Meets

The bottom three rows are this lesson's real result. For wrong type, silent wrong value, and
below-limit slowness, the `pattern triggered` column reads `none`, and all five calls pass. The
pattern stack does not see these modes, because every pattern triggers on a symptom in the
transport layer: the connection couldn't be made, time ran out, the status code was wrong, the
body couldn't be parsed. In these three modes the transport layer works flawlessly: connection
made, response on time, status 200, body valid JSON.

The results, though, are not flawless. When the wrong-type field arrives, the late-fee calculation
produces `12.505`: text arrives where a number was expected, and addition turns into
concatenation. What comes out is not an error but a **wrong fee**, written to the member's
account. Under the silent wrong value, the fee comes out `5` instead of the genuine `17.5`. Under
below-limit slowness the value is correct, and the only cost is time spent; to the stack, this
mode is indistinguishable from a healthy call.

The reason the patterns do not meet these modes is not a design flaw, it is a **missing input.** A
pattern needs a symptom to trigger on; in these three modes, the symptom is not in the transport
layer, it is in the data itself. What would see it is not a resilience pattern but a validation
that checks the response's schema and value — and that validation was never written in this topic.

#### Zero's Three Meanings

One last measure sits in the column of the value reaching the call site. In six of the eight
modes, the `balance` value is zero. In five of those, zero is a **degradation fallback**: a
pattern triggered, the failure was seen, and the value put in its place was zero. In the sixth,
zero is the value the service **actually returned**, and behind it there is no pattern, no record,
no symptom at all.

The previous lesson measured that a fallback value is indistinguishable from a genuine one; the
table here goes one level deeper. **The same number tells three stories: genuine data, the
fallback a pattern put in place, and the wrong value a dependency silently produced.** Code
reading the stack's output cannot tell these apart. The only way is to write down, next to the
value, where it came from: a fallback marker provides that in five modes, and in the sixth there
is nothing to provide it.

Up to this point, the topic counted where the patterns sit: wrapped and uncovered call sites,
paths left outside the limit, the behavior a wrong key produces. This last measurement gives the
coverage as a whole. Of eight modes, five are seen and three are not, and two of the three unseen
carry a wrong value to the call site. **Patterns do not make a failure visible; they only make
visible the symptom that triggers them.** A failure with no symptom is just as invisible with
patterns as without.

#### Summary

- The failure-mode vocabulary was built in the Resilience and Reliability course; the measure here
  is which identity each mode shows in application code.
- Eight modes produce five separate identities at the call site, read from four places: error
  class (`TimeoutError`), error cause (`ECONNREFUSED`, `UND_ERR_SOCKET`), status code (HTTP 500),
  and parse result (`SyntaxError`).
- The pattern stack does not distinguish these five identities; the same four patterns trigger on
  all five and every call falls back. The difference between modes disappears once it passes
  through the stack.
- Coverage per pattern is not equal: timeout triggers on 1 of 8 modes, retry, breaker, and the
  degradation fallback on 5. Coverage width is not the same thing as importance.
- 3 modes trigger no pattern — wrong type, silent wrong value, and below-limit slowness — because
  in all three the transport layer works flawlessly. Two carry a wrong value to the call site: the
  late fee comes out `12.505` and `5` instead of `17.5`.
- In 6 of 8 modes, zero reaches the call site; in 5, it is a fallback value carrying a marker, and
  in 1, it is the dependency's silent wrong value, carrying no marker at all.

#### Next Step

This topic built the patterns, counted where they sit in the code, and measured what falls outside
their coverage. Every number measured describes one system: two of fourteen call sites unwrapped,
two of eight path-access pairs outside every limit, three of eight failure modes seen by no
pattern. None of these gaps was learned when written; each was learned when a failure touched it.

Learning itself, though, is not yet a mechanism. How a timeline gets built once an incident
happens, which signals from the first three topics fill that timeline, what the outcome tells
whom, and how the lesson learned makes its way back into the code — none of that has been
designed. The next lesson takes up this mechanism and asks: which of this topic's numbers can the
text written after an incident actually change.

### Post-Incident Analysis

Source: https://academia.sh/en/courses/observability/post-incident-analysis

Rebuilding an incident from the signals this course produces: which signal answers each of six questions and in how many steps, counting the questions no signal can answer, actions returning to code, and how many of a human-driven analysis's actions make a question answerable in the next incident.

The previous lesson left this behind: every gap it measured was learned not when it was written,
but when a failure touched it. Learning itself, though, is not yet a mechanism.

Root-cause analysis and the causal-chain model were built in **M21/K05 Testing Process and
Automation Infrastructure** and are not repeated here. The subject here is narrower: **rebuilding
the incident from telemetry.** This lesson asks a real incident six questions, counts which signal
answers each and in how many steps, then applies the resulting actions to the code and asks the
same questions again. A step was defined earlier in this course: one pass over a signal store, or
a join matching two stores through a key.

#### The Incident

The notification service takes 200 ms instead of 5 ms from request 80 through request 140.
**RS17 — what the notification call site writes changes with three flags**; each corresponds to an
action coming out of this incident. With the flags off, the system is in the state measured in the
Telemetry topic: a call site carrying no id leaves notification off the map.

```js
// services.mjs — 8501 loan, 8502 notification, 8503 membership. Three signals are produced in application code.
// RS17: what the notification call site writes changes with three flags; each flag is a post-incident action.
import { createServer, request } from "node:http";
const S = { log: [], metric: [], trace: [] }, FLAG = { span: 0, id: 0, delivery: 0 };
const EVENT = [80, 140], TICK = 20, SLOW = 200, NORMAL = 5;  // the event starts at request 80, ends at 140
const metric = (tick, service, duration) => {        // counter + duration sum, carries no id
  const o = S.metric.find((x) => x.tick === tick && x.service === service)
    ?? S.metric[S.metric.push({ tick, service, requests: 0, total: 0 }) - 1];
  o.requests += 1; o.total += duration;
};
const call = (port, path, id) => new Promise((resolve) => {
  const t0 = Date.now();
  const r = request({ port, path, agent: false, headers: id ? { "x-id": id } : {} },
    (y) => { y.resume(); y.on("end", () => resolve(Date.now() - t0)); });
  r.on("error", () => resolve(Date.now() - t0)); r.end();
});
createServer((req, res) => { setTimeout(() => res.end("{}"), NORMAL); }).listen(8503);   // membership: always fast
createServer((req, res) => {                          // 8502 notification: slows during the event window
  const n = Number(new URL(req.url, "http://x").searchParams.get("n"));
  const slow = n >= EVENT[0] && n < EVENT[1], duration = slow ? SLOW : NORMAL;
  setTimeout(() => {
    const id = req.headers["x-id"] ?? null;          // action 2: does the call site carry the id
    const k = { t: n, service: "notification", id, duration };
    if (FLAG.delivery) k.delivery = !slow;            // action 3: is the delivery result written to the signal
    S.log.push(k);
    if (FLAG.span) S.trace.push({ id, service: "notification", duration });   // action 1: is the span written
    metric(Math.floor(n / TICK), "notification", duration);
    res.end("{}");
  }, duration);
}).listen(8502);
createServer(async (req, res) => {                    // 8501 loan
  const q = new URL(req.url, "http://x").searchParams;
  if (req.url === "/signal") return res.end(JSON.stringify(S));
  if (req.url.startsWith("/configure")) {
    for (const k of Object.keys(FLAG)) FLAG[k] = Number(q.get(k));
    for (const d of Object.keys(S)) S[d] = [];
    return res.end("{}");
  }
  const n = Number(q.get("n")), id = `loan-${n}`, t0 = Date.now();
  S.trace.push({ id, service: "membership", duration: await call(8503, "/member", id) });
  if (n % 3 === 0) await call(8502, `/send?n=${n}`, FLAG.id ? id : null);
  const duration = Date.now() - t0;
  S.log.push({ t: n, service: "loan", id, member: `u${n % 25}`, duration });
  S.trace.push({ id, service: "loan", duration });
  metric(Math.floor(n / TICK), "loan", duration);
  res.end("{}");
}).listen(8501);
```

#### Analysis

Six questions are six functions. Each works on real signal stores and increments the step counter
on every pass; the number is not asserted, it comes out of the run. A missing field or join key
leaves the question unanswered.

```js
// analyze.mjs — rebuilding the incident from telemetry: which signal answers each question, in how many steps.
import { request } from "node:http";
const N = 200, THRESHOLD = 40, HEAVY = 100;          // RS18: event if the tick average exceeds THRESHOLD, a single call exceeds HEAVY
const call = (path) => new Promise((c) => request({ port: 8501, path, agent: false }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(g === "{}" ? null : JSON.parse(g)));
}).end());
let step = 0;
const scan = (store, f) => { step += 1; return store.filter(f); };   // one pass over one store
const join = (a, b) => {                                             // matching two stores through the id
  step += 1; const m = new Map(b.map((x) => [x.id, x])); return a.map((x) => m.get(x.id));
};
const slow = (x) => x.duration > HEAVY;
const notificationLog = (S) => scan(S.log, (x) => x.service === "notification" && slow(x));
const eventTicks = (S) => scan(S.metric, (x) => x.service === "loan" && x.total / x.requests > THRESHOLD).map((x) => x.tick);
const QUESTION = [
  ["which tick did the event start", "metric", (S) => { const t = eventTicks(S); return t.length ? `tick ${Math.min(...t)}` : null; }],
  ["which tick did the event end", "metric", (S) => { const t = eventTicks(S); return t.length ? `tick ${Math.max(...t) + 1}` : null; }],
  ["which dependency slowed", "trace", (S) => {
    const a = scan(S.trace, (x) => x.service !== "loan" && slow(x));
    return a.length ? [...new Set(a.map((x) => x.service))].join("+") : null;
  }],
  ["how many requests were affected", "log", (S) => {
    const b = notificationLog(S);
    if (b.length && b[0].id !== null) return `${new Set(b.map((x) => x.id)).size} requests`;
    const o = scan(S.log, (x) => x.service === "loan" && slow(x));   // no id: stores are scanned separately
    return scan(S.metric, (x) => x.service === "notification").length ? `${o.length} requests` : null;
  }],
  ["which members were affected", "log", (S) => {
    const b = notificationLog(S);
    if (!b.length || b[0].id === null) return null;                 // no key to join on
    const u = join(b, scan(S.log, (x) => x.service === "loan"));
    return `${new Set(u.map((x) => x.member)).size} members`;
  }],
  ["was the notification delivered", "log", (S) => {
    const b = scan(S.log, (x) => x.service === "notification" && x.delivery !== undefined);
    return b.length ? `${b.filter((x) => x.delivery === false).length} not delivered` : null;
  }],
];
const run = async (flags) => {
  await call(`/configure?${flags}`);
  for (let n = 0; n < N; n += 1) await call(`/request?n=${n}`);
  const S = await call("/signal");
  return QUESTION.map(([name, signal, f]) => { step = 0; const y = f(S); return { name, signal, y, step: y === null ? null : step }; });
};

const SETUP = [["at incident time", "span=0&id=0&delivery=0"], ["+ trace span", "span=1&id=0&delivery=0"],
  ["+ correlated id", "span=1&id=1&delivery=0"], ["+ delivery signal", "span=1&id=1&delivery=1"]];
const print = (a, ...r) => console.log(String(a).padEnd(28) + r.map((x) => String(x).padStart(16)).join(""));
const results = {};
console.log(`event: ${N} requests, notification takes 200 ms instead of 5 ms on requests 80-139; tick = 20 requests`);
print("configuration", "answered", "unanswered", "steps");
for (const [name, flags] of SETUP) {
  const r = (results[name] = await run(flags)), v = r.filter((x) => x.y !== null);
  print(name, v.length, r.length - v.length, v.reduce((a, x) => a + x.step, 0));
}
const format = (x) => (x.y === null ? "-" : `${x.y} (${x.step} steps)`);
console.log(`\n${"question".padEnd(34)}${"signal".padEnd(9)}${"at incident time".padEnd(30)}after the actions`);
results["at incident time"].forEach((x, i) =>
  console.log(x.name.padEnd(34) + x.signal.padEnd(9) + format(x).padEnd(30) + format(results["+ delivery signal"][i])));

const ACTION = { human: ["talk to the developer", "train the on-call engineer", "write an owner into the report"],
  system: ["write the trace span", "add the correlated id", "write the delivery result"] };
console.log(`\n${"analysis".padEnd(28)}${"actions".padStart(16)}${"returned to code".padStart(18)}${"question answered".padStart(20)}`);
for (const [kind, name, at] of [["human", "human-driven", "at incident time"], ["system", "system-driven", "+ delivery signal"]])
  print(name, ACTION[kind].length, kind === "system" ? ACTION[kind].length : 0, results[at].filter((x) => x.y !== null).length);
process.exit(0);
```

```bash
node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1; node analyze.mjs; kill $SP
```

```
event: 200 requests, notification takes 200 ms instead of 5 ms on requests 80-139; tick = 20 requests
configuration                       answered      unanswered           steps
at incident time                           3               3               5
+ trace span                               4               2               6
+ correlated id                            5               1               7
+ delivery signal                          6               0               8

question                          signal   at incident time              after the actions
which tick did the event start    metric   tick 4 (1 steps)              tick 4 (1 steps)
which tick did the event end      metric   tick 7 (1 steps)              tick 7 (1 steps)
which dependency slowed           trace    -                             notification (1 steps)
how many requests were affected   log      20 requests (3 steps)         20 requests (1 steps)
which members were affected       log      -                             20 members (3 steps)
was the notification delivered    log      -                             20 not delivered (1 steps)

analysis                             actions  returned to code   question answered
human-driven                               3               0               3
system-driven                              3               3               6
```

#### How Many Steps It Takes to Build the Timeline

With the signals at incident time, three of six questions can be answered: start and end each take
one step from the metric, how many requests were affected takes three steps from the log. The
remaining three are empty.

The `how many requests were affected` row shows what the correlation id does: the same answer, 20,
takes 1 step instead of 3. Without an id, the notification record cannot say which request it
belongs to, so the loan log and metric get scanned separately, and the scan count grows with the
service count. Timing questions' cost never changes, since the metric needs no join. **The
timeline's skeleton comes from the cheapest signal, its detail from the most expensive one.**

#### The Telemetry Gap

Three questions cannot be answered by any signal at incident time, and their reasons differ.

**Which dependency slowed** stays unanswered, because the notification call site writes no trace
span. The loan span in the trace store looks slow, but it only says it was waiting; a service's
own slowness cannot be told apart from waiting on a dependency without a child span.

**Which members were affected** stays unanswered: the member field lives only in the loan record,
the slowness only in the notification record. Both stores exist, both fields exist; **what's
missing is only the common key** — the data was collected, it just cannot be joined.

**Was the notification delivered** stays unanswered because the delivery result is never written
anywhere. All three gaps are the result of a **writing** decision, closable only at write time,
not at read time.

#### Blamelessness Ties to a Measure

Blamelessness is not etiquette, it is a measure tied to the type of action. The last table
compares two analyses of the same incident. The human-driven analysis produced three actions;
**none returns to code**, and in the next similar incident the same three questions stay
unanswered. The system-driven analysis's three actions are each a writing decision; all three
returned to code, and all six questions became answerable.

The measure is this: **an action has returned to code if it changes the answer to a question in
the next similar incident.** Its cost is in the first table: steps went from 5 to 8, one step per
question closed.

#### Summary

- The incident's timeline was built from the three signals this course produces; root-cause
  analysis and the causal-chain model were built in M21/K05 and are not repeated here.
- At incident time, three of six questions were answered (5 steps), three stayed unanswered; after
  the actions, all six were answered (8 steps). The timing questions take 1 step under every
  configuration; the correlation id changed the cost, not the answer: the same number 20 came out
  in 1 step instead of 3.
- All three gaps are writing decisions: the unwritten trace span hid the dependency, the uncarried
  id blocked the join between the two stores, and the unrecorded delivery result closed off the
  question.
- 0 of the human-driven analysis's 3 actions returned to code, and answered questions stayed at 3;
  3 of the system-driven analysis's 3 returned, and answers rose to 6. The cost is three steps.

#### Course Wrap-Up

| Lesson | Where it sits in the code | Wrapped / uncovered call site | Cost of the wrong setting |
|---|---|---|---|
| telemetry/01 The Three Signals of Observability | all three, same request, same process | metric 6, log 11–12 steps; 5 of 9 attempts unanswered | the metric closes the identity question for good |
| telemetry/02 Structured Logging | field-based record; id lives in the client helper | 11 → 7 steps; id takes the earlier unanswered question to 8 | 3 of 4 call sites carry the id; one service off the map |
| telemetry/03 Metric Types | counter, gauge, histogram at the call site | three types, three questions; none answers another | gauge saw 20 of 2000 observations, peak 199 instead of 293 |
| telemetry/04 Distributed Tracing | trace context in one header, crosses process boundary | 100 requests → 100 complete traces, 400 linked spans | one context-free call site breaks 100 traces; lowest rate, 0 of 40 |
| telemetry/05 Instrumentation Standards | behind the collector's interface | collector name: 5 application files → 0 | a switch touches 5 files bound, 1 behind the interface |
| telemetry/06 Correlation | three signals, one id, one event | 20 steps / 3 signals → 2 steps / 1 signal | at ratio 0.50, 51 of 100 events stay untraced |
| service-level/01 Indicator, Objective, and Agreement | exclusion list as a filter array | 12 numbers from 1,188,000 records, three definitions, four filters | filter array drift from agreement text splits measured from promised |
| service-level/02 Golden Signals | saturation as queue depth in the request log | 6 event classes; a single signal catches at most 3 | error rate reads 0 in traffic collapse; 136 of 162 empty triggers at low traffic |
| service-level/03 Error Budget | release gate as a single-line condition | a 3,060-bad-request budget over 30 days; floor eats 20.6% | 10 releases spent 180.5% of budget; gate overruns fall 2,463 → 471 |
| service-level/04 Alert Design | whether the rule reads cause or symptom | 12 rules, 81 alerts; 4 rules, 9 alerts; both catch 4/6 events | 67 wasted alerts in the cause-based set, 3.5 per event |
| service-level/05 Health Endpoints | liveness/readiness split at one condition term | a 25-tick outage: 0 against 4 restarts | misclassification dropped 48 unrelated requests, recovery 0 → 4 ticks |
| resilience/01 Timeouts | 53 lines at the call site, 28 in the wrapper | 7 / 0 and 6 / 1 | inner limit exceeding outer: 10 of 10 requests orphaned |
| resilience/02 Retry and Backoff | wrapper 6, middle layer 2, gateway 6 lines | 1 of 2 call sites in the body gets retried | three layers open: 1 request becomes 27 calls, 27 deliveries |
| resilience/03 Circuit Breaker | state key: 2 per dependency, 4 per call site | 3 call sites; at threshold 10, 1 breaker never opens | same threshold: calls to dependency 5 → 13; rare path unprotected |
| resilience/04 Bulkhead Pattern | pool definition 5 lines, one place | 2 call sites unchanged; 8 of 40 calls bypass the pool | pool says 4, dependency sees 12; 40 slots binds nothing |
| resilience/05 Graceful Degradation | schema object: 7 lines at the call site, 12 in the wrapper | 6 of 14 call sites wrapped, 2 uncovered | unmarked fallback indistinguishable from a genuinely zero balance |
| resilience/06 Rate Limiting and Throttling | at the gateway and the service's middle layer | 2 / 6 of 8 path-access pairs; 2 in neither | cap 20 under fixed key, 100,000 under member key |
| resilience/07 Failure Modes | error identity read from four places | 5 of 8 modes in the pattern stack, 3 never seen | zero reaches the call site in 6 of 8 modes, 1 unmarked |
| resilience/08 Post-Incident Analysis | action returns to code as a new signal, id, span | 3 of 6 questions unanswered → 0 | 0 of the human-driven analysis's 3 actions return to code |

The table is one rule applied nineteen times. **A pattern's effect was measured in Resilience and
Reliability; measured here is where that pattern sits in the code and what it skips.** The same
rule held for signals: their categories were built elsewhere, and what got counted here was how
each signal is produced in code and how many steps it takes to answer a question. The third column
says almost the same thing on nearly every row — the uncovered spot never shows up in the
configuration, only in the source.

The system is now visible and withstands failure: where a request slows down can be traced,
indicators turn into decisions, patterns contain the failure, and a timeline can be rebuilt after
an outage. Yet no measurement here ever asked one question: what if the request itself is
malicious. Every number assumed the caller was not trying to wear the system down; the release
itself was never addressed either. **M16/K08 Server Security and Going to Production** starts with
exactly these two questions.