Health Endpoints

Every rule set up in the previous lesson pages a person, and the person responds within minutes. Some decisions are never put to a person at all. Whether one of the loan system’s processes is alive and whether it should keep receiving traffic is decided by a machine, within seconds and thousands of times a day. That decision looks at two separate questions; when the two are wired to a single endpoint, the answer given is not just wrong, it builds a self-feeding loop.

The Resilience and Reliability course measured the health endpoint’s content and check interval; that survey is not repeated here. The question here is single: are the two endpoints kept separate in code, and what happens when they are not.

Two Questions, Two Actions

The liveness endpoint’s question is this: is this process dead, and does restarting it fix the situation. If the answer is no, the controller takes exactly one action — it kills the process and restarts it.

The readiness endpoint’s question is different: can this process take work right now. If the answer is no, the action is different too — traffic is cut, but the process keeps living and is expected to recover on its own.

The rule for the distinction follows from these two actions: for a check to belong in the liveness endpoint, restarting has to be the fix when that check fails. A process’s own lock, its own memory, its own event loop meet this test; they get cleaned up when the process is reborn. A dependency does not. Restarting the loan process while the membership service is unreachable does not bring the membership service back; the check fails in the new process too, the controller kills that one as well, and the loop continues until the dependency comes back.

The Node: The Distinction Is Two Lines

// health/node.mjs — the loan node: two health endpoints (/liveness, /readiness) and two
// work endpoints. In "separate" mode, liveness only reads the process's own state; in
// "combined" mode, liveness also checks the dependency on the membership service. The
// entire distinction is these two lines.
import { createServer } from "node:http";
import { readFileSync } from "node:fs";
// SD1 (assumption): a restarted process rebuilds its local directory over 24 requests.
export const WARMUP = 24;

export function start(mode) {
  let processed = 0, locked = false;
  const dependencyOk = () => readFileSync("dependency.status", "utf8").trim() === "up";
  const s = createServer((req, res) => {
    let code = 200;
    if (req.url === "/liveness") code = locked || (mode === "combined" && !dependencyOk()) ? 500 : 200;
    else if (req.url === "/readiness") code = dependencyOk() && processed >= WARMUP ? 200 : 503;
    else {
      processed += 1;
      if (processed <= WARMUP) code = 503;                          // local directory not yet rebuilt
      else if (req.url === "/loan" && !dependencyOk()) code = 503;   // needs the membership service
    }
    res.writeHead(code).end();
  });
  s.listen(0, "127.0.0.1", () => process.send({ port: s.address().port }));
}

if (process.argv[2]) start(process.argv[2]);
else console.log(`endpoints: /liveness /readiness /search /loan; warmup ${WARMUP} requests; ` +
  `/search does not need the dependency, /loan does`);

endpoints: /liveness /readiness /search /loan; warmup 24 requests; /search does not need the dependency, /loan does

The entire difference between the two endpoints is which line calls dependencyOk(). It belongs in the readiness endpoint, because the process cannot take work without the dependency. It should not appear in the liveness endpoint; the combined mode makes exactly this mistake, adding a single condition term. The process serves two work endpoints: /search answers from the local directory and does not need the membership service, /loan does. A process that just restarted loses its local directory and cannot answer either one for twenty-four requests.

The Same Scenario, Two Modes

// health/controller.mjs — the same scenario is run in two modes: the membership service goes
// down at tick 15, comes back up at tick 40. The controller polls the liveness and readiness
// endpoints every 2 ticks, and restarts the process after 3 consecutive liveness failures.
// Every tick sends 2 search requests and 2 loan requests.
import { fork } from "node:child_process";
import { writeFileSync } from "node:fs";
import { request } from "node:http";
// SD2 (assumption): poll interval 2 ticks, restart threshold 3 consecutive liveness failures.
const TICKS = 60, DOWN = [15, 40], POLL = 2, THRESHOLD = 3, WARMUP = 24;

const call = (port, path) => new Promise((c) => {
  const req = request({ host: "127.0.0.1", port, path }, (res) => { res.resume(); c(res.statusCode); });
  req.on("error", () => c(0));
  req.end();
});
const spawn = (mode) => new Promise((c) => {
  const p = fork(new URL("node.mjs", import.meta.url), [mode],
    { stdio: ["ignore", "ignore", "ignore", "ipc"] });
  p.once("message", (m) => c([p, m.port]));
});

async function run(mode) {
  writeFileSync("dependency.status", "up");
  let [p, port] = await spawn(mode);
  for (let i = 0; i < WARMUP; i += 1) await call(port, "/search");   // the first warmup does not count
  const s = { restarts: 0, search: 0, loan: 0, notReady: 0, recovery: -1 };
  let consecutive = 0;
  for (let t = 0; t < TICKS; t += 1) {
    writeFileSync("dependency.status", t >= DOWN[0] && t < DOWN[1] ? "down" : "up");
    if (t % POLL === 0) {
      consecutive = (await call(port, "/liveness")) === 200 ? 0 : consecutive + 1;
      if (consecutive >= THRESHOLD) {
        consecutive = 0; s.restarts += 1;
        p.kill(); await new Promise((c) => p.once("exit", c));
        [p, port] = await spawn(mode);
      }
      if ((await call(port, "/readiness")) !== 200) s.notReady += 1;
    }
    let good = 0;
    for (const path of ["/search", "/search", "/loan", "/loan"]) {
      if ((await call(port, path)) === 200) good += 1;
      else if (path === "/search") s.search += 1; else s.loan += 1;
    }
    if (t >= DOWN[1] && good === 4 && s.recovery < 0) s.recovery = t - DOWN[1];
  }
  p.kill();
  return s;
}

const results = [];
for (const mode of ["separate", "combined"]) results.push([mode, await run(mode)]);
console.log(`${TICKS} ticks, 4 requests per tick; membership service down between ticks ${DOWN[0]}-` +
  `${DOWN[1]} (${DOWN[1] - DOWN[0]} ticks), warmup ${WARMUP} requests`);
console.log(`\n${"mode".padEnd(10)}${"restarts".padStart(18)}${"dropped /search".padStart(17)}` +
  `${"dropped /loan".padStart(15)}${"not ready".padStart(13)}${"recovery".padStart(12)}`);
for (const [mode, s] of results)
  console.log(`${mode.padEnd(10)}${String(s.restarts).padStart(18)}${String(s.search).padStart(17)}` +
    `${String(s.loan).padStart(15)}${(s.notReady + " polls").padStart(13)}` +
    `${(s.recovery + " ticks").padStart(12)}`);
const [a, b] = results.map(([, s]) => s);
console.log(`\nin combined mode, ${b.search} dropped search requests never needed the membership ` +
  `service; in separate mode that count is ${a.search}`);
console.log(`total dropped requests: separate ${a.search + a.loan}, combined ${b.search + b.loan} ` +
  `(${b.search + b.loan - a.search - a.loan} requests from misclassification alone)`);

60 ticks, 4 requests per tick; membership service down between ticks 15-40 (25 ticks), warmup 24 requests

mode                restarts  dropped /search  dropped /loan    not ready    recovery
separate                   0                0             50     12 polls     0 ticks
combined                   4               48             58     14 polls     4 ticks

in combined mode, 48 dropped search requests never needed the membership service; in separate mode that count is 0
total dropped requests: separate 50, combined 106 (56 requests from misclassification alone)

The Cost of Misclassification

In separate mode, there are no restarts. While the membership service was down for twenty-five ticks, the readiness endpoint answered negatively on twelve polls — meaning the traffic router could have taken this node out of rotation — but the process stayed alive. The 50 dropped requests are all /loan requests: work that could not have been done without the membership service anyway. None of the search requests dropped. When the dependency came back at tick 40, recovery happened on that same tick, because the process had never lost its in-memory directory.

In combined mode, once the liveness endpoint looked at the dependency, a chain formed. After three consecutive negative polls, the controller killed the process; the new process could not find the same dependency either, so it got killed too: four restarts across the twenty-five ticks the dependency was down. Every restart reset the local directory, and /search could not answer for twenty-four requests either. The result: 48 search requests dropped, even though those requests never needed the membership service at all. The /loan drop count also rose from 50 to 58, because the process was still warming up after the dependency came back.

Two numbers are this lesson’s measure. Total dropped requests rose from 50 to 106; 56 requests dropped for misclassification alone. And recovery time rose from 0 ticks to 4: even after the dependency came back, the system could not answer until it had paid off the last restart’s warm-up debt. That second number shows the loop’s real character — restarting extends the failure, it does not shorten it.

Keeping the Distinction in Code

Reducing the two endpoints to a single helper function is the most common mistake: a function named healthStatus() is written, it runs all the checks, and both endpoints call it. In this shape, the distinction is lost the next day, because whoever wants to add a new dependency check adds it to a single place, and that place feeds both endpoints at once. In the node above, the two endpoints are separate lines, and the only thing shared between them is the dependencyOk() reader; which endpoint calls it is spelled out explicitly every time.

A three-item test is enough for the decision. The process’s own state — lock, memory, event loop — belongs in liveness, because restarting fixes them. Dependencies and warm-up belong in readiness, because waiting fixes them, killing does not. Resources that run out — disk, connection count — belong in neither; they belong to the previous lesson’s alert rule, because neither restarting nor cutting traffic fixes them.

Summary

• Liveness answers “is this process dead” and triggers a restart; readiness answers “can it take work right now” and only cuts traffic.
• A check belongs in the liveness endpoint only if restarting is the fix when it fails; a dependency check does not meet that test.
• Over the same twenty-five-tick dependency outage, keeping the endpoints separate produced zero restarts; combining them produced four.
• Misclassification dropped 48 search requests that never needed the dependency at all; total dropped requests rose from 50 to 106, and 56 of those came from this mistake alone.
• Restarting extended the failure: recovery took 4 ticks instead of 0 after the dependency returned, because every restart reset the local directory.
• The distinction in code is a single condition term, and reducing the two endpoints to a shared healthStatus() function loses it the first time someone adds to it.

Next Step

This topic turned telemetry into a decision: an indicator was defined, four signals were computed, the objective’s complement turned into a spendable budget, an alert rule was written from symptoms, and the two health endpoints were separated. A breach of a threshold can now be stated. Nothing changes just from stating it, though. While the membership service was down, the loan node kept sending the same request the same way, waiting the same duration, and getting the same error; the readiness endpoint reported this, but no code stopped the request, shortened it, or routed it another way. Measuring changes observation, not behavior. The next topic places the patterns that change behavior into the code itself, and counts which layer each one sits in, how many call sites it wraps, and which call site it skips.