---
title: 'Health Endpoints'
source: 'https://academia.sh/en/courses/observability/health-endpoints'
course: 'Observability and Reliability'
language: en
updated: '2026-08-23T07:00:28+00:00'
license: 'CC BY-SA 4.0'
---

# Health Endpoints

Keeping liveness and readiness separate in code: the question each endpoint answers, the restart loop produced when a dependency check lands in the liveness endpoint, and running the same scenario in two modes to count the dropped requests and recovery time each one produces.

Every rule set up in the previous lesson pages a person, and the person responds within minutes.
Some decisions are never put to a person at all. Whether one of the loan system's processes is
alive and whether it should keep receiving traffic is decided by a machine, within seconds and
thousands of times a day. That decision looks at two separate questions; when the two are wired
to a single endpoint, the answer given is not just wrong, it builds a self-feeding loop.

The Resilience and Reliability course measured the health endpoint's content and check interval;
that survey is not repeated here. The question here is single: are the two endpoints kept
separate in code, and what happens when they are not.

## Two Questions, Two Actions

The **liveness** endpoint's question is this: is this process dead, and does restarting it fix
the situation. If the answer is no, the controller takes exactly one action — it kills the
process and restarts it.

The **readiness** endpoint's question is different: can this process take work right now. If the
answer is no, the action is different too — traffic is cut, but the process keeps living and is
expected to recover on its own.

The rule for the distinction follows from these two actions: **for a check to belong in the
liveness endpoint, restarting has to be the fix when that check fails.** A process's own lock,
its own memory, its own event loop meet this test; they get cleaned up when the process is
reborn. A dependency does not. Restarting the loan process while the membership service is
unreachable does not bring the membership service back; the check fails in the new process too,
the controller kills that one as well, and the loop continues until the dependency comes back.

## The Node: The Distinction Is Two Lines

```js
// health/node.mjs — the loan node: two health endpoints (/liveness, /readiness) and two
// work endpoints. In "separate" mode, liveness only reads the process's own state; in
// "combined" mode, liveness also checks the dependency on the membership service. The
// entire distinction is these two lines.
import { createServer } from "node:http";
import { readFileSync } from "node:fs";
// SD1 (assumption): a restarted process rebuilds its local directory over 24 requests.
export const WARMUP = 24;

export function start(mode) {
  let processed = 0, locked = false;
  const dependencyOk = () => readFileSync("dependency.status", "utf8").trim() === "up";
  const s = createServer((req, res) => {
    let code = 200;
    if (req.url === "/liveness") code = locked || (mode === "combined" && !dependencyOk()) ? 500 : 200;
    else if (req.url === "/readiness") code = dependencyOk() && processed >= WARMUP ? 200 : 503;
    else {
      processed += 1;
      if (processed <= WARMUP) code = 503;                          // local directory not yet rebuilt
      else if (req.url === "/loan" && !dependencyOk()) code = 503;   // needs the membership service
    }
    res.writeHead(code).end();
  });
  s.listen(0, "127.0.0.1", () => process.send({ port: s.address().port }));
}

if (process.argv[2]) start(process.argv[2]);
else console.log(`endpoints: /liveness /readiness /search /loan; warmup ${WARMUP} requests; ` +
  `/search does not need the dependency, /loan does`);
```

```
endpoints: /liveness /readiness /search /loan; warmup 24 requests; /search does not need the dependency, /loan does
```

The entire difference between the two endpoints is which line calls `dependencyOk()`. It belongs
in the readiness endpoint, because the process cannot take work without the dependency. It should
not appear in the liveness endpoint; the `combined` mode makes exactly this mistake, adding a
single condition term. The process serves two work endpoints: `/search` answers from the local
directory and does not need the membership service, `/loan` does. A process that just restarted
loses its local directory and cannot answer either one for twenty-four requests.

## The Same Scenario, Two Modes

```js
// health/controller.mjs — the same scenario is run in two modes: the membership service goes
// down at tick 15, comes back up at tick 40. The controller polls the liveness and readiness
// endpoints every 2 ticks, and restarts the process after 3 consecutive liveness failures.
// Every tick sends 2 search requests and 2 loan requests.
import { fork } from "node:child_process";
import { writeFileSync } from "node:fs";
import { request } from "node:http";
// SD2 (assumption): poll interval 2 ticks, restart threshold 3 consecutive liveness failures.
const TICKS = 60, DOWN = [15, 40], POLL = 2, THRESHOLD = 3, WARMUP = 24;

const call = (port, path) => new Promise((c) => {
  const req = request({ host: "127.0.0.1", port, path }, (res) => { res.resume(); c(res.statusCode); });
  req.on("error", () => c(0));
  req.end();
});
const spawn = (mode) => new Promise((c) => {
  const p = fork(new URL("node.mjs", import.meta.url), [mode],
    { stdio: ["ignore", "ignore", "ignore", "ipc"] });
  p.once("message", (m) => c([p, m.port]));
});

async function run(mode) {
  writeFileSync("dependency.status", "up");
  let [p, port] = await spawn(mode);
  for (let i = 0; i < WARMUP; i += 1) await call(port, "/search");   // the first warmup does not count
  const s = { restarts: 0, search: 0, loan: 0, notReady: 0, recovery: -1 };
  let consecutive = 0;
  for (let t = 0; t < TICKS; t += 1) {
    writeFileSync("dependency.status", t >= DOWN[0] && t < DOWN[1] ? "down" : "up");
    if (t % POLL === 0) {
      consecutive = (await call(port, "/liveness")) === 200 ? 0 : consecutive + 1;
      if (consecutive >= THRESHOLD) {
        consecutive = 0; s.restarts += 1;
        p.kill(); await new Promise((c) => p.once("exit", c));
        [p, port] = await spawn(mode);
      }
      if ((await call(port, "/readiness")) !== 200) s.notReady += 1;
    }
    let good = 0;
    for (const path of ["/search", "/search", "/loan", "/loan"]) {
      if ((await call(port, path)) === 200) good += 1;
      else if (path === "/search") s.search += 1; else s.loan += 1;
    }
    if (t >= DOWN[1] && good === 4 && s.recovery < 0) s.recovery = t - DOWN[1];
  }
  p.kill();
  return s;
}

const results = [];
for (const mode of ["separate", "combined"]) results.push([mode, await run(mode)]);
console.log(`${TICKS} ticks, 4 requests per tick; membership service down between ticks ${DOWN[0]}-` +
  `${DOWN[1]} (${DOWN[1] - DOWN[0]} ticks), warmup ${WARMUP} requests`);
console.log(`\n${"mode".padEnd(10)}${"restarts".padStart(18)}${"dropped /search".padStart(17)}` +
  `${"dropped /loan".padStart(15)}${"not ready".padStart(13)}${"recovery".padStart(12)}`);
for (const [mode, s] of results)
  console.log(`${mode.padEnd(10)}${String(s.restarts).padStart(18)}${String(s.search).padStart(17)}` +
    `${String(s.loan).padStart(15)}${(s.notReady + " polls").padStart(13)}` +
    `${(s.recovery + " ticks").padStart(12)}`);
const [a, b] = results.map(([, s]) => s);
console.log(`\nin combined mode, ${b.search} dropped search requests never needed the membership ` +
  `service; in separate mode that count is ${a.search}`);
console.log(`total dropped requests: separate ${a.search + a.loan}, combined ${b.search + b.loan} ` +
  `(${b.search + b.loan - a.search - a.loan} requests from misclassification alone)`);
```

```
60 ticks, 4 requests per tick; membership service down between ticks 15-40 (25 ticks), warmup 24 requests

mode                restarts  dropped /search  dropped /loan    not ready    recovery
separate                   0                0             50     12 polls     0 ticks
combined                   4               48             58     14 polls     4 ticks

in combined mode, 48 dropped search requests never needed the membership service; in separate mode that count is 0
total dropped requests: separate 50, combined 106 (56 requests from misclassification alone)
```

## The Cost of Misclassification

In `separate` mode, there are no restarts. While the membership service was down for twenty-five
ticks, the readiness endpoint answered negatively on twelve polls — meaning the traffic router
could have taken this node out of rotation — but the process stayed alive. The 50 dropped
requests are all `/loan` requests: work that could not have been done without the membership
service anyway. None of the search requests dropped. When the dependency came back at tick 40,
recovery happened **on that same tick**, because the process had never lost its in-memory
directory.

In `combined` mode, once the liveness endpoint looked at the dependency, a chain formed. After
three consecutive negative polls, the controller killed the process; the new process could not
find the same dependency either, so it got killed too: **four restarts** across the twenty-five
ticks the dependency was down. Every restart reset the local directory, and `/search` could not
answer for twenty-four requests either. The result: 48 search requests dropped, even though those
requests never needed the membership service at all. The `/loan` drop count also rose from 50 to
58, because the process was still warming up after the dependency came back.

Two numbers are this lesson's measure. **Total dropped requests rose from 50 to 106; 56 requests
dropped for misclassification alone.** And recovery time rose from 0 ticks to 4: even after the
dependency came back, the system could not answer until it had paid off the last restart's
warm-up debt. That second number shows the loop's real character — restarting extends the
failure, it does not shorten it.

## Keeping the Distinction in Code

Reducing the two endpoints to a single helper function is the most common mistake: a function
named `healthStatus()` is written, it runs all the checks, and both endpoints call it. In this
shape, the distinction is lost the next day, because whoever wants to add a new dependency check
adds it to a single place, and that place feeds both endpoints at once. In the node above, the
two endpoints are separate lines, and the only thing shared between them is the `dependencyOk()`
reader; which endpoint calls it is spelled out explicitly every time.

A three-item test is enough for the decision. The process's own state — lock, memory, event loop
— belongs in liveness, because restarting fixes them. Dependencies and warm-up belong in
readiness, because waiting fixes them, killing does not. Resources that run out — disk,
connection count — belong in neither; they belong to the previous lesson's alert rule, because
neither restarting nor cutting traffic fixes them.

## Summary

- Liveness answers "is this process dead" and triggers a restart; readiness answers "can it take
  work right now" and only cuts traffic.
- A check belongs in the liveness endpoint only if restarting is the fix when it fails; a
  dependency check does not meet that test.
- Over the same twenty-five-tick dependency outage, keeping the endpoints separate produced zero
  restarts; combining them produced four.
- Misclassification dropped 48 search requests that never needed the dependency at all; total
  dropped requests rose from 50 to 106, and 56 of those came from this mistake alone.
- Restarting extended the failure: recovery took 4 ticks instead of 0 after the dependency
  returned, because every restart reset the local directory.
- The distinction in code is a single condition term, and reducing the two endpoints to a shared
  `healthStatus()` function loses it the first time someone adds to it.

## Next Step

This topic turned telemetry into a decision: an indicator was defined, four signals were
computed, the objective's complement turned into a spendable budget, an alert rule was written
from symptoms, and the two health endpoints were separated. A breach of a threshold can now be
**stated.** Nothing changes just from stating it, though. While the membership service was down,
the loan node kept sending the same request the same way, waiting the same duration, and getting
the same error; the readiness endpoint reported this, but no code stopped the request, shortened
it, or routed it another way. Measuring changes observation, not behavior. The next topic places
the patterns that change behavior into the code itself, and counts which layer each one sits in,
how many call sites it wraps, and which call site it skips.