Health Endpoint Monitoring

Every decision in the previous lesson rested on a single input: the yes or no the health check returns. How the threshold was chosen, when failback happened, even whether failover happened at all depended on this one bit’s accuracy. In the model’s half-failed window, 3,888.80 writes were lost while the system looked healthy — because the check only asked “are you responding”.

The health check that feeds failover and the load balancer pool was established in the Relational Database Administration and The Traffic Layer courses; the endpoint’s form on the serving side was introduced in the Server-Side Fundamentals course. This lesson designs that endpoint’s content: what question it asks, whether it puts its dependencies in scope, and how often it asks. Where its collected metrics go, the dashboard, and the alert threshold are not this course’s subject; they belong to the Performance Anti-Patterns and Monitoring course.

The Four Questions the Endpoint Can Ask

The shallow check only tests that the process is up and can respond. It produces its answer without touching any dependency, so it is cheap and never gives a false negative.

The readiness check is a separate question: the process may be up but not ready to take traffic. A newly started replica may not have filled its cache, read its configuration, or finished its migration. Being up and being ready are two separate states, and merging them into one endpoint loses one of the two.

The deep check touches the replica’s dependencies. Two kinds of dependency are separated here, and the separation is this lesson’s main result: a dependency on the replica’s own path (whether this replica can write to the state store) versus a shared dependency (whether the state store itself is up). The first differs between replicas, the second is the same for all of them.

The setup below is a model: three replicas, one shared state store, and a billing store that feeds only the batch flow. A round is an abstract step. The request mix per round is chosen close to K01’s read–write ratio: 8 reads, 2 writes, 1 batch.

// health/model.mjs — the content of the health endpoint is a MODEL. Three replicas, one shared
// state store, and a store used only by the batch feed. A round is an abstract step. The request
// mix per round is chosen close to K01's read/write ratio: 8 reads, 2 writes, 1 batch.
export const ROUND = 130;
export const K3_STOPPED = 82;   // k3 stops in this round; detection delay is measured from here
export const REPLICA = ["k1", "k2", "k3"];
const P = (t, a) => a.some(([b, c]) => t >= b && t <= c);

export const schedule = {
  halfFailed: { k1: [[20, 45]], k2: [], k3: [] },       // looks like it is writing, never reaches the store
  warming: { k1: [], k2: [[1, 8], [60, 66]], k3: [] },  // process is up, not ready for traffic
  stopped: { k1: [], k2: [], k3: [[82, 95]] },          // process does not respond
  brief: { k1: [], k2: [], k3: [[50, 50], [58, 59]] },  // brief unresponsiveness, recovers on its own
  stateStore: [[100, 110]],                             // shared dependency goes down
  billingStore: [[115, 125]],                           // affects only the batch feed
};

export const stateStoreUp = (t) => !P(t, schedule.stateStore);
export const billingStoreUp = (t) => !P(t, schedule.billingStore);
export const processUp = (k, t) => !P(t, schedule.stopped[k]) && !P(t, schedule.brief[k]);
export const warming = (k, t) => P(t, schedule.warming[k]);
export const pathHealthy = (k, t) => !P(t, schedule.halfFailed[k]);   // the replica's own write path
export const canRead = (k, t) => processUp(k, t) && !warming(k, t);

// Five endpoint contents. Each returns a single yes/no per replica.
export const ENDPOINT = {
  shallow: (k, t) => processUp(k, t),
  readiness: (k, t) => canRead(k, t),
  deepLocal: (k, t) => canRead(k, t) && pathHealthy(k, t),
  deepShared: (k, t) => canRead(k, t) && pathHealthy(k, t) && stateStoreUp(t),
  deepAll: (k, t) => canRead(k, t) && pathHealthy(k, t) && stateStoreUp(t)
    && billingStoreUp(t),
};

const READS = 8, WRITES = 2, BATCH = 1;   // requests per round: close to K01's read/write ratio

export function run(endpointName, { interval = 1, round: N = ROUND } = {}) {
  const endpoint = ENDPOINT[endpointName];
  const s = { outOfPoolRounds: 0, droppedReads: 0, droppedWrites: 0, droppedBatch: 0, silent: 0,
    wastedReject: 0, checks: 0, falseEviction: 0, detection: null };
  let round = 0, decision = Object.fromEntries(REPLICA.map((k) => [k, true])), previousDecision = { ...decision };

  for (let t = 1; t <= N; t += 1) {
    if ((t - 1) % interval === 0) {                     // check round
      previousDecision = { ...decision };
      for (const k of REPLICA) decision[k] = endpoint(k, t);
      s.checks += REPLICA.length;
      for (const k of REPLICA)                          // did a brief outage evict it from the pool
        if (previousDecision[k] && decision[k] === false && P(t, schedule.brief[k])) s.falseEviction += 1;
      if (s.detection === null && decision.k3 === false && P(t, schedule.stopped.k3))
        s.detection = t - K3_STOPPED;
    }
    const pool = REPLICA.filter((k) => decision[k]);
    s.outOfPoolRounds += REPLICA.length - pool.length;
    const someCanRead = REPLICA.some((k) => canRead(k, t));

    for (let i = 0; i < READS; i += 1) {
      if (pool.length === 0) { s.droppedReads += 1; if (someCanRead) s.wastedReject += 1; continue; }
      const k = pool[round++ % pool.length];
      if (canRead(k, t) === false) s.droppedReads += 1;
    }
    for (let i = 0; i < WRITES; i += 1) {
      if (pool.length === 0) { s.droppedWrites += 1; continue; }
      const k = pool[round++ % pool.length];
      if (canRead(k, t) === false) { s.droppedWrites += 1; continue; }
      if (pathHealthy(k, t) === false) { s.silent += 1; continue; }   // accepted, never reached the store
      if (stateStoreUp(t) === false) s.droppedWrites += 1;
    }
    for (let i = 0; i < BATCH; i += 1) {
      if (pool.length === 0 || billingStoreUp(t) === false) { s.droppedBatch += 1; continue; }
      round += 1;
    }
  }
  return s;
}

// health/measure.mjs — five endpoint contents in the same failure schedule, then the effect of the check interval
import { run, ROUND, REPLICA } from "./model.mjs";

const s = (x, n) => String(x).padStart(n);
console.log(`${ROUND} rounds, ${REPLICA.length} replicas, 8 reads + 2 writes + 1 batch request per round.`);
console.log("schedule: k1 20-45 half-failed, k2 1-8 and 60-66 warming, k3 50 and 58-59 brief " +
  "unresponsive, 82-95 stopped, state store 100-110, billing store 115-125 down.");
console.log();
console.log("endpoint content | out of pool | dropped reads | dropped writes | dropped batch | silent lost update | wasted reject");
console.log("-----------------|-------------|----------------|-----------------|----------------|---------------------|--------------");
for (const name of ["shallow", "readiness", "deepLocal", "deepShared", "deepAll"]) {
  const r = run(name);
  console.log(`${name.padEnd(17)} | ${s(r.outOfPoolRounds, 11)} | ${s(r.droppedReads, 14)} | ` +
    `${s(r.droppedWrites, 15)} | ${s(r.droppedBatch, 14)} | ${s(r.silent, 19)} | ${s(r.wastedReject, 12)}`);
}

console.log();
console.log("interval | checks | detection delay | false eviction | dropped reads | silent lost update");
console.log("---------|--------|------------------|-----------------|----------------|--------------------");
for (const interval of [1, 2, 4, 8]) {
  const r = run("deepLocal", { interval });
  console.log(`${s(interval, 8)} | ${s(r.checks, 6)} | ${s(`${r.detection} round`, 16)} | ` +
    `${s(r.falseEviction, 15)} | ${s(r.droppedReads, 14)} | ${s(r.silent, 19)}`);
}

130 rounds, 3 replicas, 8 reads + 2 writes + 1 batch request per round.
schedule: k1 20-45 half-failed, k2 1-8 and 60-66 warming, k3 50 and 58-59 brief unresponsive, 82-95 stopped, state store 100-110, billing store 115-125 down.

endpoint content | out of pool | dropped reads | dropped writes | dropped batch | silent lost update | wasted reject
-----------------|-------------|----------------|-----------------|----------------|---------------------|--------------
shallow           |          17 |             40 |              32 |             11 |                  17 |            0
readiness         |          32 |              0 |              22 |             11 |                  17 |            0
deepLocal         |          58 |              0 |              22 |             11 |                   0 |            0
deepShared        |          91 |             88 |              22 |             22 |                   0 |           88
deepAll           |         124 |            176 |              44 |             22 |                   0 |          176

interval | checks | detection delay | false eviction | dropped reads | silent lost update
---------|--------|------------------|-----------------|----------------|--------------------
       1 |    390 |          0 round |               2 |              0 |                   0
       2 |    195 |          1 round |               1 |             11 |                   0
       4 |     99 |          3 round |               0 |             19 |                   0
       8 |     51 |          7 round |               0 |             40 |                   3

How Far Depth Should Go

The first three rows show what depth gains. The shallow check kept the warming replica in the pool, dropping 40 reads and 32 writes routed to it; adding the readiness check brought dropped reads to 0 and dropped writes to 22. The remaining 22 writes occur in the eleven rounds the shared store is down, and no endpoint content rescues them — the replica may be healthy, but there is nowhere to write.

The column’s name comes from the Scaling the Data Layer course: a silent lost update is an accepted write invalidated without ever telling its owner. There the cause was conflict; here it is the half-failed replica — the measure is the same.

The local deep check removes it: 17 to 0. The half-failed replica is now out of the pool, because the endpoint honestly answers “can I write”. Its price shows in the out-of-pool column, 32 to 58 — the answer to the previous lesson’s unsolved problem, at the sole cost of removing the broken replica from the pool.

The fourth row is where the sign flips. When the shared state store joins the check, the moment it goes down all three replicas give the same answer and leave the pool at once. Once the pool is empty, not only writes but reads are rejected too: 88 dropped reads, all wasted rejections — a request rejected while at least one replica could have answered it, which is exactly what that column means. The edge cache and the read-from-replica path were up; the check shut them out too. Dropped batch work rose from 11 to 22 alongside.

The fifth row repeats the mistake. When the store feeding only end-of-day billing joins the check, the replicas answering tracking queries also leave the pool once that store goes down: wasted rejects rise from 88 to 176 — no link existed between the billing flow’s store and the tracking query’s path, yet the check tied the two together.

The rule reads from these three rows: the endpoint should ask whether the replica is worse off than its neighbors, not whether the world is fine. A condition that answers the same on every replica, once placed in the endpoint, empties the pool the moment it breaks — the spread comes from the check itself. This is why a shared dependency’s failure is the circuit breaker’s and graceful degradation’s subject, measured in the previous topic, not the health endpoint’s.

How Often the Interval Should Run

The second table runs the same endpoint content (deepLocal) at four check intervals. At interval 1, checks number 390, detection delay is 0 rounds, dropped reads 0 — but there are two false evictions: brief self-recovering outages evicted the replica from the pool. At interval 4, false evictions zero out, and in exchange the round k3 stops is noticed 3 rounds later, dropping 19 reads routed to it in that span. At interval 8, delay rises to 7 rounds, dropped reads to 40, and the half-failed replica, left uninspected between checks, produces 3 silent lost updates.

The trade-off has the same shape as the previous lesson’s threshold trade-off but is a separate axis: the threshold sets how many consecutive checks must fail, the interval sets how infrequent checks are. The two multiply — interval 4 and threshold 3 mean a real failure can, in the worst case, go unnoticed for 12 rounds.

The Numbers for Two Days

The check’s failure-free-day cost has two line items: the check load placed on the dependency, and the requests false eviction shifts onto the remaining replicas. The first is the deep check’s direct bill.

// health/budget.mjs — the check's failure-free-day cost: the check load it puts on the
// dependency and the requests false eviction shifts onto the remaining replicas
const ROUND_SEC = 2;       // KU1 (assumption): one round is one check interval, 2 seconds
const STORE_REQ = 138.89;  // K01: requests/s reaching the store
const PEAK_EDGE = 513.89;  // K01: peak requests/s at the edge
const BRIEF_MONTH = 20;    // KU3 (assumption): 20 transient unresponsive periods a month
const b = (x, n = 2) => x.toFixed(n);

console.log(`${"replicas".padStart(9)}${"interval".padStart(10)}${"checks/s".padStart(11)}` +
  `${"store load added".padStart(19)}${"checks/day".padStart(14)}`);
for (const [replicas, interval] of [[3, 1], [3, 4], [12, 1], [12, 4], [48, 1], [48, 4]]) {
  const sec = interval * ROUND_SEC, rate = replicas / sec;
  console.log(`${String(replicas).padStart(9)}${(sec + " sec").padStart(10)}${b(rate).padStart(11)}` +
    `${("%" + b((100 * rate) / STORE_REQ)).padStart(19)}${Math.round(rate * 86_400).toLocaleString("en-US").padStart(14)}`);
}

console.log(`\nfalse eviction: when one of 3 replicas leaves the pool, the remaining two each carry ` +
  `${b(PEAK_EDGE / 2)} instead of ${b(PEAK_EDGE / 3)} req/s (${b((PEAK_EDGE / 2) / (PEAK_EDGE / 3))}x)`);
for (const interval of [1, 2]) {
  const share = interval === 1 ? 2 / 3 : 1 / 3;   // from the model: how many of the 3 brief outages were caught
  console.log(`interval ${interval}: ${b(BRIEF_MONTH * share, 1)} false evictions a month, each cutting ` +
    `capacity by ${b(100 / 3, 1)}% for up to ${interval * ROUND_SEC} sec`);
}

 replicas  interval   checks/s   store load added    checks/day
        3     2 sec       1.50              %1.08       129,600
        3     8 sec       0.38              %0.27        32,400
       12     2 sec       6.00              %4.32       518,400
       12     8 sec       1.50              %1.08       129,600
       48     2 sec      24.00             %17.28     2,073,600
       48     8 sec       6.00              %4.32       518,400

false eviction: when one of 3 replicas leaves the pool, the remaining two each carry 256.94 instead of 171.30 req/s (1.50x)
interval 1: 13.3 false evictions a month, each cutting capacity by 33.3% for up to 2 sec
interval 2: 6.7 false evictions a month, each cutting capacity by 33.3% for up to 4 sec

With three replicas and a two-second interval, the deep check adds 1.50 queries a second to the store — 1.08 percent of K01’s 138.89 req/s reaching it. The number grows linearly with replica count and runs into a limit there: at 48 replicas, the same interval means 17.28 percent added load. The deep check’s bill scales with replica count, not workload — a scaling decision also changes the check policy. Multiplying the interval by four divides the bill by four.

The second line item is false eviction. When one replica leaves a three-replica pool, the remaining two carry 256.94 req/s instead of 171.30, a 1.50x increase. At interval 1 the model catches two-thirds of the false evictions; with 20 transient unresponsive periods a month assumed (KU3, from the previous lesson), a third of capacity vanishes for two seconds 13.3 times a month. At interval 2 the count drops to 6.7, but each event’s window doubles — the failure-free day’s cost, paid with no failure at all.

The failing day’s gain sits between the first table’s shallow and deepLocal rows: in the same failure schedule, 40 dropped reads fall to 0, 32 dropped writes to 22, 17 silent lost updates to 0, and wasted rejects stay at 0. The same table also shows where the gain ends: one step too much depth produces 88 wasted rejects, two steps too much produces 176.

Summary

• The health endpoint is not one question: the shallow check tests that the process is up, the readiness check that it can take traffic, the deep check that the dependency path works.
• Adding the readiness check saved 40 reads and 10 writes routed to the warming replica; adding the local deep check brought silent lost updates from 17 to 0, at the cost of out-of-pool rounds rising from 32 to 58.
• Adding a shared dependency to the check produces spread: when the store went down all three replicas left the pool, and 88 reads were rejected while answerable; adding the flow-specific store too raised this to 176.
• The rule: the endpoint asks whether a replica is worse off than its neighbors; put a condition in it that answers the same on every replica, and the pool empties.
• The check interval is a separate axis: interval 1 gives 2 false evictions and 0 rounds of delay, interval 8 gives 0 false evictions but 7 rounds of delay, 40 dropped reads, and 3 silent lost updates.
• The failure-free day’s cost is counted: at 3 replicas and a 2-second interval, the deep check adds 1.08 percent load to the store, 17.28 percent at 48 replicas; false eviction raises the remaining replicas’ load to 1.50x, 13.3 times a month.

Next Step

These two lessons counted how long a failure takes to notice and to hand over: threshold, interval, promotion, and failback rounds — always the same quantity, time to recovery. But the writes lost in the previous lesson’s half-failed window are a different quantity neither lesson measured: how much data is left behind when the system comes back? The two questions call for separate thresholds, both paid from the same budget. The next lesson names these thresholds, ties them to backup frequency and replication lag to turn them into numbers, and compares them against K01’s 43.2-minute monthly budget.