Health Checks

Throughout the previous lesson’s model, a replica could serve the moment it entered the pool. The deployment controller advanced on that same assumption: once a replica came up, it counted; once it counted, it took a share. In reality there is an interval between a process starting and being able to answer requests — the connection pool gets built, the cache gets warmed, dependencies get polled. If nobody reports this interval, the controller opens the replica to traffic too early. The reverse also happens: it treats a slow-starting replica as failed, kills it, and repeats this at every startup.

There are two delegated decisions here, and they are often mistaken for one. The first is should this replica be restarted, the second should requests be sent to this replica. If the answer to the first is wrong, a working process gets killed; if the answer to the second is wrong, a request goes to a process that is not working. What the system needs to know to make these two decisions are separate information items: is the process itself responding, does it hold its dependencies, and is it still starting up. Three questions, three probes. This lesson measures what it costs to merge the questions.

The term readiness probe was used earlier in the context of single-machine composition; here the same concept appears at cluster scale, as the probe that determines pool membership.

• OO7. The cluster runs the fictional regional measurement network’s verification service with eight replicas, with a warm start; fifty requests arrive per round. Both the network and the figures are fictional.
• OO8. Six replicas become able to serve in six rounds, two in twenty-five rounds. The two slow ones warm a cache at startup; this difference comes from the application, not from the cluster.
• OO9. The events are hand-written; there is no randomness. Transient dependency loss resolves on its own once its duration elapses; permanent stuck state is resolved only by a restart.
• OO10. A probe’s answer is binary: a replica either answers or it does not. Network-caused false negatives, the probe endpoint’s own latency, and timeout settings are not modeled.
• OO11. A restart happens instantly. Rescheduling the pod, pulling the image, and node selection are outside this measurement; all of them would enlarge the delay without changing its direction.
• OO12. The “actually healthy” column is the average number of healthy replicas per round. The target is eight, and this number looks at the replicas’ real state, not at what the probes believe.

Measurement Setup

What is measured is not a real cluster but a model built with node. In every round, a replica is in one of four states: starting, stuck, missing a dependency, healthy. The probes do not see this state directly; each only gets a yes or no answer to the one question within its scope.

// health.mjs — round-based model of liveness, readiness, and startup probes
export const ROUND = 180, ROUND_REQUESTS = 50, STARTUP = [6, 6, 6, 6, 6, 6, 25, 25];
export const SCHEDULED = [{ replica: 6, round: 20 }, { replica: 7, round: 20 }];  // two replicas move to the new version
export const INCIDENT = [                                 // events hand-written, no randomness
  { replica: 3, round: 50, type: 'noDependency', duration: 18 }, // transient: resolves on its own
  { replica: 1, round: 80, type: 'stuck' },                       // permanent: only a restart fixes it
  { replica: 4, round: 120, type: 'noDependency', duration: 10 },
  { replica: 5, round: 140, type: 'stuck' } ];

export function run({ name, liveness, readiness, startup }) {
  const reps = STARTUP.map(() => ({ readyRound: 0, stuck: false, cMiss: 0, hMiss: 0, bMiss: 0,
    startupDone: true, inPool: true, restarts: 0 }));
  const s = { name, restarts: 0, unnecessary: 0, duringStartup: 0, early: 0, failed: 0, poolRounds: 0, detect: [] };
  const stuckRound = {};
  const status = (i, t) => t < reps[i].readyRound ? 'starting' : reps[i].stuck ? 'stuck'
    : INCIDENT.some((a) => a.replica === i && a.duration && t >= a.round && t < a.round + a.duration)
      ? 'noDependency' : 'healthy';
  const start = (i, t) => Object.assign(reps[i], { readyRound: t + STARTUP[i], stuck: false,
    cMiss: 0, hMiss: 0, bMiss: 0, startupDone: !startup, inPool: !readiness });
  for (let t = 0; t < ROUND; t += 1) {
    for (const p of SCHEDULED) if (p.round === t) start(p.replica, t);      // scheduled startup, not counted
    for (const a of INCIDENT) if (a.round === t && a.type === 'stuck') { reps[a.replica].stuck = true; stuckRound[a.replica] = t; }
    for (let i = 0; i < reps.length; i += 1) {
      const d = status(i, t);
      if (!reps[i].startupDone) {                          // other probes do not look until the startup probe finishes
        if (t % startup.p) continue;
        if (d !== 'starting') reps[i].startupDone = true;
        else if ((reps[i].bMiss += 1) >= startup.e) { s.restarts += 1; s.duringStartup += 1; s.unnecessary += 1; reps[i].restarts += 1; start(i, t); }
        continue;
      }
      if (liveness && t % liveness.p === 0) {
        const bad = liveness.scope === 'dependency' ? d !== 'healthy' : d === 'starting' || d === 'stuck';
        if (!bad) reps[i].cMiss = 0;
        else if ((reps[i].cMiss += 1) >= liveness.e) {
          s.restarts += 1; reps[i].restarts += 1;
          if (d === 'stuck') s.detect.push(t - stuckRound[i]); else s.unnecessary += 1;
          if (d === 'starting') s.duringStartup += 1;
          start(i, t);
        }
      }
      if (readiness && t % readiness.p === 0) {
        if (d === 'healthy') { reps[i].hMiss = 0; reps[i].inPool = true; }
        else if ((reps[i].hMiss += 1) >= readiness.e) reps[i].inPool = false;
      }
    }
    const pool = reps.map((_, i) => i).filter((i) => reps[i].inPool);         // the pool the probe believes in
    s.poolRounds += reps.filter((_, i) => status(i, t) === 'healthy').length; // actually able to serve
    for (let r = 0; r < ROUND_REQUESTS; r += 1) {
      if (!pool.length) { s.failed += 1; continue; }
      const d = status(pool[r % pool.length], t);
      if (d === 'healthy') continue;
      s.failed += 1;
      if (d === 'starting') s.early += 1;
    }
  }
  s.loop = reps.filter((c) => c.restarts >= 3).length;
  return s;
}

Two lines carry this model’s entire distinction. If the liveness probe’s scope field is process, the probe only fails on starting and stuck states — it does not see dependency loss. If the same field is dependency, the probe asks the same question as the readiness probe; a single probe endpoint then drives both restarts and pool membership. The second line is the difference between pool and poolRounds: the former is the pool the probe believes in, the latter is the number of replicas actually able to serve. Wherever the two can diverge, there is a wrong decision.

Four Settings, One Event Sequence

// run3.mjs — four settings are run against the same event sequence
import { run, ROUND, ROUND_REQUESTS } from './health.mjs';

const SETTING = [
  { name: 'no probes', liveness: null, readiness: null, startup: null },
  { name: 'single combined probe', liveness: { p: 5, e: 3, scope: 'dependency' },
    readiness: { p: 5, e: 3 }, startup: null },
  { name: 'liveness + readiness', liveness: { p: 5, e: 3, scope: 'process' },
    readiness: { p: 2, e: 1 }, startup: null },
  { name: 'plus startup', liveness: { p: 5, e: 3, scope: 'process' },
    readiness: { p: 2, e: 1 }, startup: { p: 5, e: 6 } },
];
console.log(`8 replicas, ${ROUND} rounds, ${ROUND_REQUESTS} requests per round; two replicas start up over 25 rounds`);
console.log('round 20: two slow replicas restart on schedule | rounds 80 and 140: permanent stuck state');
console.log('rounds 50 and 120: transient dependency loss, lasts 18 and 10 rounds, resolves on its own\n');
console.log('setting                restarts  unnecessary  during     restart  early   total   actually   detect');
console.log('                                 restarts     startup    loop     failed  failed  healthy    stuck');
for (const a of SETTING) {
  const s = run(a);
  console.log(`${s.name.padEnd(21)} ${String(s.restarts).padEnd(8)} ${String(s.unnecessary).padEnd(9)} ` +
    `${String(s.duringStartup).padEnd(9)} ${String(s.loop).padEnd(8)} ${String(s.early).padEnd(7)} ` +
    `${String(s.failed).padEnd(7)} ${(s.poolRounds / ROUND).toFixed(2).padEnd(10)} ${s.detect.join(', ') || '-'}`);
}

8 replicas, 180 rounds, 50 requests per round; two replicas start up over 25 rounds
round 20: two slow replicas restart on schedule | rounds 80 and 140: permanent stuck state
rounds 50 and 120: transient dependency loss, lasts 18 and 10 rounds, resolves on its own

setting                restarts  unnecessary  during     restart  early   total   actually   detect
                                 restarts     startup    loop     failed  failed  healthy    stuck
no probes             0        0         0         0        300     1408    6.79       -
single combined probe 23       21        20        2        0       330     5.89       10, 10
liveness + readiness  22       20        20        2        0       0       5.89       10, 10
plus startup          2        0         0         0        0       0       7.39       10, 10

The no-probes row gives the baseline: no replica was restarted, the two permanent stuck states went undetected for a hundred rounds and forty rounds, and one thousand four hundred eight requests failed. Three hundred of those went to replicas still starting up — meaning the sole cost of having no readiness probe is three hundred requests. The remaining one thousand one hundred eight are requests sent to replicas that were stuck or had lost a dependency.

The single combined probe cut failed requests from one thousand four hundred to three hundred thirty and detected the stuck state in ten rounds. Its cost is in the second and third columns: twenty-one of twenty-three restarts were unnecessary. This probe also counts dependency loss as a failure, so an outside outage ends up killing a healthy process. This is the price of being unable to distinguish permanent stuck state from transient dependency loss: a condition that would resolve on its own gets fixed with a restart instead.

The third row limits the liveness probe’s scope to the process and separates out the readiness probe. Failed requests dropped to zero — the pool now pulls a replica that has lost its dependency out within two rounds, without killing it. Unnecessary restarts are still twenty. Their source is not transient failure: all twenty are killed during startup. The liveness probe checks every five rounds and kills on three consecutive failures; that fifteen-round allowance is not enough for the two replicas that take twenty-five rounds to start. Those two replicas get killed in the middle of every startup and start over — two replicas caught in a restart loop.

This row’s real measurement is the second-to-last column. Failed requests are zero, but the actually-healthy replica count is 5.89 — in a cluster targeting eight, more than two replicas are permanently missing. Because the readiness probe keeps the looping replicas out of the pool, this loss produces not a single failure record. Anyone watching the cluster’s error rate sees everything looking fine; the loss stops at capacity and, when load rises, falls on the remaining six replicas.

The fourth row adds the startup probe: restarts drop from twenty-two to two — that is, down to only the two real stuck states — unnecessary restarts go to zero, and the healthy-replica count rises to 7.39. Detection time for the stuck state has not changed. What the startup probe does is not catch a new kind of failure; it is silencing the liveness probe for the duration of the startup window.

Threshold and Period

The third row’s killing during startup could also be removed by loosening the liveness probe’s settings. The sweep shows what that costs.

// run4.mjs — the liveness probe's period and threshold are swept, first without a startup probe
import { run, ROUND } from './health.mjs';

const READINESS = { p: 2, e: 1 };
const write = (label, startup) => {
  console.log(`\n${label}`);
  console.log('period  thresh  tolerance  detect      killed      restart  actually');
  console.log('                (rounds)   stuck       at startup  loop     healthy');
  for (const p of [2, 5, 10]) for (const e of [2, 3, 6]) {
    const s = run({ name: '', liveness: { p, e, scope: 'process' }, readiness: READINESS, startup });
    console.log(`${String(p).padEnd(8)} ${String(e).padEnd(5)} ${String(p * e).padEnd(8)} ` +
      `${(s.detect.join(', ') || '-').padEnd(11)} ${String(s.duringStartup).padEnd(10)} ` +
      `${String(s.loop).padEnd(8)} ${(s.poolRounds / ROUND).toFixed(2)}`);
  }
};
console.log('two slow replicas become ready over 25 rounds; permanent stuck state at rounds 80 and 140');
write('no startup probe:', null);
write('startup probe present (period 5, threshold 6 — a 30-round startup allowance):', { p: 5, e: 6 });

two slow replicas become ready over 25 rounds; permanent stuck state at rounds 80 and 140

no startup probe:
period  thresh  tolerance  detect      killed      restart  actually
                (rounds)   stuck       at startup  loop     healthy
2        2     4        2, 2        113        4        5.29
2        3     6        4, 4        52         2        5.96
2        6     12       10, 10      26         2        5.89
5        2     10       5, 5        32         2        5.94
5        3     15       10, 10      20         2        5.89
5        6     30       25, 25      0          0        7.22
10       2     20       10, 10      16         2        5.89
10       3     30       20, 20      2          0        7.06
10       6     60       50          0          0        7.03

startup probe present (period 5, threshold 6 — a 30-round startup allowance):
period  thresh  tolerance  detect      killed      restart  actually
                (rounds)   stuck       at startup  loop     healthy
2        2     4        2, 2        0          0        7.48
2        3     6        4, 4        0          0        7.46
2        6     12       10, 10      0          0        7.39
5        2     10       5, 5        0          0        7.44
5        3     15       10, 10      0          0        7.39
5        6     30       25, 25      0          0        7.22
10       2     20       10, 10      0          0        7.39
10       3     30       20, 20      0          0        7.28
10       6     60       50          0          0        7.03

In the top table, a single setting does two jobs at once, and the two jobs pull in opposite directions. As the tolerance window shortens, the stuck state gets detected faster — in two rounds, with a four-round window — but killing during startup jumps to one hundred thirteen and four replicas fall into the loop; the healthy-replica count drops to 5.29. As the window lengthens, killing falls off and detection latency grows. Two rows never kill during startup, with tolerances of thirty and sixty rounds: they detect the stuck state in twenty-five and fifty rounds. The last row shows a single number in the stuck-detection column — with a sixty-round window, it never detected the second stuck state, at round one hundred forty, before the measurement horizon ran out.

The bottom table repeats the same sweep with a startup probe that grants a thirty-round startup allowance. The killed-during-startup column is zero in every row, and so is the loop column. The tension is gone, because the two jobs were split into two settings: the startup probe covers the startup window, and the liveness probe now only looks at replicas that have finished starting up. Period and threshold can now be chosen by a single criterion — detection latency for the stuck state. The fastest row is the four-round window: it detects in two rounds, and its healthy-replica count of 7.48 is the sweep’s highest.

The distinction among the three probes is exactly the difference between these two tables. The same liveness setting produces 113 unnecessary kills in the top table and zero in the bottom. The only thing that changed is the system’s ability to distinguish starting up from failed. That distinction is an information item, and its cost is running one more probe; its payoff is that the liveness setting becomes independent of the application’s startup time.

Summary

• In the cluster without probes, two permanent stuck states went entirely unnoticed and 1408 requests failed; 300 of those went to replicas still starting up — the share attributable to having no readiness probe.
• The single combined probe cut failed requests to 330, but 21 of 23 restarts were unnecessary: a probe that cannot tell dependency loss apart from stuck state fixes a self-resolving condition with a restart.
• Limiting the liveness probe’s scope to the process and separating out the readiness probe cut failed requests to zero, but the two slow-starting replicas kept getting killed mid-startup and fell into a loop.
• The restart loop is invisible in the error rate: with failed requests at zero, the actually-healthy replica count was 5.89 against a target of 8; the loss shows up in capacity, not in the failure log.
• When one setting does two jobs, shortening the tolerance window speeds up detection and pushes killing during startup to 113; lengthening it ends the killing, but a 60-round window never catches the second stuck state at all.
• Once the startup probe is added, killing during startup goes to zero in all nine rows of the sweep; the liveness setting can now be chosen by detection latency alone, and the 4-round window is both the fastest and the highest-capacity row.

Next Step

The controllers in these two lessons act on the cluster without restriction: they change the deployment object, kill replicas, write pool membership. People and other automated processes use the same interface, and none of them has yet run into the question “are you authorized to do this.” The next lesson turns that question into a number: it builds a subject × verb × object privilege matrix, measures how many operations drop out going from broad privilege to least privilege and how many lines of role definition that costs — and counts the paths by which privilege granted to one subject can reach another subject.