Service Discovery

The previous lesson made the replicas each other’s equals: state moved out of the process, load became freely splittable, and the required replica count fell to 3. But one part of the measurement setup still stood written by hand — the driver script took the replicas’ ports from the command line, and the client carried the same numbers by hand. The Traffic Layer course’s load balancer had taken the replica list the same way. Nowhere is it written who updates that list when the replica count changes, when a replica drops, or when a new replica comes up.

Service discovery is finding, at runtime, the addresses of a service’s currently live replicas. It is easy to confuse with the load balancer, and the two do separate jobs: the balancer distributes the request, discovery finds the address. The balancer’s pool is the list discovery produces; without discovery that list sits in a configuration file. This lesson measures where the list comes from, how long it takes to update, and how many requests go to a dead address in that time.

Two Placements

Static list. Addresses are written into every calling service’s configuration. When a replica drops, no caller notices on its own; when the list changes, every caller’s configuration is updated separately.

Registry. Replicas register themselves with a registry when they come up and send a heartbeat at regular intervals. A registration has a time to live; when the heartbeat stops, the registration drops once that period elapses. The caller resolves the address from the registry at call time.

Both placements carry a delay, and that delay is the number this lesson measures: the detection window between a replica going silent and dropping out of the list.

Model

The setup below is a model: the registry, heartbeat, and resolution are modules, durations advance on a logical clock, and a failure and a new replica are each triggered by a parameter. No real discovery infrastructure, service mesh product, or cloud environment is set up. Two of the model’s numbers are this course’s assumption and are not added to K01’s table.

U2 — heartbeat interval 1000 ms, registration time to live 3000 ms. Rationale: the time to live is chosen as a few multiples of the interval so that a single missed heartbeat does not drop the registration; three times was chosen here. The sensitivity is computed below.

// discovery/registry.mjs — service REGISTRY model: registration, heartbeat, time to live, and graceful shutdown.
// Time is not real, it is a logical clock (ms); failure and new replica are triggered by parameters.
export const HEARTBEAT = 1000;         // assumption U2: heartbeat interval (ms)
export const TTL = 3000;               // assumption U2: registration's time to live (ms)

export function registry() {
  const records = new Map();
  return {
    heartbeat: (name, address, t) => records.set(address, { name, last: t }),
    deleteRecord: (address) => records.delete(address),
    resolve: (name, t) => [...records.entries()]
      .filter(([, r]) => r.name === name && t - r.last < TTL)
      .map(([address]) => address),
  };
}

// Resolution cache: the caller holds the resolved list for a period of time.
export function resolver(r, name, cacheMs) {
  let list = [], refreshed = -Infinity;
  return (t) => {
    if (t - refreshed >= cacheMs) { list = r.resolve(name, t); refreshed = t; }
    return list;
  };
}

The run repeats the same failure across four placements. Replica k2 goes silent at the fifth second, replica k4 comes up at the sixth second; the caller sends a request every ten milliseconds and picks an address from the list in turn.

// discovery/measure.mjs — same failure in four placements: static list, registry, resolution cache, graceful shutdown
import { registry, resolver, HEARTBEAT, TTL } from "./registry.mjs";

const REPLICAS = ["k1", "k2", "k3"];   // replicas up at the start
const FAILURE = 5000;                  // k2 goes silent at this instant (model parameter)
const NEW = 6000;                      // k4 comes up at this instant (model parameter)
const START = 5000, END = 11000, STEP = 10;   // request window and request interval (ms)
const CONSUMERS = 4;                   // number of calling services: gateway, billing, delivery-ops, batch worker

function run({ cache = 0, gracefulShutdown = false, static: isStatic = false }) {
  const r = registry();
  const resolve = resolver(r, "tracking", cache);
  let seq = 0, dropped = 0, wentToNew = 0, lastDropped = null, stillListed = false;
  for (let t = 0; t <= END; t += STEP) {
    for (const k of REPLICAS) {                     // heartbeats
      if (t % HEARTBEAT === 0 && (k !== "k2" || t <= FAILURE)) r.heartbeat("tracking", k, t);
    }
    if (t % HEARTBEAT === 0 && t >= NEW) r.heartbeat("tracking", "k4", t);
    if (gracefulShutdown && t === FAILURE) r.deleteRecord("k2");
    if (t < START) continue;
    const list = isStatic ? [...REPLICAS] : resolve(t);   // static list never learns about the failure
    stillListed = list.includes("k2");
    const chosen = list[seq++ % list.length];
    if (chosen === "k2") { dropped += 1; lastDropped = t; }
    if (chosen === "k4") wentToNew += 1;
  }
  const window = stillListed ? "manual" : lastDropped === null ? "0" : String(lastDropped - FAILURE + STEP);
  return { dropped, wentToNew, window };
}

const PLACEMENT = [
  ["static list", { static: true }, CONSUMERS],
  ["registry", {}, 1],
  ["registry + 800 cache", { cache: 800 }, 1],
  ["registry + graceful shutdown", { gracefulShutdown: true }, 1],
];

console.log(`heartbeat ${HEARTBEAT} ms, time to live ${TTL} ms; k2 goes silent at ${FAILURE} ms, ` +
  `k4 registers at ${NEW} ms`);
console.log(`request window ${START}-${END} ms, one request every ${STEP} ms = ${(END - START) / STEP + 1} requests\n`);
console.log(`${"placement".padEnd(30)}${"detection window(ms)".padStart(23)}` +
  `${"to dead address".padStart(17)}${"to new replica".padStart(19)}${"configuration points".padStart(22)}`);
for (const [name, option, point] of PLACEMENT) {
  const r = run(option);
  console.log(`${name.padEnd(30)}${r.window.padStart(23)}${String(r.dropped).padStart(17)}` +
    `${String(r.wentToNew).padStart(19)}${String(point).padStart(22)}`);
}

heartbeat 1000 ms, time to live 3000 ms; k2 goes silent at 5000 ms, k4 registers at 6000 ms
request window 5000-11000 ms, one request every 10 ms = 601 requests

placement                        detection window(ms)  to dead address     to new replica  configuration points
static list                                    manual              200                  0                     4
registry                                         2980               83                150                     1
registry + 800 cache                             3180               93                134                     1
registry + graceful shutdown                        0                0                167                     1

Measured Effect

These numbers come out of the model and are deterministic: the logical clock and the fixed request interval give the same result on every run.

The static list never detects anything. 200 of the 601 requests go to the dead address and the window never closes; closing it requires one person to change four separate configurations. The to new replica column gives the second half of the picture: k4 came up and, in the static placement, received no requests at all. The benefit of the added capacity waits on the same delay.

The registry closes the window, but not instantly. The window is 2980 milliseconds — nearly the entire time to live. Eighty-three requests going to the dead address is not a defect, it is the direct consequence of the chosen time to live: a registration sits in the list for three more seconds after the heartbeat stops. k4 receives 150 requests in this placement — the registry does not only remove what dropped, it also introduces what arrived.

The resolution cache extends the window. When the caller holds the list in hand for 800 milliseconds, the window grows from 2980 to 3180 and requests to the dead address rise from 83 to 93. The cache lowers the registry’s load, and it does this by growing the window; the two numbers are two faces of the same decision.

Graceful shutdown resets the window. The replica deletes its own registration before it shuts down, and no request ever reaches the dead address. This row matters because most replicas shut down planned — a release, a scale-down, a migration. Planned shutdowns do not have to pay the window; the time to live is only insurance for unplanned silence.

The last column is a separate measure: the configuration point count is 4 in the static placement, 1 with the registry. The number grows with the number of calling services; repeating the replica list in every consumer means the work of a replica-count change is multiplied by the number of consumers.

Back to the Calculation

The window is a duration, and K01’s and K02’s numbers turn it into a request count. The Traffic Layer course did the same calculation for the health check and found 34.26 requests per failure.

// discovery/budget.mjs — the detection window's counterpart in K01's request rate and outage budget
const PEAK_EDGE = 513.89;                        // K01 Back-of-the-Envelope Estimation: peak edge req/s
const REPLICAS = 3;                              // K02 Role of the Load Balancer: minimum replica count
const FAILURES_MONTH = 2.82;                     // K01 Availability in Numbers: failures per month that fit the budget
const MONTH_REQUESTS = (12_000_000 + 2_800_000) * 30;  // K01: daily tracking query + state event
const HEALTH = 0.2, REGISTRY = 2.98;             // K02 assumption Y3's window and the window the model measured
const WINDOW = [
  ["health check (K02 Y3)", HEALTH], ["registry (U2)", REGISTRY],
  ["registry + 800 cache", 3.18], ["graceful shutdown", 0], ["manual intervention (K01)", 600],
];

const replicaRate = PEAK_EDGE / REPLICAS;
console.log(`rate per replica = ${PEAK_EDGE} / ${REPLICAS} = ${replicaRate.toFixed(2)} req/s\n`);
console.log(`${"window source".padEnd(26)}${"s".padStart(7)}${"per failure".padStart(20)}` +
  `${"monthly".padStart(13)}${"request-based availability".padStart(31)}`);
for (const [name, s] of WINDOW) {
  const one = replicaRate * s, monthly = one * FAILURES_MONTH;
  console.log(`${name.padEnd(26)}${String(s).padStart(7)}${one.toFixed(2).padStart(20)}` +
    `${monthly.toFixed(0).padStart(13)}${`${(100 * (1 - monthly / MONTH_REQUESTS)).toFixed(5)}%`.padStart(31)}`);
}

console.log(`\ncombined: effective window = min(${HEALTH}, ${REGISTRY}) = ${Math.min(HEALTH, REGISTRY)} s ` +
  `-> ${(replicaRate * Math.min(HEALTH, REGISTRY)).toFixed(2)} requests; discovery alone is ${(REGISTRY / HEALTH).toFixed(1)}x worse`);
const heartbeatRate = (ttl) => (REPLICAS * 1000) / (ttl / 3);
console.log(`sensitivity: if U2's time to live went 3000 -> 1000 ms, window would be ${(REGISTRY - 2).toFixed(2)} s, ` +
  `${(replicaRate * (REGISTRY - 2)).toFixed(2)} requests per failure`);
console.log(`  the registry's own load ${heartbeatRate(3000).toFixed(2)} -> ${heartbeatRate(1000).toFixed(2)} beats/s ` +
  `(K02's health check load is 15 req/s)`);

rate per replica = 513.89 / 3 = 171.30 req/s

window source                   s         per failure      monthly     request-based availability
health check (K02 Y3)         0.2               34.26           97                      99.99998%
registry (U2)                2.98              510.46         1440                      99.99968%
registry + 800 cache         3.18              544.72         1536                      99.99965%
graceful shutdown               0                0.00            0                     100.00000%
manual intervention (K01)     600           102778.00       289834                      99.93472%

combined: effective window = min(0.2, 2.98) = 0.2 s -> 34.26 requests; discovery alone is 14.9x worse
sensitivity: if U2's time to live went 3000 -> 1000 ms, window would be 0.98 s, 167.87 requests per failure
  the registry's own load 3.00 -> 9.00 beats/s (K02's health check load is 15 req/s)

These numbers are in the computed value class. The table gives a ranking: the health check 34.26, the registry 510.46, manual intervention 102,778 requests. Discovery is two hundred times better than a hand-held list, and fifteen times worse than the health check. The two comparisons have to be read together, because this is where the lesson’s real argument sits: service discovery is not a failure-detection tool. Its job is making sure the list comes from the correct source; detecting a failure quickly is still the health check’s job.

When both are used together, the effective window is set by the smaller one. The balancer removes a dropped replica from its pool in 200 milliseconds; the registry’s registration sits for three more seconds, but that registration never reaches the balancer’s pool in the first place. One layer does not stand in for the other; one produces the other’s input.

The sensitivity row shows U2’s price. If the time to live is cut from 3000 milliseconds to 1000, the window falls to 0.98 seconds and the requests per failure fall to 167.87, but the heartbeat interval also has to shrink to a third: the registry’s own load rises from 3.00 beats to 9.00 beats/s. The comparison point is K02’s health check load — 15 req/s. Discovery’s observation load stays below the health check’s load, and shortening the window grows it linearly.

Summary

• Service discovery and load balancing are separate jobs: the balancer distributes the request, discovery finds the address; the list discovery produces is the balancer’s pool.
• In the static list, 200 of 601 requests went to the dead address and the window never closed on its own; the newly booted replica received no requests, and the configuration point count was as large as the consumer count (4).
• In the registry, the window was 2980 ms with 83 requests going to the dead address; the new replica received 150 requests and the configuration point count fell to 1.
• The resolution cache lowers the registry’s load while raising the window to 3180 ms and the dropped requests to 93; graceful shutdown brings the window to 0, because the replica deletes its own registration.
• Back to K01: requests per failure are 34.26 for the health check, 510.46 for the registry, and 102,778 for manual intervention; discovery is two hundred times better than a hand-held list and fifteen times worse than the health check.
• U2 sensitivity: if the time to live falls to 1000 ms, the window falls to 0.98 seconds and the registry’s own load rises from 3.00 to 9.00 beats/s (K02’s health check load is 15 req/s).

Next Step

The address is now found at runtime and the list keeps itself current. That was the last piece needed for two services to reach each other; but what they say once they reach each other was never chosen. The gateway called the delivery operations service and the billing service in parallel and merged their responses, and the shape of those calls was never discussed anywhere. The next lesson takes up that choice: the same internal call is written three separate ways and compared with three measures — the number of internal calls made for one external request, the bytes crossing the internal boundary, and the number of names the caller has to know.