Service Discovery

Both of the previous two lessons shared a silent assumption: the other side’s address is a fixed string inside the code. The loan service went to the fee service as 127.0.0.1:8751; the aggregation layers before that went to the catalog service as 127.0.0.1:8741. In a single-copy setup, this is correct. The library loan system’s catalog service, though, is not a single copy: copies are added as load grows, a copy crashes, a copy goes into maintenance. An address with no live counterpart behind a fixed string is as certain a failure as the call itself.

This lesson’s job is to build address resolution as an implementation. What service discovery and a registry are was defined in The Application Layer and Service Interaction course; here the registry actually runs as a process, records are given a lease, health information is tied to routing, and the result is compared against fixed configuration through two numbers: the file touched in deployment when a copy is added, the requests sent to a dead address when a copy dies.

Assumptions: SB16 the catalog service has three copies, and all three serve the same data. SB17 the registry’s lease duration is 900 ms and the copies’ renewal interval is 300 ms; these ratios are smaller than in a real system, to shorten the measurement time. SB18 there are three clients calling the catalog service (loan, notification, fee), and in fixed configuration each carries its own address file.

The Registry Process

The registry is a single process holding addresses and record times. A record’s validity depends on time: a record whose last renewal is older than the lease duration drops off the list. The resolution response returns two numbers — the live record count and the routable addresses. The reason the two are separate is health information.

// discovery/registry.mjs — registry 8761: leased records and health information
import { createServer } from "node:http";

const LEASE = 900;                     // ms: a record not renewed by this deadline drops off the list
const records = new Map();              // address -> { name, healthy, last renewal }

createServer((q, y) => {
  const [root, value] = q.url.split("/").filter(Boolean);
  if (root === "register") {                                   // /register/<name>|<address>|<healthy>
    const [name, address, healthy] = decodeURIComponent(value).split("|");
    records.set(address, { name, healthy: healthy === "1", last: Date.now() });
  }
  const name = root === "resolve" ? value : "catalog";
  const live = [...records.entries()]
    .filter(([, k]) => k.name === name && Date.now() - k.last < LEASE);
  y.writeHead(200, { "Content-Type": "application/json" });
  y.end(JSON.stringify({ lease: LEASE, records: live.length,
    addresses: live.filter(([, k]) => k.healthy).map(([a]) => a) }));
}).listen(8761, "127.0.0.1", () => console.log("registry 127.0.0.1:8761"));

The registry has no delete endpoint. A copy shutting down cleanly could have deleted its own record, but a copy that crashes cannot; so a record’s validity is tied to time, not to the copy’s good intentions. The registry never asks any copy a question itself.

The Registering Copy

Every copy registers itself as it comes up, and renews its record at an interval one third of the lease duration. The renewal call also carries health information. The /break endpoint exists to produce the state where the process is alive but cannot do the work: the process keeps renewing, but the health information it reports changes.

// discovery/catalog.mjs — catalog copy: node discovery/catalog.mjs 8762
import { createServer } from "node:http";

const port = Number(process.argv[2] ?? 8762);
const RENEWAL = 300;                        // ms: one third of the lease duration
let healthy = true;

const register = () => fetch("http://127.0.0.1:8761/register/" +
  encodeURIComponent(`catalog|127.0.0.1:${port}|${healthy ? "1" : "0"}`)).catch(() => {});

createServer((q, y) => {
  if (q.url === "/break") healthy = false;    // the process is alive and renewing its record, but cannot do the work
  const code = healthy ? 200 : 503;
  y.writeHead(code, { "Content-Type": "application/json" });
  y.end(JSON.stringify({ port, healthy }));
}).listen(port, "127.0.0.1", () => {
  register();
  setInterval(register, RENEWAL).unref();
});

The renewal interval must be smaller than the lease duration; the ratio determines how many renewals a copy can miss before it drops off the list. Here the ratio is three: a copy that misses two renewals is still on the list, one that also misses the third drops off.

Two Resolution Paths, the Same Job

The measurement script starts the registry and the three copies itself, writes the fixed configuration file itself, and at each of five stages distributes the same six requests through both paths. On the fixed path, addresses are read from the file; on the discovery path, the registry is asked at the start of every stage.

// discovery/measure.mjs — fixed configuration is run against the same job as the registry
import { spawn } from "node:child_process";
import { writeFileSync, readFileSync } from "node:fs";

const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const proc = {};
const start = (p) => (proc[p] = spawn("node", ["discovery/catalog.mjs", String(p)], { stdio: "ignore" }));
const REQUESTS = 6;                        // requests per stage; distributed round-robin across addresses
const CLIENTS = 3;                      // services calling the catalog: loan, notification, fee

const registry = spawn("node", ["discovery/registry.mjs"], { stdio: "ignore" });
await wait(400);
[8762, 8763, 8764].forEach(start);
writeFileSync("discovery/addresses.json",     // configuration file hand-written at deployment time
  JSON.stringify([8762, 8763, 8764].map((p) => `127.0.0.1:${p}`)));
await wait(500);

const call = async (address) => {
  try { return (await fetch(`http://${address}/book/978-0`)).status === 200; } catch { return false; }
};
const distribute = async (addresses) => {
  let ok = 0;
  for (let i = 0; i < REQUESTS && addresses.length; i += 1) if (await call(addresses[i % addresses.length])) ok += 1;
  return ok;
};
const fixed = async () => {
  const a = JSON.parse(readFileSync("discovery/addresses.json", "utf8"));
  return { ok: await distribute(a), records: a.length, addresses: a.length };
};
const discovery = async () => {
  const d = await (await fetch("http://127.0.0.1:8761/resolve/catalog")).json();
  return { ok: await distribute(d.addresses), records: d.records, addresses: d.addresses.length };
};

const stage = [];
const measure = async (name) => stage.push({ name, f: await fixed(), d: await discovery() });

await measure("all alive");
proc[8764].kill();
await measure("copy killed (immediately)");
await wait(1200);
await measure("lease expired");
await fetch("http://127.0.0.1:8763/break").catch(() => {});
await wait(400);
await measure("copy reported broken");
start(8765);
await wait(600);
await measure("new copy added");

const B = ["fixed ok/req", "fixed addrs", "registry ok/req", "registry records", "registry addrs"];
console.log(`${"stage".padEnd(26)}${B.map((b) => b.padStart(18)).join("")}`);
for (const a of stage) {
  console.log(`${a.name.padEnd(26)}${[`${a.f.ok}/${REQUESTS}`, a.f.addresses, `${a.d.ok}/${REQUESTS}`,
    a.d.records, a.d.addresses].map((v) => String(v).padStart(18)).join("")}`);
}
console.log(`\nlease duration 900 ms, renewal interval 300 ms; requests sent to a dead address ` +
  `in the discovery window = ${REQUESTS - stage[1].d.ok}/${REQUESTS}`);
console.log(`configuration files touched when a new copy is added: fixed ${CLIENTS}, registry 0; ` +
  `clients restarted: fixed ${CLIENTS}, registry 0`);
console.log(`registry = 1 extra process, 1 extra resolution request before every call`);

Object.values(proc).forEach((s) => s.kill());
registry.kill();

stage                           fixed ok/req       fixed addrs   registry ok/req  registry records    registry addrs
all alive                                6/6                 3               6/6                 3                 3
copy killed (immediately)                4/6                 3               4/6                 3                 3
lease expired                            4/6                 3               6/6                 2                 2
copy reported broken                     2/6                 3               6/6                 2                 1
new copy added                           2/6                 3               6/6                 3                 2

lease duration 900 ms, renewal interval 300 ms; requests sent to a dead address in the discovery window = 2/6
configuration files touched when a new copy is added: fixed 3, registry 0; clients restarted: fixed 3, registry 0
registry = 1 extra process, 1 extra resolution request before every call

Reading the Numbers

At the first stage, both paths give the same result: six requests, six successes, three addresses. Discovery shows no contribution at all, and that is correct — fixed configuration works fine while everything is fine. Discovery’s measure only shows up once the layout changes.

The second stage is measured immediately after a copy is killed. Both paths give 4/6. The registry is still returning three addresses, because the lease duration has not yet passed since the dead copy’s last renewal. This interval is called the discovery window, and its measured counterpart is 2/6: of every six requests sent in the window, two went to a dead address. The registry does not reduce the time it takes to learn of a death to zero; it only bounds it.

The third stage shows the window closing. Once the lease duration has passed, the registry returns two addresses, and the six requests split three each across the two: 6/6. Fixed configuration, though, is still carrying three addresses and keeps giving 4/6. This difference is the whole return on discovery: fixed configuration’s error persists until a person fixes the file.

The fourth stage shows why health information is a separate piece of information. Copy 8763 keeps living and renewing its record after the /break call; the registry’s live record count stays at 2, but the routable address count drops to 1. A process being up does not mean it can do the work. Fixed configuration cannot make this distinction at all and drops to 2/6: two requests go to a dead address, and two go to a copy returning 503.

The fifth stage measures the deployment side. When a fourth copy is started, no file is touched on the registry side at all; the copy registers itself, and at the next resolution the routable address count rises to 2. On the fixed-configuration side, the result does not change: the new copy serves no requests at all, because the three clients’ address file still carries the old three. The cost of the change is in the last lines of the output: configuration files touched, 3 against 0; clients restarted, 3 against 0.

The Registry’s Cost and the Failure Modes It Creates

The cost paid has three items. The first is one extra process, and this process is not an ordinary service: when the registry goes down, no client can resolve an address. Fixed configuration’s one advantage is exactly here — the file is read without depending on anything. This is why clients that lean on a registry usually hold on to the result of their last resolution; and then the held list can go stale, and the discovery window grows again.

The second is the resolution request. In the measurement, one resolution was done at the start of every stage; if resolution had been done before every call, every request to the catalog service would have turned into two requests. This is discovery’s counterpart to the previous lesson’s local-hop measure, and it makes caching the resolution mandatory.

The third is the lease duration itself, and both of its directions create a failure. Lengthen it, and the discovery window grows — that is, requests to dead addresses increase. Shorten it, and copies renew more often; the write load on the registry is multiplied by the copy count, and a short network interruption causes live copies to drop off the list. When a dropped copy comes back, requests redistribute again; this oscillation is registry implementations’ most familiar failure.

The common source of the failure modes born here is this: the registry does not hold the truth — it holds a delayed copy of the truth. In fixed configuration, the delay is unbounded and depends on a person fixing the file; in the registry, the delay is bounded by the lease duration. What discovery sells is not correctness — it is an upper bound placed on how long the wrongness lasts.

Summary

• The registry keeps leased records; there is no delete endpoint, because a copy that crashes cannot delete its own record. Copies renew their record at an interval one third of the lease duration.
• While everything is fine, both paths give 6/6; the difference only shows up once the layout changes.
• Immediately after a copy is killed, both paths give 4/6: two of six requests in the discovery window go to a dead address. Once the lease duration has passed, the registry returns to 6/6, fixed configuration stays at 4/6.
• Health information is separate from a live record: a copy that reports broken stays in the record count (records 2) but drops from routing (addresses 1); fixed configuration cannot make this distinction and drops to 2/6.
• When a new copy is added, configuration files touched are 3 against 0, clients restarted are 3 against 0.
• Cost paid: 1 extra process and a single point of failure, 1 extra resolution request per call, and a two-directional failure in the lease duration (a long duration grows the discovery window, a short one grows the renewal load and the oscillation).

Next Step

This topic began with where the boundary should pass, continued with how the two sides of the boundary talk, and ended with how the address gets resolved. The boundary was drawn, synchronous and asynchronous communication were built, edge responsibilities were collected, a client-specific layer was added, network concerns were pulled out of the application, and the address was resolved from outside the code. But one assumption sitting underneath everything the seven lessons measured was never tested: the data on the two sides of the boundary, in the middle of a job, was treated as if it were under the guarantee of a single transaction. The loan service wrote the record, and the fee service collected the penalty; the two happened in separate processes, on separate data, at separate times. The previous lesson’s double-collection measurement was the first crack in that assumption. The next topic takes on exactly this: when a workflow touches more than one service’s data, how is it ensured that all of them happen together or none of them happen, what happens when that cannot be ensured, and what units its cost is measured in.