---
title: 'Service Discovery'
source: 'https://academia.sh/en/courses/service-architectures/service-discovery'
course: 'Service Architectures'
language: en
updated: '2026-08-23T07:00:31+00:00'
license: 'CC BY-SA 4.0'
---

# Service Discovery

Building address resolution as an implementation: a registry process with leased records, copies registering themselves, and health information tied to routing; compared against fixed configuration through the file touched in deployment and the requests sent to a dead address.

Both of the previous two lessons shared a silent assumption: the other side's address is a fixed
string inside the code. The loan service went to the fee service as `127.0.0.1:8751`; the
aggregation layers before that went to the catalog service as `127.0.0.1:8741`. In a single-copy
setup, this is correct. The library loan system's catalog service, though, is not a single copy:
copies are added as load grows, a copy crashes, a copy goes into maintenance. An address with no
live counterpart behind a fixed string is as certain a failure as the call itself.

This lesson's job is to build address resolution as an **implementation**. What service discovery
and a registry are was defined in The Application Layer and Service Interaction course; here the
registry actually runs as a process, records are given a lease, health information is tied to
routing, and the result is compared against fixed configuration through two numbers: the file
touched in deployment when a copy is added, the requests sent to a dead address when a copy dies.

Assumptions: **SB16** the catalog service has three copies, and all three serve the same data.
**SB17** the registry's lease duration is 900 ms and the copies' renewal interval is 300 ms;
these ratios are smaller than in a real system, to shorten the measurement time. **SB18** there
are three clients calling the catalog service (loan, notification, fee), and in fixed
configuration each carries its own address file.

## The Registry Process

The registry is a single process holding addresses and record times. A record's validity depends
on time: a record whose last renewal is older than the lease duration drops off the list. The
resolution response returns two numbers — the live record count and the routable addresses. The
reason the two are separate is health information.

```js
// discovery/registry.mjs — registry 8761: leased records and health information
import { createServer } from "node:http";

const LEASE = 900;                     // ms: a record not renewed by this deadline drops off the list
const records = new Map();              // address -> { name, healthy, last renewal }

createServer((q, y) => {
  const [root, value] = q.url.split("/").filter(Boolean);
  if (root === "register") {                                   // /register/<name>|<address>|<healthy>
    const [name, address, healthy] = decodeURIComponent(value).split("|");
    records.set(address, { name, healthy: healthy === "1", last: Date.now() });
  }
  const name = root === "resolve" ? value : "catalog";
  const live = [...records.entries()]
    .filter(([, k]) => k.name === name && Date.now() - k.last < LEASE);
  y.writeHead(200, { "Content-Type": "application/json" });
  y.end(JSON.stringify({ lease: LEASE, records: live.length,
    addresses: live.filter(([, k]) => k.healthy).map(([a]) => a) }));
}).listen(8761, "127.0.0.1", () => console.log("registry 127.0.0.1:8761"));
```

The registry has no delete endpoint. A copy shutting down cleanly could have deleted its own
record, but a copy that crashes cannot; so a record's validity is tied to time, not to the copy's
good intentions. The registry never asks any copy a question itself.

## The Registering Copy

Every copy registers itself as it comes up, and renews its record at an interval one third of the
lease duration. The renewal call also carries health information. The `/break` endpoint exists to
produce the state where the process is alive but cannot do the work: the process keeps renewing,
but the health information it reports changes.

```js
// discovery/catalog.mjs — catalog copy: node discovery/catalog.mjs 8762
import { createServer } from "node:http";

const port = Number(process.argv[2] ?? 8762);
const RENEWAL = 300;                        // ms: one third of the lease duration
let healthy = true;

const register = () => fetch("http://127.0.0.1:8761/register/" +
  encodeURIComponent(`catalog|127.0.0.1:${port}|${healthy ? "1" : "0"}`)).catch(() => {});

createServer((q, y) => {
  if (q.url === "/break") healthy = false;    // the process is alive and renewing its record, but cannot do the work
  const code = healthy ? 200 : 503;
  y.writeHead(code, { "Content-Type": "application/json" });
  y.end(JSON.stringify({ port, healthy }));
}).listen(port, "127.0.0.1", () => {
  register();
  setInterval(register, RENEWAL).unref();
});
```

The renewal interval must be smaller than the lease duration; the ratio determines how many
renewals a copy can miss before it drops off the list. Here the ratio is three: a copy that
misses two renewals is still on the list, one that also misses the third drops off.

## Two Resolution Paths, the Same Job

The measurement script starts the registry and the three copies itself, writes the fixed
configuration file itself, and at each of five stages distributes the same six requests through
both paths. On the fixed path, addresses are read from the file; on the discovery path, the
registry is asked at the start of every stage.

```js
// discovery/measure.mjs — fixed configuration is run against the same job as the registry
import { spawn } from "node:child_process";
import { writeFileSync, readFileSync } from "node:fs";

const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const proc = {};
const start = (p) => (proc[p] = spawn("node", ["discovery/catalog.mjs", String(p)], { stdio: "ignore" }));
const REQUESTS = 6;                        // requests per stage; distributed round-robin across addresses
const CLIENTS = 3;                      // services calling the catalog: loan, notification, fee

const registry = spawn("node", ["discovery/registry.mjs"], { stdio: "ignore" });
await wait(400);
[8762, 8763, 8764].forEach(start);
writeFileSync("discovery/addresses.json",     // configuration file hand-written at deployment time
  JSON.stringify([8762, 8763, 8764].map((p) => `127.0.0.1:${p}`)));
await wait(500);

const call = async (address) => {
  try { return (await fetch(`http://${address}/book/978-0`)).status === 200; } catch { return false; }
};
const distribute = async (addresses) => {
  let ok = 0;
  for (let i = 0; i < REQUESTS && addresses.length; i += 1) if (await call(addresses[i % addresses.length])) ok += 1;
  return ok;
};
const fixed = async () => {
  const a = JSON.parse(readFileSync("discovery/addresses.json", "utf8"));
  return { ok: await distribute(a), records: a.length, addresses: a.length };
};
const discovery = async () => {
  const d = await (await fetch("http://127.0.0.1:8761/resolve/catalog")).json();
  return { ok: await distribute(d.addresses), records: d.records, addresses: d.addresses.length };
};

const stage = [];
const measure = async (name) => stage.push({ name, f: await fixed(), d: await discovery() });

await measure("all alive");
proc[8764].kill();
await measure("copy killed (immediately)");
await wait(1200);
await measure("lease expired");
await fetch("http://127.0.0.1:8763/break").catch(() => {});
await wait(400);
await measure("copy reported broken");
start(8765);
await wait(600);
await measure("new copy added");

const B = ["fixed ok/req", "fixed addrs", "registry ok/req", "registry records", "registry addrs"];
console.log(`${"stage".padEnd(26)}${B.map((b) => b.padStart(18)).join("")}`);
for (const a of stage) {
  console.log(`${a.name.padEnd(26)}${[`${a.f.ok}/${REQUESTS}`, a.f.addresses, `${a.d.ok}/${REQUESTS}`,
    a.d.records, a.d.addresses].map((v) => String(v).padStart(18)).join("")}`);
}
console.log(`\nlease duration 900 ms, renewal interval 300 ms; requests sent to a dead address ` +
  `in the discovery window = ${REQUESTS - stage[1].d.ok}/${REQUESTS}`);
console.log(`configuration files touched when a new copy is added: fixed ${CLIENTS}, registry 0; ` +
  `clients restarted: fixed ${CLIENTS}, registry 0`);
console.log(`registry = 1 extra process, 1 extra resolution request before every call`);

Object.values(proc).forEach((s) => s.kill());
registry.kill();
```

```
stage                           fixed ok/req       fixed addrs   registry ok/req  registry records    registry addrs
all alive                                6/6                 3               6/6                 3                 3
copy killed (immediately)                4/6                 3               4/6                 3                 3
lease expired                            4/6                 3               6/6                 2                 2
copy reported broken                     2/6                 3               6/6                 2                 1
new copy added                           2/6                 3               6/6                 3                 2

lease duration 900 ms, renewal interval 300 ms; requests sent to a dead address in the discovery window = 2/6
configuration files touched when a new copy is added: fixed 3, registry 0; clients restarted: fixed 3, registry 0
registry = 1 extra process, 1 extra resolution request before every call
```

## Reading the Numbers

At the first stage, both paths give the same result: six requests, six successes, three
addresses. Discovery shows no contribution at all, and that is correct — fixed configuration
works fine while everything is fine. Discovery's measure only shows up once the layout changes.

The second stage is measured immediately after a copy is killed. Both paths give 4/6. The
registry is still returning three addresses, because the lease duration has not yet passed since
the dead copy's last renewal. This interval is called the **discovery window**, and its measured
counterpart is 2/6: of every six requests sent in the window, two went to a dead address. The
registry does not reduce the time it takes to learn of a death to zero; it only bounds it.

The third stage shows the window closing. Once the lease duration has passed, the registry
returns two addresses, and the six requests split three each across the two: 6/6. Fixed
configuration, though, is still carrying three addresses and keeps giving 4/6. This difference is
the whole return on discovery: fixed configuration's error persists until a person fixes the
file.

The fourth stage shows why health information is a separate piece of information. Copy `8763`
keeps living and renewing its record after the `/break` call; the registry's live record count
stays at 2, but the routable address count drops to 1. A process being up does not mean it can do
the work. Fixed configuration cannot make this distinction at all and drops to 2/6: two requests
go to a dead address, and two go to a copy returning 503.

The fifth stage measures the deployment side. When a fourth copy is started, no file is touched
on the registry side at all; the copy registers itself, and at the next resolution the routable
address count rises to 2. On the fixed-configuration side, the result does not change: the new
copy serves no requests at all, because the three clients' address file still carries the old
three. The cost of the change is in the last lines of the output: configuration files touched, 3
against 0; clients restarted, 3 against 0.

## The Registry's Cost and the Failure Modes It Creates

The cost paid has three items. The first is one extra process, and this process is not an
ordinary service: when the registry goes down, no client can resolve an address. Fixed
configuration's one advantage is exactly here — the file is read without depending on anything.
This is why clients that lean on a registry usually hold on to the result of their last
resolution; and then the held list can go stale, and the discovery window grows again.

The second is the resolution request. In the measurement, one resolution was done at the start of
every stage; if resolution had been done before every call, every request to the catalog service
would have turned into two requests. This is discovery's counterpart to the previous lesson's
local-hop measure, and it makes caching the resolution mandatory.

The third is the lease duration itself, and both of its directions create a failure. Lengthen it,
and the discovery window grows — that is, requests to dead addresses increase. Shorten it, and
copies renew more often; the write load on the registry is multiplied by the copy count, and a
short network interruption causes live copies to drop off the list. When a dropped copy comes
back, requests redistribute again; this oscillation is registry implementations' most familiar
failure.

The common source of the failure modes born here is this: the registry does not hold the truth —
it holds **a delayed copy of the truth.** In fixed configuration, the delay is unbounded and
depends on a person fixing the file; in the registry, the delay is bounded by the lease duration.
What discovery sells is not correctness — it is **an upper bound placed on how long the wrongness
lasts.**

## Summary

- The registry keeps leased records; there is no delete endpoint, because a copy that crashes
  cannot delete its own record. Copies renew their record at an interval one third of the lease
  duration.
- While everything is fine, both paths give 6/6; the difference only shows up once the layout
  changes.
- Immediately after a copy is killed, both paths give 4/6: two of six requests in the discovery
  window go to a dead address. Once the lease duration has passed, the registry returns to 6/6,
  fixed configuration stays at 4/6.
- Health information is separate from a live record: a copy that reports broken stays in the
  record count (records 2) but drops from routing (addresses 1); fixed configuration cannot make
  this distinction and drops to 2/6.
- When a new copy is added, configuration files touched are 3 against 0, clients restarted are 3
  against 0.
- Cost paid: 1 extra process and a single point of failure, 1 extra resolution request per call,
  and a two-directional failure in the lease duration (a long duration grows the discovery
  window, a short one grows the renewal load and the oscillation).

## Next Step

This topic began with where the boundary should pass, continued with how the two sides of the
boundary talk, and ended with how the address gets resolved. The boundary was drawn, synchronous
and asynchronous communication were built, edge responsibilities were collected, a client-specific
layer was added, network concerns were pulled out of the application, and the address was
resolved from outside the code. But one assumption sitting underneath everything the seven
lessons measured was never tested: the data on the two sides of the boundary, in the middle of a
job, was treated as if it were under the guarantee of a single transaction. The loan service wrote
the record, and the fee service collected the penalty; the two happened in separate processes, on
separate data, at separate times. The previous lesson's double-collection measurement was the
first crack in that assumption. The next topic takes on exactly this: when a workflow touches more
than one service's data, how is it ensured that all of them happen together or none of them
happen, what happens when that cannot be ensured, and what units its cost is measured in.