Multi-Container Local Environment

The previous lesson looked inside a single container: a fifteen-entry tree, two identities, a permission matrix. The regional measurement network — fictional software that collects, verifies, and turns meter readings into invoices — is not a single process. A verifier reads the record the reading collector writes, the verified record goes to billing, and a record store sits underneath all of them. Bringing these four up locally is not just writing four separate startup commands; it is deciding which network gets set up first, which volume mounts where, and which service comes after which. Until these decisions are written down somewhere, they stay in the environment: in the shell of whoever set it up, in past commands, or in their head.

A compose file captures these decisions in a file, as a type. In this lesson, a small schema for such a file is defined and measured with a resolver; both the schema and the resolver are hand-written here, no real runtime is run.

RT31 — the local environment has four services: the record store, the reading collector, the verifier, billing. RT32 — the compose file’s schema is six fields: image, depends, network, volume, port, ready. RT33 — a service’s process comes up in one step; it takes as many more steps as the ready field to start serving. RT34 — without a readiness check, a dependent service tries to connect the moment its dependency’s process starts. RT35 — in manual setup, the network and the volume are created with separate commands, and every service is started with its own command and its own flags. RT36 — restarting a service drops the connection of every service transitively dependent on it.

Schema and Resolver

The schema itself is part of the measure: the moment fields are named, everything written in the file becomes checkable before startup. The resolver below does three things — it collects fields that do not match the schema and references that do not resolve, it searches the dependency graph for cycles, it derives the ordering as waves.

// compose.mjs — a declarative compose file's small schema and resolver (MODEL).
// The schema is defined here: fields, their types, and the sets they reference.

export const SCHEMA = { image: 'text', depends: 'list', network: 'list', volume: 'list', port: 'list', ready: 'number' };

// RT31: fictional four-service local environment. RT32: ready = steps to serve after coming up.
export const COMPOSE = {
  networks: ['measurement-net'],
  volumes: ['record-data'],
  services: {
    'record-store': { image: 'record:3', depends: [], network: ['measurement-net'],
      volume: ['record-data:/data'], port: [], ready: 3 },
    'reading-collector': { image: 'collector:7', depends: ['record-store'], network: ['measurement-net'],
      volume: [], port: ['8081'], ready: 1 },
    'verifier': { image: 'verify:4', depends: ['record-store'], network: ['measurement-net'],
      volume: [], port: [], ready: 2 },
    'billing': { image: 'invoice:2', depends: ['reading-collector', 'verifier'], network: ['measurement-net'],
      volume: ['record-data:/archive'], port: ['8082'], ready: 1 },
  },
};

export function validate(b) {
  const errors = [];
  const names = Object.keys(b.services);
  for (const [name, svc] of Object.entries(b.services)) {
    for (const field of Object.keys(SCHEMA)) if (!(field in svc)) errors.push(`${name}: missing field '${field}'`);
    for (const field of Object.keys(svc)) if (!(field in SCHEMA)) errors.push(`${name}: unknown field '${field}'`);
    for (const a of svc.network ?? []) if (!b.networks.includes(a)) errors.push(`${name}: unknown network '${a}'`);
    for (const v of svc.volume ?? []) {
      if (!b.volumes.includes(v.split(':')[0])) errors.push(`${name}: unknown volume '${v}'`);
    }
    for (const d of svc.depends ?? []) {
      if (d === name) errors.push(`${name}: depends on itself`);
      else if (!names.includes(d)) errors.push(`${name}: unknown dependency '${d}'`);
    }
  }
  return errors;
}

// Cycle: depth-first walk; a return to a gray node gives the cycle.
export function cycle(b) {
  const color = {}, path = [];
  let found = null;
  const visit = (name) => {
    if (color[name] === 'gray') { found ??= [...path.slice(path.indexOf(name)), name]; return; }
    if (color[name] === 'black') return;
    color[name] = 'gray'; path.push(name);
    for (const d of b.services[name].depends) if (b.services[d]) visit(d);
    path.pop(); color[name] = 'black';
  };
  for (const name of Object.keys(b.services)) visit(name);
  return found;
}

// The startup order is derived from the graph; there is no dependency between two in the same wave.
export function order(b) {
  const remaining = new Map(Object.entries(b.services).map(([a, s]) => [a, new Set(s.depends)]));
  const waves = [];
  while (remaining.size) {
    const available = [...remaining].filter(([, d]) => d.size === 0).map(([a]) => a);
    if (!available.length) return null;                  // there is a cycle: no order can be derived
    waves.push(available);
    for (const a of available) remaining.delete(a);
    for (const d of remaining.values()) for (const a of available) d.delete(a);
  }
  return waves;
}

// The number of orderings a person starting by hand has to choose among.
export function ordering(b) {
  const names = Object.keys(b.services);
  const perm = (a) => (a.length <= 1 ? [a]
    : a.flatMap((x, i) => perm([...a.slice(0, i), ...a.slice(i + 1)]).map((r) => [x, ...r])));
  const total = perm(names);
  const valid = total.filter((p) => p.every((name, i) => b.services[name].depends.every((d) => p.indexOf(d) < i)));
  return { total: total.length, valid: valid.length };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`compose validation: ${validate(COMPOSE).length} errors`);

  const BROKEN = structuredClone(COMPOSE);
  BROKEN.services['verifier'].network.push('report-net');
  BROKEN.services['billing'].depends.push('archiver');
  BROKEN.services['billing'].volume.push('scratch-data:/tmp');
  BROKEN.services['reading-collector'].restartPolicy = 'always';
  delete BROKEN.services['record-store'].ready;
  const errors = validate(BROKEN);
  console.log(`broken compose: ${errors.length} errors, all before startup`);
  for (const e of errors) console.log(`  ${e}`);

  const CYCLIC = structuredClone(COMPOSE);
  CYCLIC.services['record-store'].depends = ['billing'];
  console.log(`\ncycle: ${cycle(CYCLIC).join(' -> ')}`);
  console.log(`can an order be derived from the cyclic one: ${order(CYCLIC) === null ? 'no' : 'yes'}`);

  const d = ordering(COMPOSE);
  console.log(`\nstartup waves: ${order(COMPOSE).map((w) => `[${w.join(' ')}]`).join(' -> ')}`);
  console.log(`${d.valid} of ${d.total} orderings of ${Object.keys(COMPOSE.services).length} services are valid`);
}

compose validation: 0 errors
broken compose: 5 errors, all before startup
  record-store: missing field 'ready'
  reading-collector: unknown field 'restartPolicy'
  verifier: unknown network 'report-net'
  billing: unknown volume 'scratch-data:/tmp'
  billing: unknown dependency 'archiver'

cycle: record-store -> billing -> reading-collector -> record-store
can an order be derived from the cyclic one: no

startup waves: [record-store] -> [reading-collector verifier] -> [billing]
2 of 24 orderings of 4 services are valid

What the five errors have in common is that all of them are found before startup. In manual setup, the only way to find these five is five separate runs: the unrecognized network name shows up when the first service starts, the unresolved volume name on the third command, and the missing dependency only when billing tries to connect. The declarative form reduces this to a single read, because the file sees the whole setup at once. The check for a service depending on itself sits in the same place; a service waiting on itself is an infinite wait at run time, and a single line in the file.

The cycle output is the sharpest split between the two forms. When record-store is made dependent on billing, a three-edge cycle is born and no ordering can be derived. A person starting by hand does not see this as an error; they wait for the first service to come up, see that it does not, try the second, and find no answer for which one should come first while the two services wait on each other. The graph states that there is no answer, in three steps.

The last line gives the number of decisions. Of the four services’ twenty-four orderings, only two satisfy every dependency. Order is not a preference, it is a consequence falling out of the graph, and the waves make it visible: the record store alone, then two services with no dependency between them, then billing.

Races and the Readiness Check

A correct order is not enough. A service’s process having started does not mean it is ready to serve, and that gap falls on the dependent service’s connection attempt. The second run brings the same compose up twice — without a readiness check and with one.

// run.mjs — the cost of a readiness check, the spread of a restart, command and decision counts.
import { COMPOSE, SCHEMA, order, ordering } from './compose.mjs';

// RT33: a process comes up in 1 step; serving takes 'ready' more steps.
// RT34: without the check, a dependent service tries to connect the moment its dependency's process starts.
function boot(b, check) {
  const S = b.services;
  const startStep = {}, readyAt = {};
  for (const wave of order(b)) {
    for (const name of wave) {
      const deps = S[name].depends;
      startStep[name] = deps.length === 0 ? 0 : Math.max(...deps.map((x) => (check ? readyAt[x] : startStep[x] + 1)));
      readyAt[name] = startStep[name] + S[name].ready;
    }
  }
  const races = [];
  for (const [name, s] of Object.entries(S)) {
    for (const d of s.depends) if (startStep[name] < readyAt[d]) races.push([name, d, readyAt[d] - startStep[name]]);
  }
  return { startStep, races, total: Math.max(...Object.values(readyAt)) };
}

// What is transitively affected by restarting one service.
function affected(b, name) {
  const reverse = {};
  for (const [a, s] of Object.entries(b.services)) for (const d of s.depends) (reverse[d] ??= []).push(a);
  const set = new Set(), stack = [name];
  while (stack.length) {
    for (const x of reverse[stack.pop()] ?? []) if (!set.has(x)) { set.add(x); stack.push(x); }
  }
  return [...set];
}

const s = (x, n) => String(x).padStart(n);
const edges = Object.values(COMPOSE.services).reduce((a, x) => a + x.depends.length, 0);
console.log(`${'check'.padEnd(12)}${s('races', 7)}${s('edges', 7)}${s('hidden wait', 15)}${s('startup', 14)}`);
for (const check of [false, true]) {
  const r = boot(COMPOSE, check);
  const wait = r.races.reduce((a, [, , g]) => a + g, 0);
  console.log(`${(check ? 'readiness' : 'none').padEnd(12)}${s(r.races.length, 7)}${s(edges, 7)}`
    + `${s(`${wait} steps`, 15)}${s(`${r.total} steps`, 14)}`);
}
for (const [name, d, g] of boot(COMPOSE, false).races) {
  console.log(`  race: ${name.padEnd(17)} -> ${d.padEnd(17)} connects ${g} step${g === 1 ? "" : "s"} early`);
}

console.log(`\n${'restarting'.padEnd(18)}${s('affected', 10)}   ${'who'}`);
for (const name of Object.keys(COMPOSE.services)) {
  const e = affected(COMPOSE, name);
  console.log(`${name.padEnd(18)}${s(e.length, 10)}   ${e.join(', ') || '-'}`);
}

// The writing cost of manual setup versus the declarative file.
const services = Object.values(COMPOSE.services);
const flags = services.reduce((a, x) => a + x.network.length + x.volume.length + x.port.length, 0);
const commands = COMPOSE.networks.length + COMPOSE.volumes.length + services.length;
const fields = services.length * Object.keys(SCHEMA).length + 2;
const ord = ordering(COMPOSE);
console.log(`\nmanual     : ${commands} commands, ${flags} flags, ${ord.total - ord.valid} of ${ord.total} orderings`
  + ` break at least one dependency`);
console.log(`declarative: 1 command, ${fields}-field file, order is derived (0 ordering decisions)`);
console.log(`on manual restart: worst case ${Math.max(...Object.keys(COMPOSE.services)
  .map((a) => affected(COMPOSE, a).length))} more services need manual restart`);

check         races  edges    hidden wait       startup
none              3      4        5 steps       3 steps
readiness         0      4        0 steps       6 steps
  race: reading-collector -> record-store      connects 2 steps early
  race: verifier          -> record-store      connects 2 steps early
  race: billing           -> verifier          connects 1 step early

restarting          affected   who
record-store               3   reading-collector, verifier, billing
reading-collector          1   billing
verifier                   1   billing
billing                    0   -

manual     : 6 commands, 8 flags, 22 of 24 orderings break at least one dependency
declarative: 1 command, 26-field file, order is derived (0 ordering decisions)
on manual restart: worst case 3 more services need manual restart

In the unchecked layout the order is correct, and yet three of the four dependency edges give birth to a race. A correct order only guarantees that the process has started; the record store takes three steps to serve, the two services depending on it try to connect on the first step, and both arrive two steps early. Billing connects to the verifier one step early. This five-step-total earliness shows up as an error nowhere; with a retry it turns into delay, without one, into a service that drops on startup.

The check’s cost is in the same table’s last column: startup climbs from three steps to six, double. This is the most visible price the declarative form pays in this lesson, and it is paid on every local run. What it buys is three races zeroed out. Both numbers are read from the same file, so the trade-off becomes a visible choice in the file — in manual setup, the same trade-off is hidden inside wait times sprinkled between commands.

The restart table measures, for the first time, past a single service. When the record store restarts, three services are affected; when the reading collector or the verifier restarts, one service each; when billing restarts, none. In manual setup, this number is applied by hand: whoever restarts the record store also has to start the remaining three themselves, and which ones are needed is learned not from a dependency list but from errors.

What Leaves the Environment, and What Does Not

The last three lines are this lesson’s isolation account. Manual setup requires six commands, eight flags, and the decision of choosing one of twenty-four orderings; twenty-two of the twenty-four break at least one dependency. In the declarative form, in their place, there is a twenty-six-field file and a single command, and the ordering decision is zero. This is the difference removed from the environment: startup order, network and volume names, the flag set, and the spread of a restart move out of the setter-up’s shell history and into a file sitting next to the output. The cost is two entries: twenty-six fields written, and the three steps the readiness check adds.

One distinction must be kept while reading these numbers: the six commands and eight flags do not disappear, they change location. The network still gets created, the volume still gets made, the four processes still get started; it is just that what decides in which order and with which flags is no longer the person at the shell, it is the file. The gain is not in the command count, it is that the person setting up the same thing a second time avoiding the same twenty-two wrong orderings becomes provable. A compose file does not shorten a setup, it makes it repeatable; the only thing that shortens is the number of decisions.

Where isolation is punctured is written into the same file. The four services share a single network; to connect the services to each other, the compose file loosens the separation built in the container networking lesson, putting the four that need to see each other into the same name resolution space. The record-data volume is mounted into two services — at /data in the record store, at /archive in billing. The same set of bytes can be written from two separate containers, under two separate identities; the previous lesson’s permission matrix here looks not at a single tree but at the point where two trees intersect. The readiness check itself is not proof either: in this model the ready field is a number, and if the number is wrong the races come back, they just become invisible.

Summary

• Defining a schema for the compose file makes setup errors visible before startup: in the broken setup, all five of the five errors were found in a single read; in manual setup, each would have been a separate run.
• The startup order is derived from the graph; only two of the four services’ twenty-four orderings are valid, and once a cycle is added, that the ordering does not exist prints by name as a three-edge cycle.
• A correct order does not end races: without a readiness check, three of the four dependency edges connect early, giving birth to five steps of hidden wait in total.
• The readiness check brings races from three to zero, and takes startup from three steps to six; the trade-off becomes a choice written into the file.
• Restarting the record store affects three services, restarting billing affects none; in manual setup this number is learned from errors, in the file it is read from the dependency list.
• The difference removed from the environment is six commands, eight flags, and the exclusion of twenty-two wrong orderings; its cost is a twenty-six-field file plus three extra steps. The hole is the shared network and the single volume mounted into two services.

Next Step

This lesson brought the compose up once and left it there. But the local environment’s real use is not a single startup, it is a loop repeated hundreds of times over a day: a line is changed, the result is seen, another line is changed. In that loop, rebuilding the image and mounting the source into the container are two separate forms, and neither is free. The next lesson compares these two forms by counting per change — bytes copied, layers invalidated, feedback steps — and measures by name the cost the second form pays: how far the mounted source and the skipped build step move the running environment away from production, and which class of error that distance hides.