Developer Experience

The previous lesson brought four services up once and left them there. The local environment’s real use, though, is not a single startup, it is a loop repeated over the course of a day: a line is changed, the result is seen, another line is changed. The feedback loop is a quantity already measured in this curriculum’s first course; that count is not repeated here. The question here is not the loop’s duration, it is that under a containerized setup the loop has two separate forms, and the two pay different prices. The first form rebuilds the image and restarts the container. The second form mounts the source directory into the container and reloads the process in place. What gets measured is, per change, how many bytes each copies, how many layers it invalidates, how many steps it takes — and what the second one breaks in exchange.

The measurement subject is the regional measurement network’s reading collector service; it is fictional software, its source tree is really written to disk, and byte sizes and content digests are read from real files.

RT37 — the image definition consists of five steps: base layer, dependency declaration, dependency install, source copy, build. RT38 — the base layer is 41,943,040 and dependency install is 18,874,368 bytes; these two numbers are a model, the remaining bytes come from real files. RT39 — if a step’s input changes, that step and every step after it are invalidated. RT40 — the ignore rules are five patterns, and a file matching them does not enter the image. RT41 — in the mount form the whole source directory is mounted, ignore rules are not applied, and no copying happens. RT42 — the feedback step is the number of discrete steps between a file being written and the result being seen.

The Layer Chain and Two Sets

The first run builds the tree, decides which file enters the image using the ignore rules, and computes the chain of the five steps. Each step’s digest depends on the previous digest and the content of its own inputs; this is why, once the chain breaks somewhere, everything behind it breaks too.

// source.mjs — real source tree, real byte sizes, real content digests.
// Files and their sizes are real; image steps and the base layer/install byte counts are a MODEL.
import { mkdirSync, writeFileSync, rmSync, statSync, readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

export const ROOT = './source';

export const FILES = {
  'dependencies.json': '{"name":"reading-collector","version":"7.2.0",'
    + '"dependencies":{"parser":"2.4.1","record-client":"1.9.3","time":"4.0.2"}}\n',
  'app/start.mjs': "import { collect } from './collector.mjs';\n"
    + "import { CONFIG } from './config.local.mjs';\ncollect(CONFIG.source);\n",
  'app/collector.mjs': "import { format } from './format.mjs';\n"
    + 'export const collect = (k) => format(k);\n',
  'app/verifier.mjs': "import { format } from './format.mjs';\n"
    + 'export const verify = (o) => format(o).length > 0;\n',
  'app/format.mjs': 'export const format = (x) => String(x).trim();\n',
  'app/config.local.mjs': "export const CONFIG = { source: 'local-meter', log: 'verbose' };\n",
  'app/sample-reading.json': '[{"meter":"S-001","value":1420},{"meter":"S-002","value":980}]\n',
  'test/collector.test.mjs': "import { collect } from '../app/collector.mjs';\n"
    + "console.log(collect(' 12 '));\n",
  'notes.md': '# local notes\n- meter data is read from a sample file\n',
  '.secret-config': 'repo-token=local-development\n',
};

// RT40: excluded from the image. The mount form does not apply these rules, it mounts the whole directory.
export const IGNORE = [/^test\//, /\.local\./, /(^|\/)sample-/, /^notes\.md$/, /(^|\/)\.secret-/];
export const inImage = (path) => !IGNORE.some((r) => r.test(path));

// RT37/RT38: five steps. When a step names an output, it is a MODEL byte count; otherwise it is the real byte size of its inputs.
export const STEPS = [
  { name: 'base layer', prefix: [], output: 41943040 },
  { name: 'dependency declaration', prefix: ['dependencies.json'] },
  { name: 'dependency install', prefix: ['dependencies.json'], output: 18874368 },
  { name: 'source copy', prefix: ['app/'] },
  { name: 'build', prefix: ['app/'], build: true },
];

export function setup() {
  rmSync(ROOT, { recursive: true, force: true });
  for (const [path, content] of Object.entries(FILES)) {
    const dir = path.includes('/') ? `${ROOT}/${path.slice(0, path.lastIndexOf('/'))}` : ROOT;
    mkdirSync(dir, { recursive: true });
    writeFileSync(`${ROOT}/${path}`, content);
  }
  return Object.keys(FILES).map((path) => ({ path, bytes: statSync(`${ROOT}/${path}`).size }));
}

const digest = (m) => createHash('sha256').update(m).digest('hex');
const inputsOf = (step, tree) => tree.filter((d) => inImage(d.path)
  && step.prefix.some((o) => d.path === o || d.path.startsWith(o)));

// The build step really runs: the source modules entering the image are joined into a single output file.
export function build(tree) {
  const parts = inputsOf(STEPS[4], tree).filter((d) => d.path.endsWith('.mjs'))
    .map((d) => readFileSync(`${ROOT}/${d.path}`, 'utf8'));
  mkdirSync(`${ROOT}/dist`, { recursive: true });
  writeFileSync(`${ROOT}/dist/bundle.mjs`, parts.join('\n'));
  return statSync(`${ROOT}/dist/bundle.mjs`).size;
}

// Layer chain: each step's digest depends on the previous digest and the content of its own inputs.
export function chain(tree) {
  let previous = '0'.repeat(64);
  return STEPS.map((step) => {
    const inputs = inputsOf(step, tree);
    const content = inputs.map((d) => digest(readFileSync(`${ROOT}/${d.path}`))).join('');
    previous = digest(previous + step.name + content);
    const bytes = step.output ?? (step.build ? build(tree) : inputs.reduce((a, d) => a + d.bytes, 0));
    return { name: step.name, inputs: inputs.length, bytes, digest: previous.slice(0, 12) };
  });
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const tree = setup();
  const s = (x, n) => String(x).padStart(n);
  const t = (x) => x.toLocaleString('en-US');
  console.log(`${'file'.padEnd(28)}${s('bytes', 7)}${s('in image', 10)}`);
  for (const d of tree) console.log(`${d.path.padEnd(28)}${s(d.bytes, 7)}${s(inImage(d.path) ? 'yes' : 'no', 10)}`);
  const image = tree.filter((d) => inImage(d.path));
  console.log(`mounted set ${tree.length} files / ${t(tree.reduce((a, d) => a + d.bytes, 0))} bytes`
    + `   image set ${image.length} files / ${t(image.reduce((a, d) => a + d.bytes, 0))} bytes`);

  console.log(`\n${'step'.padEnd(22)}${s('inputs', 8)}${s('output bytes', 14)}${s('digest', 15)}`);
  for (const k of chain(tree)) {
    console.log(`${k.name.padEnd(22)}${s(k.inputs, 8)}${s(t(k.bytes), 14)}${s(k.digest, 15)}`);
  }
}

file                          bytes  in image
dependencies.json               120       yes
app/start.mjs                   112       yes
app/collector.mjs                80       yes
app/verifier.mjs                 90       yes
app/format.mjs                   47       yes
app/config.local.mjs             65        no
app/sample-reading.json          63        no
test/collector.test.mjs          78        no
notes.md                         54        no
.secret-config                   29        no
mounted set 10 files / 738 bytes   image set 5 files / 449 bytes

step                    inputs  output bytes         digest
base layer                   0    41,943,040   adc5f56b5389
dependency declaration       1           120   9299c9307bbf
dependency install           1    18,874,368   f08005c4bd3a
source copy                  4           329   536fde9c6bbc
build                        4           332   1558b519b9ef

The ratio between the table’s two halves sets this lesson’s whole measure. The source is four files and 329 bytes; the two steps beneath it are 60,817,408 bytes, roughly 185,000 times the source. Most changes land on the source, but most of the cost sits beneath it. This is the whole meaning of the layer chain: what is above the changed step is not rebuilt.

Two Loops Per Change

The second run really writes three change classes to a file, recomputes the chain, and compares which steps’ digests changed. Each class is measured in two forms.

// loop.mjs — two feedback loops: rebuilding the image versus mounting the source and reloading.
import { readFileSync, writeFileSync, statSync } from 'node:fs';
import { ROOT, FILES, STEPS, inImage, setup, chain, build } from './source.mjs';

const readTree = () => Object.keys(FILES).map((path) => ({ path, bytes: statSync(`${ROOT}/${path}`).size }));

const CHANGE = {
  'app/format.mjs': FILES['app/format.mjs'].replace('trim()', 'trim().slice(0, 64)'),
  'dependencies.json': FILES['dependencies.json'].replace('2.4.1', '2.5.0'),
  'notes.md': `${FILES['notes.md']}- review the ignore list\n`,
};

// RT42: a feedback step = a discrete step between a file being written and the result being seen.
const rebuildSteps = (g) => 1 + (g === 0 ? 1 : g + 3);   // write + steps + stop + start + become ready
const MOUNT_STEPS = 2;                                    // write + process reload

const baseline = chain(setup());
const s = (x, n) => String(x).padStart(n);
const t = (x) => x.toLocaleString('en-US');
console.log(`${'change'.padEnd(22)}${'form'.padEnd(23)}${s('invalid layers', 16)}`
  + `${s('bytes copied', 17)}${s('steps', 7)}`);
for (const [path, newContent] of Object.entries(CHANGE)) {
  setup();
  writeFileSync(`${ROOT}/${path}`, newContent);
  const updated = chain(readTree());
  const invalid = updated.filter((k, i) => k.digest !== baseline[i].digest);
  const bytes = invalid.reduce((a, k) => a + k.bytes, 0);
  const mountForm = path === 'dependencies.json' ? 'falls back to rebuild' : 'mount + reload';
  console.log(`${path.padEnd(22)}${'rebuild'.padEnd(23)}${s(`${invalid.length}/${STEPS.length}`, 16)}`
    + `${s(t(bytes), 17)}${s(rebuildSteps(invalid.length), 7)}`);
  console.log(`${''.padEnd(22)}${mountForm.padEnd(23)}`
    + `${s(path === 'dependencies.json' ? '-' : `0/${STEPS.length}`, 16)}`
    + `${s(path === 'dependencies.json' ? '-' : '0', 17)}`
    + `${s(path === 'dependencies.json' ? '-' : MOUNT_STEPS, 7)}`);
  if (invalid.length) console.log(`${''.padEnd(22)}invalidated: ${invalid.map((k) => k.name).join(', ')}`);
}

// The cost of the mount form: how far the running environment is from production.
const tree = setup();
const mounted = tree;                                   // RT41: the whole directory is mounted
const image = tree.filter((d) => inImage(d.path));
const IMPORT_RE = /from\s+'(\.[^']+)'/g;
function unresolved(set) {
  const paths = new Set(set.map((d) => d.path));
  const missing = [];
  for (const d of set.filter((x) => x.path.endsWith('.mjs'))) {
    const base = d.path.slice(0, d.path.lastIndexOf('/'));
    for (const [, target] of readFileSync(`${ROOT}/${d.path}`, 'utf8').matchAll(IMPORT_RE)) {
      const resolved = new URL(target, `file:///${base}/`).pathname.slice(1);
      if (!paths.has(resolved)) missing.push(`${d.path} -> ${target}`);
    }
  }
  return missing;
}
const built = build(tree);
console.log(`\n${'environment'.padEnd(22)}${s('files', 7)}${s('bytes', 7)}${s('unresolved imports', 24)}`);
for (const [label, set] of [['mounted source', mounted], ['production image', image]]) {
  console.log(`${label.padEnd(22)}${s(set.length, 7)}${s(set.reduce((a, d) => a + d.bytes, 0), 7)}`
    + `${s(unresolved(set).length, 24)}`);
}
for (const e of unresolved(image)) console.log(`  breaks only in the image: ${e}`);
console.log(`${mounted.length - image.length} files mounted but not in the image`
  + ` (${mounted.filter((d) => !inImage(d.path)).map((d) => d.path).join(', ')})`);
console.log(`the build step never runs in the mount form: a single ${built}-byte output`
  + ` from ${image.filter((d) => d.path.endsWith('.mjs')).length} sources`);

change                form                     invalid layers     bytes copied  steps
app/format.mjs        rebuild                             2/5              687      6
                      mount + reload                      0/5                0      2
                      invalidated: source copy, build
dependencies.json     rebuild                             4/5       18,875,149      8
                      falls back to rebuild                 -                -      -
                      invalidated: dependency declaration, dependency install, source copy, build
notes.md              rebuild                             0/5                0      2
                      mount + reload                      0/5                0      2

environment             files  bytes      unresolved imports
mounted source             10    738                       0
production image            5    449                       1
  breaks only in the image: app/start.mjs -> ./config.local.mjs
5 files mounted but not in the image (app/config.local.mjs, app/sample-reading.json, test/collector.test.mjs, notes.md, .secret-config)
the build step never runs in the mount form: a single 332-byte output from 4 sources

A change to a source line is the most frequent class, and the difference between the two forms is clearest here: two of the five steps invalidate, 687 bytes are copied, and six steps pass; in the mount form, zero layers, zero bytes, and two steps. The threefold gap in step count matters more than the gap in byte count, because the byte count scales with the size of the source tree and the step count does not. The 687 bytes here come from a ten-file tree; if the tree were a thousand files, the bytes would climb into the hundreds of thousands, but the step count would still stay at six.

What the six steps consist of is also part of the number: the file is written, the two invalidated steps re-run, the container stops, comes up again with the new one, and is waited on to become ready. None of this has anything to do with the content of the change; all of it is the cost of rebuilding the output and swapping the running process. For someone making forty source changes a day, the gap between the two forms is forty times four extra steps — a hundred and sixty stops, and each stop is a place waited on before the result is seen. What the mount form buys is not the 687 bytes not copied, it is these stops.

The second row draws the boundary of the mount form. When the dependency declaration changes, four of the five steps invalidate, the bytes copied climb to 18,875,149 — roughly 27,000 times the source change — and the loop takes eight steps. The mount cannot touch this class: the dependencies do not sit in the source directory, they sit in the install step’s output, so the loop falls back to a rebuild. The mount’s gain holds for a single change class, and once you step outside that class, the first form’s full cost comes back.

The third row shows the quiet one. When an ignored file changes, no layer invalidates on the image side; this is the correct behavior. But the same change is visible in the mount form, because the file sits inside the mounted directory and the process can read it. The two forms give the same numbers and do different things. This row is the subject of the next section.

The Cost of Mounting: A Drifting Environment

The second half of the output counts the price the mount form pays. The tree running locally is ten files and 738 bytes; the image going to production is five files and 449 bytes. Half of the files the locally running process sees are absent from production. This gap is not an accident, it is the deliberate result of the ignore rules — and the mount form works precisely by skipping those rules.

The concrete consequence of the gap is one line down. start.mjs imports ./config.local.mjs. In the mounted set, this import resolves and the process comes up without a problem; in the production image, it does not resolve, because the file is not there. The measurement counts this: zero unresolved imports in the mounted set, one in the image. The local loop cannot show this error in any run, because the error’s condition is exactly what the local loop removes. This is the name of the error class the mount form hides: a dependency remaining on a file the ignore rule removed from the image.

The second hidden class is in the last line. The build step never runs in the mount form; the 332-byte output produced from four sources is never executed locally. What runs locally is the source itself; what runs in production is the build’s product, and the two behaving the same is never tested anywhere. Together these two classes give the real price of developer experience: while the loop drops from six steps to two, the local environment drifts from production by five files, one unresolved dependency, and one step that never runs.

Where isolation is punctured, in this lesson, is not a flaw, it is the tool itself. A bind mount deliberately punctures the boundary the image closes; the source directory is no longer inside the output, it is in the environment. The previous two lessons’ measures connect directly here: the mounted directory’s file ownership may not match the internal identity, and the compose file declares this mount separately for every service. The hole’s measure is the three numbers above, and being measurable does not close the hole — it makes knowing when to close it possible.

Summary

• The source is four files and 329 bytes, the two steps beneath it are 60,817,408 bytes; most changes are at the top, most of the cost is at the bottom, and the layer chain does not rebuild what is above the changed step.
• For a source line change, rebuild invalidates two of five layers, copies 687 bytes, and takes six steps; in the mount form, zero layers, zero bytes, two steps.
• The byte gap grows as the tree grows, the step gap does not: the threefold step ratio is the part of this measure that is independent of the tree.
• When the dependency declaration changes, four layers invalidate and 18,875,149 bytes are copied; the mount form cannot touch this class, the loop falls back to a rebuild.
• The mounted tree is ten files and 738 bytes, the production image is five files and 449 bytes; an import that resolves locally does not resolve in the image, and the local loop cannot show this error in any run.
• The build step never runs in the mount form: the 332-byte output produced from four sources is never executed locally. While the loop drops from six steps to two, the environment drifts from production by five files, one unresolved dependency, and one step that never runs.

Next Step

By the end of this lesson, isolation stands deliberately punctured: a directory mounted from outside, the source not in the image, the build step skipped. What was measured across three lessons was always part of the same axis — who runs, how many services see each other, which file comes from where. These holes are a deliberate choice in the local environment. The same choices can be made in production too, and there their costs are different: the privileges a container can request from the kernel, the filesystem staying writable, the path every removed restriction opens. The next lesson takes up runtime security on this same axis and counts, one by one, the parts of isolation that are optional: which is open by default, what breaks when it is closed, and which path stays open when it is left open.