Image Definition File

The previous lesson’s layers were defined by hand: which file went into which layer was written in an object constant. This was enough to make the measurement, but it did not describe how an image is actually built. A real image is born from instructions applied in sequence; each instruction lays a layer on top of the previous one, and when the last one finishes, the image is ready.

This lesson measures those instructions and the set they draw from. The files to be copied come from somewhere: the set of files instructions can read is called the build context, and its size is a cost. There are three numbers to measure — how many files and bytes enter the context, how many files and bytes the exclusion rule cuts, and how many files enter the context but never reach the image?

IM7. The instruction set and definition file format below are defined for this lesson; they consist of five instructions and are much smaller than real image definition file formats. The interpreter is a model written with node. IM8. The run instruction does not run a real command; it produces files from a modeled command table — a dependency install’s output is represented this way. IM9. The tree is the fictional regional measurement network’s reading collector service’s source tree; file contents are generated text scaled by line count. IM10. The exclusion rule works by prefix matching; real rules use a pattern language, which does not change this measurement’s numbers.

Five Instructions

The definition file is a text file, and every line is an instruction. Instructions apply in order; the order is the same as the previous lesson’s stack order — the one at the bottom runs first.

Instruction	What it does	Produces a layer
base	Places a starting file set at the bottom of the stack	yes
copy	Copies a path from the build context into the image	yes
run	Runs a command and turns the files it produces into a layer	yes
env	Writes a value into the image’s metadata	no
entry	Writes the path to run when the container starts into the metadata	no

The last two instructions do not touch the filesystem; they write the image’s metadata. This distinction will be measured: an image’s layer count is not the definition file’s line count.

The Build Context

The tree is set up on disk, and what the exclusion rule cuts is counted. The context is the set instructions can read; a path caught by the exclusion rule is never read, even if no instruction wants it.

// measurement-network/project.mjs — sets up the fictional service's source tree on disk, measures the build context
import { mkdirSync, writeFileSync, readdirSync, readFileSync, rmSync } from "node:fs";

export const ROOT = "project";
export const text = (name, lines) => Array.from({ length: lines }, (_, i) => `// ${name} ${i}\n`).join("");

export const DEFINITION = `base      runtime-base
copy      config/region.json  /app/config/region.json
run       install-deps
copy      source              /app/source
env       REGION=north
entry     /app/source/entry.js
`;

const TREE = {
  "image-definition": DEFINITION,
  "config/region.json": '{"region":"north","counter":41820}\n',
  "config/local.json": '{"port":8080,"log":"verbose"}\n',
  "source/entry.js": text("entry", 20),
  "source/collect.js": text("collect", 140),
  "source/queue.js": text("queue", 95),
  "source/test/collect.test.js": text("test-collect", 210),
  "source/test/queue.test.js": text("test-queue", 160),
  "dependency/csv/index.js": text("csv", 900),
  "dependency/queue/index.js": text("queue-dep", 1400),
  "doc/architecture.md": text("doc", 240),
  "log/run-1873.log": text("log", 2600),
  "build-artifact/package-previous.js": text("leftover", 1100),
  ".local-config/editor.json": text("editor", 12),
  "reading-samples/day-1.csv": text("sample", 380),
};

// Exclusion rule: paths starting with these prefixes never enter the context.
export const EXCLUDE = ["dependency/", "log/", "build-artifact/", ".local-config/", "doc/"];

export const buildTree = () => {
  rmSync(ROOT, { recursive: true, force: true });
  for (const [path, content] of Object.entries(TREE)) {
    mkdirSync(`${ROOT}/${path}`.split("/").slice(0, -1).join("/"), { recursive: true });
    writeFileSync(`${ROOT}/${path}`, content);
  }
};

export const walk = (root, sub = "") =>
  readdirSync(`${root}${sub}`, { withFileTypes: true }).flatMap((g) =>
    g.isDirectory() ? walk(root, `${sub}/${g.name}`) : [`${sub}/${g.name}`.slice(1)]).sort();

export const size = (path) => readFileSync(`${ROOT}/${path}`).length;
export const isExcluded = (path, rule = EXCLUDE) => rule.some((d) => path.startsWith(d));
export const context = (rule = EXCLUDE) => walk(ROOT).filter((y) => !isExcluded(y, rule));
export const total = (paths) => paths.reduce((t, y) => t + size(y), 0);

// measurement-network/measure-context.mjs — tree, exclusion, and context size
import { buildTree, walk, size, isExcluded, context, total, ROOT, DEFINITION } from "./project.mjs";

buildTree();
const all = walk(ROOT);
const cut = all.filter((y) => isExcluded(y));
console.log("image definition:\n" + DEFINITION);
console.log("top dir".padEnd(16) + "files   bytes   in context");
const group = {};
for (const y of all) (group[y.includes("/") ? y.split("/")[0] + "/" : "(root)"] ??= []).push(y);
for (const [d, paths] of Object.entries(group))
  console.log(d.padEnd(16) + String(paths.length).padStart(3) + String(total(paths)).padStart(8) +
    "   " + (isExcluded(paths[0]) ? "no" : "yes"));
console.log(`\ntree total   : ${all.length} files, ${total(all)} bytes`);
console.log(`exclusion cut: ${cut.length} files, ${total(cut)} bytes`);
console.log(`build context: ${context().length} files, ${total(context())} bytes ` +
  `(%${((total(context()) * 100) / total(all)).toFixed(1)} of the tree's bytes)`);

image definition:
base      runtime-base
copy      config/region.json  /app/config/region.json
run       install-deps
copy      source              /app/source
env       REGION=north
entry     /app/source/entry.js

top dir         files   bytes   in context
.local-config/    1     146   no
build-artifact/   1   17590   no
config/           2      65   yes
dependency/       2   33880   no
doc/              1    2530   no
(root)            1     196   yes
log/              1   30090   no
reading-samples/  1    5210   yes
source/           5   10210   yes

tree total   : 15 files, 99917 bytes
exclusion cut: 6 files, 84236 bytes
build context: 9 files, 15681 bytes (%15.7 of the tree's bytes)

The tree is 15 files and 99,917 bytes; the exclusion rule cuts 6 files and 84,236 bytes, leaving the context with 9 files and 15,681 bytes — 15.7 percent of the tree’s bytes. The size of the cut share is not a coincidence: most of a source tree’s volume is installed dependencies, logs, and leftover artifacts from previous builds. If the exclusion rule were not written, this 84,236 bytes would be read on every build and would also feed into the cache decision the next lesson measures.

There is also something the rule skips, and the count shows it: the reading-samples/ directory stays in the context and carries 5,210 bytes. The exclusion rule only cuts the paths it names; every path it does not name is in the context.

The Interpreter

The interpreter reads the definition file line by line, opens a separate layer directory for each line, and applies the instruction to that directory. A record of copied paths is kept; the gap at the end of the measurement comes from here.

// measurement-network/interpret.mjs — applies instructions in sequence, each instruction produces one layer (model)
import { mkdirSync, writeFileSync, readFileSync, rmSync, statSync } from "node:fs";
import { walk, isExcluded, text, EXCLUDE, ROOT } from "./project.mjs";

const BASE = { "root/shell.js": text("shell", 60), "root/runtime.js": text("runtime", 420) };
const COMMAND = { "install-deps": { "app/dependency/csv.js": text("csv-installed", 900),
  "app/dependency/queue.js": text("queue-installed", 1400) } };
const write = (dir, files) => {
  for (const [path, content] of Object.entries(files)) {
    mkdirSync(`${dir}/${path}`.split("/").slice(0, -1).join("/"), { recursive: true });
    writeFileSync(`${dir}/${path}`, content);
  }
};

export const build = (definition, rule = EXCLUDE) => {
  rmSync("image", { recursive: true, force: true });
  const readFiles = new Set(), metadata = {}, layers = [];
  definition.trim().split("\n").forEach((line, i) => {
    const [instruction, ...arg] = line.trim().split(/\s+/);
    const dir = `image/layer-${i}`;
    mkdirSync(dir, { recursive: true });
    if (instruction === "base") write(dir, BASE);
    else if (instruction === "run") write(dir, COMMAND[arg[0]]);
    else if (instruction === "copy") {
      const [source, target] = arg;
      const single = !statSync(`${ROOT}/${source}`).isDirectory();
      const prefix = single || source === "." ? "" : `${source}/`;
      const matches = (single ? [source] : walk(ROOT, prefix ? `/${source}` : "")).filter((y) => !isExcluded(y, rule));
      for (const y of matches) {
        const dest = `${dir}/${target.slice(1)}` + (single ? "" : `/${y.slice(prefix.length)}`);
        mkdirSync(dest.split("/").slice(0, -1).join("/"), { recursive: true });
        writeFileSync(dest, readFileSync(`${ROOT}/${y}`));
        readFiles.add(y);
      }
    } else metadata[instruction] = arg.join(" ");
    const paths = walk(dir);
    layers.push({ index: i, instruction, arg: arg.join(" "), files: paths.length,
      bytes: paths.reduce((t, y) => t + statSync(`${dir}/${y}`).size, 0) });
  });
  return { layers, readFiles, metadata };
};

// measurement-network/measure-interpret.mjs — runs the interpreter, counts the gap between context and image
import { build } from "./interpret.mjs";
import { buildTree, context, total, DEFINITION } from "./project.mjs";

buildTree();
const { layers, readFiles, metadata } = build(DEFINITION);
console.log("idx instruction  argument".padEnd(58) + "type          files   bytes");
for (const k of layers)
  console.log(`  ${k.index}  ${k.instruction.padEnd(9)} ${k.arg}`.padEnd(58) +
    (k.files ? "filesystem   " : "metadata     ") + String(k.files).padStart(4) + String(k.bytes).padStart(8));

const fsLayers = layers.filter((k) => k.files > 0);
const imageBytes = fsLayers.reduce((t, k) => t + k.bytes, 0);
const missing = context().filter((y) => !readFiles.has(y));
const testFiles = [...readFiles].filter((y) => y.includes("/test/"));
console.log(`\nmetadata: ${JSON.stringify(metadata)}`);
console.log(`image: ${fsLayers.length} filesystem layers, ${layers.length - fsLayers.length} metadata instructions, ${imageBytes} bytes`);
console.log(`read from context ${readFiles.size} files, ${total([...readFiles])} bytes`);
console.log(`in context but never entering the image ${missing.length} files, ${total(missing)} bytes: ${missing.join(" ")}`);
console.log(`entering the image but not needed at runtime ${testFiles.length} files, ${total(testFiles)} bytes`);
console.log(`build context ${total(context())} bytes read, ${total([...readFiles])} bytes made it into the image`);

idx instruction  argument                                 type          files   bytes
  0  base      runtime-base                               filesystem      2    6900
  1  copy      config/region.json /app/config/region.json filesystem      1      35
  2  run       install-deps                               filesystem      2   51280
  3  copy      source /app/source                         filesystem      5   10210
  4  env       REGION=north                               metadata        0       0
  5  entry     /app/source/entry.js                       metadata        0       0

metadata: {"env":"REGION=north","entry":"/app/source/entry.js"}
image: 4 filesystem layers, 2 metadata instructions, 68425 bytes
read from context 6 files, 10245 bytes
in context but never entering the image 3 files, 5436 bytes: config/local.json image-definition reading-samples/day-1.csv
entering the image but not needed at runtime 2 files, 6860 bytes
build context 15681 bytes read, 10245 bytes made it into the image

The six-line definition file produces four filesystem layers; two lines only write metadata and add no bytes to the stack. In the previous lesson’s terms: changing the env and entry lines changes no layer’s content digest, because there is no layer there.

In the Context but Never in the Image

Lining up three numbers shows the definition file’s real cost. The context is 15,681 bytes, what is read from the context is 10,245 bytes, and the image is 68,425 bytes. All three differ, and each difference says something separate.

The gap between context and read is 3 files and 5,436 bytes: the local config file, the definition file itself, and the sample reading data enter the context, no instruction wants them, and they never reach the image. They are still within the build step’s read access, though — being in the context is a permission independent of entering the image. This shows the exclusion rule is not written only for size.

The gap between read and image runs the other way: the image is about 6.7 times bigger than what it took from the context. The source of the difference is the run instruction; a single line produces 51,280 bytes, 75 percent of the image is that layer. The 33,880-byte dependency tree the exclusion rule cut has not left the image — its source has changed: instead of being copied in the state installed on the writer’s machine, it is produced by the instruction itself. This is the difference pulled out of the environment — the installed dependencies’ version is no longer a machine’s state but one line of the definition file.

The last item of cost sits in the copy source line. When a directory is copied, the instruction does not say what goes into it: test files enter the image too, 2 files and 6,860 bytes, 10.0 percent of the image not needed at runtime. Because the instruction carries a directory name rather than a file, this cannot be seen by looking at the definition file; it can only be seen by counting layer content. This is where isolation falls short: the image records not what is needed but what was copied.

Selective Copying Versus Copying the Context

How much of the instruction selects is a decision, and the decision can be counted three ways. IM11. A second definition file copies the whole context with a single instruction; a third measurement runs the same definition with the exclusion rule empty. IM12. The files needed at runtime are listed explicitly: the source files outside the test directory and the region config. Everything else is unneeded if it enters the image.

// measurement-network/broad.mjs — compares selective copying against copying the whole context
import { build } from "./interpret.mjs";
import { buildTree, total, DEFINITION } from "./project.mjs";

const BROAD = `base      runtime-base
copy      .  /app
run       install-deps
entry     /app/source/entry.js
`;

const NEEDED = (y) => (y.startsWith("source/") && !y.includes("/test/")) || y === "config/region.json";

buildTree();
const measure = (label, definition, rule) => {
  const { layers, readFiles } = build(definition, rule);
  const k = layers.filter((x) => x.files > 0);
  const unneeded = [...readFiles].filter((y) => !NEEDED(y));
  console.log(label.padEnd(30) + String(k.length).padStart(2) + String(k.reduce((t, x) => t + x.files, 0)).padStart(7) +
    String(k.reduce((t, x) => t + x.bytes, 0)).padStart(9) + String(unneeded.length).padStart(9) +
    String(total(unneeded)).padStart(8));
};
console.log("definition".padEnd(30) + "lyr  files     bytes  unneeded  bytes");
measure("selective copy", DEFINITION);
measure("full context, exclusion on", BROAD);
measure("full context, exclusion off", BROAD, []);

definition                    lyr  files     bytes  unneeded  bytes
selective copy                 4     10    68425        2    6860
full context, exclusion on     3     13    73861        5   12296
full context, exclusion off    3     19   158097       11   96532

Three lines produce the same application, and their sizes are 68,425, 73,861, and 158,097 bytes. The gap between the first two lines is small — 5,436 bytes — because the exclusion rule has already cut the heavy directories. When the rule is emptied in the third line, the image grows to about 2.3 times its size, and the count of unneeded files inside it climbs from two to eleven, unneeded bytes from 6,860 to 96,532. About 61 percent of the image is made of files that will never run: logs, leftover build artifacts, docs, and local config.

The two definitions’ layer counts differ too: the selective definition produces four filesystem layers, the broad one three. Fewer layers is not a good outcome here — in the broad definition, the source and the config fall into the same instruction and cannot be split into separate layers, and on top of that, they are copied before the dependency install. The cost of these two details is invisible in this lesson; it is exactly what the next lesson measures.

Summary

• An image definition file is a sequence of instructions; instructions apply in order, and each lays a layer on top of the previous one. The six-line definition produced four filesystem layers; two lines only wrote metadata — layer count is not line count.
• The build context is the set of files instructions can read. From a 15-file, 99,917-byte tree, the exclusion rule cut 6 files and 84,236 bytes; the context was left with 9 files and 15,681 bytes.
• 3 files and 5,436 bytes were counted as present in the context but never entering the image. Being in the context is a permission separate from entering the image; the exclusion rule is not written only for size.
• The run instruction produced 51,280 bytes in a single line — 75 percent of the image. The excluded dependency tree did not leave the image; its source moved from a machine’s state to one line of the definition file.
• The directory-copying instruction let 2 test files and 6,860 bytes into the image. The image records not what is needed but what was copied; the difference is visible only by counting layer content.
• Built with three definitions, the same application produced images of 68,425, 73,861, and 158,097 bytes. With the exclusion rule empty, the definition that copies the whole context grew the image to about 2.3 times its size and carried the unneeded file count from two to eleven.

Next Step

This measurement counted a single build. When instructions are applied from scratch on every build, the 51,280 bytes the run line produces get regenerated every time too — yet if the dependency list has not changed, its output does not change either. Given that layers are named by their content digest, an unchanged instruction’s layer can be taken from storage instead of being regenerated. The next lesson builds this storage and gives a single decision’s number: how much does the order in which dependency install and source copy are written move the cache hit rate, the count of regenerated layers, and the total bytes copied, across the same change sequence?