--- title: Containers source: 'https://academia.sh/en/courses/containers' language: en updated: '2026-08-23T18:37:08+00:00' license: 'CC BY-SA 4.0' lessons: 21 outcomes: - 'Compares isolation levels together with their cost' - 'Produces small, reproducible, least-privileged images' - 'Brings a multi-component application up locally' --- # Containers Pulling the difference removed from the environment inside the build artifact: comparing isolation levels against their cost, what namespaces, control groups and the union filesystem actually isolate, the promise the runtime and image format specifications carry, the effect of the layer model and the layer cache on build time, multi-stage builds, the size and attack surface balance of base image selection, tagging schemes, image registries and signing, resource limits, the ephemeral filesystem, volumes, container networking, users and permissions, a multi-container local environment, and runtime security. ## Container Concept Meeting the problem of dependency and environment difference with isolation: comparing the isolation level and cost of a physical server, a virtual machine, and a container; what each of namespaces, control groups, and the union filesystem separates; the portability promise carried by runtime and image format specifications; and the resource a container holds through the steps of creation, running, stopping, and cleanup. ### Why Containers Source: https://academia.sh/en/courses/containers/why-containers Whether the environment difference can be pulled into the output is counted: the previous course's 18 differences are split into three buckets, the byte cost of what gets pulled in is measured, and the share of what must stay in the environment is written down. The previous course counted where the difference hides and stopped at one point: across ten dimensions between three environments and production there were **18 differences**, 12 of them written in the environment manifest, 6 visible only while the process was running. What the count actually said was not the size of the difference but **where it sat**: installed package versions, resource limits, file paths, timezone. The difference was not inside the software; it was in the place the software ran. This means there are two separate ways to close the difference. The first is to pull the environments closer together: draw every link's manifest toward production's, then keep it drawn there. The second is to take the difference out of the environment and put it **inside the build artifact**. The less the output demands from the place it runs, the less it matters how different that place is. The name of the second way is the **container**: placing the pieces of the environment the application needs alongside the application itself, and carrying all of it as a single output. This lesson asks how much of that way it actually closes, and the question has three parts: how many of the 18 differences can be pulled into the output, what pulling them in costs in bytes, and what remains when they cannot be pulled in. The same fiction runs throughout the course: a municipality's **regional measurement network**. The software collects readings from water meters, verifies them, turns them into invoices, and opens work orders for field crews; a nightly batch job processes the whole day's readings. It is fiction; the numbers below come from that fiction's model too. **CC1.** The previous course's difference list is this lesson's **input**; the ten dimensions and the difference count per environment are carried over from there, not recounted. **CC2.** The measured output is not a single file but a set of files; how the set is packaged is the next topic's question. #### Splitting the Difference into Three Buckets There is exactly one condition for a difference to be pulled into the output: the dimension's value must be writable as a **file**. A package version can be written; the package itself is placed inside the output and the version installed in the environment stops mattering. Network latency cannot be written; latency is not a file's content but a property of the path between two machines. There are dimensions in between too, and they need a name of their own. **CC3.** The bucket rule is single: if the dimension's value can be written into the output as a file, **included**; if the kernel, the hardware, or the processed data itself decides it, **environment**; if both are true, **boundary**. **CC4.** A dimension's bucket is the same across every environment; the bucket depends on the dimension, not the environment. ```js // measurement-network/bucket.mjs — splits the previous course's 18 differences into three buckets (model, fictional data) // Input: the dimensions on which the three environments diverge from production in the // regional measurement network fiction; carried over from the previous course, not recounted here. const DIFFERENCES = { development: ["resolverVersion", "memoryMB", "concurrentWorkers", "meterCount", "billingGateway", "workOrderQueue", "timezone", "caseSensitiveFileSystem", "networkLatencyMs"], test: ["resolverVersion", "memoryMB", "concurrentWorkers", "meterCount", "billingGateway", "timezone", "networkLatencyMs"], staging: ["meterCount", "networkLatencyMs"], }; // Bucket rule: if the dimension's value can be written into the output as a file, INCLUDED; // if the kernel, the hardware, or the processed data itself decides it, ENVIRONMENT; if both // are true, BOUNDARY. const BUCKET = { resolverVersion: ["included", "the resolver binary is carried inside the output"], rulePackageVersion: ["included", "the rule package is carried inside the output"], concurrentWorkers: ["included", "the worker count is fixed in the configuration inside the output"], timezone: ["included", "the timezone table and setting are carried inside the output"], memoryMB: ["environment", "the control group enforcing the limit sits outside the output"], meterCount: ["environment", "data volume is a property of the running system, not the output"], networkLatencyMs: ["environment", "latency is a property of the physical path"], billingGateway: ["boundary", "the client is included, the gate itself stays outside"], workOrderQueue: ["boundary", "the client is included, the queue stays outside"], caseSensitiveFileSystem: ["boundary", "the files are included, the name-matching rule comes from below"], }; const ENVIRONMENTS = Object.keys(DIFFERENCES); const DIMENSIONS = Object.keys(BUCKET); const bucketOf = (b) => BUCKET[b][0]; console.log("dimension".padEnd(26) + "bucket".padEnd(13) + "diffs".padEnd(7) + "reason"); for (const b of DIMENSIONS) { const n = ENVIRONMENTS.filter((o) => DIFFERENCES[o].includes(b)).length; console.log(b.padEnd(26) + bucketOf(b).padEnd(13) + String(n).padEnd(7) + BUCKET[b][1]); } const allDifferences = ENVIRONMENTS.flatMap((o) => DIFFERENCES[o]); const countByBucket = (k) => allDifferences.filter((b) => bucketOf(b) === k).length; const dimensionCountByBucket = (k) => DIMENSIONS.filter((b) => bucketOf(b) === k).length; console.log("\ntotal differences: " + allDifferences.length + " (across ten dimensions)"); for (const k of ["included", "environment", "boundary"]) { console.log(`${k.padEnd(13)}dimensions ${dimensionCountByBucket(k)}/10 diffs ${countByBucket(k)}/18 ` + DIMENSIONS.filter((b) => bucketOf(b) === k).join(", ")); } console.log("\ndifference remaining per environment once pulled into the output:"); for (const o of ENVIRONMENTS) { const remaining = DIFFERENCES[o].filter((b) => bucketOf(b) !== "included"); console.log(`${o.padEnd(13)}${DIFFERENCES[o].length} -> ${remaining.length} remaining: ${remaining.join(", ")}`); } const remainingTotal = ENVIRONMENTS.reduce((t, o) => t + DIFFERENCES[o].filter((b) => bucketOf(b) !== "included").length, 0); console.log(`total 18 -> ${remainingTotal}; closing the boundary bucket too gives ${remainingTotal - countByBucket("boundary")}`); ``` ``` dimension bucket diffs reason resolverVersion included 2 the resolver binary is carried inside the output rulePackageVersion included 0 the rule package is carried inside the output concurrentWorkers included 2 the worker count is fixed in the configuration inside the output timezone included 2 the timezone table and setting are carried inside the output memoryMB environment 2 the control group enforcing the limit sits outside the output meterCount environment 3 data volume is a property of the running system, not the output networkLatencyMs environment 3 latency is a property of the physical path billingGateway boundary 2 the client is included, the gate itself stays outside workOrderQueue boundary 1 the client is included, the queue stays outside caseSensitiveFileSystem boundary 1 the files are included, the name-matching rule comes from below total differences: 18 (across ten dimensions) included dimensions 4/10 diffs 6/18 resolverVersion, rulePackageVersion, concurrentWorkers, timezone environment dimensions 3/10 diffs 8/18 memoryMB, meterCount, networkLatencyMs boundary dimensions 3/10 diffs 4/18 billingGateway, workOrderQueue, caseSensitiveFileSystem difference remaining per environment once pulled into the output: development 9 -> 6 remaining: memoryMB, meterCount, billingGateway, workOrderQueue, caseSensitiveFileSystem, networkLatencyMs test 7 -> 4 remaining: memoryMB, meterCount, billingGateway, networkLatencyMs staging 2 -> 2 remaining: meterCount, networkLatencyMs total 18 -> 12; closing the boundary bucket too gives 8 ``` Buckets diverge between dimension count and difference count. The included bucket has four dimensions but only **6 differences** stand there; the rule package version was already the same across all four environments — there was no difference to close. The environment bucket has three dimensions but **8 differences** stand there; data volume and network latency diverged from production in all three environments. So inclusion and the size of the difference are lined up in opposite directions: the two dimensions that produce the most difference are the two dimensions that can never be pulled into the output. This is why bucket counting is read through differences rather than dimensions; the dimension count says what can be addressed, the difference count says how much has actually closed. The per-environment reading is sharper. The development environment drops from 9 differences to 6, test from 7 to 4, staging from **2 to 2** — no change at all. The link closest to production gains nothing, because both of the two dimensions staging diverges on sit in the environment bucket. This result pinpoints exactly where the sentence "the container removes the environment difference" goes wrong: the gain is largest at the link farthest from production and zero at the link closest to it. In total, 18 differences drop to 12; closing the four in the boundary bucket too brings it to 8. #### The Byte Cost of Pulling In Closing the six differences is not free. Every piece that is pulled in is a set of files genuinely written into the output, and the output's size travels wherever it travels: bytes stored, bytes crossing the network, bytes unpacked to disk. **CC5.** The measurement's source tree is real — the service files are written to disk by a deterministic generator (seed 20260801) and really measured with `node:fs`. **CC6.** The sizes of the pulled-in pieces, however, are **model** values — declared figures, not measurements from an actual setup — and a pulled-in piece replaces the one installed in the environment; the two copies are not counted together. ```js // measurement-network/cost.mjs — the byte cost of what is pulled in // The source tree is really written to disk and really measured; the sizes of the pieces // pulled in from the environment are MODEL values, not measurements from an actual setup. import { mkdirSync, writeFileSync, readdirSync, statSync, rmSync } from "node:fs"; import { join } from "node:path"; const ROOT = "/tmp/measurement-network-source"; let seed = 20260801; // generator seed; the same seed gives the same tree const next = () => (seed = (seed * 1103515245 + 12345) % 2147483648); // The fictional services' source tree: deterministic lines are generated for each service. const SERVICES = { collector: 210, verifier: 340, billing: 480, workOrder: 160, nightlyJob: 260 }; rmSync(ROOT, { recursive: true, force: true }); mkdirSync(join(ROOT, "src"), { recursive: true }); for (const [name, lines] of Object.entries(SERVICES)) { const body = Array.from({ length: lines }, (_, i) => `export const ${name}${i} = ${next() % 100000};`).join("\n"); writeFileSync(join(ROOT, "src", `${name}.mjs`), body + "\n"); } writeFileSync(join(ROOT, "config.json"), JSON.stringify({ concurrentWorkers: 8, timezone: "+00:00", rulePackage: "9.1.0" }, null, 2)); const bytes = (path) => statSync(path).isDirectory() ? readdirSync(path).reduce((t, a) => t + bytes(join(path, a)), 0) : statSync(path).size; const sourceBytes = bytes(ROOT); // MODEL: the declared sizes of the pieces pulled out of the environment and into the output. const PULLED_IN = { "resolver runtime": 44_800_000, "rule package and its dependencies": 31_200_000, "timezone table": 460_000, }; const BASE = { "small": 7_600_000, "medium": 31_500_000, "general-purpose": 118_000_000 }; const mb = (n) => (n / 1_000_000).toFixed(2).padStart(8) + " MB"; console.log("measured source tree (real files, seed " + 20260801 + ")"); for (const file of readdirSync(join(ROOT, "src")).sort()) console.log(" src/" + file.padEnd(20) + String(bytes(join(ROOT, "src", file))).padStart(8) + " bytes"); console.log(" config.json".padEnd(24) + String(bytes(join(ROOT, "config.json"))).padStart(8) + " bytes"); console.log(" source total".padEnd(24) + String(sourceBytes).padStart(8) + " bytes = " + mb(sourceBytes)); console.log("\nMODEL: pieces pulled into the output"); let pulledInTotal = 0; for (const [name, b] of Object.entries(PULLED_IN)) { pulledInTotal += b; console.log(" " + name.padEnd(35) + mb(b)); } console.log(" " + "total (base excluded)".padEnd(35) + mb(pulledInTotal)); console.log("\nbase root filesystem type scanned (closed difference count unchanged: 6/18)"); for (const [kind, t] of Object.entries(BASE)) { const output = sourceBytes + pulledInTotal + t; console.log(` ${kind.padEnd(17)}base ${mb(t)} output ${mb(output)} ` + `${Math.round(output / sourceBytes)} times the source per difference ${mb((output - sourceBytes) / 6)}`); } rmSync(ROOT, { recursive: true, force: true }); ``` ``` measured source tree (real files, seed 20260801) src/billing.mjs 15674 bytes src/collector.mjs 7217 bytes src/nightlyJob.mjs 9219 bytes src/verifier.mjs 11409 bytes src/workOrder.mjs 5472 bytes config.json 78 bytes source total 49069 bytes = 0.05 MB MODEL: pieces pulled into the output resolver runtime 44.80 MB rule package and its dependencies 31.20 MB timezone table 0.46 MB total (base excluded) 76.46 MB base root filesystem type scanned (closed difference count unchanged: 6/18) small base 7.60 MB output 84.11 MB 1714 times the source per difference 14.01 MB medium base 31.50 MB output 108.01 MB 2201 times the source per difference 17.99 MB general-purpose base 118.00 MB output 194.51 MB 3964 times the source per difference 32.41 MB ``` The written code comes to **49,069 bytes**. The three pieces pulled out of the environment and into the output add up to 76.46 MB; the underlying root filesystem is added on top of that, and with the smallest base the output comes to **84.11 MB**. The ratio is **1,714 times the source**. The cost per closed difference across the six is **14.01 MB**, and this is the first line item of the isolation budget: one difference has been pulled out of the environment at a cost of fourteen megabytes. Scanning the base type gives the second line item. The number of closed differences is the same across all three types — 6 — but the output ranges from 84.11 MB to 194.51 MB. The cost paid for the same gain differs by a factor of **2.3**, and the only thing producing the difference is which root filesystem the output is placed on top of. This is why the sentence "the container is lightweight" carries no decision: lightness is not a measurement result but the result of a choice, and the choice stands inside the output. The cost is not only a byte line item. The pulled-in resolver runtime used to be a component that upgraded itself automatically whenever the environment was upgraded; from the moment it is pulled in, upgrading it requires the output to be rebuilt. As the difference is taken out of the environment, the responsibility for closing that difference also passes from whoever operates the environment to whoever produces the output. This transfer is the budget's invisible line item. For the team operating the environment, a version upgrade used to be one installation that touched every application on that machine at once. Once pulled into the output, the same upgrade becomes a separate build job for **every output that carries that piece**. The fictional network has five services; upgrading the resolver version means five rebuilds instead of one installation. The byte line item is visible because it can be measured; this one stays invisible until someone measures it — and the isolation budget's real total is the sum of both. #### What Stays in the Environment and What Sits on the Boundary The three dimensions in the environment bucket are separated along with the reasons they cannot be pulled into the output. What enforces the resource limit is a **control group** outside the output; the output can declare how much memory it wants, but it cannot set the limit itself. Data volume is the number of meters being processed, and it is a property of the running system, not of the output. Network latency is a property of the physical path. What the three have in common is this: none of them can be written as a file's content. The three dimensions in the boundary bucket, on the other hand, include only part of the difference. For the billing gateway and the work order queue, the client library, the contract definition, and the timeout setting go into the output; the real gate on the other side stays outside. The difference shrinks but does not close to zero — the "contract drift" defect class stays open even after the client is included. The third boundary dimension is the one that shows most clearly where the isolation is pierced. All the files are inside the output; but the rule that decides whether two file names count as the same name does not come from inside the output — it comes from the file system the output is opened on. The same output fails to find the tariff file on a case-sensitive file system and finds it on a case-insensitive one. The same fact holds for the clock: the timezone table can be pulled in, the clock's value cannot — it is read from the kernel. The one-sentence summary of container isolation follows from here: **the output carries files, not behavior.** Part of the behavior always stays tied to the underlying kernel and file system. #### Summary - There are two ways to close the environment difference; this course measures the second one: taking the difference out of the environment and putting it inside the build artifact. The name of the way is the container. - The previous course's 18 differences were split into three buckets: 6 that can be included, 8 that must stay in the environment, 4 that sit on the boundary. Inclusion and how often a dimension produces a difference run in opposite directions. - The gain per environment is not equal: development drops from 9 differences to 6, test from 7 to 4, staging stays at 2. The link closest to production gains nothing. - The cost of pulling in was measured: 49,069 bytes of source becomes an 84.11 MB output with the smallest base — 1,714 times the source, 14.01 MB per closed difference. When the base type changes, the gain stays the same and the cost rises by a factor of 2.3. - Isolation is pierced: the resource limit is enforced by a control group outside the output, and the clock's value and the file-name-matching rule come from the underlying kernel. The output carries files, not behavior. #### Next Step This lesson showed that a difference can be pulled into the output, but it never asked one thing: where, and next to what, will this 76-megabyte piece of environment pulled in actually run? What happens if a second application on the same machine also brings its own resolver, its own rule package, and its own timezone table — do the two see each other's files, can either exhaust the other's memory, can one stop the other's processes? Closing the environment difference and separating two applications that share an environment are separate jobs, and the name of the second one is isolation. The next lesson splits isolation into levels and compares what each level separates, what it duplicates, and how much memory and startup delay it demands in return, all by the same criteria. ### Servers, Virtual Machines, and Containers Source: https://academia.sh/en/courses/containers/servers-virtual-machines-and-containers Three isolation levels are compared by the same criteria: which one separates what, duplicates what, how many megabytes of memory base and what order of magnitude of startup delay it demands in return, and how many instances fit on a single machine. The previous lesson pulled 6 of the differences out of the environment and measured the cost in bytes: an 84.11 MB output with the smallest base. That measurement dealt with a single output. In the fictional **regional measurement network**, though, there are five services — the reading collector, the verifier, billing, work order, and the nightly batch job — and all of them run on the same machine. When each one brings its own resolver, its own rule package, and its own timezone table, the question changes: once these five outputs stand side by side on the same machine, how separated are they from one another? The name of that separation is **isolation**, and it is not a single thing. One application being unable to see another's files is one thing, being unable to stop its processes is another, being unable to exhaust its memory yet another. This lesson compares three isolation levels — the shared server, the container, and the virtual machine — by the same criteria: what they separate, what they duplicate, what they demand in return. **CC7.** The comparison is at the scale of a single machine; distribution across multiple machines is not this course's subject. **CC8.** All the numbers below are model inputs, not measurements from an actual setup; what matters is the order-of-magnitude difference between two levels, not the value in a single cell. **CC9.** There are five criteria, and each criterion takes a binary value: separated or shared. Partial isolation does not count as full. #### Three Levels by the Same Criteria The criteria are the names of the places isolation can be pierced. **Kernel:** do the two applications call the same kernel. **File system:** can one see the other's files. **Network:** the same interface and the same port space. **Process table:** can one list and stop the other's processes. **User:** are file ownership and permission tied to the same account. ```js // measurement-network/levels.mjs — comparing three isolation levels by the same criteria // MODEL: every number is a declared model input, not a measurement from an actual setup. // The comparison is at the order-of-magnitude level; the ratio between two levels is // meaningful, not the value in a single cell. export const CRITERIA = ["kernel", "fileSystem", "network", "processTable", "user"]; export const LEVEL = { "shared server": { isolates: { kernel: false, fileSystem: false, network: false, processTable: false, user: true }, duplicates: { osCopies: 0, libraryCopies: 0 }, startupMs: 30, memoryBaseMB: 4, diskBaseMB: 0, }, "container": { isolates: { kernel: false, fileSystem: true, network: true, processTable: true, user: true }, duplicates: { osCopies: 0, libraryCopies: 1 }, startupMs: 300, memoryBaseMB: 12, diskBaseMB: 84, }, "virtual machine": { isolates: { kernel: true, fileSystem: true, network: true, processTable: true, user: true }, duplicates: { osCopies: 1, libraryCopies: 1 }, startupMs: 25_000, memoryBaseMB: 512, diskBaseMB: 1_240, }, }; // How many of the previous lesson's 18 differences were pulled out (input: lesson 01's bucket count). export const CLOSED = { "shared server": [0, "libraries are shared; the six differences in the included bucket stay in the environment"], "container": [6, "the six differences in the included bucket go into the output"], "virtual machine": [7, "since the kernel is included too, the name-matching rule closes as well"], }; export const NAMES = Object.keys(LEVEL); export const isolated = (d) => CRITERIA.filter((o) => LEVEL[d].isolates[o]); // The dump is printed only when this file is run directly; the next block imports the // module for its data and does not print this section again. if (import.meta.main) { console.log("isolated thing (+ separated, - shared)"); console.log("criterion".padEnd(16) + NAMES.map((d) => d.padEnd(20)).join("")); for (const o of CRITERIA) console.log(o.padEnd(16) + NAMES.map((d) => (LEVEL[d].isolates[o] ? "+" : "-").padEnd(20)).join("")); console.log("isolated/5".padEnd(16) + NAMES.map((d) => `${isolated(d).length}/5`.padEnd(20)).join("")); console.log("\nduplicated thing (copies per instance)"); for (const y of ["osCopies", "libraryCopies"]) console.log(y.padEnd(24) + NAMES.map((d) => String(LEVEL[d].duplicates[y]).padEnd(20)).join("")); console.log("\ndifference pulled out of the environment (how many of the 18):"); for (const d of NAMES) console.log(" " + d.padEnd(20) + `${CLOSED[d][0]}/18 ` + CLOSED[d][1]); console.log("\nshared thing (the hole):"); for (const d of NAMES) { const hole = CRITERIA.filter((o) => !LEVEL[d].isolates[o]); console.log(" " + d.padEnd(20) + (hole.length ? hole.join(", ") : "none of the criteria")); } } ``` ``` isolated thing (+ separated, - shared) criterion shared server container virtual machine kernel - - + fileSystem - + + network - + + processTable - + + user + + + isolated/5 1/5 4/5 5/5 duplicated thing (copies per instance) osCopies 0 0 1 libraryCopies 0 1 1 difference pulled out of the environment (how many of the 18): shared server 0/18 libraries are shared; the six differences in the included bucket stay in the environment container 6/18 the six differences in the included bucket go into the output virtual machine 7/18 since the kernel is included too, the name-matching rule closes as well shared thing (the hole): shared server kernel, fileSystem, network, processTable container kernel virtual machine none of the criteria ``` The matrix lines up the three levels along a single axis: 1/5, 4/5, 5/5. On the shared server only one of the five criteria is separated — the user account — and even that one separation depends on file permissions being set up correctly. The container separates four criteria and **shares the kernel**. The virtual machine separates all five, because it carries its own kernel. The duplicated-thing column gives the reason for this ordering. The shared server duplicates nothing; the libraries are common, so the entire environment difference stays in the environment — the previous lesson's six differences do not close. The container duplicates the library set once per instance and does not duplicate the operating system copy. The virtual machine duplicates both. The difference pulled out of the environment follows this ordering: 0, 6, and 7. The virtual machine's seventh difference is the dimension it closes by including the kernel too — the file-name-matching rule sat in the boundary bucket in the previous lesson; once the kernel is included, that closes as well. The middle level is not a compromise; it is the name for giving up on separating one specific criterion. The container buys three criteria over the shared server: file system, network, and process table. What it gives up relative to the virtual machine is a single criterion: the kernel. The difference between the three levels adds up in these two numbers, and both are written in the table. The criteria taking a binary value (CC9) is a simplification here: in reality a criterion can be partly separated too — the user account is separate but root privilege can stay shared, the network space is separate but uses the same physical interface. These in-between states show up as "separated" in the table and provide less isolation than the table reads; measuring exactly that is one of the things the next lessons do. #### The Order of Magnitude of the Cost Isolation is something purchased, and payment is made in three line items: startup delay, memory base, disk base. **CC10.** The memory base is the addition the isolation level brings on top of the application's own memory. **CC11.** The machine is 16,384 MB, 1,024 MB is reserved for the operating system, and per-instance application memory is 180 MB. **CC12.** Instances are started sequentially; concurrent startup is not modeled. ```js // measurement-network/magnitude.mjs — the order of magnitude of the isolation cost and its conversion to density import { LEVEL, NAMES, isolated } from "./levels.mjs"; const magnitude = (n) => (n <= 0 ? "-" : "10^" + Math.round(Math.log10(n))); const MACHINE_MB = 16_384, RESERVED_MB = 1_024, APP_MB = 180, INSTANCES = 25; console.log("cost and its magnitude (model input)"); console.log("level".padEnd(20) + "startup".padEnd(14) + "magnitude".padEnd(11) + "memory base".padEnd(15) + "magnitude".padEnd(11) + "disk base"); for (const d of NAMES) { const v = LEVEL[d]; console.log(d.padEnd(20) + `${v.startupMs} ms`.padEnd(14) + magnitude(v.startupMs).padEnd(11) + `${v.memoryBaseMB} MB`.padEnd(15) + magnitude(v.memoryBaseMB).padEnd(11) + `${v.diskBaseMB} MB`); } const base = NAMES.map((d) => LEVEL[d].memoryBaseMB); console.log("memory base ratio (relative to the smallest): " + NAMES.map((d, i) => `${d} ${Math.round(base[i] / base[0])}x`).join(", ")); console.log("\nmemory base per isolated dimension"); for (const d of NAMES) { const k = isolated(d).length; console.log(" " + d.padEnd(20) + `${k} dimensions ` + `${(LEVEL[d].memoryBaseMB / k).toFixed(1)} MB/dimension`); } console.log(`\na single ${MACHINE_MB} MB machine, ${RESERVED_MB} MB reserved, ` + `application per instance ${APP_MB} MB`); for (const d of NAMES) { const instanceMB = APP_MB + LEVEL[d].memoryBaseMB; const fits = Math.floor((MACHINE_MB - RESERVED_MB) / instanceMB); const overhead = ((LEVEL[d].memoryBaseMB / instanceMB) * 100).toFixed(1); console.log(" " + d.padEnd(20) + `instance ${String(instanceMB).padStart(4)} MB ` + `fits ${String(fits).padStart(3)} isolation overhead ${overhead}%`); } console.log(`\nsequential startup of ${INSTANCES} instances and the disk it carries`); for (const d of NAMES) { const s = (LEVEL[d].startupMs * INSTANCES) / 1000; console.log(" " + d.padEnd(20) + `${s.toFixed(2).padStart(8)} s ` + `disk ${String(LEVEL[d].diskBaseMB * INSTANCES).padStart(5)} MB ` + `isolated ${isolated(d).length}/5`); } ``` ``` cost and its magnitude (model input) level startup magnitude memory base magnitude disk base shared server 30 ms 10^1 4 MB 10^1 0 MB container 300 ms 10^2 12 MB 10^1 84 MB virtual machine 25000 ms 10^4 512 MB 10^3 1240 MB memory base ratio (relative to the smallest): shared server 1x, container 3x, virtual machine 128x memory base per isolated dimension shared server 1 dimensions 4.0 MB/dimension container 4 dimensions 3.0 MB/dimension virtual machine 5 dimensions 102.4 MB/dimension a single 16384 MB machine, 1024 MB reserved, application per instance 180 MB shared server instance 184 MB fits 83 isolation overhead 2.2% container instance 192 MB fits 80 isolation overhead 6.3% virtual machine instance 692 MB fits 22 isolation overhead 74.0% sequential startup of 25 instances and the disk it carries shared server 0.75 s disk 0 MB isolated 1/5 container 7.50 s disk 2100 MB isolated 4/5 virtual machine 625.00 s disk 31000 MB isolated 5/5 ``` Reading the orders of magnitude splits the table into two groups. At the memory-base magnitude, the shared server and the container sit **in the same place**: both are 10^1. The virtual machine is two orders of magnitude higher: 10^3. At startup delay, though, the container climbs one order of magnitude (10^2) and the virtual machine three (10^4). So the place the container parts ways with the shared server is not memory but startup and disk; the place the virtual machine parts ways with both is all three line items. The memory base per isolated dimension reduces this to a single number. Shared server 4.0 MB/dimension, container 3.0 MB/dimension, virtual machine 102.4 MB/dimension. The container being the cheapest level per unit comes from its being able to separate four dimensions without duplicating the kernel. The virtual machine separates all five of the five dimensions, but the cost of separating the fifth — the kernel — is by itself forty times the sum of the other four. The per-unit measure putting the shared server below the container looks backward at first glance, because the shared server's total cost is smaller. What looks backward is the divisor changing: 4 MB is divided by a single dimension, 12 MB by four. A comparison that looks at total cost answers "which level is cheapest," while a comparison that looks at per-unit cost answers "what is paid per separated dimension"; in the isolation budget the second question carries the decision, because what is purchased is not bytes but separated criteria. Both questions are read from the same table and do not substitute for each other. The density count translates this to the runtime side. The same 16 GB machine fits 83 instances on the shared server, 80 on the container, 22 on the virtual machine. The isolation overhead — how much of the memory goes to isolation instead of the application — is **2.2%, 6.3%, and 74.0%** respectively. On the virtual machine, close to three-quarters of the memory does not go to the application itself. Sequentially starting twenty-five instances takes 0.75 seconds, 7.50 seconds, and 625 seconds; the disk carried is 0, 2,100, and 31,000 MB. The number twenty-five comes from the fiction: five copies each of five services. This is exactly where startup time carries a decision. The nightly batch job starts once a night, and 625 seconds is a tolerable number for it; the reading collector, on the other hand, is the time field crews wait for when it restarts after a defect, and there the gap between 7.50 seconds and 625 seconds is a duration of downtime. The same model input carries two separate meanings for two services — this is why startup delay is read not against a single threshold but together with how often the work running at that level restarts. The disk line item is multiplied the same way: an 84 MB output climbs to 2,100 MB across twenty-five instances, and that number depends on whether the layers are shared; how sharing works is the next lesson's subject. #### Where Isolation Is Pierced The matrix's last section names each level's hole. On the shared server, four criteria are open: kernel, file system, network, and process table. In the fictional network this means — if the nightly batch job writes a tariff file to the wrong path, the verifier reads it; if the billing service holds a port, the work order service cannot open the same port; if one exhausts memory, all of them stop at once. On the container, the hole is a single one and its name is the kernel. The five services each see their own file system, their own network space, their own process table, and their own user; but all five make system calls to the same kernel. This has three consequences. The clock value the previous lesson measured is still shared: the five services read the same clock, and the timezone setting is inside the output but the clock itself is not. The kernel version is shared: the kernel behavior one service needs concerns the others too. And a kernel defect passes underneath all five isolations at once — the four separated criteria do not close this path. The zero in the disk-base column is also the mark of a hole. The shared server pays no bytes because it carries nothing; what it does not pay for is closing the environment difference. Being the cheapest level in all three line items is not an advantage, it is the result of having purchased nothing. Read on its own, the cost table wins on every row for the shared server; read together with the criteria table, it is visible that it also carries 1/5 alongside 0/18. This is why the isolation budget requires reading the two tables side by side, not separately. On the virtual machine none of the criteria is open; the hole is invisible in the criteria table and visible in the cost table. 102.4 MB/dimension and a startup magnitude of 10^4 are the price of isolation being complete. The choice between the three levels is therefore not the question "which one is better": it is the question **which criterion needs to be separated**, and every criterion left unseparated leaves a defect class open. #### Summary - Isolation is not a single thing; it is a decision made separately for each of five criteria: kernel, file system, network, process table, user. - The three levels separate 1/5, 4/5, and 5/5 criteria respectively. What is duplicated determines this ordering: the shared server duplicates nothing, the container duplicates the library set, the virtual machine duplicates the library set and the operating system copy. - The difference pulled out of the environment depends on the level: 0/18 on the shared server, 6/18 on the container, 7/18 on the virtual machine. The seventh is the file-name-matching rule that closes because the kernel is included. - The cost is written as an order of magnitude: memory base 10^1, 10^1, and 10^3; startup 10^1, 10^2, and 10^4. Memory base per isolated dimension is 4.0, 3.0, and 102.4 MB — the container is the cheapest level per unit. - The same machine fits 83, 80, and 22 instances; isolation overhead is 2.2%, 6.3%, and 74.0%. The container's hole is a single one and its name is the kernel: clock value, kernel version, and kernel defects all pass underneath the four separated criteria at once. #### Next Step This lesson wrote down that the container separates four criteria but never asked how it separates them. If two processes running on the same kernel do not see each other's processes, and the process table is a single table, what does "not seeing" mean? The same holds for the memory limit: if a control group enforces the limit, what decision is made when the limit is exceeded? And if five services carry the same library set, are the same files written to disk five times? The next lesson models the three mechanisms behind these three questions separately — the namespace as a visibility mapping, the control group as a share allocator, the union filesystem as a layer stack — and writes down where each one pierces isolation. ### Underlying Technologies Source: https://academia.sh/en/courses/containers/underlying-technologies The three mechanisms that build container isolation are modeled separately: the namespace as a visibility mapping, the control group as a share allocator, the union filesystem as a layer stack; each one's cost and the place it is pierced are counted. The previous lesson wrote down that the container separates four of the five criteria: file system, network, process table, and user. It did not ask how it separates them. The question is not empty, because the things being separated are singular on a single machine — there is one process table, one memory, one disk. "Separating" cannot mean producing a second one here; it must be something else. This lesson models that other thing as three separate mechanisms. The **namespace** places a visibility mapping on top of the same table. The **control group** distributes the same resource as shares and makes a decision when the limit is exceeded. The **union filesystem** stacks the same files on top of one another as a layer. What the three have in common is this: none of them produces a new resource, all three place a rule on top of a single resource. Wherever there is a rule there is also a hole, and each section names its hole. **CC13.** The three mechanisms are modeled separately; the state in which they work together is not modeled. **CC14.** In the namespace model there is a single global table, and a view deletes no record. #### Namespace: A Visibility Mapping ```js // measurement-network/namespace.mjs — the namespace as a visibility mapping (model) // There is a single global process table; the namespace places a visibility mapping on // top of that table. A record is not deleted, only made invisible. const TABLE = [ // global table: number, name, owning namespace [1, "manager", "root"], [17, "log-collector", "root"], [41, "scheduler", "root"], [102, "reading-collector", "A"], [103, "reading-processor", "A"], [110, "reading-queue", "A"], [121, "verifier", "A"], [140, "verification-cache", "A"], [205, "billing", "B"], [206, "invoice-writer", "B"], [219, "tariff-resolver", "B"], [233, "work-order", "B"], [301, "nightly-job", "C"], [305, "nightly-writer", "C"], ]; // Visibility rule: a namespace sees only its own records and numbers them locally, // starting at 1. Root sees every record. function view(namespace) { const records = namespace === "root" ? TABLE : TABLE.filter((k) => k[2] === namespace); return records.map(([global, name], i) => ({ local: i + 1, global, name })); } const NAMESPACES = ["root", "A", "B", "C"]; console.log(`the global table has ${TABLE.length} records; no view deletes a record`); console.log("namespace".padEnd(11) + "sees".padEnd(8) + "misses".padEnd(10) + "local -> global"); for (const a of NAMESPACES) { const v = view(a); console.log(a.padEnd(11) + `${v.length}/${TABLE.length}`.padEnd(8) + String(TABLE.length - v.length).padEnd(10) + v.slice(0, 3).map((k) => `${k.local}->${k.global}`).join(" ") + (v.length > 3 ? " ..." : "")); } console.log("\nthe same record carries two numbers:"); for (const a of ["A", "B", "C"]) { const first = view(a)[0]; console.log(` ${first.name.padEnd(20)} ${first.local} in ${a}, ${first.global} in root`); } const POOL = 32768; console.log(`\nhole: the number pool is shared (${POOL} numbers, ${TABLE.length} in use)`); const largest = NAMESPACES.slice(1) .map((a) => [a, view(a).length]).sort((x, y) => y[1] - x[1])[0]; console.log(` root sees all ${view("root").length} records; namespaces do not see each other`); console.log(` the namespace opening the most records is ${largest[0]} (${largest[1]} records); ` + `a namespace exhausting the pool stops the others too`); console.log(` an unseen record still consumes resources: in the root view, ` + `${TABLE.length - view("A").length} records sit outside A but on the same machine`); ``` ``` the global table has 14 records; no view deletes a record namespace sees misses local -> global root 14/14 0 1->1 2->17 3->41 ... A 5/14 9 1->102 2->103 3->110 ... B 4/14 10 1->205 2->206 3->219 ... C 2/14 12 1->301 2->305 the same record carries two numbers: reading-collector 1 in A, 102 in root billing 1 in B, 205 in root nightly-job 1 in C, 301 in root hole: the number pool is shared (32768 numbers, 14 in use) root sees all 14 records; namespaces do not see each other the namespace opening the most records is A (5 records); a namespace exhausting the pool stops the others too an unseen record still consumes resources: in the root view, 9 records sit outside A but on the same machine ``` A namespace is not a deletion, it is a **filter**. All fourteen records stand in the table; view A sees five of them, B four, C two. The same record carries two numbers — the reading collector is 1 inside A, 102 inside root — and this duality shows exactly what the isolation rests on: it is not the number itself that is separated, but **which table the number is read from**. The hole has three names. The root view sees all fourteen records; isolation does not work downward, only **sideways**. The number pool is shared: all 32,768 numbers are shared, and a namespace that exhausts the pool also blocks the others from opening records. And an unseen record is not a nonexistent one — the nine records A does not see keep consuming CPU and memory on the same machine. The cost of a namespace is not bytes, it is a mapping table; the environment difference it closes is zero. #### Control Group: A Share Allocator **CC15.** The hard-limited resource and the flexibly shared resource are modeled separately; requests arrive in sequence. **CC16.** The decision made when the limit is exceeded depends on the resource's kind. ```js // measurement-network/cgroup.mjs — the control group as a share allocator (model) // Two separate behaviors are modeled: the DECISION made when a hard-limited resource's // limit is exceeded, and the DISTRIBUTION made by weight for a flexibly shared resource. const GROUP = { // the fictional regional measurement network's three groups collection: { memoryLimitMB: 512, weight: 20, cpuDemand: 60 }, billing: { memoryLimitMB: 768, weight: 50, cpuDemand: 90 }, nightlyJob: { memoryLimitMB: 256, weight: 30, cpuDemand: 80 }, }; // Hard limit: every request arrives in sequence; if exceeded, the decision depends on the resource's kind. const DECISION = { memory: "the requesting process is killed", descriptor: "the request is rejected" }; const REQUEST = { // sequential memory requests in MB collection: [180, 140, 120, 90], billing: [300, 260, 150, 200], nightlyJob: [120, 100, 90], }; console.log("hard limit: memory (MB)"); console.log("group".padEnd(12) + "limit".padEnd(8) + "granted".padEnd(9) + "overflow".padEnd(12) + "decision (memory)"); let grantedTotal = 0, overflowTotal = 0; for (const [name, requests] of Object.entries(REQUEST)) { const limit = GROUP[name].memoryLimitMB; let usage = 0, overflow = []; for (const r of requests) (usage + r <= limit) ? (usage += r) : overflow.push(r); grantedTotal += usage; overflowTotal += overflow.length; console.log(name.padEnd(12) + String(limit).padEnd(8) + String(usage).padEnd(9) + `${overflow.length} requests`.padEnd(12) + (overflow.length ? DECISION.memory : "-")); } console.log(`total granted ${grantedTotal} MB, overflow ${overflowTotal} requests; ` + `sum of limits ${Object.values(GROUP).reduce((t, g) => t + g.memoryLimitMB, 0)} MB`); console.log(`the same overflow gives a different decision for a file descriptor: ${DECISION.descriptor}`); // Flexible sharing: weight is not a limit, it is the share ratio under contention. const TOTAL_SHARE = 100; const weightTotal = Object.values(GROUP).reduce((t, g) => t + g.weight, 0); const demandTotal = Object.values(GROUP).reduce((t, g) => t + g.cpuDemand, 0); console.log(`\nflexible sharing: cpu (total ${TOTAL_SHARE} units, demand ${demandTotal} units)`); console.log("group".padEnd(12) + "weight".padEnd(9) + "demand".padEnd(8) + "contested".padEnd(11) + "alone"); for (const [name, g] of Object.entries(GROUP)) { const share = Math.round((g.weight / weightTotal) * TOTAL_SHARE); console.log(name.padEnd(12) + String(g.weight).padEnd(9) + String(g.cpuDemand).padEnd(8) + String(Math.min(share, g.cpuDemand)).padEnd(11) + Math.min(TOTAL_SHARE, g.cpuDemand)); } console.log("hole: weight is not a ceiling; with no contention a group takes " + `${Math.round(Math.min(TOTAL_SHARE, GROUP.nightlyJob.cpuDemand) / ((GROUP.nightlyJob.weight / weightTotal) * TOTAL_SHARE) * 10) / 10}` + " times its share"); console.log(`hole: the sum of the limits is ${Object.values(GROUP).reduce((t, g) => t + g.memoryLimitMB, 0)}` + " MB, independent of the machine's memory; overcommitment ends in a kernel decision"); ``` ``` hard limit: memory (MB) group limit granted overflow decision (memory) collection 512 440 1 requests the requesting process is killed billing 768 710 1 requests the requesting process is killed nightlyJob 256 220 1 requests the requesting process is killed total granted 1370 MB, overflow 3 requests; sum of limits 1536 MB the same overflow gives a different decision for a file descriptor: the request is rejected flexible sharing: cpu (total 100 units, demand 230 units) group weight demand contested alone collection 20 60 20 60 billing 50 90 50 90 nightlyJob 30 80 30 80 hole: weight is not a ceiling; with no contention a group takes 2.7 times its share hole: the sum of the limits is 1536 MB, independent of the machine's memory; overcommitment ends in a kernel decision ``` The control group has two separate behaviors, and mixing them up produces a wrong expectation. At the hard limit the decision is sharp: in all three groups one request does not fit the limit, and the decision made for memory is that the requesting process is **killed** — not rejected, killed. The same overflow gives a different decision for a file descriptor; the request is **rejected** and the process keeps running. This distinction carries a direct consequence for the fictional nightly job: with a 256 MB limit, 220 MB is granted, the third request does not fit, and the job stops midway. In flexible sharing there is no limit, there is **weight**. Demand is 230 units, the machine is 100 units; the shares under contention come to 20, 50, and 30. But weight is not a ceiling: with no contention, the nightly job takes 2.7 times its share. The second hole is quieter — the sum of the three limits is 1,536 MB, and this number has nothing to do with the machine's memory. Limits can be set larger than the machine's memory; under overcommitment the decision is made not by the control group but by the **kernel**. The cost of a control group is not bytes either, it is bookkeeping; the environment difference it closes is again zero. #### Union Filesystem: A Layer Stack **CC17.** Layers are real directories and files are really written and measured; the delete marker is modeled as a file-name prefix. **CC18.** Only the top layer is writable. **CC19.** The sharing calculation assumes the lower layers are identical in content. ```js // measurement-network/layer.mjs — the union filesystem as a layer stack // Layers are REAL directories; files are really written and really measured. import { mkdirSync, writeFileSync, readdirSync, statSync, copyFileSync, rmSync } from "node:fs"; import { join, dirname } from "node:path"; const ROOT = "/tmp/measurement-network-layer"; const LAYER = ["layer0-base", "layer1-rules", "layer2-writable"]; // bottom to top const MARKER = ".deleted."; // delete-marker prefix; hides the one below const CONTENT = { "layer0-base": { "etc/tariff.json": 4096, "lib/resolver.bin": 61440, "lib/rules.bin": 20480, "app/collector.mjs": 3072, "app/old-config.json": 1024 }, "layer1-rules": { "lib/rules.bin": 24576, "etc/rules-version.txt": 32 }, "layer2-writable": { "app/collector.mjs": 3584, "app/new-rule.mjs": 2048, ["app/" + MARKER + "old-config.json"]: 0 }, }; rmSync(ROOT, { recursive: true, force: true }); for (const [k, files] of Object.entries(CONTENT)) for (const [path, size] of Object.entries(files)) { mkdirSync(dirname(join(ROOT, k, path)), { recursive: true }); writeFileSync(join(ROOT, k, path), Buffer.alloc(size, k.charCodeAt(5))); } const list = (k, sub = "") => readdirSync(join(ROOT, k, sub), { withFileTypes: true }) .flatMap((g) => g.isDirectory() ? list(k, join(sub, g.name)) : [join(sub, g.name)]); const sizeOf = (k, path) => statSync(join(ROOT, k, path)).size; const layerBytes = (k) => list(k).reduce((s, y) => s + sizeOf(k, y), 0); // Merge: the topmost layer wins; a delete marker on top hides the one below. const view = new Map(), shadowed = [], hidden = []; for (const k of [...LAYER].reverse()) for (const path of list(k)) { if (path.split("/").pop().startsWith(MARKER)) { hidden.push(path.replace(MARKER, "")); continue; } view.has(path) ? shadowed.push(`${path} (${view.get(path)} above)`) : view.set(path, k); } for (const y of hidden) view.delete(y); console.log("layer".padEnd(22) + "files".padEnd(8) + "bytes"); for (const k of LAYER) console.log(k.padEnd(22) + String(list(k).length).padEnd(8) + layerBytes(k)); console.log("\nmerged view (path -> serving layer)"); for (const [path, k] of [...view].sort()) console.log(" " + path.padEnd(28) + k); console.log(`visible ${view.size} files, ` + `${[...view].reduce((s, [y, k]) => s + sizeOf(k, y), 0)} bytes`); console.log(`shadowed ${shadowed.length}: ${shadowed.join(", ")}`); console.log(`hidden by delete marker ${hidden.length}: ${hidden.join(", ")} ` + `(still ${sizeOf(LAYER[0], hidden[0])} bytes in the layer below)`); console.log(`files copied for reading: 0 (${view.size} files read in place)`); // Copy-up: writing to a file in a lower layer first copies it to the top layer. const TARGET = "etc/tariff.json", OWNER = view.get(TARGET); mkdirSync(dirname(join(ROOT, LAYER[2], TARGET)), { recursive: true }); copyFileSync(join(ROOT, OWNER, TARGET), join(ROOT, LAYER[2], TARGET)); writeFileSync(join(ROOT, LAYER[2], TARGET), Buffer.alloc(4096 + 16, 122)); console.log(`\ncopy-up: ${TARGET} is ${sizeOf(OWNER, TARGET)} bytes in ${OWNER}; ` + `${sizeOf(OWNER, TARGET)} bytes moved for a 16-byte change`); console.log(` the copy in the lower layer is unchanged (${sizeOf(OWNER, TARGET)} bytes), ` + `the one on top is ${sizeOf(LAYER[2], TARGET)} bytes`); const N = 5; // read-only lower layers are shared const lowerBytes = layerBytes(LAYER[0]) + layerBytes(LAYER[1]), upperBytes = layerBytes(LAYER[2]); console.log(`\nif ${N} instances share the lower layers, ${lowerBytes} + ${N} x ${upperBytes} = ` + `${lowerBytes + N * upperBytes} bytes; without sharing, ${N} x ${lowerBytes + upperBytes} = ${N * (lowerBytes + upperBytes)} bytes ` + `(gain ${N * (lowerBytes + upperBytes) - (lowerBytes + N * upperBytes)} bytes)`); rmSync(ROOT, { recursive: true, force: true }); ``` ``` layer files bytes layer0-base 5 90112 layer1-rules 2 24608 layer2-writable 3 5632 merged view (path -> serving layer) app/collector.mjs layer2-writable app/new-rule.mjs layer2-writable etc/rules-version.txt layer1-rules etc/tariff.json layer0-base lib/resolver.bin layer0-base lib/rules.bin layer1-rules visible 6 files, 95776 bytes shadowed 2: app/collector.mjs (layer2-writable above), lib/rules.bin (layer1-rules above) hidden by delete marker 1: app/old-config.json (still 1024 bytes in the layer below) files copied for reading: 0 (6 files read in place) copy-up: etc/tariff.json is 4096 bytes in layer0-base; 4096 bytes moved for a 16-byte change the copy in the lower layer is unchanged (4096 bytes), the one on top is 4112 bytes if 5 instances share the lower layers, 114720 + 5 x 9744 = 163440 bytes; without sharing, 5 x 124464 = 622320 bytes (gain 458880 bytes) ``` The three layers hold ten files, the merged view holds six. The difference comes from two shadowings and one delete marker: the rules binary and the collector are read from the top layer, and the old config file is never read at all. Not a single file is copied for reading — all six of the six files are read in place, in the layer where they sit. This is the number that corrects the previous lesson's line "the library set is duplicated once per instance": when five instances share the lower layers the total is 163,440 bytes, without sharing it would be 622,320 bytes; the gain is 458,880 bytes. The cost is paid at the moment of writing. To write to a file in a lower layer, that file is first copied to the top layer: 4,096 bytes are moved for a 16-byte change. The ratio grows with file size, and if the write had gone to the resolver binary instead of the tariff file, the bytes moved would have been 61,440. The hole is in three places. A deleted file is not deleted: the old config file is absent from the view but keeps standing at 1,024 bytes in the lower layer — anyone reading the output can find it. The copy-up delay only appears at the moment of writing and is invisible beforehand. And even though all the files are inside the output, file ownership and the name-matching rule come from the underlying file system; this is exactly the dimension that sat in the first lesson's boundary bucket. #### Summary - None of the three mechanisms produces a new resource; all three place a rule on top of a single resource, and wherever there is a rule there is a hole. - A namespace is a visibility mapping: in a 14-record table, the views see 5, 4, and 2 records, and root sees all 14. The number pool is shared, and an unseen record keeps consuming resources. - A control group is a share allocator with two behaviors: a decision at the hard limit (killing the process for memory, rejecting the request for a descriptor), and a weight-based distribution under flexible sharing. Weight is not a ceiling — with no contention a group takes 2.7 times its share. - A union filesystem is a layer stack: 10 files reduce to 6, 0 files are copied for reading, and sharing across five instances saves 458,880 bytes. At the moment of writing, a 16-byte change moves 4,096 bytes. - The only mechanism that pulls a difference out of the environment is the union filesystem; the namespace and the control group close 0 differences, they separate criteria. A deleted file keeps standing in the lower layer, and file ownership and the name-matching rule come from below. #### Next Step These three mechanisms belong to a single machine's kernel and work the way that kernel offers. The measured layer stack, though, does not belong to a machine: it is a stack made of directories, and it can be copied and moved. So what guarantees it will open the same way wherever it is moved — who writes down the order the layers stack in, the configuration it runs with, and that a layer was not corrupted in transit? The next lesson takes up the common structure of the specifications that answer this question: it splits the promise carried by an image format specification into three parts, writes a small verifier that checks the digest chain, and counts how far portability actually goes — what stays the same and what changes between two runtimes that comply with the same specification. ### Container Standards Source: https://academia.sh/en/courses/containers/container-standards The promise carried by an image format specification is split into three parts and a verifier that checks the digest chain is written; how far portability actually goes is counted: what stays the same and what changes between two runtimes that comply with the same specification. The previous lesson measured the three mechanisms that build isolation and showed that all three belong to a single machine's kernel. The measured layer stack, though, does not belong to a machine: it is a stack made of directories, and it can be copied and moved. Being portable is not a guarantee that it will open the same way wherever it is moved. The order the layers stack in, the configuration it runs with, and that a layer was not corrupted along the way must be written down somewhere. That writing's type is a **specification**. This lesson names no specification, body, or product; specifications are referred to by their common structure, because what is measured is not a document's name but **the promise that document carries**. **CC20.** An image format specification defines three things: a **config object**, a **layer list**, and a content-addressed **digest chain**. A runtime specification, in turn, reads these three and defines the rules for starting a process. The three parts of the promise each do a separate job. The config object says what will run: entry point, working directory, user, environment variables. The layer list says what parts the output is made of and how many bytes each part is. The digest chain, in turn, ties the pieces' **identity and order** together; thanks to the chain, whether a layer has changed, and if so from which point in the stack the output becomes a different output, can be computed. The boundary between the two specification types is drawn right here too. The image format specification defines the output **at rest**: which bytes, in which order, under which identity. The runtime specification, in turn, defines the **running** state: which steps a runtime reading this definition will take, in which order. Where one ends is where the other begins, and the differences measured in this lesson's second section sit exactly on top of that boundary. A specification's promise can be read without knowing its name, because the promise always has the same shape: what it binds, what it leaves to interpretation. The fictional regional measurement network's output is defined by these three parts. The verifier below checks that definition. #### Checking the Chain **CC21.** The image definition is fictional; the digests are really computed with `node:crypto` and truncated to the first twelve hexadecimal digits. **CC22.** The chain rule is single: z(0) = own(0) and z(n) = digest(z(n-1) + " " + own(n)). **CC23.** The layers' content sizes are the values the previous lessons measured. ```js // measurement-network/verifier.mjs — a verifier that checks an image format's digest chain // The image definition is FICTIONAL; the digests are really computed with node:crypto. import { createHash } from "node:crypto"; const digest = (v) => "s256:" + createHash("sha256").update(v).digest("hex").slice(0, 12); // Layer contents (fictional): each layer is represented by a block of bytes. const layerContent = [ ["base-root", Buffer.alloc(7_600_000, 48)], ["resolver", Buffer.alloc(44_800_000, 49)], ["rule-package", Buffer.alloc(31_200_000, 50)], ["app", Buffer.alloc(50_787, 51)], ]; // The promise a specification carries has three parts: config object, layer list, // content-addressed digest chain. Chain: z(0)=own(0), z(n)=digest(z(n-1)+" "+own(n)). const chain = (owns) => owns.reduce((z, f, i) => [...z, i === 0 ? f : digest(z[i - 1] + " " + f)], []); function imageDefinition(contents) { const layers = contents.map(([name, block]) => ({ name, size: block.length, own: digest(block) })); const chainValues = chain(layers.map((k) => k.own)); const config = { entryPoint: "nightly-job", workingDir: "/app", user: "measure", chain: chainValues }; return { config, configDigest: digest(JSON.stringify(config)), layers }; } // Verifier: every layer's digest is recomputed, then the chain is rebuilt. function verify(definition, contents) { const broken = []; definition.layers.forEach((k, i) => { if (digest(contents[i][1]) !== k.own) broken.push(i); }); const expected = chain(definition.layers.map((k) => k.own)); const chainBreak = expected.findIndex((z, i) => z !== definition.config.chain[i]); return { broken, chainBreak, brokenCount: chainBreak < 0 ? 0 : expected.length - chainBreak }; } const DEFINITION = imageDefinition(layerContent); console.log("layer list (content-addressed)"); DEFINITION.layers.forEach((k, i) => console.log(" " + String(i).padEnd(3) + k.name.padEnd(14) + String(k.size).padStart(9) + " bytes own " + k.own + " chain " + DEFINITION.config.chain[i])); console.log("config object digest: " + DEFINITION.configDigest); console.log("sound definition: " + JSON.stringify(verify(DEFINITION, layerContent))); // A single byte is changed: where the bond breaks and how many links are affected. const brokenContent = layerContent.map(([a, b], i) => i === 1 ? [a, Buffer.concat([b.subarray(0, 5), Buffer.from("X"), b.subarray(6)])] : [a, b]); const result = verify(DEFINITION, brokenContent); console.log("\n1 byte changed in layer 1 (one of 44800000 bytes)"); console.log(" layer whose digest does not hold: " + result.broken.map((i) => `${i} (${DEFINITION.layers[i].name})`).join(", ")); console.log(" new own digest: " + digest(brokenContent[1][1]) + ", in definition: " + DEFINITION.layers[1].own); // If the broken layer is republished together with the definition, where does the chain break? const newDefinition = imageDefinition(brokenContent); const brokenAt = DEFINITION.config.chain .map((z, i) => z === newDefinition.config.chain[i]).indexOf(false); console.log(" chain breaks at link " + brokenAt + "; " + (DEFINITION.config.chain.length - brokenAt) + "/" + DEFINITION.config.chain.length + " links change, the " + brokenAt + " links below stay exactly the same"); console.log(" the config digest changes too: " + newDefinition.configDigest); console.log(" bytes that must move again: " + brokenContent.slice(brokenAt).reduce((t, [, b]) => t + b.length, 0) + "/" + layerContent.reduce((t, [, b]) => t + b.length, 0)); ``` ``` layer list (content-addressed) 0 base-root 7600000 bytes own s256:1048de0c7a68 chain s256:1048de0c7a68 1 resolver 44800000 bytes own s256:b0c7e6c28bfa chain s256:926ac9255c57 2 rule-package 31200000 bytes own s256:99a2f404c480 chain s256:08f1f6ea4359 3 app 50787 bytes own s256:3fbeb763e66c chain s256:ebdca65649c3 config object digest: s256:d1f033f46596 sound definition: {"broken":[],"chainBreak":-1,"brokenCount":0} 1 byte changed in layer 1 (one of 44800000 bytes) layer whose digest does not hold: 1 (resolver) new own digest: s256:9b5c4ad8de20, in definition: s256:b0c7e6c28bfa chain breaks at link 1; 3/4 links change, the 1 links below stay exactly the same the config digest changes too: s256:169313bf9d63 bytes that must move again: 76050787/83650787 ``` The verifier finds no fault in the sound definition: the broken-layer list is empty, there is no chain break. Then a single byte of the 44,800,000-byte layer is changed, and that layer's content digest no longer holds. This is the definition of content addressing — because the name is derived from the content itself, changing the content also changes the name, and the mismatch is caught by a single comparison. It is not necessary to know which byte the change is in, and the number of bytes searched does not change the result either: one byte changed in 44.8 million and a layer with half its bytes changed are reported the same way, with a single inequality. This is the one-sentence equivalent of naming a layer by its content rather than by its name. The chain's contribution is one step further. If the broken layer is republished together with the definition, the chain **breaks at link 1**: three of the four links change, one link below stays exactly the same. So a layer changing carries itself and everything above it to a new identity, and does not carry what is below it. This asymmetry determines the bytes that must move: of the 83,650,787-byte stack, 76,050,787 bytes must move again, and the 7,600,000-byte base stays where it is. This is why the chain also binds the **order** — if the layers' positions were swapped, the same digests would produce a different chain, and the two outputs would not count as the same. The config object's digest changes too, because the chain is written inside it. This is where the three parts tie into a single identity: a single digest names an output, and that digest covers both what will run and what bytes it is made of. Verification has a cost too, and this is the line item this lesson adds to the isolation budget: a full verification reads the entire stack — 83,650,787 bytes — and computes four content digests and three chain links. The cost grows linearly with the output's size and is paid again on every transfer. What is bought with it is that not a single byte having changed becomes **provable**; a check that looks only at a layer's name has no such proof. What the chain does not bind must also be written down. The digest binds the layer's content, not how that content was produced. There is no promise that building the same layer twice gives the same digest; the chain only says the bytes it holds have not changed, it does not say those bytes are reproducible. This distinction is concrete in the fictional network: the verifier catches a corrupted layer, but if the resolver layer, built twice on two separate machines, comes out with two different digests, it reports this not as corruption but as **two separate outputs**. The boundary between what the verifier says and what it does not say sits exactly here, and a "verified" line read without knowing that boundary promises more than it delivers. #### How Far Does Portability Go The digest chain checks that the bytes have been carried over. It does not check that the carried bytes will behave the same way. What the specification binds and what it leaves to interpretation must be counted separately. **CC24.** Ten properties are split into two sets — five the specification binds, five it leaves to interpretation — and the split is a model. **CC25.** The portability ratio is computed over the property count; no weighting is applied. ```js // measurement-network/portability.mjs — two runtimes complying with the same specification are compared (model) // Properties are split into two sets: what the specification BINDS and what it LEAVES TO INTERPRETATION. const BOUND = ["root filesystem layout", "layer order and merge rule", "content digest algorithm", "entry point and working directory", "environment variable set"]; const LEFT_TO_INTERPRETATION = ["resource limit interpretation", "where log output is written", "default network mode", "user mapping", "stop signal and grace period"]; // Two runtimes; they are required to carry the same value in the bound properties. const RUNTIME = { "runtime 1": { "root filesystem layout": "layer stack", "layer order and merge rule": "bottom to top", "content digest algorithm": "s256", "entry point and working directory": "nightly-job @ /app", "environment variable set": "6 variables written in the definition", "resource limit interpretation": "process is stopped when the limit is exceeded", "where log output is written": "file", "default network mode": "bridge", "user mapping": "one-to-one", "stop signal and grace period": "graceful, 10 s", }, "runtime 2": { "root filesystem layout": "layer stack", "layer order and merge rule": "bottom to top", "content digest algorithm": "s256", "entry point and working directory": "nightly-job @ /app", "environment variable set": "6 variables written in the definition", "resource limit interpretation": "request is rejected when the limit is exceeded", "where log output is written": "file", "default network mode": "host network", "user mapping": "shifted", "stop signal and grace period": "graceful, 10 s", }, }; const [A, B] = Object.keys(RUNTIME); const same = (o) => RUNTIME[A][o] === RUNTIME[B][o]; console.log("feature".padEnd(38) + "set".padEnd(10) + "across two runtimes"); for (const o of [...BOUND, ...LEFT_TO_INTERPRETATION]) console.log(o.padEnd(38) + (BOUND.includes(o) ? "bound" : "open").padEnd(10) + (same(o) ? "same" : `DIFF ${RUNTIME[A][o]} / ${RUNTIME[B][o]}`)); const changed = LEFT_TO_INTERPRETATION.filter((o) => !same(o)); console.log(`\nbound features ${BOUND.length}, left to interpretation ${LEFT_TO_INTERPRETATION.length}`); console.log(`unchanged ${BOUND.length + (LEFT_TO_INTERPRETATION.length - changed.length)}/10, ` + `changed ${changed.length}/10: ${changed.join(", ")}`); console.log(`portability: the output itself is 100% portable (the digest chain is checked), ` + `${(BOUND.length + LEFT_TO_INTERPRETATION.length - changed.length) * 10}% of the behavior is portable`); console.log(`by the previous lessons' count: 6 differences were pulled out of the environment, ` + `${changed.length} new dimensions came back as runtime differences`); ``` ``` feature set across two runtimes root filesystem layout bound same layer order and merge rule bound same content digest algorithm bound same entry point and working directory bound same environment variable set bound same resource limit interpretation open DIFF process is stopped when the limit is exceeded / request is rejected when the limit is exceeded where log output is written open same default network mode open DIFF bridge / host network user mapping open DIFF one-to-one / shifted stop signal and grace period open same bound features 5, left to interpretation 5 unchanged 7/10, changed 3/10: resource limit interpretation, default network mode, user mapping portability: the output itself is 100% portable (the digest chain is checked), 70% of the behavior is portable by the previous lessons' count: 6 differences were pulled out of the environment, 3 new dimensions came back as runtime differences ``` In the five bound properties, the two runtimes carry the same value and are required to: root filesystem layout, layer order together with the merge rule, content digest algorithm, entry point together with working directory, environment variable set. These are the specification's promise, and they are fixed in every compliant runtime. In the five properties left to interpretation, though, there is no binding force; in the model, three of the five come out different between the two runtimes. Which three make up the difference matters. The resource limit interpretation is the exact decision measured in the previous lesson: on one runtime the process that exceeds the limit is stopped, on the other the request is rejected. For the fictional nightly job, this means the same output stopping midway in one place and taking the error and continuing in another. Network mode and user mapping are of the same kind — the output is carried over, the behavior is not. The two properties that do carry the same value must also be read carefully. Where the log output is written and the stop signal are the same across both runtimes in the model; but both sit in the set left to interpretation, so their being the same is not a guarantee, it is a **coincidence**. An assumption built on these two — for instance, that the nightly job will wait ten seconds while stopping — can silently break on a third runtime, and the break only becomes visible at runtime. The distinction from the first lesson repeats itself exactly here: a declared difference is a matter for a decision, an undeclared difference is a surprise. All five properties left to interpretation are candidates for an undeclared difference, even while their values happen to match right now. The way to close it also follows from this count. The five properties left to interpretation move into the bound set the moment they are written explicitly in the runtime's configuration: the resource limit interpretation is chosen, the network mode is specified, the user mapping is declared. This does not change the specification; it is the side running the output filling in what the specification left blank. The cost is known too — a five-line declaration, and that declaration being verified separately for every runtime. As long as the declaration is not written, portability stays at 70%, and the remaining 30% must be re-measured on every transfer; the moment it is written, the measurement is made once and its result sits in the manifest. The number is right here too. The output itself is **100% portable**, because the digest chain checks that the bytes are the same. The behavior, though, is 70% portable. In this lesson, isolation is pierced in the specification itself: the digest chain binds bytes, it does not bind behavior, and every property it does not bind is a new difference dimension. The first lesson pulled 6 differences out of the environment; 3 dimensions come back as a runtime difference. The difference does not go to zero, it changes address: it moves from the environment manifest to the runtime configuration. This is also why the gain staying measurable depends on a new manifest — this time the runtime's own manifest. #### Summary - An image format specification defines three things: a config object, a layer list, and a content-addressed digest chain. The three bind into a single identity. - The verifier catches a single-byte change in a 44,800,000-byte layer from its content digest; this is the definition of content addressing. - The chain is asymmetric: when a layer changes, it and everything above it get a new identity, what is below does not. In the model, 3 of 4 links change and 76,050,787 of 83,650,787 bytes must move again. - Two runtimes complying with the same specification are identical in the five bound properties; three of the five properties left to interpretation differ. The output is 100% portable, the behavior 70%. - The specification binds bytes, not behavior: in exchange for the 6 differences pulled out of the environment, 3 dimensions come back as a runtime difference. The difference does not disappear, it changes manifest. #### Next Step This lesson wrote down the output's identity and the boundary of its portability; all of it was about the output **at rest**. Once the output is actually run, though, something unmeasured shows up: the running container stops, but what does it leave behind? The previous lesson showed the writable layer growing through copy-up; a network name, a volume, and a process record are also held somewhere. The next lesson builds the lifecycle as a state machine, counts the resource held at every transition, and measures how much a stopped-but-uncleaned container accumulates over a period. ### Container Lifecycle Source: https://academia.sh/en/courses/containers/container-lifecycle Creation, running, stopping, and cleanup are built as a state machine; the resource held at every transition is counted, and the layer, network name, and volume that stopped-but-uncleaned containers accumulate over a period are measured. The previous lessons measured what isolation is, what it separates into how many bytes, and how the output is named and carried. All of it looked at a single moment: the moment the output **exists**. A container, though, has separate moments when it starts, runs, stops, and disappears, and what is held between these moments is not the same each time. This lesson builds the lifecycle as a **state machine**. There are five states — nonexistent, created, running, stopped, cleaned up — and four transitions move between these states. What is measured is not the states themselves but **the resource that keeps being held** after every transition. The fictional regional measurement network suits this measurement: the nightly batch job starts and ends every night, services restart after a defect, and no run disappears on its own. This is also where it becomes clear why the question needs measurement. The previous four lessons counted the cost of isolation at the moment the output exists: how many bytes are carried, how many criteria are separated, what order of magnitude of startup delay is demanded. None of those numbers had been multiplied by time. A container, though, is not a single object but one produced **again and again**; the same output is created dozens of times over a period, and every creation leaves its own trace. The axis this lesson adds is time, and adding it makes a single question measurable: is the cost paid once, or paid again on every run? **CC26.** The lifecycle has five states, and no transition exists outside the transition table. **CC27.** The held resource is four line items: writable layer, network name, volume, process record. **CC28.** The writable layer's growth is a model value; the first run adds 340 MB, the second run adds 96 MB. #### Four Transitions and the Held Resource ```js // measurement-network/lifecycle.mjs — the container lifecycle as a state machine (model) // Every transition changes the held resource. The held resource is four line items. const ITEM = ["layerMB", "networkName", "volume", "processRecord"]; // Transition table: [transition, before, after, item changes] const TRANSITION = [ ["create", "nonexistent", "created", { layerMB: 0, networkName: 1, volume: 1 }], ["run", "created", "running", { processRecord: 1, layerMB: 340 }], ["stop", "running", "stopped", { processRecord: -1 }], ["run", "stopped", "running", { processRecord: 1, layerMB: 96 }], ["stop", "running", "stopped", { processRecord: -1 }], ["cleanup","stopped", "cleaned", { layerMB: "reset", networkName: -1 }], ]; let state = "nonexistent", stoppedState = null; const held = Object.fromEntries(ITEM.map((k) => [k, 0])); const print = (label) => console.log(label.padEnd(13) + state.padEnd(15) + ITEM.map((k) => String(held[k]).padStart(k.length)).join(" ")); console.log("transition".padEnd(13) + "state".padEnd(15) + ITEM.join(" ")); print("(start)"); for (const [name, before, after, delta] of TRANSITION) { if (state !== before) throw new Error(`${name}: expected ${before}, found ${state}`); for (const [k, d] of Object.entries(delta)) held[k] = d === "reset" ? 0 : held[k] + d; state = after; if (state === "stopped") stoppedState = { ...held }; print(name); } console.log("\nwhat a stopped-but-uncleaned container holds: " + ITEM.map((k) => `${k} ${stoppedState[k]}`).join(", ")); console.log("the only line item that drops on stopping is the process record; the other three stay in place"); console.log("the line item that is still not reset after cleanup: volume " + held.volume + " (a volume's lifecycle is separate from the container's)"); ``` ``` transition state layerMB networkName volume processRecord (start) nonexistent 0 0 0 0 create created 0 1 1 0 run running 340 1 1 1 stop stopped 340 1 1 0 run running 436 1 1 1 stop stopped 436 1 1 0 cleanup cleaned 0 0 1 0 what a stopped-but-uncleaned container holds: layerMB 436, networkName 1, volume 1, processRecord 0 the only line item that drops on stopping is the process record; the other three stay in place the line item that is still not reset after cleanup: volume 1 (a volume's lifecycle is separate from the container's) ``` The table separates two things. Creation and running are not the same step: creation reserves the network name and the volume but starts no process; the writable layer is zero bytes. Running adds the process record and grows the layer — 340 MB on the first run, 96 MB more on the second. The growth is permanent, because the writable layer carries over from run to run; on the second stop, what is held is 436 MB. The real distinction is in the stop row. **The only line item that drops on stopping is the process record.** The writable layer, network name, and volume stay in place. A container being stopped does not mean what it left behind is freed — stopping is a process operation, not a resource operation. The transition that frees resources is cleanup, and even that does not free everything: the volume stays at 1 even after cleanup, because a volume's lifecycle is separate from the container's. In the fictional network this means the volume the nightly job writes its intermediate files to keeps standing on disk even after the container that ran the job is deleted. The created-but-never-run state is a separate line item too, and it is easy to miss: the layer is zero bytes, the process record is zero, but the network name and the volume are already reserved. A container in this state is close to free in disk terms, but not in namespace terms — it holds its name, and a second container cannot be created under that name. This is exactly where measuring the cost with a single line item is misleading; unless all four line items are counted separately, the sentence "an idle container is not consuming anything" can be built, and the sentence is wrong on three of the four. The state machine's transition check also says something: there is no transition outside the table, and the model enforces the check. A stopped container can be run again, but a cleaned-up container cannot be run. The only one-way transition in the lifecycle is cleanup; there is no going back after it. #### Accumulation Over a Period A single container holding 436 MB is a small number. What the number means depends on how many containers are waiting in this state. **CC29.** A period is 30 days; the services' run frequencies are model inputs. **CC30.** The disk is 512,000 MB, the network name pool is 4,096 names. **CC31.** The cleanup rule takes the form "keep the last N periods"; partial cleanup is not modeled. ```js // measurement-network/accumulation.mjs — accumulation of stopped-but-uncleaned containers (model) // A period is 30 days. Every run leaves behind one container: writable layer, network name, volume. const PERIOD_DAYS = 30, DISK_MB = 512_000, NAME_POOL = 4_096, OUTPUT_MB = 84.11; const SERVICE = { // fictional regional measurement network: frequency and remaining layer nightlyJob: { everyDays: 1, layerMB: 436 }, billing: { everyDays: 3, layerMB: 52 }, collector: { everyDays: 4, layerMB: 26 }, verifier: { everyDays: 6, layerMB: 14 }, workOrder: { everyDays: 10, layerMB: 8 }, }; console.log(`one period = ${PERIOD_DAYS} days; held at period end if no cleanup runs`); console.log("service".padEnd(14) + "runs".padEnd(8) + "per run".padEnd(14) + "period accumulation"); let periodMB = 0, periodRuns = 0; for (const [name, s] of Object.entries(SERVICE)) { const runs = Math.floor(PERIOD_DAYS / s.everyDays); periodMB += runs * s.layerMB; periodRuns += runs; console.log(name.padEnd(14) + String(runs).padEnd(8) + `${s.layerMB} MB`.padEnd(14) + `${runs * s.layerMB} MB`); } console.log(`total: ${periodRuns} containers, ${periodMB} MB, ${periodRuns} network names, ` + `${periodRuns} volumes, 0 process records`); console.log(`the period's accumulation is ${Math.round(periodMB / OUTPUT_MB)} times a single output ` + `(${OUTPUT_MB} MB)`); console.log("\ncleanup rule scanned (last N periods kept)"); console.log("N".padEnd(6) + "layer".padEnd(13) + "network names".padEnd(15) + "volumes".padEnd(9) + "percent of disk"); for (const N of [0, 1, 2, 3, 6, 12]) { const mb = N * periodMB, count = N * periodRuns; console.log(String(N).padEnd(6) + `${mb} MB`.padEnd(13) + String(count).padEnd(15) + String(count).padEnd(9) + ((mb / DISK_MB) * 100).toFixed(1) + "%"); } const diskPeriods = DISK_MB / periodMB, namePeriods = NAME_POOL / periodRuns; console.log(`\nwith no cleanup at all, the ${DISK_MB} MB disk fills in ${diskPeriods.toFixed(1)} periods ` + `(${Math.round(diskPeriods * PERIOD_DAYS)} days)`); console.log(`the ${NAME_POOL}-name network name pool runs out in ${namePeriods.toFixed(1)} periods ` + `(${Math.round(namePeriods * PERIOD_DAYS)} days)`); console.log(`runs out first: ${diskPeriods < namePeriods ? "disk" : "name pool"}; ` + `the ratio between the two limits is ${(namePeriods / diskPeriods).toFixed(1)}`); console.log(`the N=1 rule keeps ${((periodMB / DISK_MB) * 100).toFixed(1)}% of the disk permanently occupied; ` + `the N=0 rule makes every run unrecoverable`); ``` ``` one period = 30 days; held at period end if no cleanup runs service runs per run period accumulation nightlyJob 30 436 MB 13080 MB billing 10 52 MB 520 MB collector 7 26 MB 182 MB verifier 5 14 MB 70 MB workOrder 3 8 MB 24 MB total: 55 containers, 13876 MB, 55 network names, 55 volumes, 0 process records the period's accumulation is 165 times a single output (84.11 MB) cleanup rule scanned (last N periods kept) N layer network names volumes percent of disk 0 0 MB 0 0 0.0% 1 13876 MB 55 55 2.7% 2 27752 MB 110 110 5.4% 3 41628 MB 165 165 8.1% 6 83256 MB 330 330 16.3% 12 166512 MB 660 660 32.5% with no cleanup at all, the 512000 MB disk fills in 36.9 periods (1107 days) the 4096-name network name pool runs out in 74.5 periods (2234 days) runs out first: disk; the ratio between the two limits is 2.0 the N=1 rule keeps 2.7% of the disk permanently occupied; the N=0 rule makes every run unrecoverable ``` In one period, 55 containers are left behind and hold 13,876 MB. The size of the number comes from a single source: the nightly job alone produces 13,080 MB of the total, that is, **94%**. The product of frequency and the layer left per run explains this — thirty runs, 436 MB per run. The other four services total 796 MB. Reading this product tells you where to look when writing a cleanup rule; it is not the number of services but the product of frequency and layer growth that decides. The work order service runs three times in the period and leaves 24 MB; a rule applied to it does not measurably change the total — it closes 24 of the 13,876 MB. The same rule applied to the nightly job, though, closes 13,080 MB, that is, 94% of the accumulation. The difference between writing a single rule for all five services and writing a separate rule for the biggest producer is the ratio of these two numbers. The period's accumulation is **165 times** a single output. The 84.11 MB output measured in the first lesson looks small next to a month of uncleaned runs. This is the line item this lesson adds to the isolation budget: the cost of isolation is not only the output's size, it is the **trace that output leaves on every run**. The first lesson's question was "how much does the output grow if the difference is pulled out of the environment," and the answer was 84.11 MB; this lesson asks the same question over a period, and the answer is 13,876 MB. The two numbers are two different scales of the same decision, and an accounting that looks only at the first leaves the second as a surprise. The scan gives the rule's cost directly. At N=1 the disk is kept permanently 2.7% full, at N=3 it is 8.1%, at N=12 it is 32.5%. With no cleanup at all, the disk fills in 36.9 periods — 1,107 days. The network name pool, meanwhile, runs out in 74.5 periods; the ratio between the two limits is 2.0, meaning the disk runs out twice as fast as the name pool. This ordering decides which resource a cleanup rule should be written against: a cleanup rule is put in place to protect the disk, not to protect the names. The other end of the rule must be counted too. N=0 holds no disk at all, but it leaves nothing behind from a run — the writable layer is the only evidence left after a defective run, and a cleaned-up container's layer cannot be recovered. When the fictional nightly job stops midway, the layer is exactly where you look: partial invoice files, the number of meters processed, the last record written. The N=0 rule erases this evidence the moment the run ends; the N=12 rule permanently holds a third of the disk. The number to write in between is not a preference, it is a balance between two measured quantities: how many days of evidence are kept versus what percentage of disk is held. Because the scan produces the table, the discussion stops being "should cleanup happen" and becomes **"how many periods of evidence are worth what percentage of disk."** #### Where Isolation Is Pierced The lifecycle has three holes, and all three come from the difference between **being stopped** and **not existing**. The first is the **writable layer**. A stopped container's layer stands on disk, and everything in it — the nightly job's intermediate files, partial invoice output, logs — remains readable. The delete marker measured in the previous lesson shows up here a second time: a file deleted from a layer kept standing in the lower layer; now the layer itself keeps standing after the container stops too. Isolation works while something is running; it does not work on the bytes left behind after the run ends. The second is the **network name**. A stopped container's name keeps holding a place in the namespace; a new container cannot be created under the same name. This is a direct source of defects for the fictional nightly job — if one night's job is not cleaned up, the next night's job cannot be started under the same name. The third is the **volume**, and it is the quietest one: even the cleanup transition does not release it. It is the only line item that sits outside the lifecycle, and because it is the only one of the four line items that no single transition resets, its accumulation has to be measured by its own rule. The common reason behind all three must also be written down: **nothing performs the cleanup transition on its own.** Creation, running, and stopping happen within the flow of a job — the nightly job starts, finishes, stops. Cleanup is not part of that flow; it is a transition that must be requested separately. Three of the lifecycle's four transitions are triggered by the work itself, the fourth by a rule. When the rule is not written, the cycle runs with three transitions instead of four, and the scan above counts exactly the cost of that three-transition state. #### Summary - The lifecycle is a five-state state machine; creation and running are separate steps — creation reserves the network name and the volume, running adds the process record and grows the layer. - The only line item that drops on stopping is the process record. The writable layer, network name, and volume stay in place; in the model, a stopped container holds 436 MB, 1 network name, and 1 volume. - Cleanup does not release everything either: the volume keeps standing even after cleanup, because its lifecycle is separate from the container's. Cleanup is the only one-way transition. - 55 containers and 13,876 MB accumulate in one period; the nightly job alone produces 94% of it. The accumulation is 165 times a single output. - The cleanup rule was scanned: at N=1, 2.7% of the disk is permanently held, at N=3 it is 8.1%, at N=12 it is 32.5%. With no cleanup at all the disk fills in 36.9 periods, the name pool runs out in 74.5 periods; the rule is written to protect the disk, not to protect the names. - Isolation works while something is running, not on the bytes left behind afterward: the layer stays readable, the name holds its place, the volume never leaves. Nothing performs the cleanup transition on its own; only three of the four transitions happen within the flow of the work. #### Next Step This topic measured what isolation is: which criterion is separated, how many megabytes and what order of magnitude of startup delay separation demands, which mechanism sets which rule, how the output is named and carried, and what it leaves behind after it runs. All these measurements share one common gap: **where the thing being run came from was never asked**. The measured 84.11 MB output, the shadowed layers, and the digest chain were all taken as given — yet someone produced that stack. The questions start here. How is that output produced; which step gives birth to which layer? Why a stack instead of a single file — what would be lost if it were a single file, what is gained by it being a stack? Who decides the layers' order and by what, and how much of the chain has to be recomputed when the order changes? The next topic takes up the production of the output, and its first question is: what is the measurable payoff of splitting an output into layers? ## Images Splitting the build artifact into layers: the layer model's reuse, the image definition file's instructions and build context, the effect of instruction order on cache hit rate and build time, separating build tools from the output, the base image selection's balance of size, updatability, and attack surface, the tagging scheme for version traceability, image registries and scanning, signing, and running with least privilege. ### Image and Layer Model Source: https://academia.sh/en/courses/containers/image-and-layer-model An image is measured as an ordered list of layers: how many layers two services sharing a common base share, how many bytes are stored once, and what the gain is over a flat copy are counted; when a lower layer changes, how many upper layers' chain ids invalidate and how many bytes a deleted file still occupies in the store are measured. The previous topic defined what isolation is and measured how many bytes the boundary drawn around a process's surroundings isolates. Throughout those measurements, where the thing being run came from was never asked. An isolated process opens inside a filesystem, and that filesystem comes from somewhere; this lesson measures where it comes from. The previous course defined the build artifact as a single object; the immutability principle and the content digest were measured there. A container's output is bound to the same principle — it is produced once, is never changed in place, and its identity is its content's digest — but it is not a single file. An **image** is an ordered list of **layers**, and the reason for this separation can be counted: when two services use the same base, a single-file output stores two full copies, while a layered output stores the shared portion once. The questions are these: how many layers are shared, how many bytes are stored once, and what invalidates above a lower layer when it changes? **IM1.** The layer model below is built with `node` and runs over real directories; no real container is run. A layer is a directory, an image is a list of layer names. **IM2.** The content digest is the first twelve hexadecimal digits of the sha256 value taken over the sorted path-and-byte pairs; the truncation is for readability and matches the format used in the previous course. **IM3.** Two services of the fictional regional measurement network are measured: the service that collects meter readings and the service that verifies them. The network and the services are fictional. #### Layer, Image, Chain Id The layer model has three parts. A **layer** is a set of files and is named by its own content digest. An **image** is a list of layers ordered bottom to top; the filesystem that will run is formed by stacking this list in order. The **chain id** names a layer's position within the image and is derived from the previous chain id and the layer's content digest — the same layer carries the same content digest in two different images, but it gets a different chain id if the stack beneath it differs. This distinction is what the rest of the measurement rests on. **IM4.** Layer contents are generated files scaled by line count; the sizes measured are the real byte counterpart of this generation, not a real product's actual image size. ```js // measurement-network/layer.mjs — builds layers as real directories (model) import { mkdirSync, writeFileSync, readFileSync, readdirSync, rmSync } from "node:fs"; import { createHash } from "node:crypto"; const ROOT = "layers"; const digest = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12); // Layer body: line-count-scaled, reproducibly generated file content. export const body = (name, lines) => Array.from({ length: lines }, (_, i) => `// ${name} ${i}\nexport const a${i} = ${i * 7};\n`).join(""); export const LAYER = { "base-root": { "root/shell.js": body("shell", 30), "root/clock.js": body("clock", 14) }, "shared-library": { "lib/metrics.js": body("metrics", 120), "lib/rules.js": body("rules", 46) }, "shared-config": { "config/region.json": '{"region":"north","counter":41820}\n' }, "collector-dependency": { "dependency/queue.js": body("queue", 190), "dependency/csv.js": body("csv", 62) }, "collector-source": { "app/collect.js": body("collect", 34), "app/entry.js": body("entry", 9) }, "verifier-dependency": { "dependency/rule.js": body("rule", 88) }, "verifier-source": { "app/verify.js": body("verify", 27), "app/entry.js": body("ventry", 11) }, }; export const setUp = (definition = LAYER) => { rmSync(ROOT, { recursive: true, force: true }); for (const [name, files] of Object.entries(definition)) for (const [path, content] of Object.entries(files)) { mkdirSync(`${ROOT}/${name}/${path}`.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(`${ROOT}/${name}/${path}`, content); } }; // Walks the real directory on disk; the layer's file list, bytes, and content digest come from here. export const walk = (name, sub = "") => { const root = `${ROOT}/${name}${sub}`; return readdirSync(root, { withFileTypes: true }).flatMap((g) => g.isDirectory() ? walk(name, `${sub}/${g.name}`) : [`${sub}/${g.name}`.slice(1)]).sort(); }; export const layerInfo = (name) => { const paths = walk(name); const bodies = paths.map((p) => readFileSync(`${ROOT}/${name}/${p}`)); return { name, files: paths.length, paths, bytes: bodies.reduce((t, g) => t + g.length, 0), digest: digest(paths.map((p, i) => `${p}\n${bodies[i]}`).join("\n")) }; }; // Chain id: derived from the previous chain's id and the layer's content digest. export const chain = (names) => names.reduce((z, name) => [...z, digest((z.at(-1) ?? "") + layerInfo(name).digest)], []); export const COLLECTOR = ["base-root", "shared-library", "shared-config", "collector-dependency", "collector-source"]; export const VERIFIER = ["base-root", "shared-library", "shared-config", "verifier-dependency", "verifier-source"]; ``` The stack's order is not arbitrary: the base, which changes least often, sits at the bottom; the application source, which changes most often, sits at the top. What this order buys shows up in the next measurement. #### The Measure of a Shared Layer Two services are built on the same base: the root files, the shared metrics library, and the region config are three layers. Above them sit each service's own dependency and source layers. ```js // measurement-network/measure.mjs — counts two images' layer sharing and store gain import { setUp, layerInfo, chain, COLLECTOR, VERIFIER } from "./layer.mjs"; setUp(); const all = [...new Set([...COLLECTOR, ...VERIFIER])].map(layerInfo); console.log("layer".padEnd(24) + "files bytes " + "content digest"); for (const k of all) console.log(k.name.padEnd(24) + String(k.files).padStart(3) + String(k.bytes).padStart(8) + " " + k.digest); const size = (image) => image.reduce((t, name) => t + layerInfo(name).bytes, 0); const shared = COLLECTOR.filter((name) => VERIFIER.includes(name)); const store = all.reduce((t, k) => t + k.bytes, 0); const flat = size(COLLECTOR) + size(VERIFIER); console.log(`\nreading-collector: ${COLLECTOR.length} layers, ${size(COLLECTOR)} bytes`); console.log(`verifier : ${VERIFIER.length} layers, ${size(VERIFIER)} bytes`); console.log(`shared layers ${shared.length}, stored once ${size(shared)} bytes`); console.log(`flat copy ${flat} bytes / layer store ${all.length} layers ${store} bytes`); console.log(`gain ${flat - store} bytes, %${(100 - (store * 100) / flat).toFixed(1)}`); console.log("chain ids (collector): " + chain(COLLECTOR).map((z) => z.slice(0, 8)).join(" ")); ``` ``` layer files bytes content digest base-root 2 1511 7e497ae2520f shared-library 2 6182 012f6ec862a6 shared-config 1 35 0fa5a2fe9601 collector-dependency 2 9101 48d5c4fac6a9 collector-source 2 1550 879ed761eaa7 verifier-dependency 1 3043 7d842f3affdb verifier-source 2 1336 f03b03ea3a2f reading-collector: 5 layers, 18379 bytes verifier : 5 layers, 12107 bytes shared layers 3, stored once 7728 bytes flat copy 30486 bytes / layer store 7 layers 22758 bytes gain 7728 bytes, %25.3 chain ids (collector): ae409503 9f5f4309 14a3ffd1 1c8a83e6 79a72ddd ``` The two images' total size is 30,486 bytes; what the layer store keeps is 22,758 bytes: the three shared layers' 7,728 bytes are stored once for both images, a 25.3 percent gain. What the measurement should be read for is not the ratio but where the ratio comes from: the gain equals the shared base's share of the total. As each service's own dependency layer grows, the ratio falls; as the shared base grows, it rises. If twenty services used the same base instead of two, the same 7,728 bytes would be stored once for twenty images, and the saving would multiply not by the layer count but by the **number of sharing images**. This is what is gained over the previous course's single-file build artifact. The difference pulled out of the environment is the same size and sits right here: the shared library's version and the region config are no longer something installed on the machine that runs them — they are two layers that sit inside the output and are named by their digest. Another version of that library existing in the environment does not change the measurement, because the filesystem being measured is born from the layers. #### When a Lower Layer Changes Layered output's cost is that layers are bound to one another by a chain. A single line is added to the shared library, and what changes in the two images is counted. **IM5.** The change is kept at the smallest scale — a single line is added to one file in the shared library — so that the invalidated chain's length comes only from the change's position, not its size. **IM6.** Deletion is modeled with a marker file placed in the upper layer: an empty path written with a `deleted/` prefix removes the same path underneath from the view. Real union filesystems format this marker differently; the behavior is the same. ```js // measurement-network/change.mjs — what invalidates when a lower layer changes, where a deleted file stays import { writeFileSync, readFileSync } from "node:fs"; import { setUp, layerInfo, chain, body, LAYER, COLLECTOR, VERIFIER } from "./layer.mjs"; setUp(); const before = { c: chain(COLLECTOR), v: chain(VERIFIER) }; const prevDigest = Object.fromEntries([...COLLECTOR, ...VERIFIER].map((a) => [a, layerInfo(a).digest])); const oldBytes = layerInfo("shared-library").bytes; writeFileSync("layers/shared-library/lib/metrics.js", body("metrics", 121)); // one line added const after = { c: chain(COLLECTOR), v: chain(VERIFIER) }; const digestChanged = [...new Set([...COLLECTOR, ...VERIFIER])].filter((a) => layerInfo(a).digest !== prevDigest[a]); const chainChanged = ["c", "v"].map((i) => before[i].filter((z, s) => z !== after[i][s]).length); console.log(`layer with changed content digest : ${digestChanged.length} (${digestChanged.join(", ")})`); console.log(`invalidated chain ids : ${chainChanged.reduce((a, b) => a + b)} (per image ${chainChanged.join(", ")})`); console.log(`upper layers with unchanged content: ${[...new Set([...COLLECTOR, ...VERIFIER])].length - digestChanged.length - 1}`); console.log(`store growth: ${layerInfo("shared-library").bytes} bytes (old version stays at ${oldBytes} bytes)`); // Upper layer: shadows one path, deletes one path. Deletion changes the view, not the lower layer. setUp({ ...LAYER, "collector-patch": { "config/region.json": '{"region":"north","counter":41820,"type":"patch"}\n', "deleted/lib/rules.js": "" } }); const stack = [...COLLECTOR, "collector-patch"]; const view = new Map(); let shadow = 0, deletedBytes = 0; for (const name of stack) for (const path of layerInfo(name).paths) { if (path.startsWith("deleted/")) { const target = path.slice(8); deletedBytes += readFileSync(`layers/${view.get(target)}/${target}`).length; view.delete(target); } else { if (view.has(path)) { shadow++; deletedBytes += readFileSync(`layers/${view.get(path)}/${path}`).length; } view.set(path, name); } } const stackBytes = stack.reduce((t, name) => t + layerInfo(name).bytes, 0); console.log(`\nlayers in stack ${stack.length}, total bytes in stack ${stackBytes}`); console.log(`files in union view ${view.size}, shadowed paths ${shadow}, deleted paths 1`); console.log(`bytes missing from the view but still in the store: ${deletedBytes}`); console.log("lib/rules.js in view: " + view.has("lib/rules.js")); ``` ``` layer with changed content digest : 1 (shared-library) invalidated chain ids : 8 (per image 4, 4) upper layers with unchanged content: 5 store growth: 6222 bytes (old version stays at 6182 bytes) layers in stack 6, total bytes in stack 18429 files in union view 8, shadowed paths 1, deleted paths 1 bytes missing from the view but still in the store: 1654 lib/rules.js in view: false ``` A one-line change alters one layer's content digest but invalidates eight chain ids: the changed layer in each of the two images and the three layers above it. The five separate upper layers' content stays bit-identical; what invalidates is not the content but the **position**. The lower the change sits, the longer the invalidated chain runs — a line's cost is not the line itself but how many layers sit above it. The second cost sits in the store. Under the immutability principle, the old layer is not deleted: the store keeps the 6,182-byte version in place and adds the new 6,222-byte one alongside it. The layer store only grows; a one-line difference between two versions does not change this, because a layer is named by its content digest, and a layer whose content changed is a new object. This is the pair of numbers behind putting small, frequently-changing layers on top and large, rarely-changing ones at the bottom. The same measurement reads in the opposite direction too. Tried on two extremes of the same model, the numbers come out as follows: when the topmost source layer changes, the invalidated chain id count is 1 for the collector image, 0 for the verifier image; when the bottommost root layer changes, it is 5 per image, 10 total. The cost of a change of the same size swings from one to ten depending on where it sits. The stack order is therefore not a matter of taste but a **cost decision**: each layer's position determines in advance the price paid when that layer changes. This is the measurement's decision-bearing form — the layer count alone is neither good nor bad; the product of how often a layer changes and how many layers sit above it is a cost. #### What the Union View Hides As layers stack up, a **union filesystem** view is born: if the same path exists in two layers, the upper one wins; a deletion marker in an upper layer removes the file underneath from the view. The measurement counts what the view hides. A six-layer stack carries 18,429 bytes; 8 files show up in the union view; one path is shadowed, one path is deleted. But the 1,654 bytes that drop out of the view are still in the store and are still part of the image. This is where isolation is pierced. The layer model pulls the filesystem out of the environment and takes it into the output; in exchange, it also takes the output's **history** into itself. A file deleted in an upper layer is absent from the view but present in the image: because the lower layer sits on disk and can be resolved by its content digest, anyone who gets hold of the image can read that file. Deletion is a view operation, not a store operation. ```bash # verify.sh — are the layers real directories on disk, is the deleted file still there find layers -type f | wc -l ls layers/collector-patch/deleted/lib/ wc -c layers/shared-library/lib/rules.js ``` ``` 14 rules.js 1619 layers/shared-library/lib/rules.js ``` Viewed from the shell, eight layers' fourteen files sit on disk in total; `lib/rules.js`, marked deleted in the upper layer, is still readable in the lower layer with its real content of 1,619 bytes. The union view does not show it; the filesystem does. #### What a Layer Does Not Carry The measurements so far counted what the layer model takes into the output: filesystem layout, library version, config file. The other half of the isolation budget is what does not get taken in. A layer is a set of files — it is not a process, not a kernel, not a clock. Even the `base-root` layer at the bottom of the stack is made only of files; when the image runs, the kernel sitting beneath it is the kernel of the machine it runs on. This has three concrete counterparts. First, processor architecture: the executable files inside layers are produced for one architecture, and the image does not carry this dependency inside itself — it assumes it from outside. The content digest names the files' bytes, not which architecture those bytes carry meaning in; the same digest can name a filesystem that does not run on a different architecture. Second, the kernel interface: if a file inside a layer calls a kernel capability that is absent from the environment, the image resolves flawlessly, the chain ids verify, and the process still does not start. Third, the machine manifest: the timestamp and absolute path counted among the previous course's sources that break reproducibility get mixed into a layer's content digest the moment they are written into the layer's content. The layer model does not close off these sources; it only makes visible where they were written, because they now sit inside a measurable object. The distinction comes down to one sentence: an image carries **what will run**, not **where it will run**. The 22,758-byte layer store this lesson measured is the whole of the difference pulled out of the environment; the rest stays as the machine itself. The isolation budget's account in this lesson has three entries: the difference pulled out is a filesystem, its cost is 22,758 bytes plus eight chain ids per change, and where it is pierced is the 1,654 bytes that drop out of the view and stay in the store. #### Summary - An image is not a single-file build artifact but a list of layers ordered bottom to top; each layer is named by its own content digest, and each position within the image is named by a chain id derived from the previous chain and the layer's digest. - Two images built on a shared base shared three layers: the total that was 30,486 bytes under a flat copy fell to 22,758 bytes in the layer store; 7,728 bytes were stored once, a 25.3 percent gain. The gain multiplies with the number of sharing images, not the number of layers. - A single line added to the shared library changed one layer's content digest but invalidated eight chain ids; five upper layers whose content stayed bit-identical got a new position id. The lower a change sits in the stack, the longer the invalidated chain runs. - Under immutability, the old layer is not deleted: the 6,182-byte version stays in place, and the new 6,222-byte one is added alongside it. The layer store only grows. - The union view looks like it deletes, the store does not. When one path was shadowed and one was deleted, 8 files remained in the view, but the 1,654 bytes that dropped out of the view kept sitting inside the image. #### Next Step This lesson's layers were defined by hand: which file went into which layer was written in an object constant. A real image is not built this way — layers are born from **instructions** applied in sequence, and each instruction produces a layer. That opens two new questions. First, where do the files the instructions will copy come from: how many files and how many bytes enter this set, called the **build context**, and how many of them never reach the image at all? Second, how many files and bytes does the **exclusion rule** that narrows this set cut? The next lesson defines an image definition file format, writes an interpreter that applies instructions in sequence, and measures the build context's size. ### Image Definition File Source: https://academia.sh/en/courses/containers/image-definition-file How layers are born from instructions rather than by hand is measured: a five-instruction definition file format is defined, an interpreter that applies instructions in sequence is written, and the build context's size is counted — the files and bytes entering the tree, the share the exclusion rule cuts, and the files that sit in the context but never reach the image. The previous lesson's layers were defined by hand: which file went into which layer was written in an object constant. This was enough to make the measurement, but it did not describe how an image is actually built. A real image is born from **instructions** applied in sequence; each instruction lays a layer on top of the previous one, and when the last one finishes, the image is ready. This lesson measures those instructions and the set they draw from. The files to be copied come from somewhere: the set of files instructions can read is called the **build context**, and its size is a cost. There are three numbers to measure — how many files and bytes enter the context, how many files and bytes the exclusion rule cuts, and how many files enter the context but never reach the image? **IM7.** The instruction set and definition file format below are defined for this lesson; they consist of five instructions and are much smaller than real image definition file formats. The interpreter is a model written with `node`. **IM8.** The `run` instruction does not run a real command; it produces files from a modeled command table — a dependency install's output is represented this way. **IM9.** The tree is the fictional regional measurement network's reading collector service's source tree; file contents are generated text scaled by line count. **IM10.** The exclusion rule works by prefix matching; real rules use a pattern language, which does not change this measurement's numbers. #### Five Instructions The definition file is a text file, and every line is an instruction. Instructions apply in order; the order is the same as the previous lesson's stack order — the one at the bottom runs first. | Instruction | What it does | Produces a layer | |---|---|---| | `base` | Places a starting file set at the bottom of the stack | yes | | `copy` | Copies a path from the build context into the image | yes | | `run` | Runs a command and turns the files it produces into a layer | yes | | `env` | Writes a value into the image's metadata | no | | `entry` | Writes the path to run when the container starts into the metadata | no | The last two instructions do not touch the filesystem; they write the image's **metadata**. This distinction will be measured: an image's layer count is not the definition file's line count. #### The Build Context The tree is set up on disk, and what the exclusion rule cuts is counted. The context is the set instructions can read; a path caught by the exclusion rule is never read, even if no instruction wants it. ```js // measurement-network/project.mjs — sets up the fictional service's source tree on disk, measures the build context import { mkdirSync, writeFileSync, readdirSync, readFileSync, rmSync } from "node:fs"; export const ROOT = "project"; export const text = (name, lines) => Array.from({ length: lines }, (_, i) => `// ${name} ${i}\n`).join(""); export const DEFINITION = `base runtime-base copy config/region.json /app/config/region.json run install-deps copy source /app/source env REGION=north entry /app/source/entry.js `; const TREE = { "image-definition": DEFINITION, "config/region.json": '{"region":"north","counter":41820}\n', "config/local.json": '{"port":8080,"log":"verbose"}\n', "source/entry.js": text("entry", 20), "source/collect.js": text("collect", 140), "source/queue.js": text("queue", 95), "source/test/collect.test.js": text("test-collect", 210), "source/test/queue.test.js": text("test-queue", 160), "dependency/csv/index.js": text("csv", 900), "dependency/queue/index.js": text("queue-dep", 1400), "doc/architecture.md": text("doc", 240), "log/run-1873.log": text("log", 2600), "build-artifact/package-previous.js": text("leftover", 1100), ".local-config/editor.json": text("editor", 12), "reading-samples/day-1.csv": text("sample", 380), }; // Exclusion rule: paths starting with these prefixes never enter the context. export const EXCLUDE = ["dependency/", "log/", "build-artifact/", ".local-config/", "doc/"]; export const buildTree = () => { rmSync(ROOT, { recursive: true, force: true }); for (const [path, content] of Object.entries(TREE)) { mkdirSync(`${ROOT}/${path}`.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(`${ROOT}/${path}`, content); } }; export const walk = (root, sub = "") => readdirSync(`${root}${sub}`, { withFileTypes: true }).flatMap((g) => g.isDirectory() ? walk(root, `${sub}/${g.name}`) : [`${sub}/${g.name}`.slice(1)]).sort(); export const size = (path) => readFileSync(`${ROOT}/${path}`).length; export const isExcluded = (path, rule = EXCLUDE) => rule.some((d) => path.startsWith(d)); export const context = (rule = EXCLUDE) => walk(ROOT).filter((y) => !isExcluded(y, rule)); export const total = (paths) => paths.reduce((t, y) => t + size(y), 0); ``` ```js // measurement-network/measure-context.mjs — tree, exclusion, and context size import { buildTree, walk, size, isExcluded, context, total, ROOT, DEFINITION } from "./project.mjs"; buildTree(); const all = walk(ROOT); const cut = all.filter((y) => isExcluded(y)); console.log("image definition:\n" + DEFINITION); console.log("top dir".padEnd(16) + "files bytes in context"); const group = {}; for (const y of all) (group[y.includes("/") ? y.split("/")[0] + "/" : "(root)"] ??= []).push(y); for (const [d, paths] of Object.entries(group)) console.log(d.padEnd(16) + String(paths.length).padStart(3) + String(total(paths)).padStart(8) + " " + (isExcluded(paths[0]) ? "no" : "yes")); console.log(`\ntree total : ${all.length} files, ${total(all)} bytes`); console.log(`exclusion cut: ${cut.length} files, ${total(cut)} bytes`); console.log(`build context: ${context().length} files, ${total(context())} bytes ` + `(%${((total(context()) * 100) / total(all)).toFixed(1)} of the tree's bytes)`); ``` ``` image definition: base runtime-base copy config/region.json /app/config/region.json run install-deps copy source /app/source env REGION=north entry /app/source/entry.js top dir files bytes in context .local-config/ 1 146 no build-artifact/ 1 17590 no config/ 2 65 yes dependency/ 2 33880 no doc/ 1 2530 no (root) 1 196 yes log/ 1 30090 no reading-samples/ 1 5210 yes source/ 5 10210 yes tree total : 15 files, 99917 bytes exclusion cut: 6 files, 84236 bytes build context: 9 files, 15681 bytes (%15.7 of the tree's bytes) ``` The tree is 15 files and 99,917 bytes; the exclusion rule cuts 6 files and 84,236 bytes, leaving the context with 9 files and 15,681 bytes — 15.7 percent of the tree's bytes. The size of the cut share is not a coincidence: most of a source tree's volume is installed dependencies, logs, and leftover artifacts from previous builds. If the exclusion rule were not written, this 84,236 bytes would be read on every build and would also feed into the cache decision the next lesson measures. There is also something the rule skips, and the count shows it: the `reading-samples/` directory stays in the context and carries 5,210 bytes. The exclusion rule only cuts the paths it names; every path it does not name is in the context. #### The Interpreter The interpreter reads the definition file line by line, opens a separate layer directory for each line, and applies the instruction to that directory. A record of copied paths is kept; the gap at the end of the measurement comes from here. ```js // measurement-network/interpret.mjs — applies instructions in sequence, each instruction produces one layer (model) import { mkdirSync, writeFileSync, readFileSync, rmSync, statSync } from "node:fs"; import { walk, isExcluded, text, EXCLUDE, ROOT } from "./project.mjs"; const BASE = { "root/shell.js": text("shell", 60), "root/runtime.js": text("runtime", 420) }; const COMMAND = { "install-deps": { "app/dependency/csv.js": text("csv-installed", 900), "app/dependency/queue.js": text("queue-installed", 1400) } }; const write = (dir, files) => { for (const [path, content] of Object.entries(files)) { mkdirSync(`${dir}/${path}`.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(`${dir}/${path}`, content); } }; export const build = (definition, rule = EXCLUDE) => { rmSync("image", { recursive: true, force: true }); const readFiles = new Set(), metadata = {}, layers = []; definition.trim().split("\n").forEach((line, i) => { const [instruction, ...arg] = line.trim().split(/\s+/); const dir = `image/layer-${i}`; mkdirSync(dir, { recursive: true }); if (instruction === "base") write(dir, BASE); else if (instruction === "run") write(dir, COMMAND[arg[0]]); else if (instruction === "copy") { const [source, target] = arg; const single = !statSync(`${ROOT}/${source}`).isDirectory(); const prefix = single || source === "." ? "" : `${source}/`; const matches = (single ? [source] : walk(ROOT, prefix ? `/${source}` : "")).filter((y) => !isExcluded(y, rule)); for (const y of matches) { const dest = `${dir}/${target.slice(1)}` + (single ? "" : `/${y.slice(prefix.length)}`); mkdirSync(dest.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(dest, readFileSync(`${ROOT}/${y}`)); readFiles.add(y); } } else metadata[instruction] = arg.join(" "); const paths = walk(dir); layers.push({ index: i, instruction, arg: arg.join(" "), files: paths.length, bytes: paths.reduce((t, y) => t + statSync(`${dir}/${y}`).size, 0) }); }); return { layers, readFiles, metadata }; }; ``` ```js // measurement-network/measure-interpret.mjs — runs the interpreter, counts the gap between context and image import { build } from "./interpret.mjs"; import { buildTree, context, total, DEFINITION } from "./project.mjs"; buildTree(); const { layers, readFiles, metadata } = build(DEFINITION); console.log("idx instruction argument".padEnd(58) + "type files bytes"); for (const k of layers) console.log(` ${k.index} ${k.instruction.padEnd(9)} ${k.arg}`.padEnd(58) + (k.files ? "filesystem " : "metadata ") + String(k.files).padStart(4) + String(k.bytes).padStart(8)); const fsLayers = layers.filter((k) => k.files > 0); const imageBytes = fsLayers.reduce((t, k) => t + k.bytes, 0); const missing = context().filter((y) => !readFiles.has(y)); const testFiles = [...readFiles].filter((y) => y.includes("/test/")); console.log(`\nmetadata: ${JSON.stringify(metadata)}`); console.log(`image: ${fsLayers.length} filesystem layers, ${layers.length - fsLayers.length} metadata instructions, ${imageBytes} bytes`); console.log(`read from context ${readFiles.size} files, ${total([...readFiles])} bytes`); console.log(`in context but never entering the image ${missing.length} files, ${total(missing)} bytes: ${missing.join(" ")}`); console.log(`entering the image but not needed at runtime ${testFiles.length} files, ${total(testFiles)} bytes`); console.log(`build context ${total(context())} bytes read, ${total([...readFiles])} bytes made it into the image`); ``` ``` idx instruction argument type files bytes 0 base runtime-base filesystem 2 6900 1 copy config/region.json /app/config/region.json filesystem 1 35 2 run install-deps filesystem 2 51280 3 copy source /app/source filesystem 5 10210 4 env REGION=north metadata 0 0 5 entry /app/source/entry.js metadata 0 0 metadata: {"env":"REGION=north","entry":"/app/source/entry.js"} image: 4 filesystem layers, 2 metadata instructions, 68425 bytes read from context 6 files, 10245 bytes in context but never entering the image 3 files, 5436 bytes: config/local.json image-definition reading-samples/day-1.csv entering the image but not needed at runtime 2 files, 6860 bytes build context 15681 bytes read, 10245 bytes made it into the image ``` The six-line definition file produces four filesystem layers; two lines only write metadata and add no bytes to the stack. In the previous lesson's terms: changing the `env` and `entry` lines changes no layer's content digest, because there is no layer there. #### In the Context but Never in the Image Lining up three numbers shows the definition file's real cost. The context is 15,681 bytes, what is read from the context is 10,245 bytes, and the image is 68,425 bytes. All three differ, and each difference says something separate. The gap between context and read is 3 files and 5,436 bytes: the local config file, the definition file itself, and the sample reading data enter the context, no instruction wants them, and they never reach the image. They are still within the build step's read access, though — being in the context is a permission independent of entering the image. This shows the exclusion rule is not written only for size. The gap between read and image runs the other way: the image is about 6.7 times bigger than what it took from the context. The source of the difference is the `run` instruction; a single line produces 51,280 bytes, 75 percent of the image is that layer. The 33,880-byte dependency tree the exclusion rule cut has not left the image — its source has changed: instead of being copied in the state installed on the writer's machine, it is produced by the instruction itself. This is the difference pulled out of the environment — the installed dependencies' version is no longer a machine's state but one line of the definition file. The last item of cost sits in the `copy source` line. When a directory is copied, the instruction does not say what goes into it: test files enter the image too, 2 files and 6,860 bytes, 10.0 percent of the image not needed at runtime. Because the instruction carries a directory name rather than a file, this cannot be seen by looking at the definition file; it can only be seen by counting layer content. This is where isolation falls short: the image records not what is needed but **what was copied**. #### Selective Copying Versus Copying the Context How much of the instruction selects is a decision, and the decision can be counted three ways. **IM11.** A second definition file copies the whole context with a single instruction; a third measurement runs the same definition with the exclusion rule empty. **IM12.** The files needed at runtime are listed explicitly: the source files outside the test directory and the region config. Everything else is unneeded if it enters the image. ```js // measurement-network/broad.mjs — compares selective copying against copying the whole context import { build } from "./interpret.mjs"; import { buildTree, total, DEFINITION } from "./project.mjs"; const BROAD = `base runtime-base copy . /app run install-deps entry /app/source/entry.js `; const NEEDED = (y) => (y.startsWith("source/") && !y.includes("/test/")) || y === "config/region.json"; buildTree(); const measure = (label, definition, rule) => { const { layers, readFiles } = build(definition, rule); const k = layers.filter((x) => x.files > 0); const unneeded = [...readFiles].filter((y) => !NEEDED(y)); console.log(label.padEnd(30) + String(k.length).padStart(2) + String(k.reduce((t, x) => t + x.files, 0)).padStart(7) + String(k.reduce((t, x) => t + x.bytes, 0)).padStart(9) + String(unneeded.length).padStart(9) + String(total(unneeded)).padStart(8)); }; console.log("definition".padEnd(30) + "lyr files bytes unneeded bytes"); measure("selective copy", DEFINITION); measure("full context, exclusion on", BROAD); measure("full context, exclusion off", BROAD, []); ``` ``` definition lyr files bytes unneeded bytes selective copy 4 10 68425 2 6860 full context, exclusion on 3 13 73861 5 12296 full context, exclusion off 3 19 158097 11 96532 ``` Three lines produce the same application, and their sizes are 68,425, 73,861, and 158,097 bytes. The gap between the first two lines is small — 5,436 bytes — because the exclusion rule has already cut the heavy directories. When the rule is emptied in the third line, the image grows to about 2.3 times its size, and the count of unneeded files inside it climbs from two to eleven, unneeded bytes from 6,860 to 96,532. About 61 percent of the image is made of files that will never run: logs, leftover build artifacts, docs, and local config. The two definitions' layer counts differ too: the selective definition produces four filesystem layers, the broad one three. Fewer layers is not a good outcome here — in the broad definition, the source and the config fall into the same instruction and cannot be split into separate layers, and on top of that, they are copied **before** the dependency install. The cost of these two details is invisible in this lesson; it is exactly what the next lesson measures. #### Summary - An image definition file is a sequence of instructions; instructions apply in order, and each lays a layer on top of the previous one. The six-line definition produced four filesystem layers; two lines only wrote metadata — layer count is not line count. - The build context is the set of files instructions can read. From a 15-file, 99,917-byte tree, the exclusion rule cut 6 files and 84,236 bytes; the context was left with 9 files and 15,681 bytes. - 3 files and 5,436 bytes were counted as present in the context but never entering the image. Being in the context is a permission separate from entering the image; the exclusion rule is not written only for size. - The `run` instruction produced 51,280 bytes in a single line — 75 percent of the image. The excluded dependency tree did not leave the image; its source moved from a machine's state to one line of the definition file. - The directory-copying instruction let 2 test files and 6,860 bytes into the image. The image records not what is needed but what was copied; the difference is visible only by counting layer content. - Built with three definitions, the same application produced images of 68,425, 73,861, and 158,097 bytes. With the exclusion rule empty, the definition that copies the whole context grew the image to about 2.3 times its size and carried the unneeded file count from two to eleven. #### Next Step This measurement counted a single build. When instructions are applied from scratch on every build, the 51,280 bytes the `run` line produces get regenerated every time too — yet if the dependency list has not changed, its output does not change either. Given that layers are named by their content digest, an unchanged instruction's layer can be taken from storage instead of being regenerated. The next lesson builds this storage and gives a single decision's number: how much does the order in which dependency install and source copy are written move the cache hit rate, the count of regenerated layers, and the total bytes copied, across the same change sequence? ### Layer Cache Source: https://academia.sh/en/courses/containers/layer-cache The effect of instruction order on the build is measured: the layer key is derived from the previous layer's key and the instruction's input digest, two instruction orders run over the same eight-build change sequence, and the cache hit rate, the count of regenerated layers, and bytes written are compared. The previous lesson measured that a layer is born from every instruction and counted a single `run` line producing the bulk of the image. That measurement covered a single build. If a service is built several times a day, the same line does the same job every time — yet if its input has not changed, neither has its output, and layers are already named by their content digest. This lesson builds the structure that removes that repetition. A **layer cache** stores a produced layer under its key; if the same key comes up on the next build, the layer is not rebuilt, it is taken from storage. What the key is derived from raises a decision: the **instructions' order**. That is also what gets measured — how much does whether dependency install or source copy is written first move the hit rate, the count of regenerated layers, and the bytes written, across the same change sequence? **IM13.** The cache is a model built with `node`; keys are kept in a map, and layers are really written under `cache//`. The tree is a simplified version of the previous lesson's tree for this measurement; its byte values are not compared with that lesson's numbers. **IM14.** The layer key is derived from the previous layer's key and the instruction's input digest. For `copy`, the input digest also covers the content of the files to be copied; for `run`, only the line itself. **IM15.** Duration is not measured. The quantities measured are run-independent: the count of regenerated layers and bytes written. #### The Layer Key The cache's operation rests on one rule: **a layer's key must cover everything beneath it and its own input.** If it does not, the layer above a changed lower layer hits by mistake and the image comes out inconsistent. The rule is the same as the previous lesson's chain id; the difference is that the instruction's input joins the chain too. ```js // measurement-network/cache.mjs — cached layer interpreter (model) import { mkdirSync, writeFileSync, readFileSync, readdirSync, appendFileSync, statSync, rmSync } from "node:fs"; import { createHash } from "node:crypto"; const ROOT = "project"; const digest = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12); export const text = (name, n) => Array.from({ length: n }, (_, i) => `// ${name} ${i}\n`).join(""); const TREE = { "dependency-list": "csv 2.4\nqueue 1.9\n", "source/entry.js": text("entry", 20), "source/collect.js": text("collect", 140), "source/queue.js": text("queue", 95), "source/test/collect.test.js": text("test", 210), }; export const OUTPUT = { "runtime-base": { "root/runtime.js": text("runtime", 420), "root/shell.js": text("shell", 60) }, "install-deps": { "dependency/csv.js": text("csv", 900), "dependency/queue.js": text("queue-dep", 1400) }, }; export const ORDER_A = `base runtime-base copy dependency-list /app/list run install-deps copy source /app/source`; export const ORDER_B = `base runtime-base copy source /app/source copy dependency-list /app/list run install-deps`; // Change sequence: file touched before each build; empty means no change. export const CHANGE = ["", "source/collect.js", "source/entry.js", "", "dependency-list", "source/collect.js", "source/test/collect.test.js", "source/queue.js"]; const writeFile = (path, content) => { mkdirSync(path.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(path, content); }; export const buildTree = () => { rmSync(ROOT, { recursive: true, force: true }); rmSync("cache", { recursive: true, force: true }); for (const [y, i] of Object.entries(TREE)) writeFile(`${ROOT}/${y}`, i); }; export const touch = (path, no) => appendFileSync(`${ROOT}/${path}`, `// fix ${no}\n`); const walk = (sub = "") => readdirSync(`${ROOT}/${sub}`, { withFileTypes: true }).flatMap((g) => g.isDirectory() ? walk(`${sub}/${g.name}`) : [`${sub}/${g.name}`.replace(/^\//, "")]).sort(); const paths = (source) => statSync(`${ROOT}/${source}`).isDirectory() ? walk(source) : [source]; // Layer key: previous layer's key + the instruction's input digest. // For "copy" the input digest also covers the content of the files to be copied; for "run" only the line itself. const inputDigest = (line) => { const [instruction, source] = line.trim().split(/\s+/); if (instruction !== "copy") return digest(line); return digest(line + paths(source).map((y) => y + readFileSync(`${ROOT}/${y}`)).join()); }; export const build = (definition, cache) => { let key = ""; const result = { hits: 0, builds: 0, bytes: 0, chain: [] }; for (const line of definition.trim().split("\n")) { key = digest(key + inputDigest(line)); result.chain.push(key); if (cache.has(key)) { result.hits++; continue; } const [instruction, source, target] = line.trim().split(/\s+/); const pairs = instruction === "copy" ? paths(source).map((y) => [`${target.slice(1)}/${y}`, readFileSync(`${ROOT}/${y}`)]) : Object.entries(OUTPUT[source]); let bytes = 0; for (const [y, content] of pairs) { writeFile(`cache/${key}/${y}`, content); bytes += content.length; } cache.set(key, bytes); result.builds++; result.bytes += bytes; } return result; }; ``` A direct consequence of chaining the key is this: when one instruction misses, every instruction above it misses too, even if its own input never changed. The cache works bottom-up, not top-down, and everything after the first miss point is rebuilt in full. #### Two Orders, the Same Change Sequence Two definition files carry the same four instructions, only their order differs. Order A copies the dependency list and runs the install, leaving the source for last; order B moves the source to the front. Both definitions run over the same eight-build sequence; each step in the sequence adds one line to a file, in exactly the same order in both runs. **IM16.** The change sequence represents the fictional reading collector service's one-week fix flow: a first build with a cold cache, five source changes, one dependency-list change, and one build where nothing changes. **IM17.** Each order starts with its own empty cache; the cache is not shared between runs. ```js // measurement-network/run.mjs — runs two instruction orders over the same change sequence import { buildTree, touch, build, CHANGE, ORDER_A, ORDER_B } from "./cache.mjs"; const runSequence = (label, definition) => { buildTree(); const cache = new Map(); const totals = { hits: 0, builds: 0, bytes: 0 }; console.log(`\n${label}`); console.log(" no change".padEnd(38) + "hits builds bytes"); CHANGE.forEach((d, i) => { if (d) touch(d, i); const s = build(definition, cache); for (const k of Object.keys(totals)) totals[k] += s[k]; console.log(` ${i + 1} ${d || (i ? "(no change)" : "(first build)")}`.padEnd(38) + String(s.hits).padStart(4) + String(s.builds).padStart(8) + String(s.bytes).padStart(9)); }); const attempts = CHANGE.length * definition.trim().split("\n").length; console.log(` total: ${totals.hits}/${attempts} hits (%${((totals.hits * 100) / attempts).toFixed(1)}), ` + `layers built ${totals.builds}, ${totals.bytes} bytes written, ${cache.size} entries in cache ` + `${[...cache.values()].reduce((a, b) => a + b, 0)} bytes`); return totals; }; const a = runSequence("A — dependency first, source last", ORDER_A); const b = runSequence("B — source first, dependency last", ORDER_B); console.log(`\nextra layers B builds over A: ${b.builds - a.builds}`); console.log(`extra bytes B writes over A : ${b.bytes - a.bytes} (%${(((b.bytes - a.bytes) * 100) / a.bytes).toFixed(1)} more work)`); ``` ``` A — dependency first, source last no change hits builds bytes 1 (first build) 0 4 46558 2 source/collect.js 3 1 5769 3 source/entry.js 3 1 5778 4 (no change) 4 0 0 5 dependency-list 1 3 39685 6 source/collect.js 3 1 5787 7 source/test/collect.test.js 3 1 5796 8 source/queue.js 3 1 5805 total: 20/32 hits (%62.5), layers built 12, 115178 bytes written, 12 entries in cache 115178 bytes B — source first, dependency last no change hits builds bytes 1 (first build) 0 4 46558 2 source/collect.js 1 3 39667 3 source/entry.js 1 3 39676 4 (no change) 4 0 0 5 dependency-list 2 2 33907 6 source/collect.js 1 3 39694 7 source/test/collect.test.js 1 3 39703 8 source/queue.js 1 3 39712 total: 11/32 hits (%34.4), layers built 21, 278917 bytes written, 21 entries in cache 278917 bytes extra layers B builds over A: 9 extra bytes B writes over A : 163739 (%142.2 more work) ``` #### What the Order Costs Same four instructions, same eight changes, two different outcomes. Order A hits on 20 of 32 layer attempts — 62.5 percent — and builds 12 layers. Order B, over the same sequence, gets 11 hits, 34.4 percent, and produces 21 layers. Bytes written are 115,178 against 278,917: order B writes 142.2 percent more bytes to do the same job. The entire difference collects in five lines — the builds with a source change. In order A, source copying sits on top; no layer beneath it is affected, and only a layer of around 5,800 bytes gets rebuilt. In order B, source copying sits at the bottom; the moment the source is touched, the list copy and dependency install above it invalidate too, and about 39,700 bytes get written. The roughly sevenfold difference does not come from the source files being small — it comes from **what sits above them**. The fifth build shows the reverse direction. When the dependency list changes, order A builds three layers (39,685 bytes), order B builds two layers (33,907 bytes) — on this build, B is cheaper. The decision cannot be made by looking at a single build; it is made by looking at the sequence's composition. Five of the eight builds change the source, one changes the dependency list, and in a real service tree the ratio is even sharper. The rule reads as follows: **instructions are ordered by how often they change, the least-frequently-changing at the bottom.** This is the build side's counterpart to the previous lesson's layer-order rule, and both rules come from the same measurement. The fourth build counts too: when nothing changes, both orders give 4 hits and 0 bytes. The cache's gain depends not only on the order but on whether a change exists at all — on an unchanged tree, both orders run for free. The hit rate itself must be read carefully too. The denominator is the number of layer attempts: eight builds over a four-instruction definition make 32 attempts. If the instruction count grows, the denominator grows, and the same behavior can show a higher rate; the ratio alone is not enough to compare two different definition files. The two numbers that carry the decision are **layers rebuilt** and **bytes written**, because both are a direct measure of the work done and are independent of the definition's line count. In this measurement all three point the same way, so the order decision is not in dispute; where they do not agree, the column to watch is the bytes column. The cache's own cost sits in the last column. Order A's cache accumulates 12 entries and 115,178 bytes; order B's accumulates 21 entries and 278,917 bytes. No entry is ever deleted: every miss writes a new layer to disk, and the old one stays in place. The cache is a store that buys build time at the price of bytes, and a bad order grows this store too, to about 2.4 times its size. Where the store sits is also counted. The layer cache accumulates on the machine the build runs on; the 12- and 21-entry stores are that machine's state. When the same definition file runs on a machine with an empty cache, both orders pay the first row's cost: 4 layers, 46,558 bytes. The order decision's gain shows up only from the second build onward. The difference measured under the name **environment parity** in the previous course takes a new shape here: a build's **result** is independent of the machine, its **cost** is not. This is the item the isolation budget pays in this lesson — the repeated work was not pulled out of the environment, it was moved into a store, and that store still sits in the environment. #### What the Cache Does Not Measure The key covers only the line itself for the `run` instruction. As long as the line does not change, the layer is not rebuilt — the thing to be installed having changed on the outside does not enter the key. **IM18.** The outside world changing is modeled by growing the content in the install table; the list file is left untouched. ```js // measurement-network/stale.mjs — the input the cache key does not measure import { buildTree, touch, build, text, OUTPUT, ORDER_A } from "./cache.mjs"; buildTree(); const cache = new Map(); const first = build(ORDER_A, cache); const installKey = first.chain[2]; console.log(`first build: ${first.builds} layers, ${first.bytes} bytes`); // The outside world changed: the content to install grew, the list file stayed the same. OUTPUT["install-deps"]["dependency/csv.js"] = text("csv", 950); const updated = Object.values(OUTPUT["install-deps"]).reduce((t, i) => t + i.length, 0); const s = build(ORDER_A, cache); console.log(`\nlist file unchanged, content to install changed:`); console.log(` hits ${s.hits}, layers rebuilt ${s.builds}, ${s.bytes} bytes written`); console.log(` install layer in cache ${cache.get(installKey)} bytes, updated content ${updated} bytes`); console.log(` difference ${updated - cache.get(installKey)} bytes, same key: ${s.chain[2] === installKey}`); touch("dependency-list", 9); const t = build(ORDER_A, cache); console.log(`\nonce a line is added to the list file:`); console.log(` hits ${t.hits}, layers rebuilt ${t.builds}, ${t.bytes} bytes written`); console.log(` install layer ${cache.get(t.chain[2])} bytes`); ``` ``` first build: 4 layers, 46558 bytes list file unchanged, content to install changed: hits 4, layers rebuilt 0, 0 bytes written install layer in cache 33880 bytes, updated content 34430 bytes difference 550 bytes, same key: true once a line is added to the list file: hits 1, layers rebuilt 3, 40217 bytes written install layer 34430 bytes ``` This is where isolation is pierced. Even though the content to install grew by 550 bytes, all four instructions hit, zero layers get built, and the image keeps carrying the old content. The key is the same, because the key measures the `run` line's **text**, not its **result**. The cache is working correctly; what it measures is incomplete. This is the cache-side shape of the reproducibility problem measured in the previous course. If a `run` line's output does the work of something time-dependent — an install with no version pin written down, say — the same line produces two different layers at two different times, but the cache names both with the same key. The fix is not in the cache but in the **instruction**: if `run`'s input is written to a file, that file enters the context through `copy`, and its content joins the key. This is exactly what happens in the third block when a single line is added to the list file — the key changes, the install layer is rebuilt, and the layer in the store updates to 34,430 bytes. **Copy granularity** is also an item of shortfall. In the seventh build, the file that changes is a test file; a file that will never run at runtime rebuilds a 5,796-byte layer, because the source directory is copied with a single instruction. The unneeded files the previous lesson measured extract a cost here a second time: first in image size, then in cache hits. #### Summary - The layer key is derived from the previous layer's key and the instruction's input digest. When one instruction misses, every instruction above it misses too; the cache rebuilds everything after the first miss point entirely. - Same four instructions, same eight builds: with dependency install written first, 20 of 32 attempts hit (62.5 percent), 12 layers were built, and 115,178 bytes were written. Moving source copy first dropped hits to 11 (34.4 percent), layers rose to 21, and bytes written rose to 278,917 — 142.2 percent more work. - The difference collects in the builds with a source change: around 5,800 bytes in order A, around 39,700 bytes in order B. When the dependency list changes, the direction reverses; the decision is made not by a single build but by the change sequence's composition. - The cache's own cost is a store, and it only grows: order A accumulated 12 entries and 115,178 bytes, order B accumulated 21 entries and 278,917 bytes. - The key measures only `run`'s line text. Even though the content to install grew by 550 bytes, all four instructions hit and zero layers were built; the image kept carrying the old content. Once the input is written to a file and pulled into the context, the key changes and the layer updates. #### Next Step The cache removed repeated work but never touched the image's content. The image that emerges at the end of eight builds is identical to the stack the first build produced: everything the dependency install left behind, every source file copied, and every tool used during install sits in the final layer. Yet part of this was only needed **during the build**; the only thing running at runtime is the application itself. The next lesson measures this distinction: when the same application is built single-stage and multi-stage, how far apart are the final image's size, layer count, and count of unneeded files left inside it, and what is multi-stage building's cost on the cache side? ### Multi-Stage Build Source: https://academia.sh/en/courses/containers/multi-stage-build Separating build tools from the output is measured: the same application is built single-stage and multi-stage, the final image's size, layer count, and count of unneeded files left inside are compared, and multi-stage building's cost is counted too — total bytes produced, unshipped intermediate layers, and cache behavior. The previous lesson built the cache that removes repeated work and counted the cost of instruction order. Throughout that measurement, the image's content was never touched: the stack that emerged at the end of eight builds was identical to what the first build produced. Every file the dependency install left behind, every source file copied, and every tool used during install sat in the final layer. Yet part of these is only needed during the build. A compiler, a linker, a test runner, and dev dependencies are used to produce the output; once the output is produced, none of them run again. **Multi-stage building** sets up this separation at the instruction level: the build happens in its own stage, and only the produced files travel to the final image. The question to measure is clear — how far apart are the final image's size, layer count, and count of unneeded files left inside it, and what is the cost of this separation? **IM19.** The interpreter below is an extended form of the previous lessons' model: a `stage` instruction is added, and when a `copy` source is written as `:`, it reads from an earlier stage's view. **IM20.** The build base adds three build tools on top of the runtime base; the difference between the two bases is written explicitly in the model. **IM21.** The bundling model merges non-test source files and runtime dependencies into a single file; dev dependencies do not enter the bundle. **IM22.** The files needed at runtime are defined by hand: the runtime base's files and the bundled output. The fictional measurement network's reading collector service is measured. #### Build Time Versus Runtime In the single-stage definition, all instructions write to a single stack, and that stack both does the build and gets run. In the multi-stage definition, the `stage` instruction splits the stack: each stage starts from its own base, builds its own chain, and the final stage takes only the paths it wants from earlier stages. The final image is the final stage's layers; earlier stages' layers never enter the image at all. The separation has a counterpart on the chain too. Every `stage` instruction resets the key chain: the second stage's base layer gets a key independent of whatever happened in the first stage, and it comes from the cache even when the source changes. The only link that crosses between the two stages is the content of the files carried across. The previous lesson's rule still holds here — a layer's key covers everything beneath it — but "everything beneath it" is now cut off at the stage boundary. This is the single structural distinction that makes multi-stage building's cache behavior both better and worse: the second stage's base comes for free, but the copy layer gets rebuilt every time the carried package changes. ```js // measurement-network/stage.mjs — staged-build interpreter and layer cache (model) import { mkdirSync, writeFileSync, readFileSync, readdirSync, appendFileSync, rmSync } from "node:fs"; import { createHash } from "node:crypto"; const ROOT = "project"; const digest = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12); const text = (name, n) => Array.from({ length: n }, (_, i) => `// ${name} ${i}\n`).join(""); const TREE = { "source/entry.js": text("entry", 22), "source/collect.js": text("collect", 150), "source/queue.js": text("queue", 96), "source/test/collect.test.js": text("test", 205) }; const RUNTIME_BASE = { "root/runtime.js": text("runtime", 400), "root/shell.js": text("shell", 55) }; // Build base adds build tools on top of the runtime base. const BUILD_BASE = { ...RUNTIME_BASE, "tool/compiler.js": text("compiler", 3200), "tool/linker.js": text("linker", 1450), "tool/headers.js": text("headers", 900) }; // Bundling: non-test source and runtime dependencies merge into a single file. const bundle = (view) => [...view].filter(([y]) => (y.startsWith("work/source/") && !y.includes("/test/")) || y.startsWith("dependency/runtime/")) .sort().map(([, i]) => i).join(""); const OUTPUT = { "build-base": () => BUILD_BASE, "runtime-base": () => RUNTIME_BASE, "install-deps": () => ({ "dependency/dev/test-runner.js": text("test-runner", 2100), "dependency/dev/build-plugin.js": text("build-plugin", 1600), "dependency/runtime/csv.js": text("csv", 780), "dependency/runtime/queue.js": text("queue-dep", 640) }), "package-build": (view) => ({ "work/dist/package.js": bundle(view) }), }; export const SINGLE = `stage single base build-base copy source /work/source run install-deps run package-build entry /work/dist/package.js`; export const MULTI = `stage builder base build-base copy source /work/source run install-deps run package-build stage output base runtime-base copy builder:/work/dist /app entry /app/package.js`; // Same multi-stage definition, in the form that moves the whole work directory across. export const MULTI_WIDE = MULTI.replace("builder:/work/dist /app", "builder:/work /app") .replace("entry /app/package.js", "entry /app/dist/package.js"); const writeFile = (path, content) => { mkdirSync(path.split("/").slice(0, -1).join("/"), { recursive: true }); writeFileSync(path, content); }; export const buildTree = () => { rmSync(ROOT, { recursive: true, force: true }); rmSync("image", { recursive: true, force: true }); for (const [y, i] of Object.entries(TREE)) writeFile(`${ROOT}/${y}`, i); }; export const touch = (path, no) => appendFileSync(`${ROOT}/${path}`, `// fix ${no}\n`); const walk = (sub = "") => readdirSync(`${ROOT}/${sub}`, { withFileTypes: true }).flatMap((g) => g.isDirectory() ? walk(`${sub}/${g.name}`) : [`${sub}/${g.name}`.replace(/^\//, "")]).sort(); // Pairs to copy: from the tree, or from another stage's view. const copyPairs = (source, target, stages) => { if (!source.includes(":")) return walk(source).map((y) => [`${target.slice(1)}/${y.slice(source.length + 1)}`, readFileSync(`${ROOT}/${y}`).toString()]); const [name, path] = source.split(":"); const prefix = `${path.slice(1)}/`; return [...stages.get(name).view].filter(([y]) => y.startsWith(prefix)) .map(([y, i]) => [`${target.slice(1)}/${y.slice(prefix.length)}`, i]); }; export const build = (definition, cache = new Map()) => { const stages = new Map(); let current = null, key = ""; const counter = { builds: 0, hits: 0, bytes: 0 }; for (const raw of definition.trim().split("\n")) { const [instruction, source, target] = raw.trim().split(/\s+/); if (instruction === "stage") { current = { name: source, view: new Map(), layer: 0 }; stages.set(source, current); key = ""; continue; } if (instruction === "entry") continue; // metadata: produces no layer const input = instruction === "copy" ? copyPairs(source, target, stages).join() : ""; key = digest(key + raw + input); let pairs; if (cache.has(key)) { pairs = cache.get(key); counter.hits++; } else { pairs = instruction === "copy" ? copyPairs(source, target, stages) : Object.entries(OUTPUT[source](current.view)); for (const [y, i] of pairs) writeFile(`image/${current.name}/layer-${current.layer}/${y}`, i); cache.set(key, pairs); counter.builds++; counter.bytes += pairs.reduce((t, [, i]) => t + i.length, 0); } current.layer++; for (const [y, i] of pairs) current.view.set(y, i); } return { stages, counter, last: current }; }; // Needed at runtime: runtime base files and the packaged output. export const needed = (path) => path.startsWith("root/") || path.endsWith("package.js"); ``` #### Two Builds, One Application Both definitions produce the same bundle from the same source. What is measured is what is left sitting alongside the bundle. ```js // measurement-network/measure-image.mjs — compares single-stage and multi-stage builds' final images import { buildTree, build, needed, SINGLE, MULTI, MULTI_WIDE } from "./stage.mjs"; const inspect = (label, definition) => { buildTree(); const { stages, counter, last } = build(definition); const files = [...last.view]; const unneeded = files.filter(([y]) => !needed(y)); const bytes = files.reduce((t, [, i]) => t + i.length, 0); const mid = [...stages.values()].filter((a) => a !== last); console.log(label.padEnd(20) + String(last.layer).padStart(4) + String(files.length).padStart(7) + String(bytes).padStart(9) + String(unneeded.length).padStart(9) + String(unneeded.reduce((t, [, i]) => t + i.length, 0)).padStart(9) + String(counter.bytes).padStart(10) + String(mid.reduce((t, a) => t + a.layer, 0)).padStart(6)); return { bytes, files, unneeded }; }; console.log("build".padEnd(20) + " lyr files bytes unneeded bytes produced mid-lyr"); const t = inspect("single-stage", SINGLE); const c = inspect("multi-stage", MULTI); inspect("multi-stage, wide", MULTI_WIDE); console.log(`\nfinal image: ${t.bytes} -> ${c.bytes} bytes, ${(t.bytes / c.bytes).toFixed(1)}x smaller`); console.log(`files in final image: ${t.files.length} -> ${c.files.length}`); console.log(`unneeded at runtime: ${t.unneeded.length} -> ${c.unneeded.length}`); console.log("unneeded files in single-stage: " + t.unneeded.map(([y]) => y).join(" ")); ``` ``` build lyr files bytes unneeded bytes produced mid-lyr single-stage 4 14 215142 11 185826 215142 0 multi-stage 2 3 29316 0 0 244458 4 multi-stage, wide 2 7 35202 4 5886 250344 4 final image: 215142 -> 29316 bytes, 7.3x smaller files in final image: 14 -> 3 unneeded at runtime: 11 -> 0 unneeded files in single-stage: tool/compiler.js tool/linker.js tool/headers.js work/source/collect.js work/source/entry.js work/source/queue.js work/source/test/collect.test.js dependency/dev/test-runner.js dependency/dev/build-plugin.js dependency/runtime/csv.js dependency/runtime/queue.js ``` The final image drops from 215,142 bytes to 29,316 bytes, file count from 14 to 3, layer count from 4 to 2. The single-stage image's 11 files and 185,826 bytes never run at runtime at all — 86.4 percent of the image. Among them are three build tools, four source files (one a test file), two dev dependencies, and two runtime dependencies that have already entered the bundle. The runtime dependencies being carried twice is also counted: they sit both inside the bundle and alongside it. The drop in layer count needs a separate reading too. The multi-stage image has two layers because the final stage runs only two instructions; this also changes the sharing gain the previous lesson measured. Other services born from the same base can share the first of those two layers; the second is specific to the service. In the single-stage image, the shareable base also carries the build tools, so it is both bigger and invalidates whenever the build tool version changes — the sharing gain depends on how often something changes, and the build base changes more often than the runtime base. The difference pulled out of the environment is this lesson's biggest item. The compiler, the linker, and the header files leave the runtime environment entirely; the running container's filesystem has no build tool in it. At the instruction level, this costs two lines — a second `stage` and a single `copy` between stages. The third row separates where the gain comes from. When the same multi-stage definition carries across the whole `/work` directory instead of `/work/dist`, the final image grows to 35,202 bytes and 7 files; four of them are source files not needed at runtime and take up 5,886 bytes. The whole gain does not come from opening a second stage; it comes from **the narrowness of the path carried across**. A second stage only helps to the extent that a decision is made about what it will **not** take from the first stage. This is the stage-boundary form of the directory copying problem measured in the previous lesson: `copy` takes a path, it does not count what sits beneath that path. #### Multi-Stage's Cost The separation is not free. The same two definitions are run twice with one cache: the first build with a cold cache, the second after a single line is added to one source file. **IM23.** The change adds one line to `source/collect.js`. **IM24.** The stage view is kept in memory, layers are written to disk; the bytes measured are real file bytes. ```js // measurement-network/cost.mjs — multi-stage build's cost: intermediate layers and cache behavior import { buildTree, touch, build, SINGLE, MULTI } from "./stage.mjs"; const sequence = (label, definition) => { buildTree(); const cache = new Map(); const first = build(definition, cache).counter; touch("source/collect.js", 1); const { stages, counter, last } = build(definition, cache); const mid = [...stages.values()].filter((a) => a !== last) .reduce((t, a) => t + [...a.view.values()].reduce((u, i) => u + i.length, 0), 0); const cacheBytes = [...cache.values()].reduce((t, c) => t + c.reduce((u, [, i]) => u + i.length, 0), 0); console.log(label.padEnd(14) + String(first.builds).padStart(4) + String(first.bytes).padStart(9) + String(counter.hits).padStart(8) + String(counter.builds).padStart(8) + String(counter.bytes).padStart(9) + String(cache.size).padStart(8) + String(cacheBytes).padStart(9) + String(mid).padStart(10)); }; console.log("build".padEnd(14) + " 1st 1st-byt hits builds bytes cache bytes unshipped"); sequence("single-stage", SINGLE); sequence("multi-stage", MULTI); ``` ``` build 1st 1st-byt hits builds bytes cache bytes unshipped single-stage 4 215142 1 3 121300 7 336442 0 multi-stage 6 244458 2 4 144085 10 388543 215160 ``` The cost has four items. First is the first build: the multi-stage layout produces 6 layers and 244,458 bytes, the single-stage one 4 layers and 215,142 bytes — a difference of 29,316 bytes, exactly the size of the second stage's own base plus the carried bundle. Second is the source change: the single-stage rebuilds 3 layers and 121,300 bytes, while the multi-stage one produces 4 layers and 144,085 bytes, because when the bundle changes, the cross-stage copy invalidates too. Third is the cache: 7 entries and 336,442 bytes against 10 entries and 388,543 bytes. The fourth item is the least visible. In the multi-stage layout, 215,160 bytes are produced on every build and enter no image at all. The intermediate stage's layers are not shipped, but they are produced; they sit on the disk of the machine doing the build, take up room in the cache, and get reused on the next build. **The image got smaller, the store did not.** This is how the isolation budget closes its account in this lesson: 185,826 bytes were pulled out of the runtime environment, and in exchange, a persistent 215,160-byte stack appeared in the build environment, along with about 18.8 percent more write work per build. How do these four items turn into a decision? Two numbers are set side by side. The extra cost is paid **per build** and is 29,316 bytes. The gain is paid **per copy of the image** and is 185,826 bytes: that many bytes are missing every time the image is written to the store, pulled to a machine, and sits on disk. The ratio is 6.3 — even a single image produced by a single build more than covers the extra cost. Considering the fictional measurement network's four services, the difference multiplies, because the extra cost is paid once per build, while the gain reappears on every store and every pull. The single-stage layout is defensible where the image is never stored and is produced and run on a single machine; outside that, the measurement points one way. What the stage boundary is **not** must also be written down. The `copy builder:/work/dist` line is a copying boundary, not a verification boundary. The final stage takes the bundle the previous stage produced exactly as it is; it does not ask what is inside it. If the bundler had not left test files out, they would have entered the final image too, and the measurement would not have caught it — because the `needed` definition is written by hand in this measurement. The image itself does not know what is needed; the person who wrote the carried path guarantees it was chosen correctly. This is the here-form of the shortfall measured in the previous lesson: an instruction records not what is needed but what was copied. #### Summary - Multi-stage building separates files needed at build time from files needed at runtime: the build happens in its own stage, and only the produced files travel to the final image. - Built two ways, the same application's final image dropped from 215,142 bytes to 29,316 bytes (7.3x), file count from 14 to 3, layer count from 4 to 2. - The single-stage image's 11 files and 185,826 bytes never ran at runtime at all — 86.4 percent of the image. Three build tools, four source files, two dev dependencies, and two runtime dependencies that had already entered the bundle. - The cost has four items: 29,316 extra bytes produced on the first build, 4 layers instead of 3 and 144,085 instead of 121,300 bytes on a source change, 10 cache entries instead of 7, and 215,160 bytes produced on every build that enter no image. The image got smaller; the store on the building machine did not. - The gain comes not from opening a second stage but from the narrowness of the path carried across: when the whole `/work` was carried instead of `/work/dist`, the final image grew to 35,202 bytes and 7 files, and 4 unneeded files and 5,886 bytes came back. - The stage boundary is a copying boundary, not a verification boundary. The final stage takes what the previous stage produced exactly as it is; the definition of what is needed at runtime lives outside the instruction, in the hands of whoever wrote it. #### Next Step The final image has three files left, and two of them come from the base: the runtime base is 6,540 bytes, the bundle is 22,776 bytes. After everything the build produces has been measured and shrunk, what remains is the image's **unproduced** portion — the base placed beneath it. Throughout this lesson the base was a constant; which files the runtime base carries, which shell and which libraries it contains, was never asked. Yet the base makes up about a fifth of the final image, and its content is not something the writer produced — it is something taken from outside. The next lesson measures this choice: how many bytes and how many files separate a small base from a broad one, what does the small base leave out, and under what circumstances is that shortfall paid for? ### Base Image Selection Source: https://academia.sh/en/courses/containers/base-image-selection Three base image candidates are measured by type designation: bytes, carried binaries and library count, and the presence of a shell and a package manager are counted; how many layers really change when the base updates is calculated, and the small base's hidden cost is measured in extra debugging steps and broken dependency bytes. The previous lesson separated build tools from the output: the compiler, development headers, and intermediate files stayed outside the final output, and the output got smaller. But what got smaller were the upper layers. The final stage is still built on top of a **base**, and that base is the one part of the image that is not written inside it — it comes before the first instruction, someone else compiled its files, and someone else sets its update schedule. This lesson measures that base. The question is not "which one is good"; it is what the same quantities — bytes, carried binaries, update load — come out to for three candidate types, and which cost the smaller one pushes out of sight. **IM25.** Candidates are referred to by type, not by product: **full-featured base**, **slimmed base**, **single-binary base**. **IM26.** Inventory numbers come from our own generator; the seed `20260518` is visible, and the numbers are not a real measurement but an order of magnitude. **IM27.** The application layer on top is the same across all three candidates: the regional measurement network's — the fictional software that collects readings from water meters — reading collector service, 24 MB. #### What Is Inside the Base The base image does most of the work of taking what is installed in the environment into the output: libraries, the certificate store, timezone data, a shell. The difference among candidate types is where this set gets cut. The full-featured base carries a general-purpose operating environment's package set; a shell, a package manager, and process and network tools come with it. The slimmed base reduces the same set to a single multi-purpose binary and the required libraries, but drops the shell and the package manager. The single-binary base carries only the libraries the application links against, certificates, and timezone data. ```js // measurement-network/base.mjs — inventory of three base image candidates (model; type, not product) export const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; // [count, low bytes, high bytes]. Numbers come from the generator; they are an order of magnitude. export const CANDIDATES = [ { name: "full-featured", binary: [480, 8e3, 220e3], library: [310, 12e3, 240e3], data: [2400, 1e3, 14e3], shell: true, pkg: true, component: 148 }, { name: "slimmed", binary: [140, 3e3, 60e3], library: [95, 8e3, 120e3], data: [120, 1e3, 9e3], shell: true, pkg: true, component: 31 }, { name: "single-binary", binary: [0, 0, 0], library: [6, 30e3, 900e3], data: [40, 1e3, 30e3], shell: false, pkg: false, component: 6 }, ]; export const APPLICATION = 24_000_000; // reading collector service; the same across all three candidates export const inventory = (candidate, seed = 20260518) => { const r = generator(seed); const group = ([n, low, high]) => { let bytes = 0; for (let i = 0; i < n; i++) bytes += low + Math.round(r() * (high - low)); return { n, bytes }; }; const i = group(candidate.binary), k = group(candidate.library), v = group(candidate.data); return { ...candidate, i, k, v, files: i.n + k.n + v.n, bytes: i.bytes + k.bytes + v.bytes }; }; export const mb = (b) => (b / 1e6).toFixed(1) + " MB"; // Below runs only when this file is executed directly; other files import it as a module. if (import.meta.url === `file://${process.argv[1]}`) { const rows = CANDIDATES.map((a) => inventory(a)); console.log("candidate".padEnd(15) + "files".padEnd(8) + "binary".padEnd(9) + "library".padEnd(11) + "shell".padEnd(7) + "pkg mgr".padEnd(12) + "base".padEnd(10) + "image"); for (const s of rows) { console.log(s.name.padEnd(15) + String(s.files).padEnd(8) + String(s.i.n).padEnd(9) + String(s.k.n).padEnd(11) + (s.shell ? "yes" : "no").padEnd(7) + (s.pkg ? "yes" : "no").padEnd(12) + mb(s.bytes).padEnd(10) + mb(s.bytes + APPLICATION)); } const [full, slim, single] = rows; console.log(`\nbase's share within the image: ${(100 * full.bytes / (full.bytes + APPLICATION)).toFixed(0)}% / ` + `${(100 * slim.bytes / (slim.bytes + APPLICATION)).toFixed(0)}% / ` + `${(100 * single.bytes / (single.bytes + APPLICATION)).toFixed(0)}%`); console.log(`executable files inside the image: ${full.i.n} / ${slim.i.n} / ${single.i.n}`); console.log(`components to track for updates: ${full.component} / ${slim.component} / ${single.component}`); } ``` ``` candidate files binary library shell pkg mgr base image full-featured 3190 480 310 yes yes 111.1 MB 135.1 MB slimmed 355 140 95 yes yes 10.5 MB 34.5 MB single-binary 46 0 6 no no 4.1 MB 28.1 MB base's share within the image: 82% / 30% / 14% executable files inside the image: 480 / 140 / 0 components to track for updates: 148 / 31 / 6 ``` The same application, built on three bases, produces output ranging from 135.1 MB to 28.1 MB. In the full-featured base, 82 percent of the image is the unwritten portion: the base alone carries 3,190 files, and none of them were written for this job. The second row is not bytes. The image containing 480 executable files is the count of tools standing ready if a code execution flaw is found — a file downloader, an archiver, a compiler, a network client. The attack surface is not equal to this number, but this number is its upper bound, and it is zero in the single-binary base. There is one thing none of the three candidates change: the **kernel**. The base image carries user-space files, not the kernel interface; all three candidates depend on the system call set the kernel underneath provides, and that set is not inside the image. The single-binary base's 46 files reach the same call surface as the full-featured base's 3,190 files. The size table therefore does not measure the whole of isolation — it only measures how much of user space was taken into the output. **The difference pulled out of the environment is counted here:** installed packages and library versions are no longer in the environment — they are in the output, as 148, 31, or 6 components. The difference has not vanished; it has only changed location and become bound to another publisher's schedule. The next section counts that schedule's cost. #### How Many Layers Change When an Update Arrives Component count means not only bytes but **event frequency**. Every component produces its own updates, and the base's publisher ships them as a new base version. When the base layer changes, the image needs rebuilding; the real question is whether the layers above it change too. **IM28.** An average of 0.9 security updates per component per year falls due, and the number of running machines is 12; both are fictional. **IM29.** The image is four layers: base, dependencies, application, config. Rebuilding is measured under two disciplines — a deterministic build puts no variable stamp into layers, a stamped build writes the build time into every rebuilt layer. ```js // measurement-network/update.mjs — how many layers really change when the base updates (model) import { createHash } from "node:crypto"; import { CANDIDATES, inventory, mb } from "./base.mjs"; const digest = (s) => createHash("sha256").update(s).digest("hex").slice(0, 10); const gb = (b) => (b / 1e9).toFixed(1) + " GB"; const NODES = 12; // number of running machines in the measurement network (fictional) const ANNUAL_RATE = 0.9; // annual security updates per component (fictional) // The image is four layers: base + the application's three layers. The top three layers are the same across all three candidates. const UPPER = [["dependencies", 16.0e6], ["application", 7.2e6], ["config", 0.8e6]]; // stamp=null: deterministic rebuild. stamp=: a timestamp enters every layer. const layers = (baseVersion, baseBytes, stamp) => [ { name: "base", bytes: baseBytes, digest: digest(`base@${baseVersion}`) }, ...UPPER.map(([name, bytes]) => ({ name, bytes, digest: digest(`${name}@v1${stamp ?? ""}`) })), ]; console.log("candidate".padEnd(15) + "annual updates".padEnd(19) + "changed layers".padEnd(19) + "changed bytes".padEnd(16) + "annual pull"); for (const candidate of CANDIDATES) { const e = inventory(candidate); const old = layers("2026.05", e.bytes, null); for (const [label, stamp] of [["deterministic", null], ["stamped", "-b2"]]) { const updated = layers("2026.06", e.bytes, stamp); const changed = updated.filter((k, i) => k.digest !== old[i].digest); const bytes = changed.reduce((t, k) => t + k.bytes, 0); const annual = Math.round(candidate.component * ANNUAL_RATE); console.log((label === "deterministic" ? candidate.name : "").padEnd(15) + (label === "deterministic" ? `${annual} times` : "").padEnd(19) + `${changed.length}/4 ${label}`.padEnd(19) + mb(bytes).padEnd(16) + gb(annual * bytes * NODES)); } } console.log(`\nannual pull = annual updates x changed bytes x ${NODES} nodes`); ``` ``` candidate annual updates changed layers changed bytes annual pull full-featured 133 times 1/4 deterministic 111.1 MB 177.4 GB 4/4 stamped 135.1 MB 215.7 GB slimmed 28 times 1/4 deterministic 10.5 MB 3.5 GB 4/4 stamped 34.5 MB 11.6 GB single-binary 5 times 1/4 deterministic 4.1 MB 0.2 GB 4/4 stamped 28.1 MB 1.7 GB annual pull = annual updates x changed bytes x 12 nodes ``` The table's two columns must be read together. Under deterministic building, a base update changes only one of the four layers: because the upper three layers' content stays the same, their digests stay the same too, and they get skipped in the pull. Under stamped building, the same update changes four of four, because even when the content is the same, the build time written into the layer shifts the digest. The difference is 38.3 GB a year in the full-featured base, 1.5 GB in the single-binary base. Determinism is not a property of the base but a discipline of the build step — no matter how small the base is, a build that stamps redistributes the whole image on every update. The annual pull column shows the real gap: 177.4 GB in the full-featured base, 0.2 GB in the single-binary base. The source of the gap in scale is frequency as much as bytes: the full-featured base updates 133 times a year, the single-binary base 5 times, and the latter has only 6 components to track. The small base's most concrete gain shows up here — not the image's size, but the size of the **update surface**. This also writes down one of the places isolation is pierced: the base image has been taken into the output, but its production stays outside. 133 times a year, a change whose decision was not made here enters the output's bottom layer. It can be stopped by pinning the version — but then security updates do not enter either. #### The Small Base's Hidden Cost The small base's cost does not show up in the size table; it shows up in two places. The first is during an incident: if the shell, a process lister, or a network tool is not inside the image, a question that tool would answer cannot be answered on-site. The second is during the build: a library the base does not carry has to be added to the image by hand. **IM30.** The eight event types, the tool set each needs to be diagnosed on-site, and the service's nine local dependencies are fictional. When a tool is missing, the extra-step count is tied to a fixed rule: 2 if a package manager is present (install, then clean the image afterward), 4 if not (prepare a helper container, share its namespace, copy the output out, rerun). Each dependency only comes ready in the base up to a certain leanness. ```js // measurement-network/cost.mjs — the small base's hidden cost: extra debugging steps and compatibility (model) import { CANDIDATES, inventory, APPLICATION, mb } from "./base.mjs"; // The tool set each candidate carries (fictional, by type name). const TOOLS = { "full-featured": ["shell", "pkg", "proc", "net", "file", "search"], "slimmed": ["shell", "pkg", "proc", "file", "search"], "single-binary": [], }; // The measurement network's eight event types and the tool needed to diagnose each on-site (fictional). const EVENTS = [ ["service not responding", ["proc"]], ["queue backed up", ["proc", "search"]], ["certificate not verifying", ["file"]], ["name not resolving", ["net"]], ["disk full", ["file"]], ["config read wrong", ["shell", "search"]], ["slow response", ["net", "proc"]], ["permission denied", ["file", "shell"]], ]; // The service's nine local dependencies: [name, bytes, rank of the leanest candidate that provides it]. // Candidates are ordered rich to lean; if rank > provider, the dependency is absent from the base. const DEPENDENCY = [ ["crypto library", 3.1e6, 2], ["certificate store", 0.3e6, 2], ["math library", 1.2e6, 2], ["compression library", 0.6e6, 2], ["localization data", 2.4e6, 1], ["timezone database", 0.9e6, 1], ["name resolver plugin", 0.9e6, 1], ["version compatibility layer", 0.6e6, 0], ["shell script runner", 0.3e6, 0], ]; console.log("candidate".padEnd(15) + "on-site events".padEnd(17) + "steps".padEnd(9) + "missing deps".padEnd(18) + "added".padEnd(9) + "real image"); CANDIDATES.forEach((candidate, rank) => { const tools = TOOLS[candidate.name]; let onSite = 0, extraSteps = 0; for (const [, need] of EVENTS) { if (need.every((g) => tools.includes(g))) onSite += 1; else extraSteps += tools.includes("pkg") ? 2 : 4; } const missing = DEPENDENCY.filter(([, , provider]) => rank > provider); const added = missing.reduce((t, [, b]) => t + b, 0); const nominal = inventory(candidate).bytes + APPLICATION; console.log(candidate.name.padEnd(15) + `${onSite}/8 events`.padEnd(17) + String(extraSteps).padEnd(9) + `${missing.length}/9`.padEnd(18) + mb(added).padEnd(9) + mb(nominal + added)); }); ``` ``` candidate on-site events steps missing deps added real image full-featured 8/8 events 0 0/9 0.0 MB 135.1 MB slimmed 6/8 events 4 2/9 0.9 MB 35.4 MB single-binary 0/8 events 32 5/9 5.1 MB 33.2 MB ``` In the single-binary base, none of the eight events can be diagnosed on-site, and the eight events produce a total of 32 extra steps. These steps are not only time. Attaching a helper container to the same namespace means bringing back every tool the image removed, right then — that is, isolation is knowingly pierced for the duration of debugging. The 480 executable files the small base closed off get reopened another way at the moment of the incident — the difference is that it is now temporary, not permanent, and when that period began can go on record. The last two columns correct the size table. The single-binary base is nominally 6.4 MB smaller than the slimmed base; but five of the nine dependencies are absent from the base and get added to the image by hand, and the gap narrows to 2.2 MB. One of the missing dependencies is the shell script runner: on a base with no shell, a startup script does not run, and the service has to be directly executable. This is a small cost in bytes but a real one in how things must be written. The slimmed base holds the middle value on both sides: 6/8 events on-site, 4 extra steps, 2 missing dependencies. Looked at together, the three measured quantities turn the choice from "the smallest one" into the question of **who the extra steps fall on**. #### Summary - The same application produces 135.1 MB, 34.5 MB, and 28.1 MB on the three bases; in the full-featured base, 82 percent of the image is the unwritten portion, and the base alone carries 3,190 files. - The count of executable files inside the image is 480 / 140 / 0. This number is not equal to the attack surface, but it is its upper bound: it is the number of tools standing ready if a code execution flaw is found. - A base update changes one of four layers under deterministic building, four of four under stamped building. Annual pull is 177.4 GB in the full-featured base, 0.2 GB in the single-binary base; the source of the difference is frequency as much as bytes (133 updates against 5). - The small base's hidden cost is measured in numbers: in the single-binary base, 0 of 8 events can be diagnosed on-site, 32 extra steps are needed, and these steps knowingly pierce isolation for the duration of debugging. - The nominal 6.4 MB size gap narrows to 2.2 MB once the 5 missing dependencies are added to the image; everything the base does not carry re-enters the image anyway — just from a different place, and by hand. #### Next Step This lesson's measurement treated the three candidates as if their content were fixed. But no base is chosen by content: it is pulled by a **name**, and which content that name brings depends on time. The same holds for the produced image itself — each of the 133 updates produces new content, but what sits in deployment records is the name. M22/K01's build artifact repositories measured the distinction between a **portable tag** and an **immutable version**; that count is not repeated here. The question here sits one layer lower: an image is not a single object, it is a list of layers. The next lesson counts this — how many names point to the same content digest, how many of the four layers really change when a tag moves, and can a deployment that reaches production be traced back from its digest to its source? ### Tagging Scheme Source: https://academia.sh/en/courses/containers/tagging-scheme A tag is measured in a layered image registry: how many distinct content digests twenty builds produce and how many tags point to the same digest are counted, how many of the four layers really change when a tag moves is calculated, and the rate at which a deployment can be traced back to its source from its digest is measured under three separate record disciplines. The previous lesson treated the three base candidates as if their content were fixed. But no base is chosen by content: it is pulled by a **name**. The same holds for the produced image — a build produces new content, but what sits in a deployment record is the name. M22/K01's build artifact repositories measured the distinction between a **portable tag** and an **immutable version**; that count is not repeated here. The question here sits one layer lower: an image is not a single object, it is a list of layers. Does a tag point to this list's entirety, how many names can point to the same list at once, and when a name moves, how many of the list's rows really change? **IM31.** The registry model keeps two ledgers: image digest → manifest, and tag → image digest. Access control and the network layer are outside the model. **IM32.** The build history comes from our own generator; the seed `20260604` is visible, 20 builds are made, and one build in four is a rebuild from the same source version. **IM33.** The image is four layers — dependencies, application, and config on top of the previous lesson's slimmed base — and totals 34.5 MB. #### What a Tag Points To A tag does not point directly to bytes. The chain has three levels: a tag names an **image digest**, an image digest addresses a **manifest**, and a manifest lists layer digests in order. These three levels are what makes the same name bring back the same set of layers. **IM34.** The tagging scheme is this: every build gets an immutable version tag (`v-1`, `v-2`, …); `latest` moves on every build, `stable` moves only to builds where the nightly batch job runs clean, `branch-ready` moves on every fourth build. **IM35.** Three-quarters of builds write the source version field into the config object; builds done by hand outside the pipeline do not write it. ```js // measurement-network/image.mjs — build history and tags in a layered image registry (model) import { createHash } from "node:crypto"; export const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; export const digest = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12); // [name, bytes, probability of changing independent of source]. The application layer depends on the source. export const LAYER = [["base", 10.5e6, 0.08], ["dependencies", 16.0e6, 0.25], ["application", 7.2e6, null], ["config", 0.8e6, 0.30]]; export const history = (n = 20, seed = 20260604) => { const r = generator(seed); const version = [1, 1, 1, 1]; const list = []; let source = 1, day = 0; for (let no = 1; no <= n; no++) { day += 1 + Math.round(r() * 8); const newSource = r() > 0.25; // one in four is a rebuild from the same source if (newSource) { source += 1; version[2] += 1; } LAYER.forEach(([, , p], k) => { if (p !== null && r() < p) version[k] += 1; }); const layers = LAYER.map(([name, bytes], k) => ({ name, bytes, digest: digest(`${name}@${version[k]}`) })); list.push({ no, day, source: `s-${String(source).padStart(3, "0")}`, layers, image: digest("manifest:" + layers.map((k) => k.digest).join("|")), sourceWritten: r() > 0.25, // builds run by hand outside the pipeline do not write this field cleanRun: r() > 0.40, // did the nightly batch job run clean }); } return list; }; // Tag ledger: every build gets an immutable version tag, movable tags are conditional. export const tag = (H) => { const ledger = new Map(); // tag -> image digest for (const d of H) { ledger.set(`v-${d.no}`, d.image); ledger.set("latest", d.image); if (d.cleanRun) ledger.set("stable", d.image); if (d.no % 4 === 0) ledger.set("branch-ready", d.image); } return ledger; }; if (import.meta.url === `file://${process.argv[1]}`) { const H = history(); const seen = new Set(); console.log("no".padEnd(4) + "day".padEnd(5) + "source".padEnd(8) + "image digest".padEnd(15) + "status"); for (const d of H) { const repeat = seen.has(d.image); seen.add(d.image); console.log(String(d.no).padEnd(4) + String(d.day).padEnd(5) + d.source.padEnd(8) + d.image.padEnd(15) + (repeat ? "repeat" : "new")); } const ledger = tag(H); const fanIn = new Map(); for (const [t, image] of ledger) fanIn.set(image, [...(fanIn.get(image) ?? []), t]); const top = [...fanIn.entries()].sort((a, b) => b[1].length - a[1].length); console.log(`\n${H.length} builds produced ${seen.size} distinct image digests; ` + `${H.length - seen.size} builds reproduced a previous digest`); console.log(`${ledger.size} tags point to ${fanIn.size} distinct digests`); for (const [image, tags] of top.slice(0, 3)) { console.log(` ${image}: ${tags.length} tags -> ${tags.join(", ")}`); } console.log(`builds with the source field written: ${H.filter((d) => d.sourceWritten).length}/${H.length}`); } ``` ``` no day source image digest status 1 7 s-002 2747388d03c9 new 2 11 s-002 acff51e2a712 new 3 18 s-002 acff51e2a712 repeat 4 25 s-003 4bd4e48c8ee9 new 5 29 s-004 81139748f001 new 6 33 s-005 cd88604b9280 new 7 41 s-006 91eb7d898a53 new 8 45 s-007 cd261660f128 new 9 54 s-008 4d5ddd536da3 new 10 60 s-009 ac698a72632e new 11 66 s-010 13c70bced338 new 12 73 s-010 13c70bced338 repeat 13 78 s-010 3027786a6a54 new 14 83 s-011 a9e045075303 new 15 88 s-011 081c32254a6d new 16 91 s-012 b3d1ed4a0430 new 17 95 s-013 385c204d516c new 18 101 s-014 5ef86296e112 new 19 105 s-015 a246d27c5542 new 20 111 s-015 a246d27c5542 repeat 20 builds produced 17 distinct image digests; 3 builds reproduced a previous digest 23 tags point to 17 distinct digests a246d27c5542: 5 tags -> latest, stable, branch-ready, v-19, v-20 acff51e2a712: 2 tags -> v-2, v-3 13c70bced338: 2 tags -> v-11, v-12 builds with the source field written: 14/20 ``` Twenty builds produced **seventeen** distinct image digests. Three builds reproduced a previous build's digest: because neither the source nor the layers changed, the manifest came out the same. This is visible proof of the deterministic build discipline from the previous lesson — under a build that stamps, twenty builds would give twenty distinct digests. The second number runs the other way: **23 tags point to 17 digests.** One digest has five names — `latest`, `stable`, `branch-ready`, `v-19`, `v-20`. Two of them are immutable, three will point to a different digest on the next build. Being able to look at the same object through five names is not a flaw, it is the registry's design; the flaw is whether **which of these names gets written to the deployment record** determines the object. #### What Changes When a Tag Moves When a tag moves, a new image is addressed, but the image is not entirely new. A manifest is a layer list, and two manifests can share some of the list's rows. ```js // measurement-network/move.mjs — how many layers really change when a tag moves (model) import { history, LAYER } from "./image.mjs"; const G = history(); const FULL = LAYER.reduce((t, [, b]) => t + b, 0); const mb = (b) => (b / 1e6).toFixed(1) + " MB"; // Builds each tag targets, in order. const targets = { latest: G, stable: G.filter((d) => d.cleanRun), "branch-ready": G.filter((d) => d.no % 4 === 0), }; console.log("tag".padEnd(15) + "moves".padEnd(8) + "0 layers".padEnd(10) + "1 layer".padEnd(10) + "2+ layers".padEnd(11) + "new bytes".padEnd(11) + "full bytes"); const detail = []; for (const [tag, sequence] of Object.entries(targets)) { const distribution = [0, 0, 0]; let newTotal = 0; for (let i = 1; i < sequence.length; i++) { const old = sequence[i - 1].layers, updated = sequence[i].layers; const changed = updated.filter((k, j) => k.digest !== old[j].digest); const bytes = changed.reduce((t, k) => t + k.bytes, 0); newTotal += bytes; distribution[Math.min(changed.length, 2)] += 1; if (tag === "stable" && changed.length !== 1) { // 1 layer is the ordinary case; it is in the summary detail.push(`s-${sequence[i - 1].no} -> s-${sequence[i].no}`.padEnd(16) + `${changed.length}/4`.padEnd(8) + (changed.map((k) => k.name).join(", ") || "-").padEnd(39) + mb(bytes)); } } const moves = sequence.length - 1; console.log(tag.padEnd(15) + String(moves).padEnd(8) + String(distribution[0]).padEnd(10) + String(distribution[1]).padEnd(10) + String(distribution[2]).padEnd(11) + mb(newTotal).padEnd(11) + mb(moves * FULL)); } console.log('\n"stable" moves with a layer count other than one:'); console.log("move".padEnd(16) + "layer".padEnd(8) + "changed".padEnd(39) + "new bytes"); for (const s of detail) console.log(s); // Per layer: how many of the 19 "latest" moves changed it? const count = {}; for (let i = 1; i < G.length; i++) G[i].layers.forEach((k, j) => { if (k.digest !== G[i - 1].layers[j].digest) count[k.name] = (count[k.name] ?? 0) + 1; }); console.log("\nchange per layer (across 19 moves): " + LAYER.map(([name]) => `${name} ${count[name] ?? 0}`).join(", ")); console.log(`full image ${mb(FULL)}; immutable version tags ${G.length}, movable tags ` + `${Object.keys(targets).length}`); ``` ``` tag moves 0 layers 1 layer 2+ layers new bytes full bytes latest 19 3 10 6 141.7 MB 655.5 MB stable 14 1 8 5 120.1 MB 483.0 MB branch-ready 4 0 0 4 58.5 MB 138.0 MB "stable" moves with a layer count other than one: move layer changed new bytes s-2 -> s-3 0/4 - 0.0 MB s-3 -> s-5 3/4 base, application, config 18.5 MB s-6 -> s-8 2/4 application, config 8.0 MB s-8 -> s-10 3/4 dependencies, application, config 24.0 MB s-16 -> s-17 2/4 application, config 8.0 MB s-18 -> s-20 2/4 application, config 8.0 MB change per layer (across 19 moves): base 1, dependencies 2, application 13, config 7 full image 34.5 MB; immutable version tags 20, movable tags 3 ``` The `latest` tag has moved nineteen times. Of these nineteen moves, ten changed only one layer, three changed no layer at all. A zero-layer move is a case where the tag moved but the content stayed the same: the name points to a new digest, the digest addresses the same manifest, and there are zero bytes to pull. From the outside, though, it looks like "a new image was published," and a deployment can get triggered. The bytes columns give the moves' real weight: across nineteen moves, the total that changed is 141.7 MB, while the same nineteen images' total size is 655.5 MB. Layer sharing removes 78 percent of the moves' load. The `branch-ready` tag sits at the opposite end — in all four of its four moves, two or more layers changed, because four builds pile up between them. **The less often a tag moves, the more layers change per move.** The last row shows where the change comes from: across nineteen moves, the application layer changed 13 times, config 7, dependencies 2, the base only 1. This looks the same direction as the previous lesson's update measurement — lower layers change rarely, upper layers change on almost every build. #### Back From Digest to Source Everything measured so far ran forward: from source to build, from build to digest, from digest to tag. The question asked during an incident runs backward — **which source did this object running in production come from?** **IM36.** The 24 deployments are fictional; the seed `20260707` is visible, and the kind of requested name — portable tag, immutable version tag, or direct digest — comes from the generator. The chain has two links: from the deployment record to the image digest, from the image digest to the source version. ```js // measurement-network/traceability.mjs — can a deployment's source be traced back from its digest (model) import { history, generator } from "./image.mjs"; const G = history(); const r = generator(20260707); const FIELD_BYTES = 96; // source version + build id + timestamp (fictional) const FULL = 34.5e6; // 24 deployments: day and requested name. The name is either a portable tag, a version tag, or a direct digest. const deployments = []; let day = 3; while (deployments.length < 24) { day += 1 + Math.round(r() * 7); const p = r(); deployments.push({ day, kind: p < 0.60 ? "portable" : p < 0.85 ? "version" : "digest" }); } // The tag ledger advances by day: what resolves at deployment time is that moment's target. const ledger = new Map(); let pos = 0; for (const d of deployments) { while (pos < G.length && G[pos].day <= d.day) { const b = G[pos++]; ledger.set("latest", b); ledger.set(`s-${b.no}`, b); if (b.cleanRun) ledger.set("stable", b); } const versionTags = [...ledger.keys()].filter((k) => k.startsWith("s-")); d.name = d.kind === "portable" ? (r() < 0.6 ? "stable" : "latest") : d.kind === "version" ? versionTags[versionTags.length - 1] : "digest:direct"; d.target = d.kind === "digest" ? ledger.get("latest") : ledger.get(d.name); } // An immutable version tag determines the digest as long as the ledger refuses to move it; a // portable tag does not. Three disciplines: (1) the record writes only the requested name, (2) it // also writes the resolved digest, (3) every build additionally writes the source field. const digestKnown = (d) => d.kind === "digest" || d.kind === "version"; const traceable = (allDigest, allSource) => deployments.filter((d) => (allDigest || digestKnown(d)) && (allSource || d.target.sourceWritten)).length; const N = deployments.length; console.log("discipline".padEnd(34) + "digest known".padEnd(16) + "source readable"); console.log("name only in record".padEnd(34) + `${deployments.filter(digestKnown).length}/${N}`.padEnd(16) + `${traceable(false, false)}/${N}`); console.log("record also has digest".padEnd(34) + `${N}/${N}`.padEnd(16) + `${traceable(true, false)}/${N}`); console.log("every build also writes source".padEnd(34) + `${N}/${N}`.padEnd(16) + `${traceable(true, true)}/${N}`); // The other direction: how many distinct image digests did the same source version produce? const fromSource = new Map(); for (const b of G) fromSource.set(b.source, new Set([...(fromSource.get(b.source) ?? []), b.image])); const multi = [...fromSource.entries()].filter(([, s]) => s.size > 1); console.log(`\ndeployment kind: portable ${deployments.filter((d) => d.kind === "portable").length}, ` + `version tag ${deployments.filter((d) => d.kind === "version").length}, ` + `direct digest ${deployments.filter((d) => d.kind === "digest").length}`); console.log(`${fromSource.size} source versions, ${multi.length} of them produced more than one image digest: ` + multi.map(([k, s]) => `${k} -> ${s.size}`).join(", ")); console.log(`traceability's cost: ${FIELD_BYTES} bytes per image, 1 in ` + `${Math.round(FULL / FIELD_BYTES)} of the full image; ${N} x 1 field in the deployment record`); ``` ``` discipline digest known source readable name only in record 14/24 9/24 record also has digest 24/24 17/24 every build also writes source 24/24 24/24 deployment kind: portable 10, version tag 10, direct digest 4 14 source versions, 3 of them produced more than one image digest: s-002 -> 2, s-010 -> 2, s-011 -> 2 traceability's cost: 96 bytes per image, 1 in 359375 of the full image; 24 x 1 field in the deployment record ``` The three rows form a ladder. Under the current setup, **nine** of twenty-four deployments' sources can be read. The loss is in two places: ten deployments were made with a portable name, so which digest they resolved to is not in the record, and part of the deployments with a known digest also targeted a build whose source field was not written. Also writing the resolved digest to the record raises the rate to **17/24**; writing the source field on every build raises it to **24/24**. The ten deployments made with an immutable version tag determining the digest rests on one assumption: the ledger **refuses to move** the `v-N` tag. This refusal is not a feature, it is a rule the registry enforces — the counterpart here of the immutability check measured in M22/K01. If the rule were not enforced, the 14 in the first row would drop too, and the ladder's first step would fall to four. The second result runs the other way. Three of fourteen source versions produced more than one image digest: the dependency or config layer changed without the source changing. Tracing back leads all the way to the source; it does not lead forward from the source to a single object. **This isolation's cost is small in number.** The source field is 96 bytes per image — 1 in 359,375 of the full image — and a single field added to the deployment record. The gain measured is a rate that climbs from 9/24 to 24/24. **Where isolation is pierced** is the tag ledger itself: the ledger is not inside the image, it is inside the registry. The image is a portable output; its **name** is not — an image copied to another registry does not carry its tags along, only its digest. #### Summary - A tag is a three-level chain: a tag names an image digest, an image digest names a manifest, and a manifest names layer digests. A tag points to a list, not to bytes. - 20 builds produced 17 distinct image digests; 3 builds reproduced a previous digest. This is visible proof of deterministic building — under stamped building, the count would be 20. - 23 tags point to 17 digests; one digest has five names. The flaw is not having many names — it is whether the name written to the record determines the object. - 10 of the `latest` tag's 19 moves changed a single layer, 3 changed none. The changed total is 141.7 MB, the same images' total is 655.5 MB: sharing removes 78 percent of the load. The less often a tag moves, the more layers change per move. - The rate a deployment can be traced back to its source depends on record discipline: 9/24; 17/24 once the resolved digest is also written; 24/24 once the source field is written on every build. The cost is 96 bytes per image and a single field in the record. #### Next Step This lesson's measurement silently assumed one thing: that the counterpart of every row in the manifest sits somewhere. That place was never measured. Against the 141.7 MB that changed across nineteen moves, there are 513.8 MB that did not, and how many copies of those bytes exist depends on whether the layers are shared. The next lesson builds that place and counts it: when multiple images reference the same layer, how much does the store really grow; when an image is deleted, how many layers can really be deleted; what happens when a retention rule runs into layer references; and how much does downloaded traffic drop when a layer already present locally gets skipped? ### Image Registries Source: https://academia.sh/en/courses/containers/image-repositories A layer-shared image registry is built with real files and measured from disk: bytes sharing removes, layers that cannot be deleted because of reference counting once a retention rule applies, the drop in pull traffic when a locally present layer is skipped, and the layer count public and private images share are counted. The previous lesson tied a tag to a layer list and assumed the counterpart of every row in the list sits somewhere. That place is this lesson's subject: the **image registry**. The registry is the layered form of the build artifact repository measured in M22/K01 — it does the same two jobs, storing content and resolving names, and the difference is that what it stores is not a single file but a set of layers. This difference changes every measure: growth, deletion, and download are now computed **per layer**, not per object. **IM37.** The registry in this lesson is built with real files: every layer is a real file named by its own content digest, every manifest is a real file, and every byte measured is read from disk. The scale is 1/1000 — the 34.5 MB image from previous lessons is built here with 34.5 KB real files. **IM38.** The build history comes from the previous lesson's generator; the seed `20260604` is visible, and the same 20 builds give the same 17 distinct images. **IM39.** Part of the images are placed in a private registry; this split comes from a second generator, seeded `20260812`. #### What the Registry Stores In a content-addressed registry, a layer file's name is its own digest. This one rule determines the whole of storage: when the same content is written a second time, it is written under the same name — that is, it takes up no new space. ```js // measurement-network/repo.mjs — layer-shared image registry: real files, real bytes import { createHash } from "node:crypto"; import { mkdirSync, rmSync, writeFileSync, readdirSync, statSync } from "node:fs"; import { join } from "node:path"; export const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-repo"); // Scale 1/1000: the 34.5 MB image from previous lessons is built here with 34.5 KB real files. export const LAYER = [["base", 10500, 0.08], ["dependencies", 16000, 0.25], ["application", 7200, null], ["config", 800, 0.30]]; export const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; export const digest = (b) => createHash("sha256").update(b).digest("hex").slice(0, 12); export const history = (n = 20, seed = 20260604, hiddenSeed = 20260812) => { const r = generator(seed), g = generator(hiddenSeed); const version = [1, 1, 1, 1], list = []; let day = 0; for (let no = 1; no <= n; no++) { day += 1 + Math.round(r() * 8); if (r() > 0.25) version[2] += 1; LAYER.forEach(([, , p], k) => { if (p !== null && r() < p) version[k] += 1; }); r(); r(); // the previous lesson's two fields: pulled so the schedule stays the same list.push({ no, day, hidden: g() < 0.35, layers: LAYER.map(([name, bytes], k) => ({ name, bytes, version: version[k] })) }); } return list; }; // The registry is content-addressed: a layer file's name is its own digest, the same layer is never written twice. export const setUpRepo = (G) => { rmSync(ROOT, { recursive: true, force: true }); mkdirSync(join(ROOT, "layer"), { recursive: true }); mkdirSync(join(ROOT, "manifest"), { recursive: true }); for (const d of G) { d.digests = d.layers.map(({ name, bytes, version }) => { const content = Buffer.alloc(bytes, `${name}@${version}|`); const h = digest(content); writeFileSync(join(ROOT, "layer", h), content); return h; }); d.image = digest(d.digests.join("|")); writeFileSync(join(ROOT, "manifest", d.image), d.digests.join("\n")); } return G; }; export const dirBytes = (path) => readdirSync(path).reduce((t, f) => t + statSync(join(path, f)).size, 0); export const kb = (b) => (b / 1000).toFixed(1) + " KB"; if (import.meta.url === `file://${process.argv[1]}`) { const G = setUpRepo(history()); const actual = dirBytes(join(ROOT, "layer")); const unshared = G.reduce((t, d) => t + d.layers.reduce((s, k) => s + k.bytes, 0), 0); const distinctLayers = readdirSync(join(ROOT, "layer")).length; const distinctImages = readdirSync(join(ROOT, "manifest")).length; console.log(`${G.length} builds, ${distinctImages} distinct manifests, ${G.length * 4} layer references`); console.log(`distinct layer files: ${distinctLayers}`); console.log(`repo actual (measured from disk): ${kb(actual)}`); console.log(`if unshared : ${kb(unshared)}`); console.log(`what sharing removes : ${kb(unshared - actual)} ` + `(%${(100 * (1 - actual / unshared)).toFixed(1)})`); const count = new Map(); for (const d of G) for (const h of new Set(d.digests)) count.set(h, (count.get(h) ?? 0) + 1); const sorted = [...count.values()].sort((a, b) => b - a); console.log(`references per layer: most ${sorted[0]}, median ` + `${sorted[Math.floor(sorted.length / 2)]}, referenced once ` + `${sorted.filter((v) => v === 1).length}/${distinctLayers}`); } ``` ``` 20 builds, 17 distinct manifests, 80 layer references distinct layer files: 27 repo actual (measured from disk): 176.2 KB if unshared : 690.0 KB what sharing removes : 513.8 KB (%74.5) references per layer: most 16, median 2, referenced once 11/27 ``` Eighty layer references correspond to **twenty-seven** files on disk. The registry is 176.2 KB; if every image carried its own layers, it would be 690.0 KB. Sharing removes 74.5 percent of the bytes. The 513.8 KB removed is the same size as the unchanged byte total in the previous lesson's move measurement — the two measurements are two faces of the same fact. The reference distribution shows the real structure: one layer appears in **sixteen** images, the median is two, but **eleven** of twenty-seven layers appear in only a single image. The registry is not homogeneous; at one end sit lower layers that almost never change, at the other sit upper layers renewed on nearly every build. This distribution is also the source of the next measurement. #### Deletion Runs Into Reference Counting M22/K01 scanned the retention rule through bytes, the rollback window, and unreachable deployments; that scan is not repeated. In a layered registry, the rule runs into a new obstacle: **deleting an image does not mean deleting its layers.** A layer can only be deleted if no manifest references it. **IM40.** The retention rule is a count — "the last K images are kept" — and K is counted over distinct manifests. The scan is calculated first, then one rule is really applied: files are deleted and the directory is re-measured. ```js // measurement-network/delete.mjs — what a retention rule really deletes when it runs into layer references (real files) import { readdirSync, statSync, unlinkSync } from "node:fs"; import { join } from "node:path"; import { ROOT, history, setUpRepo, dirBytes, kb } from "./repo.mjs"; const G = setUpRepo(history()); const distinctImages = []; // distinct manifests, in build order for (const d of G) if (!distinctImages.some((x) => x.image === d.image)) distinctImages.push(d); const sizeOf = (h) => statSync(join(ROOT, "layer", h)).size; const allLayers = readdirSync(join(ROOT, "layer")); const totalRepo = dirBytes(join(ROOT, "layer")); console.log("rule".padEnd(16) + "kept".padEnd(10) + "referenced".padEnd(13) + "deletable".padEnd(13) + "freed".padEnd(11) + "repo left"); for (const K of [3, 5, 8, 12, 17]) { const kept = distinctImages.slice(-K); const referenced = new Set(kept.flatMap((d) => d.digests)); const deletable = allLayers.filter((h) => !referenced.has(h)); const freed = deletable.reduce((t, h) => t + sizeOf(h), 0); console.log(`last ${K} images`.padEnd(16) + `${K}/${distinctImages.length}`.padEnd(10) + `${referenced.size}/${allLayers.length}`.padEnd(13) + `${deletable.length}/${allLayers.length}`.padEnd(13) + kb(freed).padEnd(11) + kb(totalRepo - freed)); } // The "last 8 images" rule is really applied: files are deleted, the directory is re-measured. const kept = distinctImages.slice(-8); const referenced = new Set(kept.flatMap((d) => d.digests)); const size = new Map(allLayers.map((h) => [h, sizeOf(h)])); const dropped = distinctImages.slice(0, -8); const droppedLayers = new Set(dropped.flatMap((d) => d.digests)); const before = dirBytes(join(ROOT, "layer")); let deleted = 0; for (const h of allLayers) if (!referenced.has(h)) { unlinkSync(join(ROOT, "layer", h)); deleted++; } for (const d of dropped) unlinkSync(join(ROOT, "manifest", d.image)); const after = dirBytes(join(ROOT, "layer")); const surviving = [...droppedLayers].filter((h) => referenced.has(h)); const droppedBytes = [...droppedLayers].reduce((t, h) => t + size.get(h), 0); console.log(`\n"last 8 images" applied: ${dropped.length} manifests and ${deleted} layer files deleted`); console.log(`layer directory ${kb(before)} -> ${kb(after)} (measured from disk): ` + `${dropped.length} of ${distinctImages.length} images dropped, bytes fell %${(100 * (before - after) / before).toFixed(0)}`); console.log(`of the dropped images' ${droppedLayers.size} distinct layers, ${surviving.length} are still ` + `referenced: ${kb(droppedBytes)} of bytes, ${kb(surviving.reduce((t, h) => t + size.get(h), 0))} of it could not be deleted`); ``` ``` rule kept referenced deletable freed repo left last 3 images 3/17 7/27 20/27 126.5 KB 49.7 KB last 5 images 5/17 10/27 17/27 111.3 KB 64.9 KB last 8 images 8/17 13/27 14/27 102.5 KB 73.7 KB last 12 images 12/17 20/27 7/27 56.1 KB 120.1 KB last 17 images 17/17 27/27 0/27 0.0 KB 176.2 KB "last 8 images" applied: 9 manifests and 14 layer files deleted layer directory 176.2 KB -> 73.7 KB (measured from disk): 9 of 17 images dropped, bytes fell %58 of the dropped images' 17 distinct layers, 3 are still referenced: 129.8 KB of bytes, 27.3 KB of it could not be deleted ``` The table is not linear. When three of seventeen images are kept, the registry drops to 49.7 KB; when eight are kept, it is 73.7 KB. While the kept image count rises two and a half times, the registry only grows by about half, because most of the newly kept images' layers are already there. The applied rule gives the real point. Nine manifests were deleted, but three of the seventeen distinct layers these nine images referenced could not be deleted, because they still appear in kept images: 27.3 KB of 129.8 KB stayed on disk. The retention rule is written over manifests, it does not operate over bytes. The sentence "delete this image" means, in a layered registry, "delete this manifest and **collect** the layers whose reference count dropped to zero" — and there is a collecting step between these two jobs. This is also a shortfall in isolation, in the deletion direction. An image being deleted does not mean the bytes inside it have left the registry; for anyone who knows its digest, those layers are still readable. #### Pull: When What Is Local Gets Skipped Sharing in the registry is not only a disk gain; the same arithmetic works on pulls too. A node does not redownload a layer it already has. **IM41.** A node pulls seventeen images in sequence, and its local layer cache is never cleared. The copying is real: layer files are copied from the registry to a local directory, and bytes pulled are measured from disk. **IM42.** Authentication itself is outside the model; what is measured is what level access control can be enforced at. ```js // measurement-network/pull.mjs — bytes pulled when a locally present layer is skipped (real copying) import { mkdirSync, rmSync, existsSync, copyFileSync, statSync } from "node:fs"; import { join } from "node:path"; import { ROOT, history, setUpRepo, dirBytes, kb } from "./repo.mjs"; const G = setUpRepo(history()); const LOCAL = join(ROOT, "local"); rmSync(LOCAL, { recursive: true, force: true }); mkdirSync(LOCAL, { recursive: true }); const distinct = []; for (const d of G) if (!distinct.some((x) => x.image === d.image)) distinct.push(d); // A node pulls images in order: a layer file already local is not copied. let pulled = 0, full = 0, skipped = 0, firstPull = 0; const rows = []; for (const d of distinct) { let pullBytes = 0, fresh = 0; for (const h of d.digests) { const dest = join(LOCAL, h); if (existsSync(dest)) { skipped += statSync(join(ROOT, "layer", h)).size; continue; } copyFileSync(join(ROOT, "layer", h), dest); pullBytes += statSync(dest).size; fresh += 1; } const imageBytes = d.layers.reduce((t, k) => t + k.bytes, 0); pulled += pullBytes; full += imageBytes; if (rows.length === 0) firstPull = pullBytes; rows.push(`${String(d.no).padStart(3)} ${fresh}/4 ${kb(pullBytes).padStart(8)} ${kb(imageBytes)}`); } console.log("first pull and last three pulls"); console.log(" no fresh pulled full image"); for (const s of [rows[0], ...rows.slice(-3)]) console.log(s); console.log(`\n${distinct.length} pulls: pulled ${kb(pulled)}, full sizes total ${kb(full)}, ` + `skipped ${kb(skipped)} (%${(100 * skipped / full).toFixed(1)})`); console.log(`first (cold) pull ${kb(firstPull)}; average of the next ${distinct.length - 1} pulls is ` + kb((pulled - firstPull) / (distinct.length - 1))); console.log(`local cache on disk ${kb(dirBytes(LOCAL))}, repo ${kb(dirBytes(join(ROOT, "layer")))}`); // Access control: public and private images share a single content-addressed pool. const publicImages = distinct.filter((d) => !d.hidden), privateImages = distinct.filter((d) => d.hidden); const publicLayers = new Set(publicImages.flatMap((d) => d.digests)); const privateLayers = new Set(privateImages.flatMap((d) => d.digests)); const common = [...privateLayers].filter((h) => publicLayers.has(h)); const commonBytes = common.reduce((t, h) => t + statSync(join(ROOT, "layer", h)).size, 0); const privateOnly = [...privateLayers].filter((h) => !publicLayers.has(h)); console.log(`\npublic images ${publicImages.length}, private images ${privateImages.length}`); console.log(`of the private images' ${privateLayers.size} layers, ${common.length} also appear in ` + `public images (${kb(commonBytes)}); private-only layers ${privateOnly.length}`); ``` ``` first pull and last three pulls no fresh pulled full image 1 4/4 34.5 KB 34.5 KB 17 2/4 8.0 KB 34.5 KB 18 1/4 7.2 KB 34.5 KB 19 2/4 8.0 KB 34.5 KB 17 pulls: pulled 176.2 KB, full sizes total 586.5 KB, skipped 410.3 KB (%70.0) first (cold) pull 34.5 KB; average of the next 16 pulls is 8.9 KB local cache on disk 176.2 KB, repo 176.2 KB public images 10, private images 7 of the private images' 16 layers, 9 also appear in public images (63.4 KB); private-only layers 7 ``` Seventeen pulls download 176.2 KB in total, against the images' combined size of 586.5 KB. The 410.3 KB skipped is 70 percent of the traffic. The first pull downloads a full image (34.5 KB); the average of the next sixteen pulls is 8.9 KB — about a quarter of an image. It is not a coincidence that the total downloaded equals the registry's size: a node that pulls every image downloads the distinct layer set exactly once. This means pull traffic's upper bound is not the image count but the **distinct layer count**. The number behind the cost is the local cache: 176.2 KB is kept on the node, and that is about five times a single image currently running. This is isolation's counterpart on the pull side — download shrinks, storage grows. #### Public Registry, Private Registry The output's last two lines measure access control. Ten of seventeen images are public, seven are private. Authentication is enforced at the **manifest level**: requesting a private image's manifest requires identity. Layers, though, sit in a single content-addressed pool. The outcome is measured: nine of the private images' sixteen layers also appear in public images — 63.4 KB. These nine layers' digests can be read from a public manifest, and once read, those bytes can be pulled without needing identity. Only **seven layers** are specific to private images; that is what is actually secret. This is this lesson's place where isolation is pierced, and it has two sides. On one hand, the loss is real: the phrase "private registry" suggests all the bytes are secret, and the measurement says it is limited to seven layers. On the other hand, the cost of closing it can be counted: removing sharing and giving private images a separate pool spreads secrecy over every layer, but gives back part of the 513.8 KB the registry gained from sharing. Isolation is a budget, and this item comes out of it too. #### Summary - In a content-addressed registry, 80 layer references correspond to 27 files: the registry is 176.2 KB on disk, 690.0 KB if unshared. Sharing removes 74.5 percent of the bytes. - The reference distribution is not homogeneous: one layer appears in 16 images, the median is 2, and 11 of 27 layers appear in only one image. - The retention rule is written over manifests, it does not operate over bytes. When "last 8 images" is applied, 9 manifests are deleted, and because 3 of the dropped images' 17 distinct layers are still referenced, 27.3 KB of 129.8 KB stays on disk. - When a locally present layer is skipped, 17 pulls download 176.2 KB instead of 586.5 KB; what is skipped is 70 percent. Pull traffic's upper bound is the distinct layer count, not the image count. The cost is the 176.2 KB local cache kept on the node. - Access control is enforced at the manifest level, layers share the pool: of the 16 layers across 7 private images, 9 (63.4 KB) also appear in public images and can be read without identity by knowing the digest. Only 7 layers are actually secret. #### Next Step The registry has been measured, but one question stays open. If a private image's nine layers are readable in the public pool, what is inside those layers? This lesson's measurement looked at layers as stacks of bytes and never looked inside them. The next lesson looks inside and closes the topic: does a file deleted in an upper layer really still sit in the lower layer, and how many bytes were really never deleted; how many does a scan running against known defects catch, and how many does it miss; what does a signing chain verify, and what path does an unsigned pull open? ### Image Security Source: https://academia.sh/en/courses/containers/image-security Image security is measured in three parts: how many bytes a file deleted in an upper layer really still occupies in the lower layer is counted in real directories, the caught, missed, and false-positive counts for a scan running against a known defect set are extracted, signing is built as a chain with a real key pair, and what gets accepted at every unverified link is counted. The previous lesson measured the registry and closed with a question: if a private image's nine layers are readable in the public pool, what is inside those layers? Up to now, layers have been looked at as stacks of bytes. This lesson looks inside and closes the topic. There are three measurements, and all three are parts of the same question: how is it known what sits inside an output at rest? The first is a leak that comes from the image's own structure, the second is an attempt to find known defects, the third is verifying who the output came from. **IM43.** Layers are real directories, and the union view is built bottom to top; a file's deletion is shown by a marker file prefixed `.deleted-` in the upper layer. **IM44.** Secret values are fictional; no real secret is used. #### A Secret Left Sitting in a Layer M22/K01 scanned how many separate ways a secret can leak out; that scan is not repeated here. The fact here is single and layer-specific: **a layer is an immutable object.** An instruction that deletes a file produces a new layer; the lower layer stays exactly as it was, because its digest also appears in other images. Deletion is a view operation, not a store operation. ```js // measurement-network/leak.mjs — does a file deleted in an upper layer still sit in the lower layer (real directories) import { mkdirSync, rmSync, writeFileSync, readdirSync, readFileSync, statSync } from "node:fs"; import { join, dirname } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-layer"); rmSync(ROOT, { recursive: true, force: true }); // Fictional secret values. No real secret is used. const SECRET_A = "fictional-access-key-A1B2C3D4E5F6"; const SECRET_B = "fictional-db-password-Z9Y8X7"; const body = (secret, pad) => Buffer.concat([Buffer.from(`${secret}\n`), Buffer.alloc(pad, "-")]); const LAYERS = [ { name: "l1-base", write: { "bin/collector": 4096, "lib/crypto": 8192 } }, { name: "l2-dependency", write: { "lib/client": 6144, "lib/queue": 3072 } }, { name: "l3-application", write: { "app/collector.js": 5120, "app/secret/access.key": body(SECRET_A, 1664), "app/temp/setup.log": body(SECRET_B, 2048) }, sameLayerDelete: ["app/temp/setup.log"] }, // removed before the layer closes { name: "l4-config", write: { "app/settings.json": 512 }, upperLayerDelete: ["app/secret/access.key"] }, // a deletion marker is left in the upper layer ]; // Layers are written to real directories; a deletion marker is an empty file prefixed ".deleted-". for (const k of LAYERS) { for (const [path, value] of Object.entries(k.write)) { if (k.sameLayerDelete?.includes(path)) continue; const full = join(ROOT, k.name, path); mkdirSync(dirname(full), { recursive: true }); writeFileSync(full, typeof value === "number" ? Buffer.alloc(value, "x") : value); } for (const path of k.upperLayerDelete ?? []) { const full = join(ROOT, k.name, dirname(path), ".deleted-" + path.split("/").pop()); mkdirSync(dirname(full), { recursive: true }); writeFileSync(full, ""); } } const walk = (root, sub = "") => readdirSync(join(root, sub), { withFileTypes: true }) .flatMap((g) => g.isDirectory() ? walk(root, join(sub, g.name)) : [join(sub, g.name)]); // Union view: bottom to top; a deletion marker removes the visible file. const view = new Map(); for (const k of LAYERS) { for (const path of walk(join(ROOT, k.name))) { const name = path.split("/").pop(); if (name.startsWith(".deleted-")) view.delete(join(dirname(path), name.slice(9))); else view.set(path, statSync(join(ROOT, k.name, path)).size); } } const onDisk = LAYERS.flatMap((k) => walk(join(ROOT, k.name)) .filter((y) => !y.split("/").pop().startsWith(".deleted-")) .map((y) => [k.name, y, statSync(join(ROOT, k.name, y)).size])); const viewBytes = [...view.values()].reduce((t, b) => t + b, 0); const diskBytes = onDisk.reduce((t, [, , b]) => t + b, 0); const hidden = onDisk.filter(([, y]) => !view.has(y)); console.log(`union view : ${view.size} files, ${viewBytes} bytes`); console.log(`really in layers : ${onDisk.length} files, ${diskBytes} bytes`); console.log(`absent from view, still in a layer: ${hidden.length} files, ` + `${hidden.reduce((t, [, , b]) => t + b, 0)} bytes`); for (const [name, secret] of [["SECRET_A", SECRET_A], ["SECRET_B", SECRET_B]]) { const found = onDisk.filter(([k, y]) => readFileSync(join(ROOT, k, y)).includes(secret)); console.log(`${name}: in ${found.length} layer files, ` + `${found.reduce((t, [, , b]) => t + b, 0)} bytes` + (found.length ? ` -> ${found.map(([k, y]) => `${k}/${y}`).join(", ")}` : "")); } ``` ``` union view : 6 files, 27136 bytes really in layers : 7 files, 28834 bytes absent from view, still in a layer: 1 files, 1698 bytes SECRET_A: in 1 layer files, 1698 bytes -> l3-application/app/secret/access.key SECRET_B: in 0 layer files, 0 bytes ``` The union view shows six files and 27,136 bytes. There are seven files and 28,834 bytes on disk. The 1,698-byte gap is the key file that appears deleted in the upper layer, and it sits whole in the `l3-application` layer. An audit that looks at the view cannot find it; whoever opens the layer reads it directly. The difference between the two secrets completes the measurement. `SECRET_B` was written too, but removed **before the same layer closed**: it appears in no layer file, zero bytes. `SECRET_A` was deleted in an upper layer and sits at its full size. The only difference is which layer the deletion happened in, and the result is a binary difference — 1,698 bytes against 0 bytes. This gets heavier when combined with the previous lesson's measurement. A layer in the registry could only be deleted if no manifest referenced it; in the measurement, three of the dropped images' seventeen layers were referenced, so they stayed on disk. A layer that carries a secret is subject to the same rule: if another image references it, the bytes stay in the registry even if the image carrying the secret is deleted. **This is where isolation is pierced** — the difference taken into the output cannot be taken back out of it. The way to close it off is written into the measurement itself. `SECRET_B`'s zero bytes were achieved by removing the value **before the layer closed**: a file written and deleted within the same instruction never enters the layer. A sturdier approach is to never write it at all — reading the value from a source visible only during the build step and never copying it into the output. The two paths' measured outcome is the same, 1,698 bytes against 0 bytes; but the first depends on instruction order, the second on the output's structure, and the first silently comes back the moment the order breaks. #### What a Scan Sees The second measurement compares the components inside the image against a known defect set. A scan is a matching process, and the matching has error in both directions. **IM45.** The defect set is fictional: a component's probability of having an open record is 0.30, the probability that the record's fix was backported to the version is 0.25; the seed `20260909` is visible. **IM46.** Visibility depends on the candidate — on a base with a package manager, the inventory can be read; on one without, the component comes embedded inside the binary, and its version cannot be counted. ```js // measurement-network/scan.mjs — scanning against a known defect set: caught, missed, false positive (model) const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const P_DEFECT = 0.30; // the component has an open record in the defect set const P_BACKPORTED = 0.25; // it has a record, but the fix was backported to the version: not open // visibility: probability that the component's version can be read from the inventory. On a base // with no package manager there is no inventory; the component comes embedded inside the binary. const CANDIDATE = [["full-featured", 148, 0.92], ["slimmed", 31, 0.82], ["single-binary", 6, 0.35]]; const scan = (components, visibility, seed = 20260909) => { const r = generator(seed); const s = { components, open: 0, warning: 0, caught: 0, missed: 0, falsePositive: 0 }; for (let i = 0; i < components; i++) { const recorded = r() < P_DEFECT, visible = r() < visibility, backported = r() < P_BACKPORTED; const open = recorded && !backported; // a genuinely open defect const flags = recorded && visible; // the warning the scan produces if (open) s.open += 1; if (flags) s.warning += 1; if (open && flags) s.caught += 1; if (open && !flags) s.missed += 1; if (flags && !open) s.falsePositive += 1; } return s; }; console.log("candidate".padEnd(15) + "components".padEnd(12) + "visibility".padEnd(12) + "open defect".padEnd(13) + "warning".padEnd(9) + "caught".padEnd(11) + "missed".padEnd(8) + "false positive"); const results = []; for (const [name, n, g] of CANDIDATE) { const s = scan(n, g); results.push([name, s]); console.log(name.padEnd(15) + String(n).padEnd(12) + `%${(100 * g).toFixed(0)}`.padEnd(12) + String(s.open).padEnd(13) + String(s.warning).padEnd(9) + String(s.caught).padEnd(11) + String(s.missed).padEnd(8) + String(s.falsePositive)); } console.log(""); const pct = (a, b) => (b ? `%${(100 * a / b).toFixed(0)}` : "unmeasured"); for (const [name, s] of results) { console.log(`${name.padEnd(15)} catch rate ${pct(s.caught, s.open)}, ` + `warning accuracy ${pct(s.caught, s.warning)}`); } ``` ``` candidate components visibility open defect warning caught missed false positive full-featured 148 %92 31 37 29 2 8 slimmed 31 %82 7 7 6 1 1 single-binary 6 %35 1 0 0 1 0 full-featured catch rate %94, warning accuracy %78 slimmed catch rate %86, warning accuracy %86 single-binary catch rate %0, warning accuracy unmeasured ``` The full-featured base has 31 open defects; the scan produces 37 warnings, 29 of which are real, **8 are false positives**, and **2 open defects are missed**. The source of the false positives is a record whose fix was backported: the version number appears in the flawed range, the code does not. The source of the misses is a component that cannot be read from the inventory. The last row is this lesson's harshest number. **In the single-binary base, the scan produces zero warnings, but there is one open defect.** A clean report does not mean a defect-free image; a component that cannot be counted goes uncounted, and nothing marks the gap. The trade-off measured in the previous lesson shows up here a second time — the small base really carries fewer defects (148 components against 6), but it also lowers the visibility of what it does carry. **The scan's cost can be counted:** 8 of 37 warnings were looked at for nothing, and each one requires manually comparing a component's version against the defect record by hand. The way to raise the catch rate is not more warnings but a readable inventory — carrying the component list alongside the image as a separate document closes this gap, and its size is a few kilobytes. #### The Signing Chain The third measurement is the identity question. Scanning says "what is inside"; a signature says "did we really produce this." The chain has three links: the content digest, that digest's signature, and verification at pull time. **IM47.** Signing is done with a real key pair, but the key is regenerated on every run; that is why what is printed is the signature's length and the verification result, not the signature itself. **IM48.** Recomputing a pulled layer's digest at pull time is an optional step. ```js // measurement-network/signature.mjs — signing chain: digest, signature, verification (with a real key pair) import { generateKeyPairSync, sign, verify, createHash } from "node:crypto"; const digest = (b) => createHash("sha256").update(b).digest(); // The key pair is regenerated on every run. Signature bytes depend on the run; that is // why what is printed is the signature's length and the verification result, not the signature itself. const { publicKey, privateKey } = generateKeyPairSync("ed25519"); // The previous lesson's 17 distinct images: every manifest is a list of four layer digests. const manifest = (i) => Buffer.from([1, 2, 3, 4] .map((k) => digest(Buffer.from(`layer-${k}-${i}`)).toString("hex").slice(0, 12)).join("\n")); const MANIFEST = Array.from({ length: 17 }, (_, i) => manifest(i)); const SIGNATURE = MANIFEST.map((m) => sign(null, digest(m), privateKey)); console.log(`signed manifests ${MANIFEST.length}, signature length ${SIGNATURE[0].length} bytes, ` + `total ${MANIFEST.length * SIGNATURE[0].length} bytes`); console.log(`registry 176200 bytes; signatures' share is ` + `${(1000 * MANIFEST.length * SIGNATURE[0].length / 176200).toFixed(1)} per thousand`); // The chain's links are tested one at a time. const m0 = MANIFEST[0]; const tampered = Buffer.from(m0.toString().replace(/^.{12}/, "ffffffffffff")); const { publicKey: otherKey } = generateKeyPairSync("ed25519"); console.log("\ncorrect manifest, correct signature :", verify(null, digest(m0), publicKey, SIGNATURE[0])); console.log("tampered manifest :", verify(null, digest(tampered), publicKey, SIGNATURE[0])); console.log("verification with another key :", verify(null, digest(m0), otherKey, SIGNATURE[0])); // The layer bytes themselves: even if the manifest is correct, the downloaded bytes can still change. const layer = Buffer.alloc(7200, "u"); const expected = digest(layer).toString("hex"); const corrupted = Buffer.from(layer); corrupted[42] = 0x41; // one byte changed console.log("downloaded layer's digest holds:", digest(corrupted).toString("hex") === expected); // 24 pulls, three disciplines: what would each accept? const PULLS = 24, LAYERS = 4; console.log("\ndiscipline".padEnd(36) + "forged manifest accepted".padEnd(29) + "unverified layer"); for (const [label, checkSignature, checkLayer] of [ ["signature not verified", false, false], ["only signature verified", true, false], ["signature and layer digest verified", true, true], ]) { console.log(label.padEnd(36) + `${checkSignature ? 0 : PULLS}/${PULLS}`.padEnd(29) + `${checkLayer ? 0 : PULLS * LAYERS}/${PULLS * LAYERS}`); } ``` ``` signed manifests 17, signature length 64 bytes, total 1088 bytes registry 176200 bytes; signatures' share is 6.2 per thousand correct manifest, correct signature : true tampered manifest : false verification with another key : false downloaded layer's digest holds: false discipline forged manifest accepted unverified layer signature not verified 24/24 96/96 only signature verified 0/24 96/96 signature and layer digest verified 0/24 0/96 ``` **The signature's cost is the smallest item:** 1,088 bytes total for seventeen manifests, 6.2 per thousand of the registry. Verification itself is nothing more than a digest comparison. The middle three rows show the chain working: the correct manifest verifies with the correct signature, verification fails once the manifest's first twelve characters are changed, and it fails when the signature is tested with a different key too. The fourth row shows where the chain ends — when a single byte of the downloaded layer changes, the digest does not hold, **but this is only seen if the digest is recomputed.** The last table counts the path each skipped verification step opens. With no verification, 24 of 24 pulls accept a modified manifest. When only the signature is verified, forged manifests drop to zero, but **96 of 96 layers** get accepted with their digest untested: the manifest is correct, but whether the downloaded bytes match the manifest is unknown. When both links are tested, both numbers are zero. There is something that sits outside the chain, and it is where isolation is pierced here: a signature does not say "this image is safe"; it says "this manifest was signed with this key." Who the key belongs to, where it is stored, and who is allowed to sign are outside the model. Signing determines where trust gets **moved to**; it does not eliminate it. The chain also connects to the previous lesson's traceability ladder. The source version field says which source the image came from; the signature says that declaration did not change after it was produced. If the field is not signed, it is a note that can be written after the fact; without a signature, the traceability rate measures a record discipline, not a guarantee. #### Summary - A layer is an immutable object; deletion changes the view, not the bytes. In the measurement, the union view shows 6 files and 27,136 bytes while there are 7 files and 28,834 bytes on disk. - The gap is 1,698 bytes and is the whole of the key file that appears deleted in the upper layer. The second secret, removed before the same layer closed, appears in no layer file at all: 1,698 bytes against 0 bytes. - If a layer carrying a secret is referenced, it stays in the registry even if the image carrying the secret is deleted; a difference taken into the output cannot be taken back out of it. - Scanning is wrong in both directions: in the full-featured base, 29 of 31 open defects are caught, 2 are missed, and 8 of 37 warnings are false positives. In the single-binary base, the scan produces 0 warnings but there is 1 open defect — a component that cannot be counted goes uncounted, and nothing marks the gap. - Signing is cheap: 1,088 bytes for 17 manifests, 6.2 per thousand of the registry. With no verification, 24 of 24 pulls accept a forged manifest; with only the signature verified, 96 of 96 layers get accepted with their digest untested. A signature does not eliminate trust, it moves it to the key. #### Next Step Across this topic, the output got smaller, split into layers, shared, named, scanned, and signed. Every number measured — bytes, layers, references, warnings, signatures — was measured on an object **at rest**. The object sat on disk and did nothing. Yet this output's reason to exist is to run. The moment it starts running, the files inside it become files a process sees, the layers stack up and turn into a writable surface, and what was verified by the signature gets loaded into memory. Which part of isolation survives this transition? Are the layers measured as read-only still read-only, how much of the difference taken into the image comes back from the environment the moment it runs, and what now determines what a process is allowed to do? ## Runtime What isolation becomes at run time: run options through resource limits and environment variables, the ephemeral filesystem losing data, persistence through volumes and bind mounts, network drivers and service naming, the risk of running as the root user, a multi-container local environment through declarative compose files, hot reload and debugging's effect on developer experience, and runtime security through capability restriction and a read-only filesystem. ### Run Options Source: https://academia.sh/en/courses/containers/run-options The two settings a container receives from outside: the cost of the throttle, kill, and degrade decisions when a resource limit is exceeded, measured in latency, lost work, and restarts; how a too-tight limit lands on the container's own work and a too-loose limit lands on the neighboring container; the difference environment variables leave outside the image; and how many variables being left undefined brings the service down. The previous topic measured the output **at rest**: how many layers, how many bytes, which digest, which signature. All of those measurements were over a set of files that was never running. The moment it starts running, what gets measured changes. Part of the isolation persists — the filesystem is still what the image says, the library version is still frozen. Part of it drops away: CPU, memory, and configuration now come not from inside the image, but from **outside** it. This lesson counts the first two of the parts that drop away: the **resource limit** given to the container and its **environment variables**. What they have in common is this — neither is written inside the image. The same image shows two different behaviors under two different limits; the same image never comes up at all with a missing variable. From the isolation budget's perspective, this is a gap the image cannot close, and its size can be measured. The measurements run through a fictional **regional measurement network**: software that collects readings from water meters, verifies them, turns them into invoices, and opens work orders for field crews. The system is fictional; the numbers come from the run below. #### The Decision Made When a Limit Is Exceeded A resource limit requires something to be **done** once it is exceeded. The limit itself is a number; the decision is made once that number is hit, and there are three choices: throttle the work, kill the container, degrade the work. The run below is a model of a **control group** — it splits the CPU share the machine distributes per period between two containers and counts the cost of the three decisions separately. - **RT1** — The measurement network's four services run on a single machine, as four separate containers. - **RT2** — The machine's capacity is normalized to 100 shares per period; share and period are abstract units, not seconds or cores. - **RT3** — The verifier requests 40-50 shares every period; the nightly batch job starts at period 60 and ends at period 140, requesting 60-80 shares in that window. - **RT4** — The kill decision is made when accumulated work exceeds 300 units; the restart takes 5 periods, and work arriving during those periods is lost. - **RT5** — The degrade decision keeps 0.6 of the work that exceeds the limit and drops the rest. ```js // runtime/control-group.mjs — the cost of the decision made when a limit is exceeded. // MODEL: period is an abstract step, share an abstract unit; demand is drawn from a seeded generator. const PERIOD = 200, MACHINE = 100; // RT2: machine capacity is 100 shares/period const START = 60, END = 140, SEED = 20260801; // RT3: the nightly job's window const QUEUE_LIMIT = 300, RESTART = 5; // RT4: kill threshold and restart const DEGRADE = 0.6; // RT5: share kept on degrade const rng = (t) => { let s = t % 2147483647; return () => (s = (s * 48271) % 2147483647) / 2147483647; }; const demand = (r, t) => ({ verifier: 40 + Math.round(r() * 10), nightly: t >= START && t < END ? 60 + Math.round(r() * 20) : 4, }); function run({ decision, limit }) { const r = rng(SEED); const names = ["verifier", "nightly"]; const d = Object.fromEntries(names.map((a) => [a, { queue: 0, done: 0, lost: 0, wait: 0, killed: 0, down: 0, own: 0, machine: 0 }])); for (let t = 0; t < PERIOD; t += 1) { const incoming = demand(r, t), request = {}, share = {}; for (const a of names) { if (d[a].down > 0) { d[a].down -= 1; d[a].lost += incoming[a]; request[a] = 0; share[a] = 0; continue; } request[a] = incoming[a] + d[a].queue; share[a] = Math.min(request[a], limit[a]); // clipping at own limit d[a].own += request[a] - share[a]; } const total = share.verifier + share.nightly; for (const a of names) { const granted = total > MACHINE ? Math.floor((MACHINE * share[a]) / total) : share[a]; d[a].machine += share[a] - granted; // clipping at the machine: the part that lands on the neighbor d[a].done += granted; if (d[a].down > 0) continue; let remainder = request[a] - granted; if (decision === "degrade") { const kept = Math.round(remainder * DEGRADE); d[a].lost += remainder - kept; remainder = kept; } d[a].queue = remainder; if (decision === "kill" && d[a].queue > QUEUE_LIMIT) { d[a].killed += 1; d[a].lost += d[a].queue; d[a].queue = 0; d[a].down = RESTART; } d[a].wait += d[a].queue; } } return d; } const fmt = (x, n = 2) => x.toFixed(n); const LIMIT = { verifier: 50, nightly: 60 }; console.log(`model: ${PERIOD} periods, machine ${MACHINE} shares/period, limits ${LIMIT.verifier}+${LIMIT.nightly}=` + `${LIMIT.verifier + LIMIT.nightly} (over-commitment), nightly work periods ${START}-${END}, seed ${SEED}`); console.log(`\n${"decision".padEnd(11)}${"work".padEnd(13)}${"done".padStart(9)}${"lost".padStart(8)}` + `${"avg wait".padStart(14)}${"kills".padStart(9)}${"down periods".padStart(14)}`); const results = {}; for (const decision of ["throttle", "kill", "degrade"]) { const k = run({ decision, limit: LIMIT }); results[decision] = k; for (const a of ["verifier", "nightly"]) console.log(`${decision.padEnd(11)}${a.padEnd(13)}${String(k[a].done).padStart(9)}${String(k[a].lost).padStart(8)}` + `${fmt(k[a].wait / k[a].done).padStart(14)}${String(k[a].killed).padStart(9)}${String(k[a].killed * RESTART).padStart(14)}`); } console.log(`\n${"nightly limit".padEnd(15)}${"nightly done".padStart(15)}${"nightly wait".padStart(17)}` + `${"verifier done".padStart(19)}${"verifier wait".padStart(21)}${"machine clip".padStart(17)}`); for (const lim of [20, 40, 60, 80, 100]) { const k = run({ decision: "throttle", limit: { verifier: 50, nightly: lim } }); console.log(`${String(lim).padEnd(15)}${String(k.nightly.done).padStart(15)}${fmt(k.nightly.wait / k.nightly.done).padStart(17)}` + `${String(k.verifier.done).padStart(19)}${fmt(k.verifier.wait / k.verifier.done).padStart(21)}` + `${String(k.verifier.machine + k.nightly.machine).padStart(17)}`); } ``` ``` model: 200 periods, machine 100 shares/period, limits 50+60=110 (over-commitment), nightly work periods 60-140, seed 20260801 decision work done lost avg wait kills down periods throttle verifier 9012 0 0.12 0 0 throttle nightly 6092 0 10.54 0 0 kill verifier 9012 0 0.06 0 0 kill nightly 4124 1968 2.15 3 15 degrade verifier 8891 121 0.02 0 0 degrade nightly 4986 1106 0.33 0 0 nightly limit nightly done nightly wait verifier done verifier wait machine clip 20 3040 122.72 9012 0.00 0 40 5840 30.08 9012 0.00 0 60 6092 10.54 9012 0.12 1033 80 6092 5.48 8625 6.00 2817 100 6092 2.49 8306 10.38 4215 ``` These numbers are in the **measurement** class and depend on the seed. #### Three Decisions, Three Separate Costs The nightly job's total demand is 6092 units in all three runs; the decisions distribute this 6092 to different places, and the total is conserved. **Throttle** loses no work: 0 lost, 6092 completed. Its cost is entirely time — every completed unit waits an average of **10.54 periods** in the queue. The verifier's wait in the same run is 0.12 periods; the throttle decision's cost is paid almost entirely by the work actually hitting the limit. **Kill** brings the wait down to 2.15 periods, because the queue is wiped once it exceeds 300 units. This drop is an accounting trick: the queue got shorter because the work inside it was thrown away. The cost shows up in two entries — **1968 units lost** (32% of demand) and **3 kills**, each with 5 periods down, for **15 periods** total with no work done. Completed work drops from 6092 to 4124. **Degrade** sits between the two, and its numbers are the strangest of all: wait drops to 0.33 periods, loss stays at 1106 units, no kills. Because 40% of the work exceeding the limit is dropped, the queue never grows. But the number that stands out is not the nightly job's — it is the **verifier's** loss: 121 units. The verifier never once hit its own 50-share limit; what clips it is the machine. Because the degrade decision applies to every container, 40% of what the machine clips is also dropped for the verifier. The decision is made in one container; its result shows up in two. The choice depends on the data class. For a reading going into an invoice, a loss of 1968 units is unacceptable; throttle is the only option, at a price of 10.54 periods of delay. For a summary feeding a trend chart, degrade is cheapest: wait drops from 10.54 to 0.33 for a loss of 1106 units. Kill is not a decision anyone **chooses**; it is the decision made at a memory limit, and not being chosen does not change its cost. #### Sweeping the Limit: The Tight End, the Loose End The second table runs the same work under five different limits and shows that the choice of limit loads cost onto two separate directions. **At the tight end**, the cost lands on the container's own work. At a limit of 20 shares, the nightly job finishes only 3040 units over 200 periods — not even half of demand; when the run ends, **3052 units are still in the queue**, and wait per unit climbs to 122.72 periods. At this limit the nightly job never finishes, it carries over to the next day. The neighbor, by contrast, is untouched: verifier wait 0.00, machine clipping 0. **A tight limit builds isolation in full and bills its own container.** **At the loose end**, the cost changes location. At a limit of 60, the nightly job already finishes all of its work: 6092. Raising the limit to 80, then 100, **does not increase** completed work at all — 6092 in all three rows. The only thing that changes is wait: 10.54 → 5.48 → 2.49. The gain from loosening is only in latency, and the neighbor pays for it. The verifier's completed work drops from 9012 to 8306 (706 units short), and its wait climbs from 0.12 to 10.38 periods. Machine clipping climbs from 1033 to 4215 units. This is the most misunderstood side of a resource limit, and the table lays it bare: **a limit is a ceiling, not a guarantee.** The limits of 50 and 100 sum to 150, and the machine can only give 100. Once the total is exceeded, what decides who gets how much is not the container's limit but the distributor — and it sits outside the container. This is where isolation is punctured: the namespace separates processes, the limit sets a ceiling, but **the machine itself continues to be shared**, and under over-commitment the neighbor's delay can climb to 86 times its baseline. #### The Difference That Stays in the Environment The second setting item is environment variables. The image pulled the filesystem difference inside itself: installed packages, library versions, file paths were all frozen. What does not freeze is the values that **let the same image behave differently in different places** — the data endpoint, the password, the tariff table, the time zone. These sit outside the image because, if they went into it, the image would have to be separate for every place. - **RT6** — The four services' environment variable inventory is fictional; the names and required classes are model data. The missing-definition check really runs. ```js // runtime/environment-variables.mjs — counting the difference that cannot be pulled into the image. // MODEL: the four services' environment variable inventory is fictional; the missing-definition check really runs. const SERVICE = { // RT6: fictional inventory collector: { required: ["METER_NETWORK_ENDPOINT", "READING_QUEUE", "COLLECTION_INTERVAL"], defaulted: { LOG_LEVEL: "info", CONCURRENCY: "4" } }, verifier: { required: ["READING_QUEUE", "RULE_VERSION", "DATA_ENDPOINT", "DATA_PASSWORD"], defaulted: { DEVIATION_THRESHOLD: "0.15", LOG_LEVEL: "info" } }, billing: { required: ["DATA_ENDPOINT", "DATA_PASSWORD", "TARIFF_TABLE", "TIME_ZONE"], defaulted: { ROUNDING: "2", LOG_LEVEL: "info" } }, workOrder: { required: ["DATA_ENDPOINT", "DATA_PASSWORD", "FIELD_ENDPOINT"], defaulted: { RETRY_COUNT: "3" } }, }; function start(name, env) { // service does not come up if a required definition is missing const s = SERVICE[name], missing = s.required.filter((k) => env[k] === undefined); if (missing.length) throw new Error(`${name}: undefined ${missing.join(", ")}`); return { ...s.defaulted, ...Object.fromEntries(s.required.map((k) => [k, env[k]])) }; } const names = Object.keys(SERVICE); const required = new Set(names.flatMap((a) => SERVICE[a].required)); const defaulted = new Set(names.flatMap((a) => Object.keys(SERVICE[a].defaulted))); const all = new Set([...required, ...defaulted]); const defs = names.reduce((t, a) => t + SERVICE[a].required.length + Object.keys(SERVICE[a].defaulted).length, 0); const defaultOnly = [...defaulted].filter((k) => !required.has(k)).length; console.log(`model: ${names.length} services, ${all.size} distinct variable names, ${defs} definition lines; ` + `required ${required.size} names, default-only ${defaultOnly} names`); const filled = Object.fromEntries([...all].map((k) => [k, "value"])); const countFailing = (env) => names.filter((a) => { try { start(a, env); return false; } catch { return true; } }); console.log(`services failing to start with empty env: ${countFailing({}).length}/${names.length}; ` + `failing to start with full env: ${countFailing(filled).length}/${names.length}`); try { start("billing", { DATA_ENDPOINT: "x", DATA_PASSWORD: "y" }); } catch (e) { console.log(`example: ${e.message}`); } console.log(`\n${"variable".padEnd(24)}${"in services".padStart(14)}${"drops when removed".padStart(19)}${"class".padStart(15)}`); let dropped = 0, fragile = 0; for (const k of [...all].sort()) { const count = names.filter((a) => SERVICE[a].required.includes(k) || k in SERVICE[a].defaulted).length; const d = countFailing(Object.fromEntries(Object.entries(filled).filter(([x]) => x !== k))).length; dropped += d; if (d > 0) fragile += 1; console.log(`${k.padEnd(24)}${String(count).padStart(14)}${String(d).padStart(19)}${(required.has(k) ? "required" : "defaulted").padStart(15)}`); } console.log(`\ntotal: ${fragile} of the ${all.size} variables individually bring down at least one service; ` + `the sum of one-at-a-time removals is ${dropped} service-drops. The output did not change, ${all.size} names stayed in the environment.`); ``` ``` model: 4 services, 14 distinct variable names, 21 definition lines; required 9 names, default-only 5 names services failing to start with empty env: 4/4; failing to start with full env: 0/4 example: billing: undefined TARIFF_TABLE, TIME_ZONE variable in services drops when removed class COLLECTION_INTERVAL 1 1 required CONCURRENCY 1 0 defaulted DATA_ENDPOINT 3 3 required DATA_PASSWORD 3 3 required DEVIATION_THRESHOLD 1 0 defaulted FIELD_ENDPOINT 1 1 required LOG_LEVEL 3 0 defaulted METER_NETWORK_ENDPOINT 1 1 required READING_QUEUE 2 2 required RETRY_COUNT 1 0 defaulted ROUNDING 1 0 defaulted RULE_VERSION 1 1 required TARIFF_TABLE 1 1 required TIME_ZONE 1 1 required total: 9 of the 14 variables individually bring down at least one service; the sum of one-at-a-time removals is 14 service-drops. The output did not change, 14 names stayed in the environment. ``` The number of the difference that cannot be pulled out of the environment reads here: **14 variable names**, 21 definition lines. File paths and library versions were pulled into the output; these 14 names were not. Nine are required, and the service never comes up if they are left undefined — with an empty environment, all four services go down. Five have a default, and removing them drops no service; they carry the difference but produce no fragility. The right-hand column shows where the fragility is concentrated. `DATA_ENDPOINT` and `DATA_PASSWORD` are required in three services at once; forgetting a single definition keeps three containers from coming up. These two are 14% of the 14 names, but alone they produce 6 of the 14 service-drops in the one-at-a-time removal sum — 43% of it. `TIME_ZONE` is another class: the image pulled in the time zone data but could not settle **which time zone to operate in**, because that decision belongs to the run, not the image. For billing, this one variable determines where a day boundary falls. Here the cost of isolation is measured in writing: 21 definition lines, once per run site. What it buys is the same image running with four different behaviors in four different places. #### Summary - One of three decisions is made when a resource limit is exceeded, and all three distribute the same 6092 units of work to different places: throttle keeps loss at 0 but loads 10.54 periods of delay per unit; kill brings the delay down to 2.15 periods but loses 1968 units and spends 15 periods down; degrade brings delay down to 0.33 periods in exchange for a loss of 1106 units. Kill's short queue is not a gain: the queue got short because the work inside it was deleted and the container restarted 3 times. - The degrade decision's result shows up in two containers: while the nightly job loses 1106 units, the verifier — which never once hit its own limit — also loses 121 units, because what clips it is the machine, not the container's own limit. - A tight limit bills its own container: at a limit of 20 shares, the nightly job finishes only 3040 units over 200 periods, 3052 units stay in the queue, and wait climbs to 122.72 periods; the neighbor's wait stays at 0.00. - A loose limit bills the neighbor: raising the limit from 60 to 100 does not increase the 6092 units the nightly job finishes at all, it only brings its wait down to 2.49; in exchange, the verifier finishes 706 units short and its wait climbs from 0.12 to 10.38 periods. - **Where isolation is punctured:** a limit is a ceiling, not a guarantee. When the sum of the limits exceeds the machine's capacity, the distribution decision is made outside the container; machine clipping climbs from 1033 to 4215 units. - The image pulled the filesystem difference inside itself; it could not pull in 14 variable names. Nine of them are required, and with an empty environment all four of the four services fail to come up; two of them, being required in three services at once, produce 6 of the 14 service-drops in the one-at-a-time removal sum. #### Next Step Both the resource limit and the environment variables were settings given from **outside** the container; both were known before the container started, and neither changed while the container ran. But a running container does not sit idle: it writes logs, fills a cache, saves the state of an unfinished batch job somewhere, puts a downloaded file in a directory. This data was not in the image, and it did not come from the environment either — the container itself produced it. So where does it get written, and what happens to it when the container stops? The next lesson builds the writable layer with real directories and counts this: how many bytes accumulate over the course of a run, how many records go away when the container is deleted, why a one-byte change does not take up one byte of space, and which data class this loss is acceptable in. ### Ephemeral Filesystem Source: https://academia.sh/en/courses/containers/ephemeral-filesystem The fate of the data a container produces while running: building the writable layer with real directories, the distribution across data classes of the bytes accumulating over a thirty-night run, a one-byte change taking up an entire file's worth of space through copy-up, the number of records lost when the container is deleted, and which data class that loss is acceptable in. The previous lesson measured two settings given to the container **from outside**: the resource limit and the environment variables. Both were known before the container started. But a running container does not sit idle. The verifier writes a log line for every reading, puts the tariff value it computed into a cache, records where the nightly run has gotten to somewhere, appends the operator's hand-entered correction to a file. None of this data was in the image, and none of it came from the environment either — the container itself produced it. The question is: where did it get written, how much space did it take up, and what happened to it once the container went away. The answer is isolation's quietest cost, because it throws no error and exceeds no limit. Data is written, read, used for a while, and then one day it is gone. #### The Writable Layer The image is read-only, and this is not a preference but a consequence of the layer model: layers are addressed by content digest, if the content changes the digest changes, and if the digest changes the layer becomes a different layer. When a container needs to write, a **writable layer** is added on top of this stack. Reads resolve from top to bottom; writes always land on top. The run below builds this arrangement with real directories. The union filesystem is a **model** — the read resolution and copy-up rule are hand-written — but the directories, files, and byte counts are real; all of it is written to disk and measured with `node:fs`. - **RT7** — The image's lower layers consist of four files: application code, rule set, tariff table, meter list. The sizes are fictional; the files are really written. - **RT8** — The nightly run processes 480 meter readings. - **RT9** — Every reading produces a log line; every fourth reading a cache write; every sixtieth reading a progress write; every fortieth an operator correction. - **RT10** — At the end of every night, a one-byte change is made to the tariff table. - **RT11** — The container runs for 30 nights without ever restarting, then is deleted. Deletion removes the writable layer and does not touch the lower layers. ```js // runtime/writable-layer.mjs — counting what accumulates in the writable layer and what leaves when the container is deleted. // MODEL: the union filesystem is modeled with node; directories, files, and byte counts are real. import { mkdirSync, writeFileSync, appendFileSync, copyFileSync, existsSync, statSync, readdirSync, rmSync } from "node:fs"; import { join, dirname } from "node:path"; const LOWER = "root/lower", UPPER = "root/upper"; // read-only image layers / writable layer const READINGS = 480, NIGHTS = 30, SEED = 20260801; // RT8: nightly run, RT11: container lifetime rmSync("root", { recursive: true, force: true }); const put = (root, path, content) => { mkdirSync(join(root, dirname(path)), { recursive: true }); writeFileSync(join(root, path), content); }; for (const [path, n] of [["app/verifier.js", 12288], ["app/rules.json", 49152], ["config/tariff.csv", 98304], ["data/meter-list.csv", 204800]]) put(LOWER, path, "x".repeat(n)); let copiedBytes = 0, copiedCount = 0; function appendToUpper(path, line) { // absent in upper, present in lower: copy-up const u = join(UPPER, path); if (!existsSync(u) && existsSync(join(LOWER, path))) { mkdirSync(dirname(u), { recursive: true }); copyFileSync(join(LOWER, path), u); copiedBytes += statSync(u).size; copiedCount += 1; } mkdirSync(dirname(u), { recursive: true }); appendFileSync(u, line); } let s = SEED % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const counts = { log: 0, cache: 0, state: 0, user: 0, config: 0 }; function nightlyRun() { for (let i = 1; i <= READINGS; i += 1) { const value = (100 + rand() * 900).toFixed(3); appendToUpper("log/verification.log", `reading ${i} meter ${1000 + i} value ${value} status ok\n`); counts.log += 1; if (i % 4 === 0) { put(UPPER, `cache/tariff-${i / 4}.json`, `{"meter":${1000 + i},"unit":"${value}"}`); counts.cache += 1; } if (i % 60 === 0) { put(UPPER, "state/progress.json", `{"processed":${i},"remaining":${READINGS - i}}`); counts.state += 1; } if (i % 40 === 0) { appendToUpper("data/corrections.csv", `${1000 + i},operator,${value}\n`); counts.user += 1; } } appendToUpper("config/tariff.csv", "#"); // one-byte change, whole file gets copied counts.config += 1; } const measure = (root, on = "") => { let b = 0, n = 0; const walk = (d) => { for (const e of readdirSync(d, { withFileTypes: true })) { const p = join(d, e.name); if (e.isDirectory()) walk(p); else if (p.includes(on)) { b += statSync(p).size; n += 1; } } }; if (existsSync(root)) walk(root); return { bytes: b, files: n }; }; const a = measure(LOWER); const PREFIX = { log: "log/", cache: "cache/", state: "state/", user: "data/", config: "config/" }; console.log(`model: ${READINGS} readings/night, ${NIGHTS} nights (seed ${SEED}); lower layers ` + `${a.bytes} bytes / ${a.files} files, all read-only`); console.log(`\n${"night".padEnd(6)}${"total bytes".padStart(12)}${"files".padStart(7)}` + Object.keys(PREFIX).map((k) => k.padStart(11)).join("") + `${"ratio to lower".padStart(19)}`); for (let g = 1; g <= NIGHTS; g += 1) { nightlyRun(); if (![1, 2, 7, 30].includes(g)) continue; const u = measure(UPPER); console.log(`${String(g).padEnd(6)}${String(u.bytes).padStart(12)}${String(u.files).padStart(7)}` + Object.values(PREFIX).map((o) => String(measure(UPPER, o).bytes).padStart(11)).join("") + `${(u.bytes / a.bytes).toFixed(2).padStart(19)}`); } const u = measure(UPPER); console.log(`\ncopy-up ${copiedCount} files, ${copiedBytes} bytes; share of the layer first night ` + `${((100 * (copiedBytes + 1)) / 124772).toFixed(1)}%, night thirty ${((100 * copiedBytes) / u.bytes).toFixed(1)}%`); rmSync(UPPER, { recursive: true, force: true }); // container is deleted console.log(`container deleted: remaining ${measure(LOWER).bytes} bytes / ${measure(LOWER).files} files (lower layers), ` + `gone ${u.bytes} bytes / ${u.files} files`); console.log(`records gone: ${counts.log} log lines, ${counts.cache} cache writes, ` + `${counts.state} progress writes, ${counts.user} operator corrections, ${counts.config} config changes`); console.log(`if two containers ran from the same image: shared ${a.bytes + 2 * u.bytes} bytes, ` + `unshared ${2 * (a.bytes + u.bytes)} bytes; lower-layer sharing saves ${a.bytes} bytes`); rmSync("root", { recursive: true, force: true }); ``` ``` model: 480 readings/night, 30 nights (seed 20260801); lower layers 364544 bytes / 4 files, all read-only night total bytes files log cache state user config ratio to lower 1 124772 124 22452 3720 31 264 98305 0.34 2 147489 124 44904 3720 31 528 98306 0.40 7 261074 124 157164 3720 31 1848 98311 0.72 30 783565 124 673560 3720 31 7920 98334 2.15 copy-up 1 files, 98304 bytes; share of the layer first night 78.8%, night thirty 12.5% container deleted: remaining 364544 bytes / 4 files (lower layers), gone 783565 bytes / 124 files records gone: 14400 log lines, 3600 cache writes, 240 progress writes, 360 operator corrections, 30 config changes if two containers ran from the same image: shared 1931674 bytes, unshared 2296218 bytes; lower-layer sharing saves 364544 bytes ``` These numbers are in the **measurement** class. #### Copy-Up: One Byte Costing 98,304 Bytes The first row looks wrong at first glance: the first night's `config` column is 98,305 bytes. What was changed was a single byte. The reason lies in the write rule. The writable layer **cannot modify** a file in the lower layer, because the lower layer is read-only and addressed by its digest. If a lower-layer file needs to be written to, the whole file is first copied to the upper layer, and then the copy is modified. This is called **copy-up**, and its cost is measured not by the size of the change but by the size of **the file**. The first night's makeup is skewed as a result. The genuine new data the run produces is 26,468 bytes — what is left once the 98,304 bytes copied are subtracted from the layer's total. What sits in the layer is 124,772 bytes, that is, **4.7 times** as much. The entire difference is a single copied file, and it fills 78.8% of the layer that night. The cost of isolation is stark here: what giving up the right to modify a file in place costs is 98,304 bytes of space and that much copying work, for a one-byte change. The cost is paid once. The second night's `config` column is 98,306: the file is now in the upper layer, the second byte is appended to its end, and nothing gets copied. The practical consequence is this — in the writable layer, **which** file is touched for the first time is more expensive than how much gets written to it afterward. #### Over Thirty Nights, the Layer's Shape Changes The layer's makeup inverts as the table's rows go down. On the thirtieth night, the total is 783,565 bytes; the `config` column still sits at 98,334, and its share has dropped from 78.8% to 12.5%. Taking its place is the `log` column: 673,560 bytes, 86% of the layer. Log grows linearly because it adds 22,452 bytes every night; copy-up, on the other hand, stays constant because it is paid once. The right-hand column gives the real fact. The writable layer's ratio to the lower layers is 0.34 on the first night and climbs to **2.15** by the thirtieth. By the end of thirty nights, more than two-thirds of the space the container holds on disk is not the image's — it is data the run produced. The phrase "the container is light" has a number behind it here, and the number that is true on day one is false on day thirty. Two columns do not grow at all, and their reasons are different. `cache` stays fixed at 3,720 bytes, because the same 120 tariff keys are overwritten every night. `state` stays fixed at 31 bytes, because the progress file is overwritten on every write — of the 240 writes, only the last one remains. This is also why the file count stays at 124 over the thirty nights: the run does not open new files, it grows the ones that already exist. #### What Leaves When the Container Is Deleted After deletion, two numbers remain: 364,544 bytes / 4 files stand in the lower layers, and 783,565 bytes / 124 files left the upper layer. The lower layer is untouched — the image is whatever it is. What left is what the container **produced itself**, and its breakdown by record is: 14,400 log lines, 3,600 cache writes, 240 progress writes, 360 operator corrections, 30 config changes. The gap between record counts and byte counts is itself informative. The cache was written 3,600 times but left 120 files and 3,720 bytes in the layer: 97% of the writes were overwritten by a later write. The writable layer is not a log, it is a **snapshot** — it holds not what was written, but what was written last. This is why what gets deleted is not the sum of what was written either, but its final state. The progress row should also be read separately: 31 bytes sat in the layer, but the write count was 240. The information on how far the nightly run had gotten was overwritten every time and went away with the container. A new container starts that night from zero; all 480 readings are processed again. Which class of loss is acceptable is the measurement network's operating rule (**RT12**) and, with the thirty-night numbers, gives the table below: | Data class | Records | Bytes | Is loss acceptable | Rationale | |---|---|---|---|---| | Log | 14,400 lines | 673,560 | Conditional | Acceptable if lines are shipped out; if the only copy is on the container, an incident review goes with it too | | Cache | 3,600 writes | 3,720 | Yes | Reproducible; the cost is slower initial requests | | Job state | 240 writes | 31 | No | An unfinished run starts over, 480 readings get reprocessed | | User data | 360 corrections | 7,920 | No | Not reproducible; the operator's hand entry is the only source | | Config copy | 30 changes | 98,334 | Yes | Comes back from the image or the config source | The table reads backward: the two unacceptable classes hold only 1% of the layer, 7,951 bytes. The remaining 99% is reproducible data. The problem is not size, it is **mixture**: 7,951 bytes of unrecoverable data sit inside 775,614 bytes of recoverable data, and both get deleted at the same instant. #### The Difference Removed from the Environment This loss also has a counterpart, and the course's measure requires asking about it: what difference does the ephemeral layer remove from the environment. The difference it removes is **cross-run residue**. A container deleted and recreated every night runs every night from the same 364,544-byte, four-file starting point; its writable layer never exceeds 124,772 bytes, and the difference between two nights' starting states is exactly zero bytes. A half-finished log, a bloated cache, the previous night's hand-corrected config do not carry over to the next night. This is the measure by which a container counts as "stateless": zero difference between runs. The container that lives thirty nights does not remove this difference. It walks into the thirtieth night with its layer already 2.15 times the lower layers, and that night's behavior depends on the residue of the previous twenty-nine nights. Where a problem needs to be looked for is no longer just the image, it is that layer's contents too — in other words, the difference thought to have been removed from the environment has come back. The cost is the two rows in the table. Whatever brings the cross-run difference to zero also deletes the 240 progress writes and the 360 operator corrections at the same time. **Statelessness is not free: the price of zeroing the cross-run residue is also zeroing the 7,951 bytes of unrecoverable data inside that residue.** This is exactly the problem the next lesson solves. #### Where Isolation Is Punctured There are two holes, and one of them is deliberate. The deliberate one is lower-layer sharing. If two containers run from the same image, total disk usage is not 2,296,218 bytes, it is 1,931,674 bytes; the lower layers' 364,544 bytes are held once and visible to both containers at the same time. Isolation is deliberately punctured here, because without the hole the same content would be stored twice. Because it is read-only, sharing has no effect on behavior — but the source is single. The second hole is unintentional, and its name comes from a misunderstanding. **Ephemerality is not a filesystem property, it is a lifecycle promise.** The writable layer is not in memory; it is a directory on the host machine's disk. When the container is *stopped*, that directory keeps standing; 783,565 bytes stay in place, and they come back if the container is restarted. Data goes away only when the container is **deleted**. This has two consequences: a stopped container still holds on disk the data it is assumed to have lost; and a container that is never deleted grows the layer it assumes is temporary without limit — to 2.15 times over thirty nights. #### Summary - The writable layer is a separate unit added on top of the read-only lower layers; reads resolve from top to bottom, writes always land on top. - **Copy-up cost is measured by file size:** a one-byte change in the tariff table caused 98,304 bytes to be copied and filled 78.8% of the layer on the first night. The genuine new data produced that night is 26,468 bytes; what sits in the layer is 124,772 bytes. The cost is paid once: no copying on the second night. - Over thirty nights the layer's shape turns over: the total climbs to 783,565 bytes, the log's share rises to 86% while the copy-up share drops to 12.5%. The writable layer's ratio to the lower layers climbs from 0.34 to 2.15. - When the container is deleted, 14,400 log lines, 3,600 cache writes, 240 progress writes, 360 operator corrections, and 30 config changes go away; the lower layer's 364,544 bytes remain. - The acceptability of loss depends on the data class, and mixture is dangerous: the two unacceptable classes hold only 1% of the layer, 7,951 bytes, but they get deleted at the same instant as the remaining 775,614 bytes. - **Where isolation is punctured:** the lower layer is shared between two containers, and 364,544 bytes are held once; also, ephemerality is not a filesystem property — the writable layer is a directory on the host machine's disk, and it goes away not when the container is stopped, but when it is deleted. #### Next Step Loss was unacceptable in two classes: job state and user data. Together they hold 7,951 bytes, and both get deleted because they sit inside the writable layer. The fix is obvious — put those two files outside the layer, somewhere not tied to the container's lifecycle. But outside the layer is the host machine's filesystem, and every path opened onto it takes something away from isolation. The next lesson builds two mount forms with real directories and counts this: how many bytes remain when the container is deleted, how many files cannot be written when the container's internal user identity does not match the directory's owner, and which of the two forms breaks when the machine changes. ### Volumes and Bind Mounts Source: https://academia.sh/en/courses/containers/volumes-and-bind-mounts Moving persistent data outside the writable layer: building a managed volume and a direct directory mount with real directories, the bytes remaining when the container is deleted, the number of nodes that cannot be written when the container's internal user identity does not match the host machine's file owner, and an absolute-path-bound mount breaking when the machine changes. The previous lesson measured that loss was unacceptable in two classes: job state and user data, 7,951 bytes in total. Both went away on deletion because both sat inside the writable layer. The fix fits in one sentence — put those two files outside the layer. But outside the layer is the host machine's filesystem, and every path opened onto it spends an entry from the isolation budget. This lesson builds two mount forms with real directories and counts three things separately: bytes remaining when the container is deleted, nodes that cannot be written when the container's internal user identity does not match the directory's owner, and which form breaks when the machine changes. The third measurement also names, in this course, the place where isolation is punctured most plainly. #### Two Mount Forms The filesystem a container sees is a **mount table**: every path inside the container corresponds to somewhere on the host machine. The writable layer is itself a row in this table, the only difference being that it is deleted with the container. There are two ways to get outside the layer, and the table distinguishes the two by **how they are referred to**. A **managed volume** is referred to by name. The runtime resolves the name to a location under its own root directory; only the name sits in the config, not a path. A **bind mount** is referred to directly by an absolute path; the host directory path is written into the config, and there is no resolution step — the path is whatever it is. The two share a side effect: if the mount point lands on a directory already populated in the image, the content there becomes **invisible** for as long as the mount lasts — **mount shadowing**. The bytes in the lower layer stay put, unerased; it is just that a different directory now sits at the end of the path the container sees. A default setting baked into the image thus silently disappears at run time; no error is raised, the file looks "gone". Which data gets moved out is also a budget decision. Of the five classes in lesson 02, only two were mounted; the log was left in the writable layer, and its 673,560 bytes went away again. Had it been mounted, it would not have been lost, but 673,560 bytes would have accumulated on persistent disk over thirty nights, adding the question of who deletes it. Every class moved out gains persistence, and in exchange loads a lifecycle responsibility. - **RT13** — The mount table is a model: three in-container paths correspond to three separate host machine targets. The directories and bytes are real. - **RT14** — The run from lesson 02 is repeated exactly: 30 nights, 480 readings a night, the same seed. The only thing that changes is where job state and corrections are written; the log stays in the writable layer. - **RT15** — The managed volume is referred to by the name `reading-state` under the runtime's own root directory; the bind mount is referred to by an absolute path on the host machine. - **RT16** — The user identity inside the container does not match the file owner on the host machine; this is why the permission class applied is "other". The mode bits are read from real files. - **RT17** — The mode distribution of the eight nodes is fictional; it represents the layout the field crew left behind. - **RT18** — The machine change is modeled by renaming the root directory. ```js // runtime/mounts.mjs — two mount forms: persistence, ownership conflict, portability. // MODEL: the mount table and the container's user identity are modeled; directories, bytes, and mode bits are real. import { mkdirSync, writeFileSync, appendFileSync, chmodSync, statSync, existsSync, readdirSync, renameSync, rmSync } from "node:fs"; import { join, dirname, resolve } from "node:path"; const ROOT = "root", RUNTIME = `${ROOT}/runtime`, UPPER = `${ROOT}/container/upper`, FIELD = `${ROOT}/field`; const READINGS = 480, NIGHTS = 30, SEED = 20260801; // same run as lesson 02 rmSync(ROOT, { recursive: true, force: true }); rmSync("root-b", { recursive: true, force: true }); const MOUNTS = { // RT13: in-container path -> host machine "/log": { kind: "writable layer", target: `${UPPER}/log` }, "/data/state": { kind: "managed volume", name: "reading-state", target: `${RUNTIME}/volumes/reading-state` }, "/data/corrections": { kind: "direct mount", hostPath: resolve(`${FIELD}/corrections`), target: `${FIELD}/corrections` }, }; const write = (mountPath, file, content, append) => { const p = join(MOUNTS[mountPath].target, file); mkdirSync(dirname(p), { recursive: true }); (append ? appendFileSync : writeFileSync)(p, content); }; let s = SEED % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; for (let g = 1; g <= NIGHTS; g += 1) for (let i = 1; i <= READINGS; i += 1) { const value = (100 + rand() * 900).toFixed(3); write("/log", "verification.log", `reading ${i} meter ${1000 + i} value ${value} status ok\n`, true); if (i % 60 === 0) write("/data/state", "progress.json", `{"processed":${i},"remaining":${READINGS - i}}`, false); if (i % 40 === 0) write("/data/corrections", "corrections.csv", `${1000 + i},operator,${value}\n`, true); } const measure = (k) => { let b = 0, n = 0; const walk = (d) => { for (const e of readdirSync(d, { withFileTypes: true })) { const p = join(d, e.name); if (e.isDirectory()) walk(p); else { b += statSync(p).size; n += 1; } } }; if (existsSync(k)) walk(k); return { bytes: b, files: n }; }; const before = Object.fromEntries(Object.entries(MOUNTS).map(([mountPath, t]) => [mountPath, measure(t.target)])); rmSync(`${ROOT}/container`, { recursive: true, force: true }); // container is deleted console.log(`${"mount".padEnd(22)}${"kind".padEnd(21)}${"before delete".padStart(14)}${"after delete".padStart(16)}`); let remaining = 0, gone = 0; for (const [mountPath, t] of Object.entries(MOUNTS)) { const after = measure(t.target).bytes; remaining += after; gone += before[mountPath].bytes - after; console.log(`${mountPath.padEnd(22)}${t.kind.padEnd(21)}${String(before[mountPath].bytes).padStart(14)}${String(after).padStart(16)}`); } console.log(`container deleted: gone ${gone} bytes, remaining ${remaining} bytes (lesson 02 had 0 bytes remaining)`); // RT16: the identity inside the container does not match the host machine's owner; the applied class is "other". const NODES = [[`${RUNTIME}/volumes/reading-state`, 0o755, "d"], [`${RUNTIME}/volumes/reading-state/progress.json`, 0o644, "f"], [`${RUNTIME}/volumes/reading-state/lock`, 0o644, "f"], [`${FIELD}/corrections`, 0o775, "d"], [`${FIELD}/corrections/corrections.csv`, 0o664, "f"], [`${FIELD}/archive/2026-07.csv`, 0o600, "f"], [`${FIELD}/share`, 0o777, "d"], [`${FIELD}/share/transfer.csv`, 0o666, "f"]]; for (const [p, m, kind] of NODES) { if (kind === "d") mkdirSync(p, { recursive: true }); else if (!existsSync(p)) { mkdirSync(dirname(p), { recursive: true }); writeFileSync(p, "x"); } chmodSync(p, m); } console.log(`\n${"node".padEnd(46)}${"mode".padStart(6)}${"owner writable".padStart(17)}${"other writable".padStart(17)}`); let ownerW = 0, otherW = 0; for (const [p] of NODES) { const m = statSync(p).mode & 0o777; const s1 = (m & 0o200) !== 0, s2 = (m & 0o002) !== 0; ownerW += s1 ? 1 : 0; otherW += s2 ? 1 : 0; console.log(`${p.replace(`${ROOT}/`, "").padEnd(46)}${m.toString(8).padStart(6)}` + `${(s1 ? "yes" : "no").padStart(17)}${(s2 ? "yes" : "no").padStart(17)}`); } console.log(`when identities do not match, ${NODES.length - otherW} of ${NODES.length} nodes cannot be written; ` + `when identities are equal, ${NODES.length - ownerW} cannot be written`); renameSync(ROOT, "root-b"); // machine changed: layout is different console.log(`\n${"mount".padEnd(22)}${"reference in config".padEnd(30)}${"on new machine".padStart(14)}`); let broken = 0, absolute = 0; for (const [mountPath, t] of Object.entries(MOUNTS)) { if (t.kind === "writable layer") continue; const newPath = t.name ? join("root-b/runtime/volumes", t.name) : t.hostPath; const resolved = existsSync(newPath); if (!resolved) broken += 1; if (!t.name) absolute += 1; console.log(`${mountPath.padEnd(22)}${(t.name ? `name: ${t.name}` : "absolute path").padEnd(30)}${(resolved ? "resolved" : "broken").padStart(14)}`); } console.log(`${absolute} of the two mounts is tied to an absolute path; when the machine changed, ${broken} broke`); rmSync("root-b", { recursive: true, force: true }); ``` ``` mount kind before delete after delete /log writable layer 673560 0 /data/state managed volume 31 31 /data/corrections direct mount 7920 7920 container deleted: gone 673560 bytes, remaining 7951 bytes (lesson 02 had 0 bytes remaining) node mode owner writable other writable runtime/volumes/reading-state 755 yes no runtime/volumes/reading-state/progress.json 644 yes no runtime/volumes/reading-state/lock 644 yes no field/corrections 775 yes no field/corrections/corrections.csv 664 yes no field/archive/2026-07.csv 600 yes no field/share 777 yes yes field/share/transfer.csv 666 yes yes when identities do not match, 6 of 8 nodes cannot be written; when identities are equal, 0 cannot be written mount reference in config on new machine /data/state name: reading-state resolved /data/corrections absolute path broken 1 of the two mounts is tied to an absolute path; when the machine changed, 1 broke ``` These numbers are in the **measurement** class. #### Persistence: A Deleted Container, Bytes That Remain The first table shows that the problem is solved, and it also gives the boundary of that solution. After the container is deleted, the 673,560-byte log in the writable layer is gone; the 31 bytes in the managed volume and the 7,920 bytes in the bind mount are still there. **7,951 bytes remained in total** — exactly the previous lesson's unrecoverable number, and in that lesson this number was zero. From a persistence standpoint, there is **no difference at all** between the two forms. Both sit outside the writable layer, and neither is tied to the container's lifetime. This measure does not distinguish between them; the measures that do are the ones that follow. Here the cost of isolation is measured not in run terms but in **surface**. In the previous lesson, the container's write surface onto the host machine was a single directory, and it went away with the container. Now two more directories have been added, and both outlive the container. The container no longer starts from where it started before; its starting state depends on the 7,951 bytes sitting on disk. Statelessness, in the sense measured in lesson 02, is broken here — deliberately, and in exchange for something. The second entry in the cost is not immediately visible. There is no longer a lifecycle that deletes the remaining 7,951 bytes. The container was deleted, the layer went away, the volume and the mount stayed put; what removes them is a separate decision, not the container's death. Every directory moved outside isolation also demands its own retention rule — who deletes it, when, by what criterion. Left undecided, the measure becomes this: the container count stays constant while the number of directories sitting on disk keeps climbing, because every new name creates a new volume and nobody goes looking for the old ones. #### Ownership: Where Isolation Is Punctured The second table gives the place where isolation is most concretely punctured in containers. The user running inside the container has an identity number. A directory opened through a bind mount, though, is on the host machine's filesystem, and its files belong to a host machine user. Write permission is decided by comparing these two identities, and an **identity mismatch** between them is the default, not the exception. **The namespace separates processes, the network, and the filesystem view; it does not separate the user identity's number space.** The identity inside the container and the owner on the host machine live in the same number space, and file permission looks only at that number. In numbers: **six of the eight nodes cannot be written**. The two that can are the ones whose mode bits grant write to the "other" class — the 777 directory and the 666 file. The remaining six nodes are closed to the container even though their owner can write: 755, 644, 644, 775, 664, and 600. When the identities are made equal, the number drops to **zero**; all eight nodes become writable, because the class applied is no longer "other" but "owner". Two rows deserve a separate reading. `field/corrections/corrections.csv` is in mode 664; its owner and its **group** can write to it. This is usually assumed to be the fix — put the container user in the right group and it can write. But the group is also a number, and if the container's internal group identity does not match the host machine's, the class applied is again "other"; that row's answer in the table is `no`. `field/archive/2026-07.csv`, meanwhile, is in mode 600 and open only to its owner: a past period's archive, sitting inside the mounted directory, is both unwritable and unreadable for the container. A mount makes the directory **visible**; it does not grant access to what is inside it. This table shows both wrong fixes as well. The first is loosening permissions: the 777 and 666 rows really are writable, but those two nodes are also open to **every** host machine user; the container's write access costs write access for everyone with nothing to do with the container. The second is equating identities: the number drops to zero, but then the container behaves like a host machine user and can do anything that user can do in the mounted directory. From the isolation budget's perspective, both spend the same entry — the authority surface opened onto the host machine's filesystem. There is a third way, and its measure can be read indirectly in the table: treat the mounted directory as read-only for the container. Then the six closed nodes stop being a problem, because writing was never wanted from them in the first place. In the measurement network this distinction is clean — the tariff table and the meter list are read, job state and corrections are written. Asked how many of the eight nodes genuinely need to be written, the number drops to two, and the conflict narrows to those two as well. A managed volume does not eliminate this conflict, it only controls the **starting state**: because the volume is created by the runtime, its ownership can be set from the start to the identity the container expects. There is no such moment with a bind mount; the directory already exists, its owner is already fixed, and the container has to conform to it. #### Portability: The Cost of an Absolute Path The third table is the measure that separates the two forms. When the machine changes — the run models this by renaming the root directory — the managed volume **resolved**, and the bind mount **broke**. The reason is the kind of reference in the config. The volume is referred to by name, and the name resolves under the runtime's own root on every machine; wherever the root is, `reading-state` is found. The mount is referred to by an absolute path, and that path belongs to one machine's directory layout. One of the two mounts is tied to an absolute path, and when the machine changed **one broke** — that is, the portability ratio is one in two. The difference removed from the environment has a number here, and the table reads in two directions at once. Looked at **from inside** the container, the difference is removed entirely: the application writes to `/data/state` and `/data/corrections` on every machine, both paths are the same everywhere, both are embedded in the code, and neither changes with the machine. The difference has been pushed onto the host machine's side, onto the mount table's right-hand column, where one of the two references is tied to the machine. So the removed difference did not disappear, **it collapsed into a single row** — and that row now belongs to the config, not the application. This is not a flaw of the bind mount, it is its definition: its purpose is precisely to open a **specific** host machine directory into the container. A development setup that mounts the directory holding the source code does this deliberately; being tied to the machine is the wanted behavior. The problem is the same mount carried over into the run environment. The number says this: if a machine-independent config is wanted, only one of the two forms makes that promise. #### Summary - Both mount forms sit outside the writable layer: after the container is deleted, 31 bytes remain in the managed volume and 7,920 bytes in the bind mount; 7,951 bytes in total, which was zero in the previous lesson. The 673,560-byte log in the writable layer went away again. - The persistence measure does not distinguish the two forms; the measures that do are ownership and portability. - **Where isolation is punctured:** the namespace does not separate the user identity's number space. When the identity inside the container does not match the file owner on the host machine, six of the eight nodes cannot be written; when the identities are made equal, the number drops to zero. - Both wrong fixes spend the same entry: loosening permissions (777, 666) opens the node to everyone on the host machine, and equating identities gives the container the host machine user's authority. - A managed volume does not remove the conflict, it controls the starting state: its ownership can be set the moment the volume is created; there is no such moment with a bind mount. - The two forms diverge in portability: one of the two mounts is tied to an absolute path, and it broke when the machine changed; the volume referred to by name resolved. #### Next Step Persistent data has settled into place: job state and corrections now outlive the container, paid for with two directories opened onto the host machine's filesystem. Everything measured here was within a single container's own boundaries — its resources, its filesystem, its identity. But the measurement network is not a single program: the reading collector feeds the verifier, the verifier feeds billing, billing feeds the work order service, and the four run in separate containers. Once the filesystem is isolated, how do these four find each other, what name do they call each other by, and what happens when one is meant to be unreachable from another? The next lesson builds a name resolution and reachability matrix and counts this: how many paths close when four services are split across two networks, and how far the surface opened to the outside drops, from how many endpoints to how many. ### Container Networking Source: https://academia.sh/en/courses/containers/container-networking Containers finding each other, and failing to: which boundary a service name's resolution is bound to, the number of paths that close when four services are split across two networks, separating published ports from network-only access, the outbound surface dropping from eight endpoints to one, and network membership not being an authorization. Everything measured up to this point was within a single container's own boundaries: its resources, its environment settings, its writable layer, the directories it mounts. But the measurement network is not a single program. The reading collector feeds data to the verifier, the verifier feeds billing, billing and the work order service look at the same data, and the four run in separate containers. Once the filesystem is isolated, how do these four find each other. The interesting part of the problem is this: isolation works in two directions here. One container needs to be able to **reach** another, but it also needs to be unable to reach a third. This lesson builds reachability as a matrix and measures two numbers separately: how many paths network separation closes, and how far the surface opened to the outside drops, from how many endpoints to how many. #### A Name Instead of an Address For one service to reach another, it needs to know its address. An address is a machine name and a port number; it is tied to the machine, it does not travel, and it is part of the **environment difference** counted in lesson 01 — four of that lesson's 14 variables (`METER_NETWORK_ENDPOINT`, `READING_QUEUE`, `DATA_ENDPOINT`, `FIELD_ENDPOINT`) carried exactly this. This is the difference container networking removes. Every container that joins a network is referred to by a **service name**, and that name can be resolved from inside another container on the same network. The config no longer holds a machine name, it holds the service's name; the name is the same on every machine. The difference has been removed from the environment — but not entirely, because **whether** the name resolves depends on network membership, and membership is defined in the environment. - **RT19** — There are four services, each listening on two ports: an interface port and a metrics port. - **RT20** — The access rule is single: two endpoints reach each other only when they are members of a common network. - **RT21** — Name resolution is tied to the same boundary as access; a service name resolves only on a common network. - **RT22** — In the split layout, the front network holds the collector and the verifier; the back network holds the verifier, billing, and the work order service. The verifier is a member of both. - **RT23** — A published port opens an endpoint on the host machine's network interface; an unpublished port is reachable only from inside the network. - **RT24** — Network driver classes are modeled as membership arrangements: in the machine-shared layout, all services are in a single set and their ports sit directly on the host machine's interface; in the networkless layout, every service is alone in its own set. ```js // runtime/network-matrix.mjs — name resolution, reachability matrix, and outbound surface. // MODEL: services, networks, and ports are fictional; the access rule is defined in the model as "common network". const SERVICE = { // RT19: four services, two ports (interface + metrics) collector: { ports: [8081, 9101], nets: ["front"] }, verifier: { ports: [8082, 9102], nets: ["front", "back"] }, billing: { ports: [8083, 9103], nets: ["back"] }, workOrder: { ports: [8084, 9104], nets: ["back"] }, }; const NAMES = Object.keys(SERVICE), ALL = NAMES.flatMap((a) => SERVICE[a].ports), DEFAULT = 8080; const LAYOUT = { // RT22: membership set changes with the layout "single network": () => ["shared"], "two networks": (a) => SERVICE[a].nets, "host-shared": () => ["host"], "networkless": (a) => [a], }; const OUTSIDE = { "single network": 1, "two networks": 1, "host-shared": ALL.length, "networkless": 0 }; // RT23 const shareNet = (a, b, d) => LAYOUT[d](a).some((x) => LAYOUT[d](b).includes(x)); // RT20 function count(d) { // tally over directed pairs let open = 0, closed = 0; for (const a of NAMES) for (const b of NAMES) { if (a === b) continue; if (shareNet(a, b, d)) open += 1; else closed += 1; } return { open, closed, resolved: open, endpoints: open * 2 }; // RT21: name resolution tied to the same boundary as access } console.log(`model: ${NAMES.length} services, ${SERVICE.collector.ports.length} ports per service, ` + `${ALL.length} endpoints total; in the two-network layout front = collector + verifier, back = verifier + billing + workOrder`); console.log(`\n${"source \\ target".padEnd(16)}${NAMES.map((x) => x.padStart(13)).join("")}${"resolved".padStart(12)}`); for (const a of NAMES) { const row = NAMES.map((b) => (a === b ? "-" : shareNet(a, b, "two networks") ? "resolves" : "closed").padStart(13)).join(""); console.log(`${a.padEnd(16)}${row}` + `${String(NAMES.filter((b) => b !== a && shareNet(a, b, "two networks")).length).padStart(12)}`); } console.log(`\n${"layout".padEnd(16)}${"open paths".padStart(11)}${"closed paths".padStart(13)}` + `${"resolved".padStart(11)}${"in-network".padStart(12)}${"from outside".padStart(14)}`); const r = {}; for (const d of Object.keys(LAYOUT)) { const k = count(d); r[d] = k; console.log(`${d.padEnd(16)}${String(k.open).padStart(11)}${String(k.closed).padStart(13)}` + `${String(k.resolved).padStart(11)}${String(k.endpoints).padStart(12)}${String(OUTSIDE[d]).padStart(14)}`); } const single = r["single network"], split = r["two networks"]; console.log(`network separation closes ${single.open - split.open} paths (${((100 * (single.open - split.open)) / single.open).toFixed(1)}%); ` + `closed pairs: ${NAMES.flatMap((a) => NAMES.filter((b) => a !== b && !shareNet(a, b, "two networks")).map((b) => `${a}->${b}`)).join(", ")}`); console.log(`if all four services listened on the framework default ${DEFAULT}: ${NAMES.length - new Set(NAMES.map(() => DEFAULT)).size} ` + `collisions in the host-shared layout, 0 collisions in the separate-namespace layout`); const PUBLISH = { "all published": ALL, "field interface only": [8084], "none": [] }; console.log(`\n${"publish option".padEnd(22)}${"from outside".padStart(14)}${"in-network".padStart(13)}` + `${"open fraction".padStart(16)}`); for (const [label, ports] of Object.entries(PUBLISH)) console.log(`${label.padEnd(22)}${String(ports.length).padStart(14)}${String(split.endpoints).padStart(13)}` + `${`${ports.length}/${ALL.length}`.padStart(16)}`); console.log(`the outbound surface drops from ${ALL.length} endpoints to ${PUBLISH["field interface only"].length}; ` + `the number of endpoints reachable from within the network stays ${split.endpoints} across all three, none of them asks for identity`); ``` ``` model: 4 services, 2 ports per service, 8 endpoints total; in the two-network layout front = collector + verifier, back = verifier + billing + workOrder source \ target collector verifier billing workOrder resolved collector - resolves closed closed 1 verifier resolves - resolves resolves 3 billing closed resolves - resolves 2 workOrder closed resolves resolves - 2 layout open paths closed paths resolved in-network from outside single network 12 0 12 24 1 two networks 8 4 8 16 1 host-shared 12 0 12 24 8 networkless 0 12 0 0 0 network separation closes 4 paths (33.3%); closed pairs: collector->billing, collector->workOrder, billing->collector, workOrder->collector if all four services listened on the framework default 8080: 3 collisions in the host-shared layout, 0 collisions in the separate-namespace layout publish option from outside in-network open fraction all published 8 16 8/8 field interface only 1 16 1/8 none 0 16 0/8 the outbound surface drops from 8 endpoints to 1; the number of endpoints reachable from within the network stays 16 across all three, none of them asks for identity ``` These numbers are in the **measurement** class. #### Reading the Matrix The first table gives what each service can see in the split layout. The rightmost column counts how many names each service can resolve, and the four services give four different numbers: collector 1, verifier 3, billing 2, work order 2. The collector's row is the narrowest, and that narrowness is the design itself. The collector takes in readings arriving from meters in the field; it is the measurement network's most outward-facing part, and so it should see the least. In the matrix, the billing and work order columns show `closed` — from inside the collector, resolving either of those two names is not even possible. Because the name does not resolve, a connection attempt ends before it finds an address; the obstacle, in other words, is not a rejection, it is a state of **not knowing**. The verifier's row is the widest, and that too is by design: it sits at the intersection of both networks, takes readings from the front network, and gives verified data to the back network. It resolves all three names. This split also says something about how name resolution works. The name is a fixed string inside the application, and it is the same on every machine; what turns it into an address is the network the container is attached to. The same code, the same image, and the same name give two different results under two different network memberships — resolving on one, not resolving on the other. The only thing that changes in the config is the membership line, and every cell of the matrix derives from that line. The billing and work order rows are identical. Both are on the back network, both see each other and the verifier, neither sees the collector. This symmetry shows that network membership is a **set**: there is no directional distinction between two endpoints in the same set, each sees the other. #### Network Separation Closes Four Paths The second table's first two rows give the gain from separation. On a single network there are 12 directed paths between the four services and all are open; all 12 names resolve, and the reachable endpoint count is 24 (12 paths times 2 ports per service). Split into two networks, open paths drop to 8, resolved names to 8, reachable endpoints to 16. **Network separation closes 4 of the 12 paths, that is, a third of them.** The four closed paths are two symmetric pairs: collector with billing, collector with work order. These are pairs that were never talking to each other in the first place; what separation does is make the ones that do not talk **unable** to talk. This separation is not free, and its cost is in writing: a single-network layout writes four membership lines, a two-network layout writes five — the verifier is counted twice. In exchange for one network definition and one extra membership line, four paths close. The second entry in the cost is on the design side. Making the separation requires knowing in advance which services each network will cover, and that knowledge sits in the run config, not inside the application. When a new service is added, which network it joins is a separate decision; forget it, and the service comes up, runs, and finds nobody, without ever throwing an error. #### Three Driver Classes, Three Separate Rows How a container attaches to the network is a **driver** choice, and the table's last two rows show the extremes of that choice. Drivers are referred to here by type name; each corresponds to one membership arrangement. **Separate-namespace driver** — the first two rows. Every container has its own network namespace and its own address; service names resolve, and only published endpoints open to the outside: 1. **Host-shared driver** — the third row. No separate network namespace is given; containers listen directly in the machine's address space. Its results show up in three columns at once: open paths climb back to 12, in-network endpoints return to 24, and endpoints reachable from outside jump from 1 to **8**. There is no longer a publish decision to make either, because there is no boundary left to publish across — all eight of the eight ports are already on the machine's interface. This driver's second cost is worked out in the last row. Without a separate namespace, port numbers come from a single shared space. If all four services listened on their framework's default of 8080, the host-shared layout would have **3 collisions**: one takes the port, three cannot come up. In the separate-namespace layout the collision count is 0, because every container's 8080 sits in its own space. This is namespace isolation's most concrete gain — services do not have to know each other's port numbers. **Networkless driver** — the fourth row. There is no membership at all: 0 open paths, 0 resolved names, 0 endpoints, 0 from outside. This is the one row where every measurement is zero, and it is not useless. For a container like the nightly batch job that only reads and writes to a mounted directory and calls nobody, this is the isolation budget's cheapest point: the network surface is entirely closed, and closing it costs nothing, because it was never used. #### Published Ports and In-Network Access The third table measures an entirely different boundary. The first two tables counted containers' access **to each other**; this table counts access coming from the host machine's network interface — that is, from entirely outside the container network. The four services listen on 8 ports in total. If all are published, 8 endpoints open on the host machine's interface; if only the interface port the field crews use is published, 1 endpoint opens; if none are published, 0. **The outbound surface drops from 8 endpoints to 1**, and while it drops, the number of in-network reachable endpoints stays constant: 16 in all three options. This is what the table is really saying. The publish decision and in-network access are **independent of each other**. A service remains exactly as reachable as before by everyone on its own network whether or not it is published; publishing only adds an endpoint to the host machine's interface. The metrics ports are a good example of this: none of the four metrics ports is published in any option, yet all of them are reachable from within the network. In practice the distinction settles here: in-network access is for services to talk to each other, publishing is for a user or tool outside the network to talk to them. Managing the two with a single decision ends with all eight of the eight endpoints left open. The surface dropping from 8 to 1 is a security gain, and its price is paid on the operations side. An unpublished port cannot be looked at from outside; if understanding the cause of a problem requires asking that port a question, whatever is asking also has to be inside the network. #### Where Isolation Is Punctured There are three holes, and all three show up in the numbers. **First:** network membership is not authorization. In the split layout there are 16 endpoints reachable from within the network, and none of them asks for identity. A container on the same network reaches not only its neighbor's interface port but its metrics port too. Network separation roughly decides **who** can reach whom; it does not restrict at all what a reaching endpoint may ask for. **Second:** the four closed paths are only the **direct** ones. The verifier is a member of both networks; a request coming from the front network reaches the verifier, and the verifier reaches everything on the back network. So the path closed between the collector and billing can still be covered in two hops. The number network separation closes is 4; the number that can be routed around through the service sitting at the intersection is also 4. **Third:** the network namespace is separate, but the host machine's network stack is shared. Every published port opens a real endpoint on the host machine's interface, and that endpoint is now under the machine's rules, not the container network's. Port 8084 inside the container and the endpoint on the machine's interface are not the same thing; the second is outside isolation. #### Summary - Container networking removes the address from the environment: the config holds a service name instead of a machine name, and the name is the same on every machine. Four of lesson 01's 14 variables carried exactly this difference. - Name resolution is tied to the same boundary as access: in the split layout the collector resolves 1 name, the verifier 3, billing 2, work order 2. The obstacle is not a rejection, it is the name failing to resolve. - **The gain from network separation:** where a single network has 12 directed paths, 12 resolved names, and 24 reachable endpoints, two networks bring these down to 8, 8, and 16; 4 paths, a third of the total, close. The cost is one network definition and one extra membership line — five instead of four. - Three driver classes are three separate rows: the separate-namespace layout has 1 endpoint from outside and 0 port collisions; the host-shared layout has 8 endpoints from outside, 12 open paths, and 3 collisions if all four services listened on 8080; the networkless layout has every measurement at zero. - **Published ports and in-network access are independent:** the outbound surface drops from 8 endpoints to 1 while the number of endpoints reachable from within the network stays at 16 across all three options. - **Where isolation is punctured:** network membership is not authorization — none of the 16 endpoints asks for identity; the 4 closed direct paths can still be covered in two hops through the verifier, which is a member of both networks; and a published port opens an endpoint outside isolation on the host machine's interface. #### Next Step The network matrix answered one question and left another wide open. The matrix says **which container** can reach which endpoint; it does not say **who** stands behind the request that reaches it. The same gap showed up in the previous lesson: what decided write permission in a mounted directory was not the container's name but the numeric identity of the user running inside it, and that is why six of the eight nodes were closed. Two measurements are looking at the same place. Which user is the process running inside the container, what does that user correspond to on the host machine, and which files does it access when left at the default? The next lesson measures this identity: how many files stay open with the privileged identity, which operations drop when the identity changes, and how many entries the cost of moving to least privilege requires an ownership fix in. The identity's **capabilities** are left to the end of the topic; a separate set of numbers stands there. ### Users and Permissions Source: https://academia.sh/en/courses/containers/users-and-permissions Three measures of running rootless: how many of fifteen entries stay open to the privileged identity, how many entries the switch to a rootless identity requires an ownership fix in, and how many paths in the bind mount stay closed after the fix because of ownership coming from the environment. The previous lesson built network separation: which name resolves to which address, which port opens outward, and which one, left unopened, stays unreachable, were all counted. All of those measurements counted what the process **does**. Who runs inside was never asked, and because it was never asked, the default stood: while an image is being built, every file written into it is owned by the privileged identity running the build, and the process starts with the same identity. The regional measurement network's reading collector service — fictional software that collects, verifies, and hands off meter data to billing — is this lesson's measurement subject. Application-level authorization questions are outside this lesson. Where a secret sits, how many places it is repeated in, and what breaks when it is rotated were measured in the Secrets Management lesson. The question here is one boundary lower: which identity can reach the **file** that secret sits in, and whether that reach comes from the image definition or from the environment. **RT25 — the service's image consists of fifteen entries** (files and directories), and the service performs twelve operations on them while it runs: read, write, execute. **RT26 — every entry written during the build passes into the ownership of the privileged identity (`0:0`).** **RT27 — in the rootless layout the process identity is `10001:10001`.** **RT28 — the mounted host directory appears inside with `501:20` ownership**; this ownership is read not from the image but from wherever the mount comes from. **RT29 — the privileged identity bypasses discretionary access control for reading and writing**; for execution it still looks for an `x` bit in the mode. **RT30 — without user namespace mapping, the numeric identity inside the container is the same number on the host.** #### Permission Matrix The run below really writes files to disk, sets mode bits with `chmod`, and reads them back with `statSync`; the mode column in the table comes from disk. Ownership, on the other hand, is a **model**: because changing ownership requires privilege, owner and group numbers are kept in the table, and the access decision is computed from those numbers over the real mode bits. That access to a path also requires an execute bit on every ancestor directory is accounted for too. ```js // tree.mjs — the fictional reading collector service's file tree. // Files and mode bits are REAL: written to disk, set with chmod, read back with statSync. // Ownership is a MODEL: chown requires privilege, so owner/group are kept in the table. import { mkdirSync, writeFileSync, chmodSync, statSync, rmSync } from 'node:fs'; export const PRIVILEGED = { name: 'privileged', uid: 0, gid: 0 }; // RT26 export const ROOTLESS = { name: 'rootless', uid: 10001, gid: 10001 }; // RT27 export const HOST = { uid: 501, gid: 20 }; // RT28 // [path, kind, mode, owner, group] — mounted/ and below appear with host ownership. export const IMAGE = [ ['app', 'd', 0o755, 0, 0], ['app/start.mjs', 'f', 0o755, 0, 0], ['app/collector.mjs', 'f', 0o644, 0, 0], ['app/verifier.mjs', 'f', 0o644, 0, 0], ['config', 'd', 0o755, 0, 0], ['config/service.json', 'f', 0o644, 0, 0], ['config/signing-key', 'f', 0o600, 0, 0], ['state', 'd', 0o755, 0, 0], ['state/queue.data', 'f', 0o644, 0, 0], ['state/lock', 'f', 0o644, 0, 0], ['tmp', 'd', 0o777, 0, 0], ['mounted', 'd', 0o755, HOST.uid, HOST.gid], ['mounted/reading-01.csv', 'f', 0o644, HOST.uid, HOST.gid], ['mounted/reading-02.csv', 'f', 0o640, HOST.uid, HOST.gid], ['mounted/output', 'd', 0o755, HOST.uid, HOST.gid], ]; export function setup(root, nodes) { rmSync(root, { recursive: true, force: true }); mkdirSync(root, { recursive: true }); for (const [path, kind, mode] of nodes) { const full = `${root}/${path}`; if (kind === 'd') mkdirSync(full); else writeFileSync(full, `# ${path}\n`); chmodSync(full, mode); } return nodes.map(([path, kind, , uid, gid]) => ({ path, kind, uid, gid, mode: statSync(`${root}/${path}`).mode & 0o777, })); } // Discretionary access control: one of owner / group / other is selected. // RT29: the privileged identity bypasses the triple; it still looks for an x bit to execute. export function permission(node, identity) { const { mode } = node; if (identity.uid === 0) return { r: true, w: true, x: (mode & 0o111) !== 0 }; const u = identity.uid === node.uid ? mode >> 6 : identity.gid === node.gid ? mode >> 3 : mode; return { r: (u & 4) !== 0, w: (u & 2) !== 0, x: (u & 1) !== 0 }; } // Access to a path also requires the x bit on every ancestor directory. export function access(tree, path, identity) { const p = path.split('/'); for (let i = 1; i < p.length; i += 1) { const ancestor = tree.find((d) => d.path === p.slice(0, i).join('/')); if (ancestor && !permission(ancestor, identity).x) return { r: false, w: false, x: false }; } return permission(tree.find((d) => d.path === path), identity); } export const format = (e) => `${e.r ? 'r' : '-'}${e.w ? 'w' : '-'}${e.x ? 'x' : '-'}`; if (import.meta.url === `file://${process.argv[1]}`) { const tree = setup('./root', IMAGE); const s = (x, n) => String(x).padStart(n); console.log(`${'path'.padEnd(26)}${s('mode', 5)}${s('owner', 8)}${s('privileged', 13)}${s('rootless', 10)}`); for (const d of tree) { console.log(`${(d.kind === 'd' ? `${d.path}/` : d.path).padEnd(26)}${s(d.mode.toString(8), 5)}` + `${s(`${d.uid}:${d.gid}`, 8)}${s(format(access(tree, d.path, PRIVILEGED)), 13)}` + `${s(format(access(tree, d.path, ROOTLESS)), 10)}`); } for (const k of [PRIVILEGED, ROOTLESS]) { const e = tree.map((d) => access(tree, d.path, k)); console.log(`${k.name.padEnd(13)} readable ${s(e.filter((x) => x.r).length, 2)}/${tree.length}` + ` writable ${s(e.filter((x) => x.w).length, 2)}/${tree.length}` + ` executable ${s(e.filter((x) => x.x).length, 2)}/${tree.length}`); } } ``` ``` path mode owner privileged rootless app/ 755 0:0 rwx r-x app/start.mjs 755 0:0 rwx r-x app/collector.mjs 644 0:0 rw- r-- app/verifier.mjs 644 0:0 rw- r-- config/ 755 0:0 rwx r-x config/service.json 644 0:0 rw- r-- config/signing-key 600 0:0 rw- --- state/ 755 0:0 rwx r-x state/queue.data 644 0:0 rw- r-- state/lock 644 0:0 rw- r-- tmp/ 777 0:0 rwx rwx mounted/ 755 501:20 rwx r-x mounted/reading-01.csv 644 501:20 rw- r-- mounted/reading-02.csv 640 501:20 rw- --- mounted/output/ 755 501:20 rwx r-x privileged readable 15/15 writable 15/15 executable 7/15 rootless readable 13/15 writable 1/15 executable 7/15 ``` The difference between the two columns shows how much decision-making weight the mode bits carry. For the privileged identity, fifteen of the fifteen entries are readable and fifteen are writable; both the signing key protected by mode `600` and the measurement file arriving with host ownership are open. In that column, mode bits carry **no decision**, they only keep a record. In the rootless column, the same bits do make the decision: thirteen readable, one writable. The one writable entry is the `tmp/` directory, and the reason is that its mode is `777`. That the execute column comes out at seven for both identities is a separate distinction. The privileged identity bypasses permission checks, but it cannot invent an execute bit that is not there; whether a file is executable comes from its mode, not from privilege. Of all this lesson's numbers, this is the one column privilege does not change. #### The Cost of Switching to Rootless The matrix alone does not make a decision; without knowing which access is **needed**, "closed" is not a number. The second run lists the twelve operations the service actually performs, runs them under both identities, and separates the dropped operations: the ones fixable inside the image from the ones coming from the mount. ```js // matrix.mjs — the cost of switching to rootless and the surface left on the host side. import { setup, permission, access, format, IMAGE, PRIVILEGED, ROOTLESS } from './tree.mjs'; // Operations the service actually performs: [path, op] const REQUIRED = [ ['app/start.mjs', 'x'], ['app/collector.mjs', 'r'], ['app/verifier.mjs', 'r'], ['config/service.json', 'r'], ['config/signing-key', 'r'], ['state', 'w'], ['state/queue.data', 'w'], ['state/lock', 'w'], ['tmp', 'w'], ['mounted/reading-01.csv', 'r'], ['mounted/reading-02.csv', 'r'], ['mounted/output', 'w'], ]; const dropped = (tree, identity) => REQUIRED.filter(([y, i]) => !access(tree, y, identity)[i]); const surface = (tree, identity) => tree.filter((d) => permission(d, identity).w).length; const tree = setup('./root', IMAGE); const s = (x, n) => String(x).padStart(n); console.log(`privileged: dropped ops ${dropped(tree, PRIVILEGED).length}/${REQUIRED.length}` + `, writable entries ${surface(tree, PRIVILEGED)}/${tree.length}`); console.log(`rootless : dropped ops ${dropped(tree, ROOTLESS).length}/${REQUIRED.length}` + `, writable entries ${surface(tree, ROOTLESS)}/${tree.length}`); for (const [y, i] of dropped(tree, ROOTLESS)) { const d = tree.find((x) => x.path === y); console.log(` ${y.padEnd(24)} wants ${i} mode ${d.mode.toString(8)} owner ${d.uid}:${d.gid}` + ` ${d.path.startsWith('mounted/') ? 'bind mount — cannot be fixed in the image' : 'in the image'}`); } // Two fix forms: move ownership, or loosen mode bits. // OUTSIDER: a third identity that benefits from the same bits (another process, or the host when there is no mapping). const OUTSIDER = { uid: 20002, gid: 20002 }; function fix(fixKind) { const t = setup('./root', IMAGE); let touched = 0; for (const [y] of dropped(t, ROOTLESS)) { const d = t.find((x) => x.path === y); if (y.startsWith('mounted/')) continue; if (fixKind === 'ownership') { d.uid = ROOTLESS.uid; d.gid = ROOTLESS.gid; } else d.mode |= d.kind === 'd' ? 0o077 : 0o066; touched += 1; } return { t, touched, remaining: dropped(t, ROOTLESS) }; } console.log(`\n${'fix'.padEnd(16)}${s('touched', 11)}${s('still dropped', 15)}` + `${s('rootless writes', 17)}${s('outsider writes', 17)}${s('outsider reads', 16)}`); let remaining = []; for (const b of ['ownership', 'mode-loosen']) { const r = fix(b); remaining = r.remaining; console.log(`${b.padEnd(16)}${s(r.touched, 11)}${s(r.remaining.length, 15)}` + `${s(`${surface(r.t, ROOTLESS)}/${r.t.length}`, 17)}${s(`${surface(r.t, OUTSIDER)}/${r.t.length}`, 17)}` + `${s(`${r.t.filter((d) => permission(d, OUTSIDER).r).length}/${r.t.length}`, 16)}`); } // Host surface: entries the same numeric identity can touch outside the container (RT30). const HOST_TREE = [ ['data', 'd', 0o755, 0, 0], ['data/reading-archive', 'f', 0o644, 10001, 10001], ['data/service.log', 'f', 0o664, 0, 10001], ['config', 'd', 0o755, 0, 0], ['config/network.json', 'f', 0o644, 0, 0], ['home', 'd', 0o755, 10001, 10001], ['home/key', 'f', 0o600, 10001, 10001], ['home/collector.mjs', 'f', 0o755, 10001, 10001], ]; const hostTree = setup('./host', HOST_TREE); const SHIFT = 100000; // user namespace shift console.log(`\n${'host identity'.padEnd(16)}${s('readable', 12)}${s('writable', 13)}${s('home/key', 12)}`); for (const k of [{ label: 'unmapped 10001', uid: ROOTLESS.uid, gid: ROOTLESS.gid }, { label: `shift +${SHIFT}`, uid: ROOTLESS.uid + SHIFT, gid: ROOTLESS.gid + SHIFT }]) { const e = hostTree.map((d) => access(hostTree, d.path, k)); console.log(`${k.label.padEnd(16)}${s(`${e.filter((x) => x.r).length}/${hostTree.length}`, 12)}` + `${s(`${e.filter((x) => x.w).length}/${hostTree.length}`, 13)}` + `${s(format(access(hostTree, 'home/key', k)), 12)}`); } console.log(`${remaining.length} paths in the bind mount must be fixed on the host side;` + ` with the shift on, the target identity is not ${ROOTLESS.uid} but ${ROOTLESS.uid + SHIFT}`); ``` ``` privileged: dropped ops 0/12, writable entries 15/15 rootless : dropped ops 6/12, writable entries 1/15 config/signing-key wants r mode 600 owner 0:0 in the image state wants w mode 755 owner 0:0 in the image state/queue.data wants w mode 644 owner 0:0 in the image state/lock wants w mode 644 owner 0:0 in the image mounted/reading-02.csv wants r mode 640 owner 501:20 bind mount — cannot be fixed in the image mounted/output wants w mode 755 owner 501:20 bind mount — cannot be fixed in the image fix touched still dropped rootless writes outsider writes outsider reads ownership 4 2 5/15 1/15 13/15 mode-loosen 4 2 5/15 5/15 14/15 host identity readable writable home/key unmapped 10001 8/8 5/8 rw- shift +100000 7/8 0/8 --- 2 paths in the bind mount must be fixed on the host side; with the shift on, the target identity is not 10001 but 110001 ``` With the privileged identity, zero of the twelve operations drop. This explains in one line why privileged operation persists: nothing breaks, and so nothing is questioned. Switching to the rootless identity drops six of the twelve operations, and the distribution of what drops is the real information — four sit inside the image, two come from the mount. The four drops inside the image close with a single fix: moving the ownership of those four entries to the rootless identity. This lesson's cost of isolation is this number — **an ownership fix on four entries**, that is, four paths that need to be written into the image definition. What actually makes the cost heavier is that these paths cannot be found by reading the code: which file needs it only surfaces once the service runs rootless and drops, and every dropped operation is a separate breakage. After the fix, the writable entry count drops not from ten to five but **from fifteen to five**: the ten entries the privileged identity could write to close. Among the ten entries that close are one config file (`config/service.json`) and the application code itself — three `.mjs` files; a compromised process modifying its own code to persist is eliminated by these ten entries closing. The same four entries can be opened two separate ways, and the table puts them side by side. Both moving ownership and loosening mode bits touch four entries, both bring the still-dropped count down to two, and in both the rootless identity can write to five entries. The service comes up the same way under either fix. The difference is in the table's last two columns: when ownership is moved, a third identity can write to only one of the fifteen entries; when mode is loosened, it can write to five. The read column states the difference more sharply — loosening turns the signing key's mode `600` into `666`, and the entries an outsider identity can read climb from thirteen to fourteen. The two fixes silence the same error, write the same number of lines into the image definition, and **leave a different surface**; which one was chosen shows up in neither the error message nor the service's behavior. The remaining two operations do not close by writing to the image definition. `mounted/reading-02.csv` and `mounted/output` come from the mount; their ownership is on record not in the image but wherever the mount comes from. **This is the first place isolation is punctured:** the ownership of eleven of the fifteen entries can be pulled into the output, but four of them come from outside at run time, and the right number for those four can only be settled by agreement between both sides. #### The Sharing of the Numeric Identity The lower half of the output measures the second hole. The rootless identity `10001` is nothing but a number, and that number is read from a single kernel table. On the host side, in a fictional set of eight entries, the same number can write to five entries; `home/key` is in mode `600`, meaning the host equivalent of the signing key, unreachable from inside, is open `rw-` to that identity. Running rootless is not "running unprivileged"; **it is running with the authority of a different number**, and what that number means on the host is not written in the image. User namespace mapping closes this hole: with a shift of one hundred thousand, the internal `10001` becomes `110001` on the host, is the owner or group of no host entry, the writable count drops from five to **zero**, and `home/key` becomes unreadable. Its cost is in the last line. The target of the ownership fix to be made on the host side for the two mount paths is no longer `10001` but `110001`; every new file written from inside also shows up on the host under that identity. While mapping closes the authority leaking outward, it grows the identity mismatch between the two ends of the mount. Isolation is still not complete here either — only its cost gets written to a different entry. #### Summary - The permission matrix carries no decision for the privileged identity: fifteen of fifteen entries are readable, fifteen are writable; the `600`-mode signing key is open too. The one column mode does not change is execute — seven under both identities. - With the rootless identity, six of the twelve operations drop; four are inside the image, two are in the bind mount. - The cost of switching is an ownership fix on four entries, and these four paths are found not by reading the code but by the service dropping; in exchange, the writable entry count drops from fifteen to five. Opening the same four by loosening mode also runs the service, but it takes an outsider identity's writable entries from one to five and its readable entries from thirteen to fourteen. - The ownership of eleven of the fifteen entries can be pulled into the output; four come from the environment at run time and cannot be fixed through the image definition. - The rootless identity is the same number on the host: in the fictional host set it can write to five of eight entries and can read the `600`-mode host key. - User namespace mapping brings the host writable count from five to zero; its cost is the identity mismatch at the two ends of the mount deepening, with the fix target shifting to `110001`. #### Next Step Everything measured up to this point was on a single container: a file tree, an identity, a permission matrix. But the regional measurement network is not a single process — a verifier reads the queue the reading collector writes, the verified record goes to billing, and a record store sits underneath all of them. Starting these services by hand means setting up each one's network, volume, and order with separate decisions; which ones are affected when one of them restarts is also buried inside those decisions. The next lesson moves this setup into a declarative compose file and compares the two forms in numbers: how the startup order is derived from the dependency graph in a four-service setup, how many races are born without a readiness check, and how many commands and decisions disappear compared to manual startup. ### Multi-Container Local Environment Source: https://academia.sh/en/courses/containers/multi-container-local-environment Three measures of a declarative compose file: deriving the startup order from a dependency graph, the number of races born without a readiness check against the steps the check adds to startup, and the commands, flags, and ordering decisions that disappear compared to manual setup. The previous lesson looked inside a single container: a fifteen-entry tree, two identities, a permission matrix. The regional measurement network — fictional software that collects, verifies, and turns meter readings into invoices — is not a single process. A verifier reads the record the reading collector writes, the verified record goes to billing, and a record store sits underneath all of them. Bringing these four up locally is not just writing four separate startup commands; it is deciding which network gets set up first, which volume mounts where, and which service comes after which. Until these decisions are written down somewhere, they stay in the **environment**: in the shell of whoever set it up, in past commands, or in their head. A **compose file** captures these decisions in a file, as a type. In this lesson, a small schema for such a file is defined and measured with a resolver; both the schema and the resolver are hand-written here, no real runtime is run. **RT31 — the local environment has four services:** the record store, the reading collector, the verifier, billing. **RT32 — the compose file's schema is six fields:** image, depends, network, volume, port, ready. **RT33 — a service's process comes up in one step; it takes as many more steps as the `ready` field to start serving.** **RT34 — without a readiness check, a dependent service tries to connect the moment its dependency's process starts.** **RT35 — in manual setup, the network and the volume are created with separate commands, and every service is started with its own command and its own flags.** **RT36 — restarting a service drops the connection of every service transitively dependent on it.** #### Schema and Resolver The schema itself is part of the measure: the moment fields are named, everything written in the file becomes checkable before startup. The resolver below does three things — it collects fields that do not match the schema and references that do not resolve, it searches the dependency graph for cycles, it derives the ordering as waves. ```js // compose.mjs — a declarative compose file's small schema and resolver (MODEL). // The schema is defined here: fields, their types, and the sets they reference. export const SCHEMA = { image: 'text', depends: 'list', network: 'list', volume: 'list', port: 'list', ready: 'number' }; // RT31: fictional four-service local environment. RT32: ready = steps to serve after coming up. export const COMPOSE = { networks: ['measurement-net'], volumes: ['record-data'], services: { 'record-store': { image: 'record:3', depends: [], network: ['measurement-net'], volume: ['record-data:/data'], port: [], ready: 3 }, 'reading-collector': { image: 'collector:7', depends: ['record-store'], network: ['measurement-net'], volume: [], port: ['8081'], ready: 1 }, 'verifier': { image: 'verify:4', depends: ['record-store'], network: ['measurement-net'], volume: [], port: [], ready: 2 }, 'billing': { image: 'invoice:2', depends: ['reading-collector', 'verifier'], network: ['measurement-net'], volume: ['record-data:/archive'], port: ['8082'], ready: 1 }, }, }; export function validate(b) { const errors = []; const names = Object.keys(b.services); for (const [name, svc] of Object.entries(b.services)) { for (const field of Object.keys(SCHEMA)) if (!(field in svc)) errors.push(`${name}: missing field '${field}'`); for (const field of Object.keys(svc)) if (!(field in SCHEMA)) errors.push(`${name}: unknown field '${field}'`); for (const a of svc.network ?? []) if (!b.networks.includes(a)) errors.push(`${name}: unknown network '${a}'`); for (const v of svc.volume ?? []) { if (!b.volumes.includes(v.split(':')[0])) errors.push(`${name}: unknown volume '${v}'`); } for (const d of svc.depends ?? []) { if (d === name) errors.push(`${name}: depends on itself`); else if (!names.includes(d)) errors.push(`${name}: unknown dependency '${d}'`); } } return errors; } // Cycle: depth-first walk; a return to a gray node gives the cycle. export function cycle(b) { const color = {}, path = []; let found = null; const visit = (name) => { if (color[name] === 'gray') { found ??= [...path.slice(path.indexOf(name)), name]; return; } if (color[name] === 'black') return; color[name] = 'gray'; path.push(name); for (const d of b.services[name].depends) if (b.services[d]) visit(d); path.pop(); color[name] = 'black'; }; for (const name of Object.keys(b.services)) visit(name); return found; } // The startup order is derived from the graph; there is no dependency between two in the same wave. export function order(b) { const remaining = new Map(Object.entries(b.services).map(([a, s]) => [a, new Set(s.depends)])); const waves = []; while (remaining.size) { const available = [...remaining].filter(([, d]) => d.size === 0).map(([a]) => a); if (!available.length) return null; // there is a cycle: no order can be derived waves.push(available); for (const a of available) remaining.delete(a); for (const d of remaining.values()) for (const a of available) d.delete(a); } return waves; } // The number of orderings a person starting by hand has to choose among. export function ordering(b) { const names = Object.keys(b.services); const perm = (a) => (a.length <= 1 ? [a] : a.flatMap((x, i) => perm([...a.slice(0, i), ...a.slice(i + 1)]).map((r) => [x, ...r]))); const total = perm(names); const valid = total.filter((p) => p.every((name, i) => b.services[name].depends.every((d) => p.indexOf(d) < i))); return { total: total.length, valid: valid.length }; } if (import.meta.url === `file://${process.argv[1]}`) { console.log(`compose validation: ${validate(COMPOSE).length} errors`); const BROKEN = structuredClone(COMPOSE); BROKEN.services['verifier'].network.push('report-net'); BROKEN.services['billing'].depends.push('archiver'); BROKEN.services['billing'].volume.push('scratch-data:/tmp'); BROKEN.services['reading-collector'].restartPolicy = 'always'; delete BROKEN.services['record-store'].ready; const errors = validate(BROKEN); console.log(`broken compose: ${errors.length} errors, all before startup`); for (const e of errors) console.log(` ${e}`); const CYCLIC = structuredClone(COMPOSE); CYCLIC.services['record-store'].depends = ['billing']; console.log(`\ncycle: ${cycle(CYCLIC).join(' -> ')}`); console.log(`can an order be derived from the cyclic one: ${order(CYCLIC) === null ? 'no' : 'yes'}`); const d = ordering(COMPOSE); console.log(`\nstartup waves: ${order(COMPOSE).map((w) => `[${w.join(' ')}]`).join(' -> ')}`); console.log(`${d.valid} of ${d.total} orderings of ${Object.keys(COMPOSE.services).length} services are valid`); } ``` ``` compose validation: 0 errors broken compose: 5 errors, all before startup record-store: missing field 'ready' reading-collector: unknown field 'restartPolicy' verifier: unknown network 'report-net' billing: unknown volume 'scratch-data:/tmp' billing: unknown dependency 'archiver' cycle: record-store -> billing -> reading-collector -> record-store can an order be derived from the cyclic one: no startup waves: [record-store] -> [reading-collector verifier] -> [billing] 2 of 24 orderings of 4 services are valid ``` What the five errors have in common is that all of them are found **before startup**. In manual setup, the only way to find these five is five separate runs: the unrecognized network name shows up when the first service starts, the unresolved volume name on the third command, and the missing dependency only when billing tries to connect. The declarative form reduces this to a single read, because **the file sees the whole setup at once**. The check for a service depending on itself sits in the same place; a service waiting on itself is an infinite wait at run time, and a single line in the file. The cycle output is the sharpest split between the two forms. When `record-store` is made dependent on billing, a three-edge cycle is born and no ordering can be derived. A person starting by hand does not see this as an error; they wait for the first service to come up, see that it does not, try the second, and find no answer for which one should come first while the two services wait on each other. The graph states that there **is** no answer, in three steps. The last line gives the number of decisions. Of the four services' twenty-four orderings, only two satisfy every dependency. Order is not a preference, it is a consequence falling out of the graph, and the waves make it visible: the record store alone, then two services with no dependency between them, then billing. #### Races and the Readiness Check A correct order is not enough. A service's process having started does not mean it is ready to serve, and that gap falls on the dependent service's connection attempt. The second run brings the same compose up twice — without a readiness check and with one. ```js // run.mjs — the cost of a readiness check, the spread of a restart, command and decision counts. import { COMPOSE, SCHEMA, order, ordering } from './compose.mjs'; // RT33: a process comes up in 1 step; serving takes 'ready' more steps. // RT34: without the check, a dependent service tries to connect the moment its dependency's process starts. function boot(b, check) { const S = b.services; const startStep = {}, readyAt = {}; for (const wave of order(b)) { for (const name of wave) { const deps = S[name].depends; startStep[name] = deps.length === 0 ? 0 : Math.max(...deps.map((x) => (check ? readyAt[x] : startStep[x] + 1))); readyAt[name] = startStep[name] + S[name].ready; } } const races = []; for (const [name, s] of Object.entries(S)) { for (const d of s.depends) if (startStep[name] < readyAt[d]) races.push([name, d, readyAt[d] - startStep[name]]); } return { startStep, races, total: Math.max(...Object.values(readyAt)) }; } // What is transitively affected by restarting one service. function affected(b, name) { const reverse = {}; for (const [a, s] of Object.entries(b.services)) for (const d of s.depends) (reverse[d] ??= []).push(a); const set = new Set(), stack = [name]; while (stack.length) { for (const x of reverse[stack.pop()] ?? []) if (!set.has(x)) { set.add(x); stack.push(x); } } return [...set]; } const s = (x, n) => String(x).padStart(n); const edges = Object.values(COMPOSE.services).reduce((a, x) => a + x.depends.length, 0); console.log(`${'check'.padEnd(12)}${s('races', 7)}${s('edges', 7)}${s('hidden wait', 15)}${s('startup', 14)}`); for (const check of [false, true]) { const r = boot(COMPOSE, check); const wait = r.races.reduce((a, [, , g]) => a + g, 0); console.log(`${(check ? 'readiness' : 'none').padEnd(12)}${s(r.races.length, 7)}${s(edges, 7)}` + `${s(`${wait} steps`, 15)}${s(`${r.total} steps`, 14)}`); } for (const [name, d, g] of boot(COMPOSE, false).races) { console.log(` race: ${name.padEnd(17)} -> ${d.padEnd(17)} connects ${g} step${g === 1 ? "" : "s"} early`); } console.log(`\n${'restarting'.padEnd(18)}${s('affected', 10)} ${'who'}`); for (const name of Object.keys(COMPOSE.services)) { const e = affected(COMPOSE, name); console.log(`${name.padEnd(18)}${s(e.length, 10)} ${e.join(', ') || '-'}`); } // The writing cost of manual setup versus the declarative file. const services = Object.values(COMPOSE.services); const flags = services.reduce((a, x) => a + x.network.length + x.volume.length + x.port.length, 0); const commands = COMPOSE.networks.length + COMPOSE.volumes.length + services.length; const fields = services.length * Object.keys(SCHEMA).length + 2; const ord = ordering(COMPOSE); console.log(`\nmanual : ${commands} commands, ${flags} flags, ${ord.total - ord.valid} of ${ord.total} orderings` + ` break at least one dependency`); console.log(`declarative: 1 command, ${fields}-field file, order is derived (0 ordering decisions)`); console.log(`on manual restart: worst case ${Math.max(...Object.keys(COMPOSE.services) .map((a) => affected(COMPOSE, a).length))} more services need manual restart`); ``` ``` check races edges hidden wait startup none 3 4 5 steps 3 steps readiness 0 4 0 steps 6 steps race: reading-collector -> record-store connects 2 steps early race: verifier -> record-store connects 2 steps early race: billing -> verifier connects 1 step early restarting affected who record-store 3 reading-collector, verifier, billing reading-collector 1 billing verifier 1 billing billing 0 - manual : 6 commands, 8 flags, 22 of 24 orderings break at least one dependency declarative: 1 command, 26-field file, order is derived (0 ordering decisions) on manual restart: worst case 3 more services need manual restart ``` In the unchecked layout the order is correct, and yet three of the four dependency edges give birth to a race. A correct order only guarantees that the process has **started**; the record store takes three steps to serve, the two services depending on it try to connect on the first step, and both arrive two steps early. Billing connects to the verifier one step early. This five-step-total earliness shows up as an error nowhere; with a retry it turns into delay, without one, into a service that drops on startup. The check's cost is in the same table's last column: startup climbs from three steps to six, **double**. This is the most visible price the declarative form pays in this lesson, and it is paid on every local run. What it buys is three races zeroed out. Both numbers are read from the same file, so the trade-off becomes a visible choice in the file — in manual setup, the same trade-off is hidden inside wait times sprinkled between commands. The restart table measures, for the first time, **past a single service**. When the record store restarts, three services are affected; when the reading collector or the verifier restarts, one service each; when billing restarts, none. In manual setup, this number is applied by hand: whoever restarts the record store also has to start the remaining three themselves, and which ones are needed is learned not from a dependency list but from errors. #### What Leaves the Environment, and What Does Not The last three lines are this lesson's isolation account. Manual setup requires six commands, eight flags, and the decision of choosing one of twenty-four orderings; twenty-two of the twenty-four break at least one dependency. In the declarative form, in their place, there is a twenty-six-field file and a single command, and the ordering decision is zero. **This is the difference removed from the environment:** startup order, network and volume names, the flag set, and the spread of a restart move out of the setter-up's shell history and into a file sitting next to the output. The cost is two entries: twenty-six fields written, and the three steps the readiness check adds. One distinction must be kept while reading these numbers: the six commands and eight flags do not **disappear**, they change location. The network still gets created, the volume still gets made, the four processes still get started; it is just that what decides in which order and with which flags is no longer the person at the shell, it is the file. The gain is not in the command count, it is that the person setting up the same thing a second time avoiding the same twenty-two wrong orderings becomes **provable**. A compose file does not shorten a setup, it makes it repeatable; the only thing that shortens is the number of decisions. **Where isolation is punctured** is written into the same file. The four services share a single network; to connect the services to each other, the compose file **loosens** the separation built in the container networking lesson, putting the four that need to see each other into the same name resolution space. The `record-data` volume is mounted into two services — at `/data` in the record store, at `/archive` in billing. The same set of bytes can be written from two separate containers, under two separate identities; the previous lesson's permission matrix here looks not at a single tree but at the point where two trees intersect. The readiness check itself is not proof either: in this model the `ready` field is a number, and if the number is wrong the races come back, they just become invisible. #### Summary - Defining a schema for the compose file makes setup errors visible before startup: in the broken setup, all five of the five errors were found in a single read; in manual setup, each would have been a separate run. - The startup order is derived from the graph; only two of the four services' twenty-four orderings are valid, and once a cycle is added, that the ordering **does not exist** prints by name as a three-edge cycle. - A correct order does not end races: without a readiness check, three of the four dependency edges connect early, giving birth to five steps of hidden wait in total. - The readiness check brings races from three to zero, and takes startup from three steps to six; the trade-off becomes a choice written into the file. - Restarting the record store affects three services, restarting billing affects none; in manual setup this number is learned from errors, in the file it is read from the dependency list. - The difference removed from the environment is six commands, eight flags, and the exclusion of twenty-two wrong orderings; its cost is a twenty-six-field file plus three extra steps. The hole is the shared network and the single volume mounted into two services. #### Next Step This lesson brought the compose up once and left it there. But the local environment's real use is not a single startup, it is a loop repeated hundreds of times over a day: a line is changed, the result is seen, another line is changed. In that loop, rebuilding the image and mounting the source into the container are two separate forms, and neither is free. The next lesson compares these two forms by counting per change — bytes copied, layers invalidated, feedback steps — and measures by name the cost the second form pays: how far the mounted source and the skipped build step move the running environment away from production, and which class of error that distance hides. ### Developer Experience Source: https://academia.sh/en/courses/containers/developer-experience Comparing two feedback loops per change: the layers rebuilding the image invalidates, the bytes it copies, and its step count, against mounting the source and reloading the process zeroing out those same three numbers, and the environment distance that mounting costs. The previous lesson brought four services up once and left them there. The local environment's real use, though, is not a single startup, it is a loop repeated over the course of a day: a line is changed, the result is seen, another line is changed. The **feedback loop** is a quantity already measured in this curriculum's first course; that count is not repeated here. The question here is not the loop's duration, it is that under a containerized setup the loop has **two separate forms**, and the two pay different prices. The first form rebuilds the image and restarts the container. The second form mounts the source directory into the container and reloads the process in place. What gets measured is, per change, how many bytes each copies, how many layers it invalidates, how many steps it takes — and what the second one breaks in exchange. The measurement subject is the regional measurement network's reading collector service; it is fictional software, its source tree is really written to disk, and byte sizes and content digests are read from real files. **RT37 — the image definition consists of five steps:** base layer, dependency declaration, dependency install, source copy, build. **RT38 — the base layer is 41,943,040 and dependency install is 18,874,368 bytes**; these two numbers are a model, the remaining bytes come from real files. **RT39 — if a step's input changes, that step and every step after it are invalidated.** **RT40 — the ignore rules are five patterns**, and a file matching them does not enter the image. **RT41 — in the mount form the whole source directory is mounted**, ignore rules are not applied, and no copying happens. **RT42 — the feedback step is the number of discrete steps between a file being written and the result being seen.** #### The Layer Chain and Two Sets The first run builds the tree, decides which file enters the image using the ignore rules, and computes the chain of the five steps. Each step's digest depends on the previous digest and the content of its own inputs; this is why, once the chain breaks somewhere, everything behind it breaks too. ```js // source.mjs — real source tree, real byte sizes, real content digests. // Files and their sizes are real; image steps and the base layer/install byte counts are a MODEL. import { mkdirSync, writeFileSync, rmSync, statSync, readFileSync } from 'node:fs'; import { createHash } from 'node:crypto'; export const ROOT = './source'; export const FILES = { 'dependencies.json': '{"name":"reading-collector","version":"7.2.0",' + '"dependencies":{"parser":"2.4.1","record-client":"1.9.3","time":"4.0.2"}}\n', 'app/start.mjs': "import { collect } from './collector.mjs';\n" + "import { CONFIG } from './config.local.mjs';\ncollect(CONFIG.source);\n", 'app/collector.mjs': "import { format } from './format.mjs';\n" + 'export const collect = (k) => format(k);\n', 'app/verifier.mjs': "import { format } from './format.mjs';\n" + 'export const verify = (o) => format(o).length > 0;\n', 'app/format.mjs': 'export const format = (x) => String(x).trim();\n', 'app/config.local.mjs': "export const CONFIG = { source: 'local-meter', log: 'verbose' };\n", 'app/sample-reading.json': '[{"meter":"S-001","value":1420},{"meter":"S-002","value":980}]\n', 'test/collector.test.mjs': "import { collect } from '../app/collector.mjs';\n" + "console.log(collect(' 12 '));\n", 'notes.md': '# local notes\n- meter data is read from a sample file\n', '.secret-config': 'repo-token=local-development\n', }; // RT40: excluded from the image. The mount form does not apply these rules, it mounts the whole directory. export const IGNORE = [/^test\//, /\.local\./, /(^|\/)sample-/, /^notes\.md$/, /(^|\/)\.secret-/]; export const inImage = (path) => !IGNORE.some((r) => r.test(path)); // RT37/RT38: five steps. When a step names an output, it is a MODEL byte count; otherwise it is the real byte size of its inputs. export const STEPS = [ { name: 'base layer', prefix: [], output: 41943040 }, { name: 'dependency declaration', prefix: ['dependencies.json'] }, { name: 'dependency install', prefix: ['dependencies.json'], output: 18874368 }, { name: 'source copy', prefix: ['app/'] }, { name: 'build', prefix: ['app/'], build: true }, ]; export function setup() { rmSync(ROOT, { recursive: true, force: true }); for (const [path, content] of Object.entries(FILES)) { const dir = path.includes('/') ? `${ROOT}/${path.slice(0, path.lastIndexOf('/'))}` : ROOT; mkdirSync(dir, { recursive: true }); writeFileSync(`${ROOT}/${path}`, content); } return Object.keys(FILES).map((path) => ({ path, bytes: statSync(`${ROOT}/${path}`).size })); } const digest = (m) => createHash('sha256').update(m).digest('hex'); const inputsOf = (step, tree) => tree.filter((d) => inImage(d.path) && step.prefix.some((o) => d.path === o || d.path.startsWith(o))); // The build step really runs: the source modules entering the image are joined into a single output file. export function build(tree) { const parts = inputsOf(STEPS[4], tree).filter((d) => d.path.endsWith('.mjs')) .map((d) => readFileSync(`${ROOT}/${d.path}`, 'utf8')); mkdirSync(`${ROOT}/dist`, { recursive: true }); writeFileSync(`${ROOT}/dist/bundle.mjs`, parts.join('\n')); return statSync(`${ROOT}/dist/bundle.mjs`).size; } // Layer chain: each step's digest depends on the previous digest and the content of its own inputs. export function chain(tree) { let previous = '0'.repeat(64); return STEPS.map((step) => { const inputs = inputsOf(step, tree); const content = inputs.map((d) => digest(readFileSync(`${ROOT}/${d.path}`))).join(''); previous = digest(previous + step.name + content); const bytes = step.output ?? (step.build ? build(tree) : inputs.reduce((a, d) => a + d.bytes, 0)); return { name: step.name, inputs: inputs.length, bytes, digest: previous.slice(0, 12) }; }); } if (import.meta.url === `file://${process.argv[1]}`) { const tree = setup(); const s = (x, n) => String(x).padStart(n); const t = (x) => x.toLocaleString('en-US'); console.log(`${'file'.padEnd(28)}${s('bytes', 7)}${s('in image', 10)}`); for (const d of tree) console.log(`${d.path.padEnd(28)}${s(d.bytes, 7)}${s(inImage(d.path) ? 'yes' : 'no', 10)}`); const image = tree.filter((d) => inImage(d.path)); console.log(`mounted set ${tree.length} files / ${t(tree.reduce((a, d) => a + d.bytes, 0))} bytes` + ` image set ${image.length} files / ${t(image.reduce((a, d) => a + d.bytes, 0))} bytes`); console.log(`\n${'step'.padEnd(22)}${s('inputs', 8)}${s('output bytes', 14)}${s('digest', 15)}`); for (const k of chain(tree)) { console.log(`${k.name.padEnd(22)}${s(k.inputs, 8)}${s(t(k.bytes), 14)}${s(k.digest, 15)}`); } } ``` ``` file bytes in image dependencies.json 120 yes app/start.mjs 112 yes app/collector.mjs 80 yes app/verifier.mjs 90 yes app/format.mjs 47 yes app/config.local.mjs 65 no app/sample-reading.json 63 no test/collector.test.mjs 78 no notes.md 54 no .secret-config 29 no mounted set 10 files / 738 bytes image set 5 files / 449 bytes step inputs output bytes digest base layer 0 41,943,040 adc5f56b5389 dependency declaration 1 120 9299c9307bbf dependency install 1 18,874,368 f08005c4bd3a source copy 4 329 536fde9c6bbc build 4 332 1558b519b9ef ``` The ratio between the table's two halves sets this lesson's whole measure. The source is four files and 329 bytes; the two steps beneath it are 60,817,408 bytes, roughly 185,000 times the source. Most changes land on the source, but most of the cost sits beneath it. This is the whole meaning of the layer chain: what is **above** the changed step is not rebuilt. #### Two Loops Per Change The second run really writes three change classes to a file, recomputes the chain, and compares which steps' digests changed. Each class is measured in two forms. ```js // loop.mjs — two feedback loops: rebuilding the image versus mounting the source and reloading. import { readFileSync, writeFileSync, statSync } from 'node:fs'; import { ROOT, FILES, STEPS, inImage, setup, chain, build } from './source.mjs'; const readTree = () => Object.keys(FILES).map((path) => ({ path, bytes: statSync(`${ROOT}/${path}`).size })); const CHANGE = { 'app/format.mjs': FILES['app/format.mjs'].replace('trim()', 'trim().slice(0, 64)'), 'dependencies.json': FILES['dependencies.json'].replace('2.4.1', '2.5.0'), 'notes.md': `${FILES['notes.md']}- review the ignore list\n`, }; // RT42: a feedback step = a discrete step between a file being written and the result being seen. const rebuildSteps = (g) => 1 + (g === 0 ? 1 : g + 3); // write + steps + stop + start + become ready const MOUNT_STEPS = 2; // write + process reload const baseline = chain(setup()); const s = (x, n) => String(x).padStart(n); const t = (x) => x.toLocaleString('en-US'); console.log(`${'change'.padEnd(22)}${'form'.padEnd(23)}${s('invalid layers', 16)}` + `${s('bytes copied', 17)}${s('steps', 7)}`); for (const [path, newContent] of Object.entries(CHANGE)) { setup(); writeFileSync(`${ROOT}/${path}`, newContent); const updated = chain(readTree()); const invalid = updated.filter((k, i) => k.digest !== baseline[i].digest); const bytes = invalid.reduce((a, k) => a + k.bytes, 0); const mountForm = path === 'dependencies.json' ? 'falls back to rebuild' : 'mount + reload'; console.log(`${path.padEnd(22)}${'rebuild'.padEnd(23)}${s(`${invalid.length}/${STEPS.length}`, 16)}` + `${s(t(bytes), 17)}${s(rebuildSteps(invalid.length), 7)}`); console.log(`${''.padEnd(22)}${mountForm.padEnd(23)}` + `${s(path === 'dependencies.json' ? '-' : `0/${STEPS.length}`, 16)}` + `${s(path === 'dependencies.json' ? '-' : '0', 17)}` + `${s(path === 'dependencies.json' ? '-' : MOUNT_STEPS, 7)}`); if (invalid.length) console.log(`${''.padEnd(22)}invalidated: ${invalid.map((k) => k.name).join(', ')}`); } // The cost of the mount form: how far the running environment is from production. const tree = setup(); const mounted = tree; // RT41: the whole directory is mounted const image = tree.filter((d) => inImage(d.path)); const IMPORT_RE = /from\s+'(\.[^']+)'/g; function unresolved(set) { const paths = new Set(set.map((d) => d.path)); const missing = []; for (const d of set.filter((x) => x.path.endsWith('.mjs'))) { const base = d.path.slice(0, d.path.lastIndexOf('/')); for (const [, target] of readFileSync(`${ROOT}/${d.path}`, 'utf8').matchAll(IMPORT_RE)) { const resolved = new URL(target, `file:///${base}/`).pathname.slice(1); if (!paths.has(resolved)) missing.push(`${d.path} -> ${target}`); } } return missing; } const built = build(tree); console.log(`\n${'environment'.padEnd(22)}${s('files', 7)}${s('bytes', 7)}${s('unresolved imports', 24)}`); for (const [label, set] of [['mounted source', mounted], ['production image', image]]) { console.log(`${label.padEnd(22)}${s(set.length, 7)}${s(set.reduce((a, d) => a + d.bytes, 0), 7)}` + `${s(unresolved(set).length, 24)}`); } for (const e of unresolved(image)) console.log(` breaks only in the image: ${e}`); console.log(`${mounted.length - image.length} files mounted but not in the image` + ` (${mounted.filter((d) => !inImage(d.path)).map((d) => d.path).join(', ')})`); console.log(`the build step never runs in the mount form: a single ${built}-byte output` + ` from ${image.filter((d) => d.path.endsWith('.mjs')).length} sources`); ``` ``` change form invalid layers bytes copied steps app/format.mjs rebuild 2/5 687 6 mount + reload 0/5 0 2 invalidated: source copy, build dependencies.json rebuild 4/5 18,875,149 8 falls back to rebuild - - - invalidated: dependency declaration, dependency install, source copy, build notes.md rebuild 0/5 0 2 mount + reload 0/5 0 2 environment files bytes unresolved imports mounted source 10 738 0 production image 5 449 1 breaks only in the image: app/start.mjs -> ./config.local.mjs 5 files mounted but not in the image (app/config.local.mjs, app/sample-reading.json, test/collector.test.mjs, notes.md, .secret-config) the build step never runs in the mount form: a single 332-byte output from 4 sources ``` A change to a source line is the most frequent class, and the difference between the two forms is clearest here: two of the five steps invalidate, 687 bytes are copied, and six steps pass; in the mount form, zero layers, zero bytes, and two steps. The threefold gap in step count matters more than the gap in byte count, because the byte count scales with the size of the source tree and the step count does not. The 687 bytes here come from a ten-file tree; if the tree were a thousand files, the bytes would climb into the hundreds of thousands, but the step count would still stay at six. What the six steps consist of is also part of the number: the file is written, the two invalidated steps re-run, the container stops, comes up again with the new one, and is waited on to become ready. None of this has anything to do with the content of the change; all of it is the cost of rebuilding the output and swapping the running process. For someone making forty source changes a day, the gap between the two forms is forty times four extra steps — a hundred and sixty stops, and each stop is a place waited on before the result is seen. What the mount form buys is not the 687 bytes not copied, it is these stops. The second row draws the boundary of the mount form. When the dependency declaration changes, four of the five steps invalidate, the bytes copied climb to 18,875,149 — roughly 27,000 times the source change — and the loop takes eight steps. The mount cannot touch this class: the dependencies do not sit in the source directory, they sit in the install step's output, so the loop falls back to a rebuild. The mount's gain holds **for a single change class**, and once you step outside that class, the first form's full cost comes back. The third row shows the quiet one. When an ignored file changes, no layer invalidates on the image side; this is the correct behavior. But the same change is **visible** in the mount form, because the file sits inside the mounted directory and the process can read it. The two forms give the same numbers and do different things. This row is the subject of the next section. #### The Cost of Mounting: A Drifting Environment The second half of the output counts the price the mount form pays. The tree running locally is ten files and 738 bytes; the image going to production is five files and 449 bytes. **Half** of the files the locally running process sees are absent from production. This gap is not an accident, it is the deliberate result of the ignore rules — and the mount form works precisely by skipping those rules. The concrete consequence of the gap is one line down. `start.mjs` imports `./config.local.mjs`. In the mounted set, this import resolves and the process comes up without a problem; in the production image, it does not resolve, because the file is not there. The measurement counts this: zero unresolved imports in the mounted set, one in the image. **The local loop cannot show this error in any run**, because the error's condition is exactly what the local loop removes. This is the name of the error class the mount form hides: a dependency remaining on a file the ignore rule removed from the image. The second hidden class is in the last line. The build step never runs in the mount form; the 332-byte output produced from four sources is never executed locally. What runs locally is the source itself; what runs in production is the build's product, and the two behaving the same is never tested anywhere. Together these two classes give the real price of developer experience: while the loop drops from six steps to two, the local environment drifts from production by five files, one unresolved dependency, and one step that never runs. **Where isolation is punctured**, in this lesson, is not a flaw, it is the tool itself. A bind mount deliberately punctures the boundary the image closes; the source directory is no longer inside the output, it is in the environment. The previous two lessons' measures connect directly here: the mounted directory's file ownership may not match the internal identity, and the compose file declares this mount separately for every service. The hole's measure is the three numbers above, and being measurable does not close the hole — it makes **knowing when to close it** possible. #### Summary - The source is four files and 329 bytes, the two steps beneath it are 60,817,408 bytes; most changes are at the top, most of the cost is at the bottom, and the layer chain does not rebuild what is above the changed step. - For a source line change, rebuild invalidates two of five layers, copies 687 bytes, and takes six steps; in the mount form, zero layers, zero bytes, two steps. - The byte gap grows as the tree grows, the step gap does not: the threefold step ratio is the part of this measure that is independent of the tree. - When the dependency declaration changes, four layers invalidate and 18,875,149 bytes are copied; the mount form cannot touch this class, the loop falls back to a rebuild. - The mounted tree is ten files and 738 bytes, the production image is five files and 449 bytes; an import that resolves locally does not resolve in the image, and the local loop cannot show this error in any run. - The build step never runs in the mount form: the 332-byte output produced from four sources is never executed locally. While the loop drops from six steps to two, the environment drifts from production by five files, one unresolved dependency, and one step that never runs. #### Next Step By the end of this lesson, isolation stands deliberately punctured: a directory mounted from outside, the source not in the image, the build step skipped. What was measured across three lessons was always part of the same axis — who runs, how many services see each other, which file comes from where. These holes are a deliberate choice in the local environment. The same choices can be made in production too, and there their costs are different: the privileges a container can request from the kernel, the filesystem staying writable, the path every removed restriction opens. The next lesson takes up runtime security on this same axis and counts, one by one, the parts of isolation that are optional: which is open by default, what breaks when it is closed, and which path stays open when it is left open. ### Runtime Security Source: https://academia.sh/en/courses/containers/runtime-security A running container's authority is measured with two restrictions: how many of fourteen default capabilities have a legitimate operation and what replaces the ones dropped, how many write attempts a read-only root drops, the surface the paths left writable open, and how many of twelve abuse attempts close. The previous lesson brought the development loop down from six steps to two, and it did this by **deliberately puncturing isolation**: the source was mounted, the build step was skipped, the environment drifted from production. All of that measurement was about speed. What the process running in production **can do** — which privilege it can request from the kernel, which file it can write to — was never asked. Two restrictions are measured: **capability restriction** and a **read-only root filesystem**. Both close what is open by default, and their costs are in the same entry: if what gets closed has a legitimate user, a rearrangement gets written in its place. The measurement subject is the regional measurement network's reading collector service; it is fictional, its mode bits and write attempts are real. **RT43 — the runtime gives the container fourteen open capabilities by default**; the names are written with their generic equivalents. **RT44 — the service's legitimate operations and the operation each capability wants are a fictional inventory.** **RT45 — the abuse set is twelve attempts**; every attempt wants either a capability or a write path, one wants neither. **RT46 — the read-only root is built with real mode bits.** **RT47 — the execute ban is a model: writable paths carry no execute bit.** **RT48 — a false positive is a rule-compliant operation being blocked.** #### Capability Set A capability names, on its own, one of the things the privileged identity can do. The default set is not a decision, it is a default kept broad; what gets measured is how much of it is used. ```js // capabilities.mjs — the capability set and the known abuse set (MODEL). // Capabilities are referred to by their generic names; no kernel constant is written. // [capability, legitimate operation wanting it (null if none), replacement when dropped] export const CAPABILITY = [ ['change ownership', 'data directory ownership', 'ownership is set at build time'], ['change identity', 'dropping identity at startup', 'the process starts rootless directly'], ['open raw socket', 'network diagnostics', 'diagnostics in a helper container'], ['kill process', 'stopping the nightly job', 'no capability needed under the same identity'], ['bypass file ownership', 'appending to a shared log', 'a shared group and mode bit'], ['bypass file permissions', null, '-'], ['bind privileged port', null, 'a high port is already in use'], ['trace processes', null, 'a dump is triggered from outside'], ['change time', null, 'the clock comes from the host'], ['create device node', null, '-'], ['load kernel module', null, '-'], ['read system log', null, '-'], ['exceed resource limit', null, '-'], ['perform mount', null, 'mounts are set up before start'], ]; // Abuse set (fictional): [attempt, capability it wants, write path it wants] export const ABUSE = [ ['roll back the clock', 'change time', null], ['load a module', 'load kernel module', null], ["read another identity's file", 'bypass file permissions', null], ['listen to raw packets', 'open raw socket', null], ['attach to a neighboring process', 'trace processes', null], ['access raw disk', 'create device node', null], ['open a new mount point', 'perform mount', null], ['modify application code', null, 'app/collector.mjs'], ['hook the startup script', null, 'app/start.sh'], ['move the endpoint in config', null, 'config/service.json'], ['write and run a binary in the scratch dir', null, 'tmp/payload.bin'], ['read from the mounted dir and exfiltrate', null, null], ]; // What the service actually writes in one nightly run. export const LEGIT_WRITES = ['log/verification.log', 'cache/tariff-1.json', 'data/state/progress.json', 'data/corrections/corrections.csv', 'tmp/interim-0001.tmp']; export const unused = CAPABILITY.filter(([, op]) => op === null).map(([c]) => c); export const used = CAPABILITY.filter(([, op]) => op !== null); if (import.meta.url === `file://${process.argv[1]}`) { const s = (x, n) => String(x).padStart(n); console.log(`${'capability with a legit op'.padEnd(30)}${'operation'.padEnd(32)}replacement when dropped`); for (const [c, op, replacement] of used) console.log(`${c.padEnd(30)}${op.padEnd(32)}${replacement}`); console.log(`\ndefault open ${CAPABILITY.length}, with a legit op ${used.length}` + `, needlessly open ${unused.length}`); const closes = (d) => ABUSE.filter(([, c]) => c && d.includes(c)).length; console.log(`\n${'restriction'.padEnd(32)}${s('closed', 9)}${s('escapes', 8)}${s('legit ops dropped', 20)}`); for (const [label, d] of [['drop unused capabilities', unused], ['drop all capabilities', CAPABILITY.map(([c]) => c)]]) { const k = closes(d); console.log(`${label.padEnd(32)}${s(`${k}/${ABUSE.length}`, 9)}${s(ABUSE.length - k, 8)}` + `${s(CAPABILITY.filter(([c, op]) => op && d.includes(c)).length, 20)}`); } console.log(`attempts a capability cannot close: ${ABUSE.filter(([, c]) => !c).length}, of which ` + `${ABUSE.filter(([, c, p]) => !c && p).length} want a write path`); } ``` ``` capability with a legit op operation replacement when dropped change ownership data directory ownership ownership is set at build time change identity dropping identity at startup the process starts rootless directly open raw socket network diagnostics diagnostics in a helper container kill process stopping the nightly job no capability needed under the same identity bypass file ownership appending to a shared log a shared group and mode bit default open 14, with a legit op 5, needlessly open 9 restriction closed escapes legit ops dropped drop unused capabilities 6/12 6 0 drop all capabilities 7/12 5 5 attempts a capability cannot close: 5, of which 4 want a write path ``` Of the fourteen capabilities, only **five** have a legitimate operation. The remaining nine are the one entry in the isolation budget whose cost is zero: dropping them closes **six** of the twelve attempts, and the legitimate operations dropped are **zero**. Closing an unused privilege is not a trade-off — this is the one place in this course where a cost-free gain shows up. When all five are dropped too, one more attempt closes, and five legitimate operations drop. The right-hand column writes what replaces them: ownership is set at build time, the process starts rootless directly, diagnostics move to a helper container. The five operations that drop are not a capability given up, they are **work moved somewhere else**; the cost is not bytes, it is writing. The last line draws the boundary: five attempts do not close by dropping capabilities, four of them want to write to a file. #### Read-Only Root The second restriction targets those four attempts. The run below builds the root with real mode bits — read-only directories at `0555`, the ones left writable at `0755` — and really makes nine write attempts. ```js // read-only-root.mjs — a read-only root on real directories: real mode bits, real write attempts. import { mkdirSync, writeFileSync, chmodSync, rmSync } from 'node:fs'; import { CAPABILITY, ABUSE, LEGIT_WRITES } from './capabilities.mjs'; const ROOT = './root'; const DIRS = ['app', 'config', 'data', 'data/state', 'data/corrections', 'log', 'cache', 'tmp']; const WRITABLE = ['data/state', 'data/corrections', 'log', 'cache', 'tmp']; const TARGETS = [...LEGIT_WRITES.map((y) => [y, 'legit', null]), ...ABUSE.filter(([, , y]) => y).map(([label, , y]) => [y, 'abuse', label])]; const release = () => { // mode bits are opened back up before deleting for (const d of [...DIRS].reverse()) { try { chmodSync(`${ROOT}/${d}`, 0o755); } catch { /* none */ } } }; const setup = (mode) => { release(); rmSync(ROOT, { recursive: true, force: true }); for (const d of DIRS) mkdirSync(`${ROOT}/${d}`, { recursive: true }); for (const d of DIRS) chmodSync(`${ROOT}/${d}`, mode(d)); // mode bits are really set }; const attempt = (y) => { try { writeFileSync(`${ROOT}/${y}`, 'x'); return true; } catch { return false; } }; const s = (x, n) => String(x).padStart(n); console.log(`${'layout'.padEnd(30)}${s('legit', 7)}${s('abuse', 8)}${s('closed', 9)}${s('false positive', 15)}`); let open = []; for (const [label, mode] of [['root writable', () => 0o755], ['root read-only', () => 0o555], ['read-only + writable paths', (d) => (WRITABLE.includes(d) ? 0o755 : 0o555)]]) { setup(mode); const r = TARGETS.map(([path, kind, abuser]) => [kind, attempt(path), abuser]); const m = r.filter(([t]) => t === 'legit'), k = r.filter(([t]) => t === 'abuse'); open = k.filter(([, ok]) => ok).map(([, , abuser]) => abuser); console.log(`${label.padEnd(30)}${s(`${m.filter(([, o]) => o).length}/${m.length}`, 7)}` + `${s(`${open.length}/${k.length}`, 8)}${s(k.length - open.length, 9)}` + `${s(m.filter(([, o]) => !o).length, 15)}`); } console.log(`left writable: ${WRITABLE.length}/${DIRS.length} paths; still open: ${open.join(', ')}`); console.log(`execute ban (model): 0 of the writable paths carry an x bit -> ${open.length} more attempt${open.length === 1 ? "" : "s"} close${open.length === 1 ? "s" : ""}`); // Cumulative: all capabilities dropped + read-only root + writable paths + execute ban. setup((d) => (WRITABLE.includes(d) ? 0o755 : 0o555)); const escaping = ABUSE.filter(([, c, p]) => !c && !p).map(([label]) => label); const legitCount = CAPABILITY.filter(([, op]) => op).length; console.log(`\ncumulative: closed ${ABUSE.length - escaping.length}/${ABUSE.length}, escaping ` + `${escaping.length} (${escaping.join(', ')})`); console.log(`remaining surface: ${WRITABLE.length} writable paths; open capabilities ${CAPABILITY.length} -> 0;` + ` ${legitCount} legit operations dropped, ${legitCount} replacements written`); console.log(`is the application still running: ${LEGIT_WRITES.filter((y) => attempt(y)).length}/${LEGIT_WRITES.length}` + ' legit writes succeed'); release(); rmSync(ROOT, { recursive: true, force: true }); ``` ``` layout legit abuse closed false positive root writable 5/5 4/4 0 0 root read-only 0/5 0/4 4 5 read-only + writable paths 5/5 1/4 3 0 left writable: 5/8 paths; still open: write and run a binary in the scratch dir execute ban (model): 0 of the writable paths carry an x bit -> 1 more attempt closes cumulative: closed 11/12, escaping 1 (read from the mounted dir and exfiltrate) remaining surface: 5 writable paths; open capabilities 14 -> 0; 5 legit operations dropped, 5 replacements written is the application still running: 5/5 legit writes succeed ``` The second row is this lesson's most instructive one. When the root is made entirely read-only, all four of the four write attempts close — this is the layout with the highest close count — but **all five of the five legitimate writes drop too**, and the service does not run at all. Measuring a restriction only by how many attempts it closes reports the layout the application does not run in as the best layout. The third row strikes the balance: when five paths are left writable, the legitimate writes come back, the false positive count zeroes out, and the closed count drops from four to three. The attempt that gets away is writing a binary into the `tmp` directory. **This is where isolation is punctured:** the hole a read-only root opens is the set of paths left writable, and that set is exactly as large as the application needs it to be — five of eight directories. The execute ban closes this attempt too, because none of those five paths carries an execute bit. The cumulative row stacks the two on top of each other: **eleven** of the twelve attempts close, one gets away, and the application makes all five of its five legitimate writes. The attempt that gets away wants neither a capability nor a write; read permission was granted back in the volumes lesson, and, as the networking lesson measured for the sixteen in-network endpoints, it does not ask for identity. **Restrictions narrow the authority that can be requested, they do not take back what has already been granted.** #### Summary - Five of the fourteen default capabilities have a legitimate operation. Dropping the remaining nine closes six of the twelve attempts and drops zero legitimate operations — the one cost-free entry in this course. - When all five are dropped too, one more attempt closes and five legitimate operations drop; a rearrangement is written in place of each one, so the cost is not bytes, it is writing. - When the root is made entirely read-only, all four write attempts close too, but so do all five legitimate writes: the layout that closes the most is the layout the application does not run in. - When five of eight directories are left writable, legitimate writes return to 5/5, the closed count drops to three, and the remaining attempt only closes through the execute ban. - In the cumulative layout, eleven of the twelve attempts close and the application runs; the attempt that gets away has already been granted read permission and network access. #### Course Wrap-Up | Lesson | Difference removed from the environment | Cost of isolation | Where isolation is punctured | |---|---|---|---| | Why Containers | 6 of 18 differences in the output | 84.11 MB; 14.01 MB per difference | Resource limit outside; clock and name matching in the kernel | | Servers, Virtual Machines, and Containers | 6/18 (server 0, virtual machine 7) | 12 MB memory, 300 ms, 84 MB disk; overhead 6.3% | Kernel shared: clock, version, defects | | Underlying Technologies | Only the union filesystem; namespace 0, control group 0 | 4,096-byte copy for 16 bytes | Number pool shared; weight is not a ceiling; 1,024 deleted bytes remain beneath | | Container Standards | Output 100%, behavior 70% | Verification reads 83,650,787 bytes | 3 of 5 properties left to interpretation differ | | Container Lifecycle | Zero residue if cleaned up | Stopped: 436 MB, 1 name, 1 volume; 13,876 MB per period | Only the process record drops; the volume remains | | Image and Layer Model | Filesystem layout in a 22,758-byte store | 30,486 → 22,758 bytes; one line invalidates 8 chain ids | 1,654 bytes outside the view, still in the store | | Image Definition File | Dependency version to a single line | Context 15,681 bytes; 5,436 not entering the image | Records what was copied: 6,860 excess bytes | | Layer Cache | Work moved to the store, did not leave the environment | 21 layers in the bad order, 278,917 bytes (142.2%) | Measures the key text: grew 550 bytes, 4 instructions hit | | Multi-Stage Build | 185,826 bytes out of the run (86.4%) | 215,160 bytes that never enter an image | Copy boundary: the wide path brings back 4 files | | Base Image Selection | 148 / 31 / 6 components inside | 135.1 / 34.5 / 28.1 MB; 32 extra steps | Production outside: 133 updates a year | | Tagging Scheme | Traceability 9/24 → 24/24 | 96 bytes per image, 1 in 359,375 of the image | Ledger in the store: the name does not travel | | Image Repositories | 80 references, 27 files; 690.0 → 176.2 KB | Local 176.2 KB; 27.3 KB cannot be deleted | 9 of 16 layers of 7 private images sit in the public registry | | Image Security | Inventory and signature 1,088 bytes | 8 of 37 warnings are false positives | Deleted key 1,698 bytes beneath; 0 warnings, 1 open flaw | | Run Options | 14 variables in the environment (9 required) | 21 definition lines; 10.54 periods of delay | Limit is a ceiling: clipping climbs to 4,215 units | | Ephemeral Filesystem | Cross-run residue 0 | 1 byte → 98,304 bytes; layer 2.15x | Lower layer shared; ephemerality tied to deletion, not stopping | | Volumes and Bind Mounts | Path the same on every machine | 2 persistent directories; 7,951 bytes remain | Number space shared: 6 of 8 nodes cannot be written | | Container Networking | Service name instead of machine name | 1 network definition, 1 membership line (4 → 5) | 16 endpoints ask no identity; 4 paths in two hops | | Users and Permissions | 11 of 15 entries inside, 4 from the environment | 4 ownership fixes; writable 15 → 5 | Identity shared with host: 5/8 writable on the host | | Multi-Container Local Environment | 6 commands, 8 flags, 22 orderings into a file | 26 fields; 3 extra steps (3 → 6) | 4 services 1 network, 2 services 1 volume | | Developer Experience | Image set 10 → 5 files | 2 layers, 687 bytes, 6 steps / 0, 0, 2 | Deliberate hole: 1 unresolved import | | Runtime Security | 14 capabilities → 0 | 5 operations, 5 replacements; 5 paths writable | 1 of 12 attempts does not close; 5 paths open, kernel shared | The course's rule reads the same way across twenty-one lines: **isolation is a budget.** Every difference removed comes out in exchange for a byte, a run, or a resource — 14.01 MB, 4,096 bytes, 142.2% extra work, three extra steps — and **no isolation is complete.** Four holes repeat in the last column. The first is the **shared kernel**: clock value, system call surface, and kernel defects pass through the same place in three separate lessons. The second is that **what gets pulled into the output cannot be pulled out of the output**: the 1,654 bytes deleted from the view and the 1,699-byte key that stays in the layer both keep standing in the store. The third is **shared number spaces** — process number, user identity, port number; the namespace separates the view, not the number. The fourth is **the rule standing outside isolation**: the distributor enforcing the limit, the ledger resolving the tag, the host interface opening the port, and the runtime handing out the capability set are all outside the container. All of these measurements shared one boundary: every one of them was done on a **single machine**. The output became portable, its identity was derived from its content, its authority came down to a single line. But deciding how many copies to run, on which machine, and when is still a human being. Can this decision be handed off to a system? If it is, what does that system need to know: how many copies are standing, whether a copy is actually serving, how much room is left on which machine, how many copies it is safe to change at once when moving to a new version. **M22/K03 Container Orchestration** begins with this question.