Ephemeral Filesystem

The previous lesson measured two settings given to the container from outside: the resource limit and the environment variables. Both were known before the container started. But a running container does not sit idle. The verifier writes a log line for every reading, puts the tariff value it computed into a cache, records where the nightly run has gotten to somewhere, appends the operator’s hand-entered correction to a file. None of this data was in the image, and none of it came from the environment either — the container itself produced it.

The question is: where did it get written, how much space did it take up, and what happened to it once the container went away. The answer is isolation’s quietest cost, because it throws no error and exceeds no limit. Data is written, read, used for a while, and then one day it is gone.

The Writable Layer

The image is read-only, and this is not a preference but a consequence of the layer model: layers are addressed by content digest, if the content changes the digest changes, and if the digest changes the layer becomes a different layer. When a container needs to write, a writable layer is added on top of this stack. Reads resolve from top to bottom; writes always land on top.

The run below builds this arrangement with real directories. The union filesystem is a model — the read resolution and copy-up rule are hand-written — but the directories, files, and byte counts are real; all of it is written to disk and measured with node:fs.

• RT7 — The image’s lower layers consist of four files: application code, rule set, tariff table, meter list. The sizes are fictional; the files are really written.
• RT8 — The nightly run processes 480 meter readings.
• RT9 — Every reading produces a log line; every fourth reading a cache write; every sixtieth reading a progress write; every fortieth an operator correction.
• RT10 — At the end of every night, a one-byte change is made to the tariff table.
• RT11 — The container runs for 30 nights without ever restarting, then is deleted. Deletion removes the writable layer and does not touch the lower layers.

// runtime/writable-layer.mjs — counting what accumulates in the writable layer and what leaves when the container is deleted.
// MODEL: the union filesystem is modeled with node; directories, files, and byte counts are real.
import { mkdirSync, writeFileSync, appendFileSync, copyFileSync, existsSync, statSync,
  readdirSync, rmSync } from "node:fs";
import { join, dirname } from "node:path";

const LOWER = "root/lower", UPPER = "root/upper";        // read-only image layers / writable layer
const READINGS = 480, NIGHTS = 30, SEED = 20260801;       // RT8: nightly run, RT11: container lifetime
rmSync("root", { recursive: true, force: true });

const put = (root, path, content) => { mkdirSync(join(root, dirname(path)), { recursive: true });
  writeFileSync(join(root, path), content); };
for (const [path, n] of [["app/verifier.js", 12288], ["app/rules.json", 49152],
  ["config/tariff.csv", 98304], ["data/meter-list.csv", 204800]]) put(LOWER, path, "x".repeat(n));

let copiedBytes = 0, copiedCount = 0;
function appendToUpper(path, line) {                     // absent in upper, present in lower: copy-up
  const u = join(UPPER, path);
  if (!existsSync(u) && existsSync(join(LOWER, path))) {
    mkdirSync(dirname(u), { recursive: true });
    copyFileSync(join(LOWER, path), u);
    copiedBytes += statSync(u).size; copiedCount += 1;
  }
  mkdirSync(dirname(u), { recursive: true });
  appendFileSync(u, line);
}

let s = SEED % 2147483647;
const rand = () => (s = (s * 48271) % 2147483647) / 2147483647;
const counts = { log: 0, cache: 0, state: 0, user: 0, config: 0 };
function nightlyRun() {
  for (let i = 1; i <= READINGS; i += 1) {
    const value = (100 + rand() * 900).toFixed(3);
    appendToUpper("log/verification.log", `reading ${i} meter ${1000 + i} value ${value} status ok\n`);
    counts.log += 1;
    if (i % 4 === 0) { put(UPPER, `cache/tariff-${i / 4}.json`, `{"meter":${1000 + i},"unit":"${value}"}`); counts.cache += 1; }
    if (i % 60 === 0) { put(UPPER, "state/progress.json", `{"processed":${i},"remaining":${READINGS - i}}`); counts.state += 1; }
    if (i % 40 === 0) { appendToUpper("data/corrections.csv", `${1000 + i},operator,${value}\n`); counts.user += 1; }
  }
  appendToUpper("config/tariff.csv", "#");                // one-byte change, whole file gets copied
  counts.config += 1;
}

const measure = (root, on = "") => { let b = 0, n = 0;
  const walk = (d) => { for (const e of readdirSync(d, { withFileTypes: true })) {
    const p = join(d, e.name);
    if (e.isDirectory()) walk(p); else if (p.includes(on)) { b += statSync(p).size; n += 1; } } };
  if (existsSync(root)) walk(root); return { bytes: b, files: n }; };

const a = measure(LOWER);
const PREFIX = { log: "log/", cache: "cache/", state: "state/", user: "data/", config: "config/" };
console.log(`model: ${READINGS} readings/night, ${NIGHTS} nights (seed ${SEED}); lower layers ` +
  `${a.bytes} bytes / ${a.files} files, all read-only`);
console.log(`\n${"night".padEnd(6)}${"total bytes".padStart(12)}${"files".padStart(7)}` +
  Object.keys(PREFIX).map((k) => k.padStart(11)).join("") + `${"ratio to lower".padStart(19)}`);
for (let g = 1; g <= NIGHTS; g += 1) {
  nightlyRun();
  if (![1, 2, 7, 30].includes(g)) continue;
  const u = measure(UPPER);
  console.log(`${String(g).padEnd(6)}${String(u.bytes).padStart(12)}${String(u.files).padStart(7)}` +
    Object.values(PREFIX).map((o) => String(measure(UPPER, o).bytes).padStart(11)).join("") +
    `${(u.bytes / a.bytes).toFixed(2).padStart(19)}`);
}

const u = measure(UPPER);
console.log(`\ncopy-up ${copiedCount} files, ${copiedBytes} bytes; share of the layer first night ` +
  `${((100 * (copiedBytes + 1)) / 124772).toFixed(1)}%, night thirty ${((100 * copiedBytes) / u.bytes).toFixed(1)}%`);
rmSync(UPPER, { recursive: true, force: true });          // container is deleted
console.log(`container deleted: remaining ${measure(LOWER).bytes} bytes / ${measure(LOWER).files} files (lower layers), ` +
  `gone ${u.bytes} bytes / ${u.files} files`);
console.log(`records gone: ${counts.log} log lines, ${counts.cache} cache writes, ` +
  `${counts.state} progress writes, ${counts.user} operator corrections, ${counts.config} config changes`);
console.log(`if two containers ran from the same image: shared ${a.bytes + 2 * u.bytes} bytes, ` +
  `unshared ${2 * (a.bytes + u.bytes)} bytes; lower-layer sharing saves ${a.bytes} bytes`);
rmSync("root", { recursive: true, force: true });

model: 480 readings/night, 30 nights (seed 20260801); lower layers 364544 bytes / 4 files, all read-only

night  total bytes  files        log      cache      state       user     config     ratio to lower
1           124772    124      22452       3720         31        264      98305               0.34
2           147489    124      44904       3720         31        528      98306               0.40
7           261074    124     157164       3720         31       1848      98311               0.72
30          783565    124     673560       3720         31       7920      98334               2.15

copy-up 1 files, 98304 bytes; share of the layer first night 78.8%, night thirty 12.5%
container deleted: remaining 364544 bytes / 4 files (lower layers), gone 783565 bytes / 124 files
records gone: 14400 log lines, 3600 cache writes, 240 progress writes, 360 operator corrections, 30 config changes
if two containers ran from the same image: shared 1931674 bytes, unshared 2296218 bytes; lower-layer sharing saves 364544 bytes

These numbers are in the measurement class.

Copy-Up: One Byte Costing 98,304 Bytes

The first row looks wrong at first glance: the first night’s config column is 98,305 bytes. What was changed was a single byte.

The reason lies in the write rule. The writable layer cannot modify a file in the lower layer, because the lower layer is read-only and addressed by its digest. If a lower-layer file needs to be written to, the whole file is first copied to the upper layer, and then the copy is modified. This is called copy-up, and its cost is measured not by the size of the change but by the size of the file.

The first night’s makeup is skewed as a result. The genuine new data the run produces is 26,468 bytes — what is left once the 98,304 bytes copied are subtracted from the layer’s total. What sits in the layer is 124,772 bytes, that is, 4.7 times as much. The entire difference is a single copied file, and it fills 78.8% of the layer that night. The cost of isolation is stark here: what giving up the right to modify a file in place costs is 98,304 bytes of space and that much copying work, for a one-byte change.

The cost is paid once. The second night’s config column is 98,306: the file is now in the upper layer, the second byte is appended to its end, and nothing gets copied. The practical consequence is this — in the writable layer, which file is touched for the first time is more expensive than how much gets written to it afterward.

Over Thirty Nights, the Layer’s Shape Changes

The layer’s makeup inverts as the table’s rows go down. On the thirtieth night, the total is 783,565 bytes; the config column still sits at 98,334, and its share has dropped from 78.8% to 12.5%. Taking its place is the log column: 673,560 bytes, 86% of the layer. Log grows linearly because it adds 22,452 bytes every night; copy-up, on the other hand, stays constant because it is paid once.

The right-hand column gives the real fact. The writable layer’s ratio to the lower layers is 0.34 on the first night and climbs to 2.15 by the thirtieth. By the end of thirty nights, more than two-thirds of the space the container holds on disk is not the image’s — it is data the run produced. The phrase “the container is light” has a number behind it here, and the number that is true on day one is false on day thirty.

Two columns do not grow at all, and their reasons are different. cache stays fixed at 3,720 bytes, because the same 120 tariff keys are overwritten every night. state stays fixed at 31 bytes, because the progress file is overwritten on every write — of the 240 writes, only the last one remains. This is also why the file count stays at 124 over the thirty nights: the run does not open new files, it grows the ones that already exist.

What Leaves When the Container Is Deleted

After deletion, two numbers remain: 364,544 bytes / 4 files stand in the lower layers, and 783,565 bytes / 124 files left the upper layer. The lower layer is untouched — the image is whatever it is. What left is what the container produced itself, and its breakdown by record is: 14,400 log lines, 3,600 cache writes, 240 progress writes, 360 operator corrections, 30 config changes.

The gap between record counts and byte counts is itself informative. The cache was written 3,600 times but left 120 files and 3,720 bytes in the layer: 97% of the writes were overwritten by a later write. The writable layer is not a log, it is a snapshot — it holds not what was written, but what was written last. This is why what gets deleted is not the sum of what was written either, but its final state.

The progress row should also be read separately: 31 bytes sat in the layer, but the write count was 240. The information on how far the nightly run had gotten was overwritten every time and went away with the container. A new container starts that night from zero; all 480 readings are processed again.

Which class of loss is acceptable is the measurement network’s operating rule (RT12) and, with the thirty-night numbers, gives the table below:

Data class	Records	Bytes	Is loss acceptable	Rationale
Log	14,400 lines	673,560	Conditional	Acceptable if lines are shipped out; if the only copy is on the container, an incident review goes with it too
Cache	3,600 writes	3,720	Yes	Reproducible; the cost is slower initial requests
Job state	240 writes	31	No	An unfinished run starts over, 480 readings get reprocessed
User data	360 corrections	7,920	No	Not reproducible; the operator’s hand entry is the only source
Config copy	30 changes	98,334	Yes	Comes back from the image or the config source

The table reads backward: the two unacceptable classes hold only 1% of the layer, 7,951 bytes. The remaining 99% is reproducible data. The problem is not size, it is mixture: 7,951 bytes of unrecoverable data sit inside 775,614 bytes of recoverable data, and both get deleted at the same instant.

The Difference Removed from the Environment

This loss also has a counterpart, and the course’s measure requires asking about it: what difference does the ephemeral layer remove from the environment.

The difference it removes is cross-run residue. A container deleted and recreated every night runs every night from the same 364,544-byte, four-file starting point; its writable layer never exceeds 124,772 bytes, and the difference between two nights’ starting states is exactly zero bytes. A half-finished log, a bloated cache, the previous night’s hand-corrected config do not carry over to the next night. This is the measure by which a container counts as “stateless”: zero difference between runs.

The container that lives thirty nights does not remove this difference. It walks into the thirtieth night with its layer already 2.15 times the lower layers, and that night’s behavior depends on the residue of the previous twenty-nine nights. Where a problem needs to be looked for is no longer just the image, it is that layer’s contents too — in other words, the difference thought to have been removed from the environment has come back.

The cost is the two rows in the table. Whatever brings the cross-run difference to zero also deletes the 240 progress writes and the 360 operator corrections at the same time. Statelessness is not free: the price of zeroing the cross-run residue is also zeroing the 7,951 bytes of unrecoverable data inside that residue. This is exactly the problem the next lesson solves.

Where Isolation Is Punctured

There are two holes, and one of them is deliberate.

The deliberate one is lower-layer sharing. If two containers run from the same image, total disk usage is not 2,296,218 bytes, it is 1,931,674 bytes; the lower layers’ 364,544 bytes are held once and visible to both containers at the same time. Isolation is deliberately punctured here, because without the hole the same content would be stored twice. Because it is read-only, sharing has no effect on behavior — but the source is single.

The second hole is unintentional, and its name comes from a misunderstanding. Ephemerality is not a filesystem property, it is a lifecycle promise. The writable layer is not in memory; it is a directory on the host machine’s disk. When the container is stopped, that directory keeps standing; 783,565 bytes stay in place, and they come back if the container is restarted. Data goes away only when the container is deleted. This has two consequences: a stopped container still holds on disk the data it is assumed to have lost; and a container that is never deleted grows the layer it assumes is temporary without limit — to 2.15 times over thirty nights.

Summary

• The writable layer is a separate unit added on top of the read-only lower layers; reads resolve from top to bottom, writes always land on top.
• Copy-up cost is measured by file size: a one-byte change in the tariff table caused 98,304 bytes to be copied and filled 78.8% of the layer on the first night. The genuine new data produced that night is 26,468 bytes; what sits in the layer is 124,772 bytes. The cost is paid once: no copying on the second night.
• Over thirty nights the layer’s shape turns over: the total climbs to 783,565 bytes, the log’s share rises to 86% while the copy-up share drops to 12.5%. The writable layer’s ratio to the lower layers climbs from 0.34 to 2.15.
• When the container is deleted, 14,400 log lines, 3,600 cache writes, 240 progress writes, 360 operator corrections, and 30 config changes go away; the lower layer’s 364,544 bytes remain.
• The acceptability of loss depends on the data class, and mixture is dangerous: the two unacceptable classes hold only 1% of the layer, 7,951 bytes, but they get deleted at the same instant as the remaining 775,614 bytes.
• Where isolation is punctured: the lower layer is shared between two containers, and 364,544 bytes are held once; also, ephemerality is not a filesystem property — the writable layer is a directory on the host machine’s disk, and it goes away not when the container is stopped, but when it is deleted.

Next Step

Loss was unacceptable in two classes: job state and user data. Together they hold 7,951 bytes, and both get deleted because they sit inside the writable layer. The fix is obvious — put those two files outside the layer, somewhere not tied to the container’s lifecycle. But outside the layer is the host machine’s filesystem, and every path opened onto it takes something away from isolation. The next lesson builds two mount forms with real directories and counts this: how many bytes remain when the container is deleted, how many files cannot be written when the container’s internal user identity does not match the directory’s owner, and which of the two forms breaks when the machine changes.