Underlying Technologies

The previous lesson wrote down that the container separates four of the five criteria: file system, network, process table, and user. It did not ask how it separates them. The question is not empty, because the things being separated are singular on a single machine — there is one process table, one memory, one disk. “Separating” cannot mean producing a second one here; it must be something else.

This lesson models that other thing as three separate mechanisms. The namespace places a visibility mapping on top of the same table. The control group distributes the same resource as shares and makes a decision when the limit is exceeded. The union filesystem stacks the same files on top of one another as a layer. What the three have in common is this: none of them produces a new resource, all three place a rule on top of a single resource. Wherever there is a rule there is also a hole, and each section names its hole.

CC13. The three mechanisms are modeled separately; the state in which they work together is not modeled. CC14. In the namespace model there is a single global table, and a view deletes no record.

Namespace: A Visibility Mapping

// measurement-network/namespace.mjs — the namespace as a visibility mapping (model)
// There is a single global process table; the namespace places a visibility mapping on
// top of that table. A record is not deleted, only made invisible.

const TABLE = [                       // global table: number, name, owning namespace
  [1, "manager", "root"], [17, "log-collector", "root"], [41, "scheduler", "root"],
  [102, "reading-collector", "A"], [103, "reading-processor", "A"], [110, "reading-queue", "A"],
  [121, "verifier", "A"], [140, "verification-cache", "A"],
  [205, "billing", "B"], [206, "invoice-writer", "B"], [219, "tariff-resolver", "B"],
  [233, "work-order", "B"], [301, "nightly-job", "C"], [305, "nightly-writer", "C"],
];

// Visibility rule: a namespace sees only its own records and numbers them locally,
// starting at 1. Root sees every record.
function view(namespace) {
  const records = namespace === "root" ? TABLE : TABLE.filter((k) => k[2] === namespace);
  return records.map(([global, name], i) => ({ local: i + 1, global, name }));
}

const NAMESPACES = ["root", "A", "B", "C"];
console.log(`the global table has ${TABLE.length} records; no view deletes a record`);
console.log("namespace".padEnd(11) + "sees".padEnd(8) + "misses".padEnd(10) + "local -> global");
for (const a of NAMESPACES) {
  const v = view(a);
  console.log(a.padEnd(11) + `${v.length}/${TABLE.length}`.padEnd(8) +
    String(TABLE.length - v.length).padEnd(10) +
    v.slice(0, 3).map((k) => `${k.local}->${k.global}`).join(" ") + (v.length > 3 ? " ..." : ""));
}

console.log("\nthe same record carries two numbers:");
for (const a of ["A", "B", "C"]) {
  const first = view(a)[0];
  console.log(`  ${first.name.padEnd(20)} ${first.local} in ${a}, ${first.global} in root`);
}

const POOL = 32768;
console.log(`\nhole: the number pool is shared (${POOL} numbers, ${TABLE.length} in use)`);
const largest = NAMESPACES.slice(1)
  .map((a) => [a, view(a).length]).sort((x, y) => y[1] - x[1])[0];
console.log(`  root sees all ${view("root").length} records; namespaces do not see each other`);
console.log(`  the namespace opening the most records is ${largest[0]} (${largest[1]} records); ` +
  `a namespace exhausting the pool stops the others too`);
console.log(`  an unseen record still consumes resources: in the root view, ` +
  `${TABLE.length - view("A").length} records sit outside A but on the same machine`);

the global table has 14 records; no view deletes a record
namespace  sees    misses    local -> global
root       14/14   0         1->1 2->17 3->41 ...
A          5/14    9         1->102 2->103 3->110 ...
B          4/14    10        1->205 2->206 3->219 ...
C          2/14    12        1->301 2->305

the same record carries two numbers:
  reading-collector    1 in A, 102 in root
  billing              1 in B, 205 in root
  nightly-job          1 in C, 301 in root

hole: the number pool is shared (32768 numbers, 14 in use)
  root sees all 14 records; namespaces do not see each other
  the namespace opening the most records is A (5 records); a namespace exhausting the pool stops the others too
  an unseen record still consumes resources: in the root view, 9 records sit outside A but on the same machine

A namespace is not a deletion, it is a filter. All fourteen records stand in the table; view A sees five of them, B four, C two. The same record carries two numbers — the reading collector is 1 inside A, 102 inside root — and this duality shows exactly what the isolation rests on: it is not the number itself that is separated, but which table the number is read from.

The hole has three names. The root view sees all fourteen records; isolation does not work downward, only sideways. The number pool is shared: all 32,768 numbers are shared, and a namespace that exhausts the pool also blocks the others from opening records. And an unseen record is not a nonexistent one — the nine records A does not see keep consuming CPU and memory on the same machine. The cost of a namespace is not bytes, it is a mapping table; the environment difference it closes is zero.

Control Group: A Share Allocator

CC15. The hard-limited resource and the flexibly shared resource are modeled separately; requests arrive in sequence. CC16. The decision made when the limit is exceeded depends on the resource’s kind.

// measurement-network/cgroup.mjs — the control group as a share allocator (model)
// Two separate behaviors are modeled: the DECISION made when a hard-limited resource's
// limit is exceeded, and the DISTRIBUTION made by weight for a flexibly shared resource.

const GROUP = {                          // the fictional regional measurement network's three groups
  collection: { memoryLimitMB: 512, weight: 20, cpuDemand: 60 },
  billing:    { memoryLimitMB: 768, weight: 50, cpuDemand: 90 },
  nightlyJob: { memoryLimitMB: 256, weight: 30, cpuDemand: 80 },
};

// Hard limit: every request arrives in sequence; if exceeded, the decision depends on the resource's kind.
const DECISION = { memory: "the requesting process is stopped", descriptor: "the request is rejected" };
const REQUEST = {                         // sequential memory requests in MB
  collection: [180, 140, 120, 90], billing: [300, 260, 150, 200], nightlyJob: [120, 100, 90],
};

console.log("hard limit: memory (MB)");
console.log("group".padEnd(12) + "limit".padEnd(8) + "granted".padEnd(9) +
  "overflow".padEnd(12) + "decision (memory)");
let grantedTotal = 0, overflowTotal = 0;
for (const [name, requests] of Object.entries(REQUEST)) {
  const limit = GROUP[name].memoryLimitMB;
  let usage = 0, overflow = [];
  for (const r of requests) (usage + r <= limit) ? (usage += r) : overflow.push(r);
  grantedTotal += usage; overflowTotal += overflow.length;
  console.log(name.padEnd(12) + String(limit).padEnd(8) + String(usage).padEnd(9) +
    `${overflow.length} requests`.padEnd(12) + (overflow.length ? DECISION.memory : "-"));
}
console.log(`total granted ${grantedTotal} MB, overflow ${overflowTotal} requests; ` +
  `sum of limits ${Object.values(GROUP).reduce((t, g) => t + g.memoryLimitMB, 0)} MB`);
console.log(`the same overflow gives a different decision for a file descriptor: ${DECISION.descriptor}`);

// Flexible sharing: weight is not a limit, it is the share ratio under contention.
const TOTAL_SHARE = 100;
const weightTotal = Object.values(GROUP).reduce((t, g) => t + g.weight, 0);
const demandTotal = Object.values(GROUP).reduce((t, g) => t + g.cpuDemand, 0);
console.log(`\nflexible sharing: cpu (total ${TOTAL_SHARE} units, demand ${demandTotal} units)`);
console.log("group".padEnd(12) + "weight".padEnd(9) + "demand".padEnd(8) +
  "contested".padEnd(11) + "alone");
for (const [name, g] of Object.entries(GROUP)) {
  const share = Math.round((g.weight / weightTotal) * TOTAL_SHARE);
  console.log(name.padEnd(12) + String(g.weight).padEnd(9) + String(g.cpuDemand).padEnd(8) +
    String(Math.min(share, g.cpuDemand)).padEnd(11) + Math.min(TOTAL_SHARE, g.cpuDemand));
}
console.log("hole: weight is not a ceiling; with no contention a group takes " +
  `${Math.round(Math.min(TOTAL_SHARE, GROUP.nightlyJob.cpuDemand) / ((GROUP.nightlyJob.weight / weightTotal) * TOTAL_SHARE) * 10) / 10}` +
  " times its share");
console.log(`hole: the sum of the limits is ${Object.values(GROUP).reduce((t, g) => t + g.memoryLimitMB, 0)}` +
  " MB, independent of the machine's memory; overcommitment ends in a kernel decision");

hard limit: memory (MB)
group       limit   granted  overflow    decision (memory)
collection  512     440      1 requests  the requesting process is stopped
billing     768     710      1 requests  the requesting process is stopped
nightlyJob  256     220      1 requests  the requesting process is stopped
total granted 1370 MB, overflow 3 requests; sum of limits 1536 MB
the same overflow gives a different decision for a file descriptor: the request is rejected

flexible sharing: cpu (total 100 units, demand 230 units)
group       weight   demand  contested  alone
collection  20       60      20         60
billing     50       90      50         90
nightlyJob  30       80      30         80
hole: weight is not a ceiling; with no contention a group takes 2.7 times its share
hole: the sum of the limits is 1536 MB, independent of the machine's memory; overcommitment ends in a kernel decision

The control group has two separate behaviors, and mixing them up produces a wrong expectation. At the hard limit the decision is sharp: in all three groups one request does not fit the limit, and the decision made for memory is that the requesting process is stopped — not rejected, stopped. The same overflow gives a different decision for a file descriptor; the request is rejected and the process keeps running. This distinction carries a direct consequence for the fictional nightly job: with a 256 MB limit, 220 MB is granted, the fourth request does not fit, and the job stops midway.

In flexible sharing there is no limit, there is weight. Demand is 230 units, the machine is 100 units; the shares under contention come to 20, 50, and 30. But weight is not a ceiling: with no contention, the nightly job takes 2.7 times its share. The second hole is quieter — the sum of the three limits is 1,536 MB, and this number has nothing to do with the machine’s memory. Limits can be set larger than the machine’s memory; under overcommitment the decision is made not by the control group but by the kernel. The cost of a control group is not bytes either, it is bookkeeping; the environment difference it closes is again zero.

Union Filesystem: A Layer Stack

CC17. Layers are real directories and files are really written and measured; the delete marker is modeled as a file-name prefix. CC18. Only the top layer is writable. CC19. The sharing calculation assumes the lower layers are identical in content.

// measurement-network/layer.mjs — the union filesystem as a layer stack
// Layers are REAL directories; files are really written and really measured.
import { mkdirSync, writeFileSync, readdirSync, statSync, copyFileSync, rmSync } from "node:fs";
import { join, dirname } from "node:path";

const ROOT = "/tmp/measurement-network-layer";
const LAYER = ["layer0-base", "layer1-rules", "layer2-writable"];  // bottom to top
const MARKER = ".deleted.";                    // delete-marker prefix; hides the one below
const CONTENT = {
  "layer0-base": { "etc/tariff.json": 4096, "lib/resolver.bin": 61440,
    "lib/rules.bin": 20480, "app/collector.mjs": 3072, "app/old-config.json": 1024 },
  "layer1-rules": { "lib/rules.bin": 24576, "etc/rules-version.txt": 32 },
  "layer2-writable": { "app/collector.mjs": 3584, "app/new-rule.mjs": 2048,
    ["app/" + MARKER + "old-config.json"]: 0 },
};

rmSync(ROOT, { recursive: true, force: true });
for (const [k, files] of Object.entries(CONTENT))
  for (const [path, size] of Object.entries(files)) {
    mkdirSync(dirname(join(ROOT, k, path)), { recursive: true });
    writeFileSync(join(ROOT, k, path), Buffer.alloc(size, k.charCodeAt(5)));
  }

const list = (k, sub = "") => readdirSync(join(ROOT, k, sub), { withFileTypes: true })
  .flatMap((g) => g.isDirectory() ? list(k, join(sub, g.name)) : [join(sub, g.name)]);
const sizeOf = (k, path) => statSync(join(ROOT, k, path)).size;
const layerBytes = (k) => list(k).reduce((s, y) => s + sizeOf(k, y), 0);

// Merge: the topmost layer wins; a delete marker on top hides the one below.
const view = new Map(), shadowed = [], hidden = [];
for (const k of [...LAYER].reverse())
  for (const path of list(k)) {
    if (path.split("/").pop().startsWith(MARKER)) { hidden.push(path.replace(MARKER, "")); continue; }
    view.has(path) ? shadowed.push(`${path} (${view.get(path)} above)`) : view.set(path, k);
  }
for (const y of hidden) view.delete(y);

console.log("layer".padEnd(22) + "files".padEnd(8) + "bytes");
for (const k of LAYER)
  console.log(k.padEnd(22) + String(list(k).length).padEnd(8) + layerBytes(k));
console.log("\nmerged view (path -> serving layer)");
for (const [path, k] of [...view].sort()) console.log("  " + path.padEnd(28) + k);
console.log(`visible ${view.size} files, ` +
  `${[...view].reduce((s, [y, k]) => s + sizeOf(k, y), 0)} bytes`);
console.log(`shadowed ${shadowed.length}: ${shadowed.join(", ")}`);
console.log(`hidden by delete marker ${hidden.length}: ${hidden.join(", ")} ` +
  `(still ${sizeOf(LAYER[0], hidden[0])} bytes in the layer below)`);
console.log(`files copied for reading: 0 (${view.size} files read in place)`);

// Copy-up: writing to a file in a lower layer first copies it to the top layer.
const TARGET = "etc/tariff.json", OWNER = view.get(TARGET);
mkdirSync(dirname(join(ROOT, LAYER[2], TARGET)), { recursive: true });
copyFileSync(join(ROOT, OWNER, TARGET), join(ROOT, LAYER[2], TARGET));
writeFileSync(join(ROOT, LAYER[2], TARGET), Buffer.alloc(4096 + 16, 122));
console.log(`\ncopy-up: ${TARGET} is ${sizeOf(OWNER, TARGET)} bytes in ${OWNER}; ` +
  `${sizeOf(OWNER, TARGET)} bytes moved for a 16-byte change`);
console.log(`  the copy in the lower layer is unchanged (${sizeOf(OWNER, TARGET)} bytes), ` +
  `the one on top is ${sizeOf(LAYER[2], TARGET)} bytes`);

const N = 5;                                  // read-only lower layers are shared
const lowerBytes = layerBytes(LAYER[0]) + layerBytes(LAYER[1]), upperBytes = layerBytes(LAYER[2]);
console.log(`\nif ${N} instances share the lower layers, ${lowerBytes} + ${N} x ${upperBytes} = ` +
  `${lowerBytes + N * upperBytes} bytes; without sharing, ${N} x ${lowerBytes + upperBytes} = ${N * (lowerBytes + upperBytes)} bytes ` +
  `(gain ${N * (lowerBytes + upperBytes) - (lowerBytes + N * upperBytes)} bytes)`);
rmSync(ROOT, { recursive: true, force: true });

layer                 files   bytes
layer0-base           5       90112
layer1-rules          2       24608
layer2-writable       3       5632

merged view (path -> serving layer)
  app/collector.mjs           layer2-writable
  app/new-rule.mjs            layer2-writable
  etc/rules-version.txt       layer1-rules
  etc/tariff.json             layer0-base
  lib/resolver.bin            layer0-base
  lib/rules.bin               layer1-rules
visible 6 files, 95776 bytes
shadowed 2: app/collector.mjs (layer2-writable above), lib/rules.bin (layer1-rules above)
hidden by delete marker 1: app/old-config.json (still 1024 bytes in the layer below)
files copied for reading: 0 (6 files read in place)

copy-up: etc/tariff.json is 4096 bytes in layer0-base; 4096 bytes moved for a 16-byte change
  the copy in the lower layer is unchanged (4096 bytes), the one on top is 4112 bytes

if 5 instances share the lower layers, 114720 + 5 x 9744 = 163440 bytes; without sharing, 5 x 124464 = 622320 bytes (gain 458880 bytes)

The three layers hold ten files, the merged view holds six. The difference comes from two shadowings and one delete marker: the rules binary and the collector are read from the top layer, and the old config file is never read at all. Not a single file is copied for reading — all six of the six files are read in place, in the layer where they sit. This is the number that corrects the previous lesson’s line “the library set is duplicated once per instance”: when five instances share the lower layers the total is 163,440 bytes, without sharing it would be 622,320 bytes; the gain is 458,880 bytes.

The cost is paid at the moment of writing. To write to a file in a lower layer, that file is first copied to the top layer: 4,096 bytes are moved for a 16-byte change. The ratio grows with file size, and if the write had gone to the resolver binary instead of the tariff file, the bytes moved would have been 61,440.

The hole is in three places. A deleted file is not deleted: the old config file is absent from the view but keeps standing at 1,024 bytes in the lower layer — anyone reading the output can find it. The copy-up delay only appears at the moment of writing and is invisible beforehand. And even though all the files are inside the output, file ownership and the name-matching rule come from the underlying file system; this is exactly the dimension that sat in the first lesson’s boundary bucket.

Summary

• None of the three mechanisms produces a new resource; all three place a rule on top of a single resource, and wherever there is a rule there is a hole.
• A namespace is a visibility mapping: in a 14-record table, the views see 5, 4, and 2 records, and root sees all 14. The number pool is shared, and an unseen record keeps consuming resources.
• A control group is a share allocator with two behaviors: a decision at the hard limit (stopping the process for memory, rejecting the request for a descriptor), and a weight-based distribution under flexible sharing. Weight is not a ceiling — with no contention a group takes 2.7 times its share.
• A union filesystem is a layer stack: 10 files reduce to 6, 0 files are copied for reading, and sharing across five instances saves 458,880 bytes. At the moment of writing, a 16-byte change moves 4,096 bytes.
• The only mechanism that pulls a difference out of the environment is the union filesystem; the namespace and the control group close 0 differences, they separate criteria. A deleted file keeps standing in the lower layer, and file ownership and the name-matching rule come from below.

Next Step

These three mechanisms belong to a single machine’s kernel and work the way that kernel offers. The measured layer stack, though, does not belong to a machine: it is a stack made of directories, and it can be copied and moved. So what guarantees it will open the same way wherever it is moved — who writes down the order the layers stack in, the configuration it runs with, and that a layer was not corrupted in transit? The next lesson takes up the common structure of the specifications that answer this question: it splits the promise carried by an image format specification into three parts, writes a small verifier that checks the digest chain, and counts how far portability actually goes — what stays the same and what changes between two runtimes that comply with the same specification.