Image Security

The previous lesson measured the registry and closed with a question: if a private image’s nine layers are readable in the public pool, what is inside those layers? Up to now, layers have been looked at as stacks of bytes. This lesson looks inside and closes the topic.

There are three measurements, and all three are parts of the same question: how is it known what sits inside an output at rest? The first is a leak that comes from the image’s own structure, the second is an attempt to find known defects, the third is verifying who the output came from.

IM43. Layers are real directories, and the union view is built bottom to top; a file’s deletion is shown by a marker file prefixed .deleted- in the upper layer. IM44. Secret values are fictional; no real secret is used.

A Secret Left Sitting in a Layer

M22/K01 scanned how many separate ways a secret can leak out; that scan is not repeated here. The fact here is single and layer-specific: a layer is an immutable object. An instruction that deletes a file produces a new layer; the lower layer stays exactly as it was, because its digest also appears in other images. Deletion is a view operation, not a store operation.

// measurement-network/leak.mjs — does a file deleted in an upper layer still sit in the lower layer (real directories)
import { mkdirSync, rmSync, writeFileSync, readdirSync, readFileSync, statSync } from "node:fs";
import { join, dirname } from "node:path";

const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-layer");
rmSync(ROOT, { recursive: true, force: true });

// Fictional secret values. No real secret is used.
const SECRET_A = "fictional-access-key-A1B2C3D4E5F6";
const SECRET_B = "fictional-db-password-Z9Y8X7";
const body = (secret, pad) => Buffer.concat([Buffer.from(`${secret}\n`), Buffer.alloc(pad, "-")]);

const LAYERS = [
  { name: "l1-base", write: { "bin/collector": 4096, "lib/crypto": 8192 } },
  { name: "l2-dependency", write: { "lib/client": 6144, "lib/queue": 3072 } },
  { name: "l3-application",
    write: { "app/collector.js": 5120, "app/secret/access.key": body(SECRET_A, 1664),
           "app/temp/setup.log": body(SECRET_B, 2048) },
    sameLayerDelete: ["app/temp/setup.log"] },   // removed before the layer closes
  { name: "l4-config", write: { "app/settings.json": 512 },
    upperLayerDelete: ["app/secret/access.key"] },  // a deletion marker is left in the upper layer
];

// Layers are written to real directories; a deletion marker is an empty file prefixed ".deleted-".
for (const k of LAYERS) {
  for (const [path, value] of Object.entries(k.write)) {
    if (k.sameLayerDelete?.includes(path)) continue;
    const full = join(ROOT, k.name, path);
    mkdirSync(dirname(full), { recursive: true });
    writeFileSync(full, typeof value === "number" ? Buffer.alloc(value, "x") : value);
  }
  for (const path of k.upperLayerDelete ?? []) {
    const full = join(ROOT, k.name, dirname(path), ".deleted-" + path.split("/").pop());
    mkdirSync(dirname(full), { recursive: true });
    writeFileSync(full, "");
  }
}

const walk = (root, sub = "") => readdirSync(join(root, sub), { withFileTypes: true })
  .flatMap((g) => g.isDirectory() ? walk(root, join(sub, g.name)) : [join(sub, g.name)]);

// Union view: bottom to top; a deletion marker removes the visible file.
const view = new Map();
for (const k of LAYERS) {
  for (const path of walk(join(ROOT, k.name))) {
    const name = path.split("/").pop();
    if (name.startsWith(".deleted-")) view.delete(join(dirname(path), name.slice(9)));
    else view.set(path, statSync(join(ROOT, k.name, path)).size);
  }
}

const onDisk = LAYERS.flatMap((k) => walk(join(ROOT, k.name))
  .filter((y) => !y.split("/").pop().startsWith(".deleted-"))
  .map((y) => [k.name, y, statSync(join(ROOT, k.name, y)).size]));
const viewBytes = [...view.values()].reduce((t, b) => t + b, 0);
const diskBytes = onDisk.reduce((t, [, , b]) => t + b, 0);
const hidden = onDisk.filter(([, y]) => !view.has(y));

console.log(`union view          : ${view.size} files, ${viewBytes} bytes`);
console.log(`really in layers    : ${onDisk.length} files, ${diskBytes} bytes`);
console.log(`absent from view, still in a layer: ${hidden.length} files, ` +
  `${hidden.reduce((t, [, , b]) => t + b, 0)} bytes`);

for (const [name, secret] of [["SECRET_A", SECRET_A], ["SECRET_B", SECRET_B]]) {
  const found = onDisk.filter(([k, y]) => readFileSync(join(ROOT, k, y)).includes(secret));
  console.log(`${name}: in ${found.length} layer files, ` +
    `${found.reduce((t, [, , b]) => t + b, 0)} bytes` +
    (found.length ? ` -> ${found.map(([k, y]) => `${k}/${y}`).join(", ")}` : ""));
}

union view          : 6 files, 27136 bytes
really in layers    : 7 files, 28834 bytes
absent from view, still in a layer: 1 files, 1698 bytes
SECRET_A: in 1 layer files, 1698 bytes -> l3-application/app/secret/access.key
SECRET_B: in 0 layer files, 0 bytes

The union view shows six files and 27,136 bytes. There are seven files and 28,834 bytes on disk. The 1,698-byte gap is the key file that appears deleted in the upper layer, and it sits whole in the l3-application layer. An audit that looks at the view cannot find it; whoever opens the layer reads it directly.

The difference between the two secrets completes the measurement. SECRET_B was written too, but removed before the same layer closed: it appears in no layer file, zero bytes. SECRET_A was deleted in an upper layer and sits at its full size. The only difference is which layer the deletion happened in, and the result is a binary difference — 1,698 bytes against 0 bytes.

This gets heavier when combined with the previous lesson’s measurement. A layer in the registry could only be deleted if no manifest referenced it; in the measurement, three of the dropped images’ seventeen layers were referenced, so they stayed on disk. A layer that carries a secret is subject to the same rule: if another image references it, the bytes stay in the registry even if the image carrying the secret is deleted. This is where isolation is pierced — the difference taken into the output cannot be taken back out of it.

The way to close it off is written into the measurement itself. SECRET_B‘s zero bytes were achieved by removing the value before the layer closed: a file written and deleted within the same instruction never enters the layer. A sturdier approach is to never write it at all — reading the value from a source visible only during the build step and never copying it into the output. The two paths’ measured outcome is the same, 1,698 bytes against 0 bytes; but the first depends on instruction order, the second on the output’s structure, and the first silently comes back the moment the order breaks.

What a Scan Sees

The second measurement compares the components inside the image against a known defect set. A scan is a matching process, and the matching has error in both directions.

IM45. The defect set is fictional: a component’s probability of having an open record is 0.30, the probability that the record’s fix was backported to the version is 0.25; the seed 20260909 is visible. IM46. Visibility depends on the candidate — on a base with a package manager, the inventory can be read; on one without, the component comes embedded inside the binary, and its version cannot be counted.

// measurement-network/scan.mjs — scanning against a known defect set: caught, missed, false positive (model)
const generator = (seed) => () =>
  (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;

const P_DEFECT = 0.30;    // the component has an open record in the defect set
const P_BACKPORTED = 0.25;   // it has a record, but the fix was backported to the version: not open

// visibility: probability that the component's version can be read from the inventory. On a base
// with no package manager there is no inventory; the component comes embedded inside the binary.
const CANDIDATE = [["full-featured", 148, 0.92], ["slimmed", 31, 0.82], ["single-binary", 6, 0.35]];

const scan = (components, visibility, seed = 20260909) => {
  const r = generator(seed);
  const s = { components, open: 0, warning: 0, caught: 0, missed: 0, falsePositive: 0 };
  for (let i = 0; i < components; i++) {
    const recorded = r() < P_DEFECT, visible = r() < visibility, backported = r() < P_BACKPORTED;
    const open = recorded && !backported;          // a genuinely open defect
    const flags = recorded && visible;              // the warning the scan produces
    if (open) s.open += 1;
    if (flags) s.warning += 1;
    if (open && flags) s.caught += 1;
    if (open && !flags) s.missed += 1;
    if (flags && !open) s.falsePositive += 1;
  }
  return s;
};

console.log("candidate".padEnd(15) + "components".padEnd(12) + "visibility".padEnd(12) +
  "open defect".padEnd(13) + "warning".padEnd(9) + "caught".padEnd(11) +
  "missed".padEnd(8) + "false positive");
const results = [];
for (const [name, n, g] of CANDIDATE) {
  const s = scan(n, g);
  results.push([name, s]);
  console.log(name.padEnd(15) + String(n).padEnd(12) + `%${(100 * g).toFixed(0)}`.padEnd(12) +
    String(s.open).padEnd(13) + String(s.warning).padEnd(9) + String(s.caught).padEnd(11) +
    String(s.missed).padEnd(8) + String(s.falsePositive));
}

console.log("");
const pct = (a, b) => (b ? `%${(100 * a / b).toFixed(0)}` : "unmeasured");
for (const [name, s] of results) {
  console.log(`${name.padEnd(15)} catch rate ${pct(s.caught, s.open)}, ` +
    `warning accuracy ${pct(s.caught, s.warning)}`);
}

candidate      components  visibility  open defect  warning  caught     missed  false positive
full-featured  148         %92         31           37       29         2       8
slimmed        31          %82         7            7        6          1       1
single-binary  6           %35         1            0        0          1       0

full-featured   catch rate %94, warning accuracy %78
slimmed         catch rate %86, warning accuracy %86
single-binary   catch rate %0, warning accuracy unmeasured

The full-featured base has 31 open defects; the scan produces 37 warnings, 29 of which are real, 8 are false positives, and 2 open defects are missed. The source of the false positives is a record whose fix was backported: the version number appears in the flawed range, the code does not. The source of the misses is a component that cannot be read from the inventory.

The last row is this lesson’s harshest number. In the single-binary base, the scan produces zero warnings, but there is one open defect. A clean report does not mean a defect-free image; a component that cannot be counted goes uncounted, and nothing marks the gap. The trade-off measured in the previous lesson shows up here a second time — the small base really carries fewer defects (148 components against 6), but it also lowers the visibility of what it does carry.

The scan’s cost can be counted: 8 of 37 warnings were looked at for nothing, and each one requires manually comparing a component’s version against the defect record by hand. The way to raise the catch rate is not more warnings but a readable inventory — carrying the component list alongside the image as a separate document closes this gap, and its size is a few kilobytes.

The Signing Chain

The third measurement is the identity question. Scanning says “what is inside”; a signature says “did we really produce this.” The chain has three links: the content digest, that digest’s signature, and verification at pull time.

IM47. Signing is done with a real key pair, but the key is regenerated on every run; that is why what is printed is the signature’s length and the verification result, not the signature itself. IM48. Recomputing a pulled layer’s digest at pull time is an optional step.

// measurement-network/signature.mjs — signing chain: digest, signature, verification (with a real key pair)
import { generateKeyPairSync, sign, verify, createHash } from "node:crypto";

const digest = (b) => createHash("sha256").update(b).digest();
// The key pair is regenerated on every run. Signature bytes depend on the run; that is
// why what is printed is the signature's length and the verification result, not the signature itself.
const { publicKey, privateKey } = generateKeyPairSync("ed25519");

// The previous lesson's 17 distinct images: every manifest is a list of four layer digests.
const manifest = (i) => Buffer.from([1, 2, 3, 4]
  .map((k) => digest(Buffer.from(`layer-${k}-${i}`)).toString("hex").slice(0, 12)).join("\n"));
const MANIFEST = Array.from({ length: 17 }, (_, i) => manifest(i));
const SIGNATURE = MANIFEST.map((m) => sign(null, digest(m), privateKey));

console.log(`signed manifests ${MANIFEST.length}, signature length ${SIGNATURE[0].length} bytes, ` +
  `total ${MANIFEST.length * SIGNATURE[0].length} bytes`);
console.log(`registry 176200 bytes; signatures' share is ` +
  `${(1000 * MANIFEST.length * SIGNATURE[0].length / 176200).toFixed(1)} per thousand`);

// The chain's links are tested one at a time.
const m0 = MANIFEST[0];
const tampered = Buffer.from(m0.toString().replace(/^.{12}/, "ffffffffffff"));
const { publicKey: otherKey } = generateKeyPairSync("ed25519");
console.log("\ncorrect manifest, correct signature :", verify(null, digest(m0), publicKey, SIGNATURE[0]));
console.log("tampered manifest                   :", verify(null, digest(tampered), publicKey, SIGNATURE[0]));
console.log("verification with another key       :", verify(null, digest(m0), otherKey, SIGNATURE[0]));

// The layer bytes themselves: even if the manifest is correct, the downloaded bytes can still change.
const layer = Buffer.alloc(7200, "u");
const expected = digest(layer).toString("hex");
const corrupted = Buffer.from(layer);
corrupted[42] = 0x41;                               // one byte changed
console.log("downloaded layer's digest holds:", digest(corrupted).toString("hex") === expected);

// 24 pulls, three disciplines: what would each accept?
const PULLS = 24, LAYERS = 4;
console.log("\ndiscipline".padEnd(36) + "forged manifest accepted".padEnd(29) + "unverified layer");
for (const [label, checkSignature, checkLayer] of [
  ["signature not verified", false, false],
  ["only signature verified", true, false],
  ["signature and layer digest verified", true, true],
]) {
  console.log(label.padEnd(36) + `${checkSignature ? 0 : PULLS}/${PULLS}`.padEnd(29) +
    `${checkLayer ? 0 : PULLS * LAYERS}/${PULLS * LAYERS}`);
}

signed manifests 17, signature length 64 bytes, total 1088 bytes
registry 176200 bytes; signatures' share is 6.2 per thousand

correct manifest, correct signature : true
tampered manifest                   : false
verification with another key       : false
downloaded layer's digest holds: false

discipline                         forged manifest accepted     unverified layer
signature not verified              24/24                        96/96
only signature verified             0/24                         96/96
signature and layer digest verified 0/24                         0/96

The signature’s cost is the smallest item: 1,088 bytes total for seventeen manifests, 6.2 per thousand of the registry. Verification itself is nothing more than a digest comparison.

The middle three rows show the chain working: the correct manifest verifies with the correct signature, verification fails once the manifest’s first twelve characters are changed, and it fails when the signature is tested with a different key too. The fourth row shows where the chain ends — when a single byte of the downloaded layer changes, the digest does not hold, but this is only seen if the digest is recomputed.

The last table counts the path each skipped verification step opens. With no verification, 24 of 24 pulls accept a modified manifest. When only the signature is verified, forged manifests drop to zero, but 96 of 96 layers get accepted with their digest untested: the manifest is correct, but whether the downloaded bytes match the manifest is unknown. When both links are tested, both numbers are zero.

There is something that sits outside the chain, and it is where isolation is pierced here: a signature does not say “this image is safe”; it says “this manifest was signed with this key.” Who the key belongs to, where it is stored, and who is allowed to sign are outside the model. Signing determines where trust gets moved to; it does not eliminate it.

The chain also connects to the previous lesson’s traceability ladder. The source version field says which source the image came from; the signature says that declaration did not change after it was produced. If the field is not signed, it is a note that can be written after the fact; without a signature, the traceability rate measures a record discipline, not a guarantee.

Summary

• A layer is an immutable object; deletion changes the view, not the bytes. In the measurement, the union view shows 6 files and 27,136 bytes while there are 7 files and 28,834 bytes on disk.
• The gap is 1,698 bytes and is the whole of the key file that appears deleted in the upper layer. The second secret, removed before the same layer closed, appears in no layer file at all: 1,698 bytes against 0 bytes.
• If a layer carrying a secret is referenced, it stays in the registry even if the image carrying the secret is deleted; a difference taken into the output cannot be taken back out of it.
• Scanning is wrong in both directions: in the full-featured base, 29 of 31 open defects are caught, 2 are missed, and 8 of 37 warnings are false positives. In the single-binary base, the scan produces 0 warnings but there is 1 open defect — a component that cannot be counted goes uncounted, and nothing marks the gap.
• Signing is cheap: 1,088 bytes for 17 manifests, 6.2 per thousand of the registry. With no verification, 24 of 24 pulls accept a forged manifest; with only the signature verified, 96 of 96 layers get accepted with their digest untested. A signature does not eliminate trust, it moves it to the key.

Next Step

Across this topic, the output got smaller, split into layers, shared, named, scanned, and signed. Every number measured — bytes, layers, references, warnings, signatures — was measured on an object at rest. The object sat on disk and did nothing.

Yet this output’s reason to exist is to run. The moment it starts running, the files inside it become files a process sees, the layers stack up and turn into a writable surface, and what was verified by the signature gets loaded into memory. Which part of isolation survives this transition? Are the layers measured as read-only still read-only, how much of the difference taken into the image comes back from the environment the moment it runs, and what now determines what a process is allowed to do?