Runtime Security

The previous lesson brought the development loop down from six steps to two, and it did this by deliberately puncturing isolation: the source was mounted, the build step was skipped, the environment drifted from production. All of that measurement was about speed. What the process running in production can do — which privilege it can request from the kernel, which file it can write to — was never asked.

Two restrictions are measured: capability restriction and a read-only root filesystem. Both close what is open by default, and their costs are in the same entry: if what gets closed has a legitimate user, a rearrangement gets written in its place. The measurement subject is the regional measurement network’s reading collector service; it is fictional, its mode bits and write attempts are real.

RT43 — the runtime gives the container fourteen open capabilities by default; the names are written with their generic equivalents. RT44 — the service’s legitimate operations and the operation each capability wants are a fictional inventory. RT45 — the abuse set is twelve attempts; every attempt wants either a capability or a write path, one wants neither. RT46 — the read-only root is built with real mode bits. RT47 — the execute ban is a model: writable paths carry no execute bit. RT48 — a false positive is a rule-compliant operation being blocked.

Capability Set

A capability names, on its own, one of the things the privileged identity can do. The default set is not a decision, it is a default kept broad; what gets measured is how much of it is used.

// capabilities.mjs — the capability set and the known abuse set (MODEL).
// Capabilities are referred to by their generic names; no kernel constant is written.

// [capability, legitimate operation wanting it (null if none), replacement when dropped]
export const CAPABILITY = [
  ['change ownership', 'data directory ownership', 'ownership is set at build time'],
  ['change identity', 'dropping identity at startup', 'the process starts rootless directly'],
  ['open raw socket', 'network diagnostics', 'diagnostics in a helper container'],
  ['kill process', 'stopping the nightly job', 'no capability needed under the same identity'],
  ['bypass file ownership', 'appending to a shared log', 'a shared group and mode bit'],
  ['bypass file permissions', null, '-'],
  ['bind privileged port', null, 'a high port is already in use'],
  ['trace processes', null, 'a dump is triggered from outside'],
  ['change time', null, 'the clock comes from the host'],
  ['create device node', null, '-'],
  ['load kernel module', null, '-'],
  ['read system log', null, '-'],
  ['exceed resource limit', null, '-'],
  ['perform mount', null, 'mounts are set up before start'],
];

// Abuse set (fictional): [attempt, capability it wants, write path it wants]
export const ABUSE = [
  ['roll back the clock', 'change time', null],
  ['load a module', 'load kernel module', null],
  ["read another identity's file", 'bypass file permissions', null],
  ['listen to raw packets', 'open raw socket', null],
  ['attach to a neighboring process', 'trace processes', null],
  ['access raw disk', 'create device node', null],
  ['open a new mount point', 'perform mount', null],
  ['modify application code', null, 'app/collector.mjs'],
  ['hook the startup script', null, 'app/start.sh'],
  ['move the endpoint in config', null, 'config/service.json'],
  ['write and run a binary in the scratch dir', null, 'tmp/payload.bin'],
  ['read from the mounted dir and exfiltrate', null, null],
];

// What the service actually writes in one nightly run.
export const LEGIT_WRITES = ['log/verification.log', 'cache/tariff-1.json',
  'data/state/progress.json', 'data/corrections/corrections.csv', 'tmp/interim-0001.tmp'];

export const unused = CAPABILITY.filter(([, op]) => op === null).map(([c]) => c);
export const used = CAPABILITY.filter(([, op]) => op !== null);

if (import.meta.url === `file://${process.argv[1]}`) {
  const s = (x, n) => String(x).padStart(n);
  console.log(`${'capability with a legit op'.padEnd(30)}${'operation'.padEnd(32)}replacement when dropped`);
  for (const [c, op, replacement] of used) console.log(`${c.padEnd(30)}${op.padEnd(32)}${replacement}`);
  console.log(`\ndefault open ${CAPABILITY.length}, with a legit op ${used.length}`
    + `, needlessly open ${unused.length}`);

  const closes = (d) => ABUSE.filter(([, c]) => c && d.includes(c)).length;
  console.log(`\n${'restriction'.padEnd(32)}${s('closed', 9)}${s('escapes', 8)}${s('legit ops dropped', 20)}`);
  for (const [label, d] of [['drop unused capabilities', unused],
    ['drop all capabilities', CAPABILITY.map(([c]) => c)]]) {
    const k = closes(d);
    console.log(`${label.padEnd(32)}${s(`${k}/${ABUSE.length}`, 9)}${s(ABUSE.length - k, 8)}`
      + `${s(CAPABILITY.filter(([c, op]) => op && d.includes(c)).length, 20)}`);
  }
  console.log(`attempts a capability cannot close: ${ABUSE.filter(([, c]) => !c).length}, of which `
    + `${ABUSE.filter(([, c, p]) => !c && p).length} want a write path`);
}

capability with a legit op    operation                       replacement when dropped
change ownership              data directory ownership        ownership is set at build time
change identity               dropping identity at startup    the process starts rootless directly
open raw socket               network diagnostics             diagnostics in a helper container
kill process                  stopping the nightly job        no capability needed under the same identity
bypass file ownership         appending to a shared log       a shared group and mode bit

default open 14, with a legit op 5, needlessly open 9

restriction                        closed escapes   legit ops dropped
drop unused capabilities             6/12       6                   0
drop all capabilities                7/12       5                   5
attempts a capability cannot close: 5, of which 4 want a write path

Of the fourteen capabilities, only five have a legitimate operation. The remaining nine are the one entry in the isolation budget whose cost is zero: dropping them closes six of the twelve attempts, and the legitimate operations dropped are zero. Closing an unused privilege is not a trade-off — this is the one place in this course where a cost-free gain shows up.

When all five are dropped too, one more attempt closes, and five legitimate operations drop. The right-hand column writes what replaces them: ownership is set at build time, the process starts rootless directly, diagnostics move to a helper container. The five operations that drop are not a capability given up, they are work moved somewhere else; the cost is not bytes, it is writing. The last line draws the boundary: five attempts do not close by dropping capabilities, four of them want to write to a file.

Read-Only Root

The second restriction targets those four attempts. The run below builds the root with real mode bits — read-only directories at 0555, the ones left writable at 0755 — and really makes nine write attempts.

// read-only-root.mjs — a read-only root on real directories: real mode bits, real write attempts.
import { mkdirSync, writeFileSync, chmodSync, rmSync } from 'node:fs';
import { CAPABILITY, ABUSE, LEGIT_WRITES } from './capabilities.mjs';

const ROOT = './root';
const DIRS = ['app', 'config', 'data', 'data/state', 'data/corrections', 'log', 'cache', 'tmp'];
const WRITABLE = ['data/state', 'data/corrections', 'log', 'cache', 'tmp'];

const TARGETS = [...LEGIT_WRITES.map((y) => [y, 'legit', null]),
  ...ABUSE.filter(([, , y]) => y).map(([label, , y]) => [y, 'abuse', label])];

const release = () => {                      // mode bits are opened back up before deleting
  for (const d of [...DIRS].reverse()) { try { chmodSync(`${ROOT}/${d}`, 0o755); } catch { /* none */ } }
};
const setup = (mode) => {
  release();
  rmSync(ROOT, { recursive: true, force: true });
  for (const d of DIRS) mkdirSync(`${ROOT}/${d}`, { recursive: true });
  for (const d of DIRS) chmodSync(`${ROOT}/${d}`, mode(d));   // mode bits are really set
};
const attempt = (y) => { try { writeFileSync(`${ROOT}/${y}`, 'x'); return true; } catch { return false; } };

const s = (x, n) => String(x).padStart(n);
console.log(`${'layout'.padEnd(30)}${s('legit', 7)}${s('abuse', 8)}${s('closed', 9)}${s('false positive', 15)}`);
let open = [];
for (const [label, mode] of [['root writable', () => 0o755], ['root read-only', () => 0o555],
  ['read-only + writable paths', (d) => (WRITABLE.includes(d) ? 0o755 : 0o555)]]) {
  setup(mode);
  const r = TARGETS.map(([path, kind, abuser]) => [kind, attempt(path), abuser]);
  const m = r.filter(([t]) => t === 'legit'), k = r.filter(([t]) => t === 'abuse');
  open = k.filter(([, ok]) => ok).map(([, , abuser]) => abuser);
  console.log(`${label.padEnd(30)}${s(`${m.filter(([, o]) => o).length}/${m.length}`, 7)}`
    + `${s(`${open.length}/${k.length}`, 8)}${s(k.length - open.length, 9)}`
    + `${s(m.filter(([, o]) => !o).length, 15)}`);
}
console.log(`left writable: ${WRITABLE.length}/${DIRS.length} paths; still open: ${open.join(', ')}`);
console.log(`execute ban (model): 0 of the writable paths carry an x bit -> ${open.length} more attempt${open.length === 1 ? "" : "s"} close${open.length === 1 ? "s" : ""}`);

// Cumulative: all capabilities dropped + read-only root + writable paths + execute ban.
setup((d) => (WRITABLE.includes(d) ? 0o755 : 0o555));
const escaping = ABUSE.filter(([, c, p]) => !c && !p).map(([label]) => label);
const legitCount = CAPABILITY.filter(([, op]) => op).length;
console.log(`\ncumulative: closed ${ABUSE.length - escaping.length}/${ABUSE.length}, escaping `
  + `${escaping.length} (${escaping.join(', ')})`);
console.log(`remaining surface: ${WRITABLE.length} writable paths; open capabilities ${CAPABILITY.length} -> 0;`
  + ` ${legitCount} legit operations dropped, ${legitCount} replacements written`);
console.log(`is the application still running: ${LEGIT_WRITES.filter((y) => attempt(y)).length}/${LEGIT_WRITES.length}`
  + ' legit writes succeed');
release();
rmSync(ROOT, { recursive: true, force: true });

layout                          legit   abuse   closed false positive
root writable                     5/5     4/4        0              0
root read-only                    0/5     0/4        4              5
read-only + writable paths        5/5     1/4        3              0
left writable: 5/8 paths; still open: write and run a binary in the scratch dir
execute ban (model): 0 of the writable paths carry an x bit -> 1 more attempt closes

cumulative: closed 11/12, escaping 1 (read from the mounted dir and exfiltrate)
remaining surface: 5 writable paths; open capabilities 14 -> 0; 5 legit operations dropped, 5 replacements written
is the application still running: 5/5 legit writes succeed

The second row is this lesson’s most instructive one. When the root is made entirely read-only, all four of the four write attempts close — this is the layout with the highest close count — but all five of the five legitimate writes drop too, and the service does not run at all. Measuring a restriction only by how many attempts it closes reports the layout the application does not run in as the best layout.

The third row strikes the balance: when five paths are left writable, the legitimate writes come back, the false positive count zeroes out, and the closed count drops from four to three. The attempt that gets away is writing a binary into the tmp directory. This is where isolation is punctured: the hole a read-only root opens is the set of paths left writable, and that set is exactly as large as the application needs it to be — five of eight directories. The execute ban closes this attempt too, because none of those five paths carries an execute bit.

The cumulative row stacks the two on top of each other: eleven of the twelve attempts close, one gets away, and the application makes all five of its five legitimate writes. The attempt that gets away wants neither a capability nor a write; read permission was granted back in the volumes lesson, and, as the networking lesson measured for the sixteen in-network endpoints, it does not ask for identity. Restrictions narrow the authority that can be requested, they do not take back what has already been granted.

Summary

• Five of the fourteen default capabilities have a legitimate operation. Dropping the remaining nine closes six of the twelve attempts and drops zero legitimate operations — the one cost-free entry in this course.
• When all five are dropped too, one more attempt closes and five legitimate operations drop; a rearrangement is written in place of each one, so the cost is not bytes, it is writing.
• When the root is made entirely read-only, all four write attempts close too, but so do all five legitimate writes: the layout that closes the most is the layout the application does not run in.
• When five of eight directories are left writable, legitimate writes return to 5/5, the closed count drops to three, and the remaining attempt only closes through the execute ban.
• In the cumulative layout, eleven of the twelve attempts close and the application runs; the attempt that gets away has already been granted read permission and network access.

Course Wrap-Up

Lesson	Difference removed from the environment	Cost of isolation	Where isolation is punctured
Why Containers	6 of 18 differences in the output	84.11 MB; 14.01 MB per difference	Resource limit outside; clock and name matching in the kernel
Servers, Virtual Machines, and Containers	6/18 (server 0, virtual machine 7)	12 MB memory, 300 ms, 84 MB disk; overhead 6.3%	Kernel shared: clock, version, defects
Underlying Technologies	Only the union filesystem; namespace 0, control group 0	4,096-byte copy for 16 bytes	Number pool shared; weight is not a ceiling; 1,024 deleted bytes remain beneath
Container Standards	Output 100%, behavior 70%	Verification reads 83,650,787 bytes	3 of 5 properties left to interpretation differ
Container Lifecycle	Zero residue if cleaned up	Stopped: 436 MB, 1 name, 1 volume; 13,876 MB per period	Only the process record drops; the volume remains
Image and Layer Model	Filesystem layout in a 22,758-byte store	30,486 → 22,758 bytes; one line invalidates 8 chain ids	1,654 bytes outside the view, still in the store
Image Definition File	Dependency version to a single line	Context 15,681 bytes; 5,436 not entering the image	Records what was copied: 6,860 excess bytes
Layer Cache	Work moved to the store, did not leave the environment	21 layers in the bad order, 278,917 bytes (142.2%)	Measures the key text: grew 550 bytes, 4 instructions hit
Multi-Stage Build	185,826 bytes out of the run (86.4%)	215,160 bytes that never enter an image	Copy boundary: the wide path brings back 4 files
Base Image Selection	148 / 31 / 6 components inside	135.1 / 34.5 / 28.1 MB; 32 extra steps	Production outside: 133 updates a year
Tagging Scheme	Traceability 9/24 → 24/24	96 bytes per image, 1 in 359,375 of the image	Ledger in the store: the name does not travel
Image Repositories	80 references, 27 files; 690.0 → 176.2 KB	Local 176.2 KB; 27.3 KB cannot be deleted	9 of 16 layers of 7 private images sit in the public registry
Image Security	Inventory and signature 1,088 bytes	8 of 37 warnings are false positives	Deleted key 1,698 bytes beneath; 0 warnings, 1 open flaw
Run Options	14 variables in the environment (9 required)	21 definition lines; 10.54 periods of delay	Limit is a ceiling: clipping climbs to 4,215 units
Ephemeral Filesystem	Cross-run residue 0	1 byte → 98,304 bytes; layer 2.15x	Lower layer shared; ephemerality tied to deletion, not stopping
Volumes and Bind Mounts	Path the same on every machine	2 persistent directories; 7,951 bytes remain	Number space shared: 6 of 8 nodes cannot be written
Container Networking	Service name instead of machine name	1 network definition, 1 membership line (4 → 5)	16 endpoints ask no identity; 4 paths in two hops
Users and Permissions	11 of 15 entries inside, 4 from the environment	4 ownership fixes; writable 15 → 5	Identity shared with host: 5/8 writable on the host
Multi-Container Local Environment	6 commands, 8 flags, 22 orderings into a file	26 fields; 3 extra steps (3 → 6)	4 services 1 network, 2 services 1 volume
Developer Experience	Image set 10 → 5 files	2 layers, 687 bytes, 6 steps / 0, 0, 2	Deliberate hole: 1 unresolved import
Runtime Security	14 capabilities → 0	5 operations, 5 replacements; 5 paths writable	1 of 12 attempts does not close; 5 paths open, kernel shared

The course’s rule reads the same way across twenty-one lines: isolation is a budget. Every difference removed comes out in exchange for a byte, a run, or a resource — 14.01 MB, 4,096 bytes, 142.2% extra work, three extra steps — and no isolation is complete. Four holes repeat in the last column. The first is the shared kernel: clock value, system call surface, and kernel defects pass through the same place in three separate lessons. The second is that what gets pulled into the output cannot be pulled out of the output: the 1,654 bytes deleted from the view and the 1,699-byte key that stays in the layer both keep standing in the store. The third is shared number spaces — process number, user identity, port number; the namespace separates the view, not the number. The fourth is the rule standing outside isolation: the distributor enforcing the limit, the ledger resolving the tag, the host interface opening the port, and the runtime handing out the capability set are all outside the container.

All of these measurements shared one boundary: every one of them was done on a single machine. The output became portable, its identity was derived from its content, its authority came down to a single line. But deciding how many copies to run, on which machine, and when is still a human being. Can this decision be handed off to a system? If it is, what does that system need to know: how many copies are standing, whether a copy is actually serving, how much room is left on which machine, how many copies it is safe to change at once when moving to a new version. M22/K03 Container Orchestration begins with this question.