Comparison with Containers

The previous three lessons asked the same question about a single technology: where is the guest system’s boundary, how many items cross it, and what is reached if it is crossed. There is a second way to do the same job, and in that way there is no guest system at all: compartments separated from one another run on top of the same kernel. Both arrangements are used on the regional measurement network’s branch servers — the validator in a guest system, three copies of the reading aggregator in compartments sharing the same kernel.

This lesson does not compare the two technologies on speed, size, or ease of operation; those measurements were made in the Containers course and are not repeated here. The only question here is isolation strength, and that strength is written in this course’s unit of measure: the number of items that must be wrong at the same time to cross the boundary. The two technologies are compared on the same set of targets with this number.

VR20 — The comparison is between two arrangements doing the same job: a guest system and a compartment sharing the same kernel. VR21 — The guest system’s five required shared items are an input from the previous lesson; the compartment’s list is counted with the same measure. VR22 — A namespace separates the view, not the resource; this distinction is kept as a separate column in the item count. VR23 — The target set is eight items, and for each target the required items are written by name; item names are a model, the count is real. VR24 — Concentration is how many targets the same item appears in. VR25 — The access universe is 52 items: nine items each for four workloads, the host’s thirteen items, and three config items.

Number of Shared Items

Running on the same host, the two arrangements share different numbers of items. In a guest system, the sharing is as much as the hypervisor opens; in a compartment, the sharing is the kernel itself.

// compartment.mjs — items a guest system vs. a compartment sharing the same kernel shares with the host (MODEL).
// The guest system's five required items are an input from the previous lesson; the compartment is counted with the same measure.
export const GUEST = [['scheduler', false], ['processor cache', false], ['clock source', false],
  ['disk controller emulation', false], ['network interface emulation', false]];
// [item, does its view get separated by a namespace?] — the resource is shared on every row
export const CONTAINER = [['scheduler', false], ['processor cache', false], ['clock source', false],
  ['system call surface', false], ['kernel data structures', false], ['page cache', false],
  ['kernel log buffer', false], ['kernel module set', false],
  ['process ID space', true], ['user ID space', true], ['network stack', true],
  ['file system view', true]];

if (import.meta.url === `file://${process.argv[1]}`) {
  const s = (x, n) => String(x).padStart(n);
  console.log(`${'technology'.padEnd(37)}${s('shared items', 18)}${s('view separated', 18)}`
    + `${s('resource shared', 17)}`);
  for (const [ad, liste] of [['guest system', GUEST], ['compartment sharing the same kernel', CONTAINER]])
    console.log(`${ad.padEnd(37)}${s(liste.length, 18)}`
      + `${s(liste.filter(([, a]) => a).length, 18)}${s(liste.length, 17)}`);
  const notSeparated = CONTAINER.filter(([, a]) => !a).map(([k]) => k);
  console.log(`\nin the compartment, ${notSeparated.length} items with no separated view:`);
  for (let i = 0; i < notSeparated.length; i += 4) console.log(`  ${notSeparated.slice(i, i + 4).join(', ')}`);
  console.log(`ratio: compartment / guest = ${(CONTAINER.length / GUEST.length).toFixed(1)}x shared items`);
}

technology                                 shared items    view separated  resource shared
guest system                                          5                 0                5
compartment sharing the same kernel                  12                 4               12

in the compartment, 8 items with no separated view:
  scheduler, processor cache, clock source, system call surface
  kernel data structures, page cache, kernel log buffer, kernel module set
ratio: compartment / guest = 2.4x shared items

The compartment shares 2.4 times the items. That ratio alone is not a conclusion, it is a baseline: every shared item is a path candidate, and the candidate count has risen from five to twelve. Two of the guest system’s five items are a piece of emulation code — that is, a limited, single-purpose piece of software running on the host side. The compartment’s twelve items, by contrast, are directly the kernel itself; there is no emulation layer placed in between.

The second column is more interesting. Only four of the twelve items have their view separated by a namespace: process ID, user ID, network stack, file system view. The remaining eight do not even have a separate view — the scheduler, the processor cache, the clock, the system call surface, kernel data structures, the page cache, the kernel log buffer, and the kernel module set are the same object for the compartment and for the host. The third column settles the distinction: the resource is shared in all twelve of the twelve items. Even in the four items that have a namespace, what is separated is the view; the underlying ID space and data structure are singular.

Eight Targets, Two Numbers

The number of shared items is not isolation strength; isolation strength is the number of items that must be wrong at the same time to reach a given target. The run below writes the same eight targets in both arrangements.

// targets.mjs — the same eight targets in both technologies: items that must be wrong at the same
// time to cross the boundary. Item names are a model, the count is real.
const TARGETS = [
  ["reading the neighbor's files", ['hypervisor bug', "access to the neighbor's image"], ['kernel bug']],
  ["reading the neighbor's memory", ['hypervisor bug', 'memory isolation flaw'],
    ['kernel bug', 'process-tracing privilege']],
  ["writing to the host's file system", ['hypervisor bug', 'host write privilege'], ['a writable mount']],
  ['running code in the host kernel', ['hypervisor bug', 'host kernel bug'], ['kernel bug']],
  ["changing the host's clock", ['hypervisor bug', 'host clock privilege'], ['clock-change privilege']],
  ["exhausting the neighbor's resources", ['an open resource limit'], ['an open resource limit']],
  ["reading the host's secrets", ['hypervisor bug', 'host file access'],
    ['the secret file being mounted', 'read permission']],
  ['reaching an endpoint on the host network', ['an open network path'], ['an open network path']],
];
const NAMES = ['guest system', 'compartment'];
const s = (x, n) => String(x).padStart(n);
console.log(`${'target'.padEnd(42)}${s(NAMES[0], 14)}${s(NAMES[1], 11)}`);
for (const [h, a, b] of TARGETS) console.log(`${h.padEnd(42)}${s(a.length, 14)}${s(b.length, 11)}`);
const required = (i) => TARGETS.reduce((t, h) => t + h[i + 1].length, 0);
const single = (i) => TARGETS.filter((h) => h[i + 1].length === 1).length;
console.log(`${'total required items'.padEnd(42)}${s(required(0), 14)}${s(required(1), 11)}`);
console.log(`${'targets crossed with a single item'.padEnd(42)}${s(single(0), 14)}${s(single(1), 11)}`);

for (let i = 0; i < 2; i++) {                    // concentration: the most frequent item
  const all = TARGETS.flatMap((h) => h[i + 1]);
  const [item, n] = [...new Set(all)].map((k) => [k, all.filter((x) => x === k).length])
    .sort((a, b) => b[1] - a[1])[0];
  const after = TARGETS.filter((h) => h[i + 1].filter((k) => k !== item).length <= 1).length;
  console.log(`\n${NAMES[i]}: most frequent item "${item}" in ${n}/${TARGETS.length} targets`);
  console.log(`with that item in place, targets crossed with a single item ${single(i)} -> ${after}`);
}

target                                      guest systemcompartment
reading the neighbor's files                           2          1
reading the neighbor's memory                          2          2
writing to the host's file system                      2          1
running code in the host kernel                        2          1
changing the host's clock                              2          1
exhausting the neighbor's resources                    1          1
reading the host's secrets                             2          2
reaching an endpoint on the host network               1          1
total required items                                  14         10
targets crossed with a single item                     2          6

guest system: most frequent item "hypervisor bug" in 6/8 targets
with that item in place, targets crossed with a single item 2 -> 8

compartment: most frequent item "kernel bug" in 3/8 targets
with that item in place, targets crossed with a single item 6 -> 7

The expected difference is in the table: six of the eight targets are crossed with a single item in the compartment, two in the guest system. Total required items are fourteen against ten. On average, reaching a target requires 1.75 items to be wrong at the same time in the guest system, 1.25 in the compartment.

The two rows that come out equal show where this difference does not come from. Exhausting the neighbor’s resources and reaching an endpoint on the host network both demand a single item in either arrangement, because these two targets depend not on the isolation boundary but on a configuration item: if the resource limit was left open and the network path is open, the type of boundary does not change the outcome. Choosing an isolation technology does not compensate for a configuration item being left open.

The last two sections correct the table itself. The six “two items” in the guest system are not independent: all six depend on the same second item, the hypervisor bug. With that item in place, the number of targets crossed with a single item rises from two to eight — that is, the guest system’s isolation for all eight targets at once rests on the correctness of a single item. The compartment has no such concentration: the most frequent item appears in three targets, and with that item in place, the targets crossed with a single item rise from six to seven, not eight.

This is where the two technologies part ways. The guest system gives a higher but more concentrated number: six of the eight targets depend on a single item. The compartment gives a lower but more scattered number: six separate targets depend on five separate items. In the first, defense has one clear place to go; in the second, five separate items must be kept correct, separately.

Items Reached When the Boundary Falls

The required item count is the second of three columns; the third is how many items are reached once the boundary is crossed. This number comes from layout, and it splits not between two technologies but among three arrangements.

// fall.mjs — items reached when the boundary falls, across three layout arrangements (MODEL).
// Fictional branch server: four workloads, each carrying 6 file and 3 memory items.
const WORKLOADS = 4, FILES = 6, MEMORY = 3, HOST_FS = 9, HOST_SECRET = 4, CONFIG = 3;
const TOTAL = WORKLOADS * (FILES + MEMORY) + HOST_FS + HOST_SECRET + CONFIG;
// [layout, items required to cross the boundary, items reached on first fall, patch items to maintain]
const LAYOUT = [
  ['4 workloads in a single guest system', 2, WORKLOADS * (FILES + MEMORY) + CONFIG, 2],
  ['4 compartments on a single host', 1, TOTAL, 1],
  ['guest system + compartment per workload', 1, FILES + MEMORY, 5],
  ['  same layout, up to host level', 3, WORKLOADS * (FILES + MEMORY) + CONFIG, 5],
];
const s = (x, n) => String(x).padStart(n);
console.log(`universe ${TOTAL} items: ${WORKLOADS} workloads x ${FILES + MEMORY}, host ${HOST_FS + HOST_SECRET}, config ${CONFIG}`);
console.log(`\n${'layout'.padEnd(41)}${s('required', 9)}${s('reached', 10)}${s('patch items', 13)}`);
for (const [ad, g, e, y] of LAYOUT)
  console.log(`${ad.padEnd(41)}${s(g, 9)}${s(`${e}/${TOTAL}`, 10)}${s(y, 13)}`);
const [, , narrowest] = LAYOUT.reduce((a, b) => (b[2] < a[2] ? b : a));
console.log(`\nnarrowest first fall ${narrowest}/${TOTAL} (${((100 * narrowest) / TOTAL).toFixed(1)}%), `
  + `cost ${LAYOUT[2][3] - LAYOUT[1][3]} extra patch items`);

universe 52 items: 4 workloads x 9, host 13, config 3

layout                                    required   reached  patch items
4 workloads in a single guest system             2     39/52            2
4 compartments on a single host                  1     52/52            1
guest system + compartment per workload          1      9/52            5
  same layout, up to host level                  3     39/52            5

narrowest first fall 9/52 (17.3%), cost 4 extra patch items

The first two rows repeat the previous table’s result on the access side: when four compartments run on a single host, the boundary is crossed with one item and all fifty-two of the fifty-two items are reached; when four workloads run in a guest system, the boundary demands two items and thirty-nine items are reached. The reason the compartment’s reached items equal the entire universe is structural: the layer that draws the boundary is the kernel, and that kernel is the host itself; whoever crosses it has also crossed into the host’s file system and its secrets.

The third row shows that the two technologies are not opposites — they can be stacked. When each workload runs inside a compartment within its own guest system, the items reached on the first fall are nine of fifty-two — 17.3% — because escaping that compartment only reaches into its own guest system. The items required to reach the host level rise to three: the compartment boundary’s one item is added to the guest system’s two items. When layers are stacked, the required items add up; this is the only arrangement in the course where the required item rises to three.

The right column writes the cost. Four compartments on a single host require one patch item — a single kernel. Layout number three requires five patch items: one hypervisor and four separate guest kernels. Four extra patch items are a continuous, recurring job; that is the price of bringing the items reached down from 52 to 9.

How to Read the Number

This measurement does not say to prefer one technology over the other; it describes two different defense programs. In the guest system, the program has a single item, and that item is the hypervisor: keeping it patched, separating the management interface, not running other work on it. The whole of this program watches one component, and when that component falls, all eight of the eight targets are crossed with a single item. That is both the strength and the weakness of concentration.

In the compartment, the program has six items: keeping the kernel patched, leaving the mount read-only, not granting clock privilege, closing off process-tracing privilege, setting a resource limit, closing the network path. When all six are kept correct separately, the number of targets crossed with a single item drops; when one is forgotten, that target is crossed with a single item and the others do not compensate for it.

The remaining two rows are the same in both: the resource limit and the network path. These two items are outside the isolation technology, and each is sufficient on its own in either arrangement.

The third table shows that these two programs do not exclude one another. Running the workload inside a compartment within its own guest system requires carrying out both programs — both the hypervisor and the kernel will be kept patched, five patch items — but in exchange it brings the items reached on the first fall down to nine of fifty-two and raises the items required to reach the host level to three. This is the only row in the course where the required item rises to three, and it comes not from choosing a technology but from stacking two technologies. The next lesson’s measure starts from the same item universe: the question of what is left once the boundary falls is independent of the boundary’s type.

Summary

• The guest system shares five required items with the host, the compartment sharing the same kernel shares twelve; the ratio is 2.4x.
• Only four of the twelve items in the compartment have a namespace separating the view, and all twelve share their resource; the remaining eight do not even have a separate view.
• Across eight targets, the total required items are 14 in the guest system and 10 in the compartment; the number of targets crossed with a single item is 2 against 6.
• Two targets — stopping via resource exhaustion and reaching an endpoint on the host network — demand a single item in either arrangement, because they depend not on the isolation boundary but on a configuration item.
• The guest system’s six “two items” are not independent: all six depend on the hypervisor bug, and with that item in place, the targets crossed with a single item rise from 2 to 8. In the compartment the most frequent item appears in 3 targets, and the number rises from 6 to 7.
• Items reached on the first fall are 39/52, 52/52, and 9/52 across the three layouts; in the compartment-per-workload layout, reaching host level requires three items, at a cost of four extra patch items.

Next Step

Across four lessons the boundary was always measured from the same direction: what happens if someone crosses it. That is not the only way a boundary falls. A disk fails, a file is deleted by mistake, a host becomes unbootable — and in these cases no one crosses the boundary, yet the guest is still lost. Virtualization has two separate answers to this question: the snapshot and the backup. Both are called “bringing it back,” and they do not bring back the same thing. The last lesson runs both in the same loss scenario and counts the recovered and unrecovered items separately, computes the point full, incremental, and differential backups can return to when the chain breaks, and closes the course with an eleven-lesson boundary table.