Servers, Virtual Machines, and Containers

The previous lesson pulled 6 of the differences out of the environment and measured the cost in bytes: an 84.11 MB output with the smallest base. That measurement dealt with a single output. In the fictional regional measurement network, though, there are five services — the reading collector, the verifier, billing, work order, and the nightly batch job — and all of them run on the same machine. When each one brings its own resolver, its own rule package, and its own timezone table, the question changes: once these five outputs stand side by side on the same machine, how separated are they from one another?

The name of that separation is isolation, and it is not a single thing. One application being unable to see another’s files is one thing, being unable to stop its processes is another, being unable to exhaust its memory yet another. This lesson compares three isolation levels — the shared server, the container, and the virtual machine — by the same criteria: what they separate, what they duplicate, what they demand in return.

CC7. The comparison is at the scale of a single machine; distribution across multiple machines is not this course’s subject. CC8. All the numbers below are model inputs, not measurements from an actual setup; what matters is the order-of-magnitude difference between two levels, not the value in a single cell. CC9. There are five criteria, and each criterion takes a binary value: separated or shared. Partial isolation does not count as full.

Three Levels by the Same Criteria

The criteria are the names of the places isolation can be pierced. Kernel: do the two applications call the same kernel. File system: can one see the other’s files. Network: the same interface and the same port space. Process table: can one list and stop the other’s processes. User: are file ownership and permission tied to the same account.

// measurement-network/levels.mjs — comparing three isolation levels by the same criteria
// MODEL: every number is a declared model input, not a measurement from an actual setup.
// The comparison is at the order-of-magnitude level; the ratio between two levels is
// meaningful, not the value in a single cell.

export const CRITERIA = ["kernel", "fileSystem", "network", "processTable", "user"];

export const LEVEL = {
  "shared server": {
    isolates: { kernel: false, fileSystem: false, network: false, processTable: false, user: true },
    duplicates: { osCopies: 0, libraryCopies: 0 },
    startupMs: 30, memoryBaseMB: 4, diskBaseMB: 0,
  },
  "container": {
    isolates: { kernel: false, fileSystem: true, network: true, processTable: true, user: true },
    duplicates: { osCopies: 0, libraryCopies: 1 },
    startupMs: 300, memoryBaseMB: 12, diskBaseMB: 84,
  },
  "virtual machine": {
    isolates: { kernel: true, fileSystem: true, network: true, processTable: true, user: true },
    duplicates: { osCopies: 1, libraryCopies: 1 },
    startupMs: 25_000, memoryBaseMB: 512, diskBaseMB: 1_240,
  },
};

// How many of the previous lesson's 18 differences were pulled out (input: lesson 01's bucket count).
export const CLOSED = {
  "shared server": [0, "libraries are shared; the six differences in the included bucket stay in the environment"],
  "container": [6, "the six differences in the included bucket go into the output"],
  "virtual machine": [7, "since the kernel is included too, the name-matching rule closes as well"],
};

export const NAMES = Object.keys(LEVEL);
export const isolated = (d) => CRITERIA.filter((o) => LEVEL[d].isolates[o]);

// The dump is printed only when this file is run directly; the next block imports the
// module for its data and does not print this section again.
if (import.meta.main) {
console.log("isolated thing (+ separated, - shared)");
console.log("criterion".padEnd(16) + NAMES.map((d) => d.padEnd(20)).join(""));
for (const o of CRITERIA)
  console.log(o.padEnd(16) + NAMES.map((d) => (LEVEL[d].isolates[o] ? "+" : "-").padEnd(20)).join(""));
console.log("isolated/5".padEnd(16) +
  NAMES.map((d) => `${isolated(d).length}/5`.padEnd(20)).join(""));

console.log("\nduplicated thing (copies per instance)");
for (const y of ["osCopies", "libraryCopies"])
  console.log(y.padEnd(24) + NAMES.map((d) => String(LEVEL[d].duplicates[y]).padEnd(20)).join(""));

console.log("\ndifference pulled out of the environment (how many of the 18):");
for (const d of NAMES)
  console.log("  " + d.padEnd(20) + `${CLOSED[d][0]}/18   ` + CLOSED[d][1]);

console.log("\nshared thing (the hole):");
for (const d of NAMES) {
  const hole = CRITERIA.filter((o) => !LEVEL[d].isolates[o]);
  console.log("  " + d.padEnd(20) + (hole.length ? hole.join(", ") : "none of the criteria"));
}
}

isolated thing (+ separated, - shared)
criterion       shared server       container           virtual machine     
kernel          -                   -                   +                   
fileSystem      -                   +                   +                   
network         -                   +                   +                   
processTable    -                   +                   +                   
user            +                   +                   +                   
isolated/5      1/5                 4/5                 5/5                 

duplicated thing (copies per instance)
osCopies                0                   0                   1                   
libraryCopies           0                   1                   1                   

difference pulled out of the environment (how many of the 18):
  shared server       0/18   libraries are shared; the six differences in the included bucket stay in the environment
  container           6/18   the six differences in the included bucket go into the output
  virtual machine     7/18   since the kernel is included too, the name-matching rule closes as well

shared thing (the hole):
  shared server       kernel, fileSystem, network, processTable
  container           kernel
  virtual machine     none of the criteria

The matrix lines up the three levels along a single axis: 1/5, 4/5, 5/5. On the shared server only one of the five criteria is separated — the user account — and even that one separation depends on file permissions being set up correctly. The container separates four criteria and shares the kernel. The virtual machine separates all five, because it carries its own kernel.

The duplicated-thing column gives the reason for this ordering. The shared server duplicates nothing; the libraries are common, so the entire environment difference stays in the environment — the previous lesson’s six differences do not close. The container duplicates the library set once per instance and does not duplicate the operating system copy. The virtual machine duplicates both. The difference pulled out of the environment follows this ordering: 0, 6, and 7. The virtual machine’s seventh difference is the dimension it closes by including the kernel too — the file-name-matching rule sat in the boundary bucket in the previous lesson; once the kernel is included, that closes as well.

The middle level is not a compromise; it is the name for giving up on separating one specific criterion. The container buys three criteria over the shared server: file system, network, and process table. What it gives up relative to the virtual machine is a single criterion: the kernel. The difference between the three levels adds up in these two numbers, and both are written in the table. The criteria taking a binary value (CC9) is a simplification here: in reality a criterion can be partly separated too — the user account is separate but root privilege can stay shared, the network space is separate but uses the same physical interface. These in-between states show up as “separated” in the table and provide less isolation than the table reads; measuring exactly that is one of the things the next lessons do.

The Order of Magnitude of the Cost

Isolation is something purchased, and payment is made in three line items: startup delay, memory base, disk base.

CC10. The memory base is the addition the isolation level brings on top of the application’s own memory. CC11. The machine is 16,384 MB, 1,024 MB is reserved for the operating system, and per-instance application memory is 180 MB. CC12. Instances are started sequentially; concurrent startup is not modeled.

// measurement-network/magnitude.mjs — the order of magnitude of the isolation cost and its conversion to density
import { LEVEL, NAMES, isolated } from "./levels.mjs";

const magnitude = (n) => (n <= 0 ? "-" : "10^" + Math.round(Math.log10(n)));
const MACHINE_MB = 16_384, RESERVED_MB = 1_024, APP_MB = 180, INSTANCES = 25;

console.log("cost and its magnitude (model input)");
console.log("level".padEnd(20) + "startup".padEnd(14) + "magnitude".padEnd(11) +
  "memory base".padEnd(15) + "magnitude".padEnd(11) + "disk base");
for (const d of NAMES) {
  const v = LEVEL[d];
  console.log(d.padEnd(20) + `${v.startupMs} ms`.padEnd(14) + magnitude(v.startupMs).padEnd(11) +
    `${v.memoryBaseMB} MB`.padEnd(15) + magnitude(v.memoryBaseMB).padEnd(11) + `${v.diskBaseMB} MB`);
}

const base = NAMES.map((d) => LEVEL[d].memoryBaseMB);
console.log("memory base ratio (relative to the smallest): " +
  NAMES.map((d, i) => `${d} ${Math.round(base[i] / base[0])}x`).join(", "));

console.log("\nmemory base per isolated dimension");
for (const d of NAMES) {
  const k = isolated(d).length;
  console.log("  " + d.padEnd(20) + `${k} dimensions  ` +
    `${(LEVEL[d].memoryBaseMB / k).toFixed(1)} MB/dimension`);
}

console.log(`\na single ${MACHINE_MB} MB machine, ${RESERVED_MB} MB reserved, ` +
  `application per instance ${APP_MB} MB`);
for (const d of NAMES) {
  const instanceMB = APP_MB + LEVEL[d].memoryBaseMB;
  const fits = Math.floor((MACHINE_MB - RESERVED_MB) / instanceMB);
  const overhead = ((LEVEL[d].memoryBaseMB / instanceMB) * 100).toFixed(1);
  console.log("  " + d.padEnd(20) + `instance ${String(instanceMB).padStart(4)} MB   ` +
    `fits ${String(fits).padStart(3)}   isolation overhead ${overhead}%`);
}

console.log(`\nsequential startup of ${INSTANCES} instances and the disk it carries`);
for (const d of NAMES) {
  const s = (LEVEL[d].startupMs * INSTANCES) / 1000;
  console.log("  " + d.padEnd(20) + `${s.toFixed(2).padStart(8)} s   ` +
    `disk ${String(LEVEL[d].diskBaseMB * INSTANCES).padStart(5)} MB   ` +
    `isolated ${isolated(d).length}/5`);
}

cost and its magnitude (model input)
level               startup       magnitude  memory base    magnitude  disk base
shared server       30 ms         10^1       4 MB           10^1       0 MB
container           300 ms        10^2       12 MB          10^1       84 MB
virtual machine     25000 ms      10^4       512 MB         10^3       1240 MB
memory base ratio (relative to the smallest): shared server 1x, container 3x, virtual machine 128x

memory base per isolated dimension
  shared server       1 dimensions  4.0 MB/dimension
  container           4 dimensions  3.0 MB/dimension
  virtual machine     5 dimensions  102.4 MB/dimension

a single 16384 MB machine, 1024 MB reserved, application per instance 180 MB
  shared server       instance  184 MB   fits  83   isolation overhead 2.2%
  container           instance  192 MB   fits  80   isolation overhead 6.3%
  virtual machine     instance  692 MB   fits  22   isolation overhead 74.0%

sequential startup of 25 instances and the disk it carries
  shared server           0.75 s   disk     0 MB   isolated 1/5
  container               7.50 s   disk  2100 MB   isolated 4/5
  virtual machine       625.00 s   disk 31000 MB   isolated 5/5

Reading the orders of magnitude splits the table into two groups. At the memory-base magnitude, the shared server and the container sit in the same place: both are 10^1. The virtual machine is two orders of magnitude higher: 10^3. At startup delay, though, the container climbs one order of magnitude (10^2) and the virtual machine three (10^4). So the place the container parts ways with the shared server is not memory but startup and disk; the place the virtual machine parts ways with both is all three line items.

The memory base per isolated dimension reduces this to a single number. Shared server 4.0 MB/dimension, container 3.0 MB/dimension, virtual machine 102.4 MB/dimension. The container being the cheapest level per unit comes from its being able to separate four dimensions without duplicating the kernel. The virtual machine separates all five of the five dimensions, but the cost of separating the fifth — the kernel — is by itself forty times the sum of the other four. The per-unit measure putting the shared server below the container looks backward at first glance, because the shared server’s total cost is smaller. What looks backward is the divisor changing: 4 MB is divided by a single dimension, 12 MB by four. A comparison that looks at total cost answers “which level is cheapest,” while a comparison that looks at per-unit cost answers “what is paid per separated dimension”; in the isolation budget the second question carries the decision, because what is purchased is not bytes but separated criteria. Both questions are read from the same table and do not substitute for each other.

The density count translates this to the runtime side. The same 16 GB machine fits 83 instances on the shared server, 80 on the container, 22 on the virtual machine. The isolation overhead — how much of the memory goes to isolation instead of the application — is 2.2%, 6.3%, and 74.0% respectively. On the virtual machine, more than three-quarters of the memory does not go to the application itself. Sequentially starting twenty-five instances takes 0.75 seconds, 7.50 seconds, and 625 seconds; the disk carried is 0, 2,100, and 31,000 MB.

The number twenty-five comes from the fiction: five copies each of five services. This is exactly where startup time carries a decision. The nightly batch job starts once a night, and 625 seconds is a tolerable number for it; the reading collector, on the other hand, is the time field crews wait for when it restarts after a defect, and there the gap between 7.50 seconds and 625 seconds is a duration of downtime. The same model input carries two separate meanings for two services — this is why startup delay is read not against a single threshold but together with how often the work running at that level restarts. The disk line item is multiplied the same way: an 84 MB output climbs to 2,100 MB across twenty-five instances, and that number depends on whether the layers are shared; how sharing works is the next lesson’s subject.

Where Isolation Is Pierced

The matrix’s last section names each level’s hole. On the shared server, four criteria are open: kernel, file system, network, and process table. In the fictional network this means — if the nightly batch job writes a tariff file to the wrong path, the verifier reads it; if the billing service holds a port, the work order service cannot open the same port; if one exhausts memory, all of them stop at once.

On the container, the hole is a single one and its name is the kernel. The five services each see their own file system, their own network space, their own process table, and their own user; but all five make system calls to the same kernel. This has three consequences. The clock value the previous lesson measured is still shared: the five services read the same clock, and the timezone setting is inside the output but the clock itself is not. The kernel version is shared: the kernel behavior one service needs concerns the others too. And a kernel defect passes underneath all five isolations at once — the four separated criteria do not close this path.

The zero in the disk-base column is also the mark of a hole. The shared server pays no bytes because it carries nothing; what it does not pay for is closing the environment difference. Being the cheapest level in all three line items is not an advantage, it is the result of having purchased nothing. Read on its own, the cost table wins on every row for the shared server; read together with the criteria table, it is visible that it also carries 1/5 alongside 0/18. This is why the isolation budget requires reading the two tables side by side, not separately.

On the virtual machine none of the criteria is open; the hole is invisible in the criteria table and visible in the cost table. 102.4 MB/dimension and a startup magnitude of 10^4 are the price of isolation being complete. The choice between the three levels is therefore not the question “which one is better”: it is the question which criterion needs to be separated, and every criterion left unseparated leaves a defect class open.

Summary

• Isolation is not a single thing; it is a decision made separately for each of five criteria: kernel, file system, network, process table, user.
• The three levels separate 1/5, 4/5, and 5/5 criteria respectively. What is duplicated determines this ordering: the shared server duplicates nothing, the container duplicates the library set, the virtual machine duplicates the library set and the operating system copy.
• The difference pulled out of the environment depends on the level: 0/18 on the shared server, 6/18 on the container, 7/18 on the virtual machine. The seventh is the file-name-matching rule that closes because the kernel is included.
• The cost is written as an order of magnitude: memory base 10^1, 10^1, and 10^3; startup 10^1, 10^2, and 10^4. Memory base per isolated dimension is 4.0, 3.0, and 102.4 MB — the container is the cheapest level per unit.
• The same machine fits 83, 80, and 22 instances; isolation overhead is 2.2%, 6.3%, and 74.0%. The container’s hole is a single one and its name is the kernel: clock value, kernel version, and kernel defects all pass underneath the four separated criteria at once.

Next Step

This lesson wrote down that the container separates four criteria but never asked how it separates them. If two processes running on the same kernel do not see each other’s processes, and the process table is a single table, what does “not seeing” mean? The same holds for the memory limit: if a control group enforces the limit, what decision is made when the limit is exceeded? And if five services carry the same library set, are the same files written to disk five times? The next lesson models the three mechanisms behind these three questions separately — the namespace as a visibility mapping, the control group as a share allocator, the union filesystem as a layer stack — and writes down where each one pierces isolation.