Run Options

The previous topic measured the output at rest: how many layers, how many bytes, which digest, which signature. All of those measurements were over a set of files that was never running. The moment it starts running, what gets measured changes. Part of the isolation persists — the filesystem is still what the image says, the library version is still frozen. Part of it drops away: CPU, memory, and configuration now come not from inside the image, but from outside it.

This lesson counts the first two of the parts that drop away: the resource limit given to the container and its environment variables. What they have in common is this — neither is written inside the image. The same image shows two different behaviors under two different limits; the same image never comes up at all with a missing variable. From the isolation budget’s perspective, this is a gap the image cannot close, and its size can be measured.

The measurements run through a fictional regional measurement network: software that collects readings from water meters, verifies them, turns them into invoices, and opens work orders for field crews. The system is fictional; the numbers come from the run below.

The Decision Made When a Limit Is Exceeded

A resource limit requires something to be done once it is exceeded. The limit itself is a number; the decision is made once that number is hit, and there are three choices: throttle the work, kill the container, degrade the work. The run below is a model of a control group — it splits the CPU share the machine distributes per period between two containers and counts the cost of the three decisions separately.

• RT1 — The measurement network’s four services run on a single machine, as four separate containers.
• RT2 — The machine’s capacity is normalized to 100 shares per period; share and period are abstract units, not seconds or cores.
• RT3 — The verifier requests 40-50 shares every period; the nightly batch job starts at period 60 and ends at period 140, requesting 60-80 shares in that window.
• RT4 — The kill decision is made when accumulated work exceeds 300 units; the restart takes 5 periods, and work arriving during those periods is lost.
• RT5 — The degrade decision keeps 0.6 of the work that exceeds the limit and drops the rest.

// runtime/control-group.mjs — the cost of the decision made when a limit is exceeded.
// MODEL: period is an abstract step, share an abstract unit; demand is drawn from a seeded generator.
const PERIOD = 200, MACHINE = 100;               // RT2: machine capacity is 100 shares/period
const START = 60, END = 140, SEED = 20260801;    // RT3: the nightly job's window
const QUEUE_LIMIT = 300, RESTART = 5;            // RT4: kill threshold and restart
const DEGRADE = 0.6;                             // RT5: share kept on degrade

const rng = (t) => { let s = t % 2147483647; return () => (s = (s * 48271) % 2147483647) / 2147483647; };
const demand = (r, t) => ({
  verifier: 40 + Math.round(r() * 10),
  nightly: t >= START && t < END ? 60 + Math.round(r() * 20) : 4,
});

function run({ decision, limit }) {
  const r = rng(SEED);
  const names = ["verifier", "nightly"];
  const d = Object.fromEntries(names.map((a) => [a, { queue: 0, done: 0, lost: 0, wait: 0, killed: 0, down: 0, own: 0, machine: 0 }]));

  for (let t = 0; t < PERIOD; t += 1) {
    const incoming = demand(r, t), request = {}, share = {};
    for (const a of names) {
      if (d[a].down > 0) { d[a].down -= 1; d[a].lost += incoming[a]; request[a] = 0; share[a] = 0; continue; }
      request[a] = incoming[a] + d[a].queue;
      share[a] = Math.min(request[a], limit[a]);        // clipping at own limit
      d[a].own += request[a] - share[a];
    }
    const total = share.verifier + share.nightly;
    for (const a of names) {
      const granted = total > MACHINE ? Math.floor((MACHINE * share[a]) / total) : share[a];
      d[a].machine += share[a] - granted;               // clipping at the machine: the part that lands on the neighbor
      d[a].done += granted;
      if (d[a].down > 0) continue;
      let remainder = request[a] - granted;
      if (decision === "degrade") { const kept = Math.round(remainder * DEGRADE); d[a].lost += remainder - kept; remainder = kept; }
      d[a].queue = remainder;
      if (decision === "kill" && d[a].queue > QUEUE_LIMIT) {
        d[a].killed += 1; d[a].lost += d[a].queue; d[a].queue = 0; d[a].down = RESTART;
      }
      d[a].wait += d[a].queue;
    }
  }
  return d;
}

const fmt = (x, n = 2) => x.toFixed(n);
const LIMIT = { verifier: 50, nightly: 60 };
console.log(`model: ${PERIOD} periods, machine ${MACHINE} shares/period, limits ${LIMIT.verifier}+${LIMIT.nightly}=` +
  `${LIMIT.verifier + LIMIT.nightly} (over-commitment), nightly work periods ${START}-${END}, seed ${SEED}`);
console.log(`\n${"decision".padEnd(11)}${"work".padEnd(13)}${"done".padStart(9)}${"lost".padStart(8)}` +
  `${"avg wait".padStart(14)}${"kills".padStart(9)}${"down periods".padStart(14)}`);
const results = {};
for (const decision of ["throttle", "kill", "degrade"]) {
  const k = run({ decision, limit: LIMIT });
  results[decision] = k;
  for (const a of ["verifier", "nightly"])
    console.log(`${decision.padEnd(11)}${a.padEnd(13)}${String(k[a].done).padStart(9)}${String(k[a].lost).padStart(8)}` +
      `${fmt(k[a].wait / k[a].done).padStart(14)}${String(k[a].killed).padStart(9)}${String(k[a].killed * RESTART).padStart(14)}`);
}
console.log(`\n${"nightly limit".padEnd(15)}${"nightly done".padStart(15)}${"nightly wait".padStart(17)}` +
  `${"verifier done".padStart(19)}${"verifier wait".padStart(21)}${"machine clip".padStart(17)}`);
for (const lim of [20, 40, 60, 80, 100]) {
  const k = run({ decision: "throttle", limit: { verifier: 50, nightly: lim } });
  console.log(`${String(lim).padEnd(15)}${String(k.nightly.done).padStart(15)}${fmt(k.nightly.wait / k.nightly.done).padStart(17)}` +
    `${String(k.verifier.done).padStart(19)}${fmt(k.verifier.wait / k.verifier.done).padStart(21)}` +
    `${String(k.verifier.machine + k.nightly.machine).padStart(17)}`);
}

model: 200 periods, machine 100 shares/period, limits 50+60=110 (over-commitment), nightly work periods 60-140, seed 20260801

decision   work              done    lost      avg wait    kills  down periods
throttle   verifier          9012       0          0.12        0             0
throttle   nightly           6092       0         10.54        0             0
kill       verifier          9012       0          0.06        0             0
kill       nightly           4124    1968          2.15        3            15
degrade    verifier          8891     121          0.02        0             0
degrade    nightly           4986    1106          0.33        0             0

nightly limit     nightly done     nightly wait      verifier done        verifier wait     machine clip
20                        3040           122.72               9012                 0.00                0
40                        5840            30.08               9012                 0.00                0
60                        6092            10.54               9012                 0.12             1033
80                        6092             5.48               8625                 6.00             2817
100                       6092             2.49               8306                10.38             4215

These numbers are in the measurement class and depend on the seed.

Three Decisions, Three Separate Costs

The nightly job’s total demand is 6092 units in all three runs; the decisions distribute this 6092 to different places, and the total is conserved.

Throttle loses no work: 0 lost, 6092 completed. Its cost is entirely time — every completed unit waits an average of 10.54 periods in the queue. The verifier’s wait in the same run is 0.12 periods; the throttle decision’s cost is paid almost entirely by the work actually hitting the limit.

Kill brings the wait down to 2.15 periods, because the queue is wiped once it exceeds 300 units. This drop is an accounting trick: the queue got shorter because the work inside it was thrown away. The cost shows up in two entries — 1968 units lost (32% of demand) and 3 kills, each with 5 periods down, for 15 periods total with no work done. Completed work drops from 6092 to 4124.

Degrade sits between the two, and its numbers are the strangest of all: wait drops to 0.33 periods, loss stays at 1106 units, no kills. Because 40% of the work exceeding the limit is dropped, the queue never grows. But the number that stands out is not the nightly job’s — it is the verifier’s loss: 121 units. The verifier never once hit its own 50-share limit; what clips it is the machine. Because the degrade decision applies to every container, 40% of what the machine clips is also dropped for the verifier. The decision is made in one container; its result shows up in two.

The choice depends on the data class. For a reading going into an invoice, a loss of 1968 units is unacceptable; throttle is the only option, at a price of 10.54 periods of delay. For a summary feeding a trend chart, degrade is cheapest: wait drops from 10.54 to 0.33 for a loss of 1106 units. Kill is not a decision anyone chooses; it is the decision made at a memory limit, and not being chosen does not change its cost.

Sweeping the Limit: The Tight End, the Loose End

The second table runs the same work under five different limits and shows that the choice of limit loads cost onto two separate directions.

At the tight end, the cost lands on the container’s own work. At a limit of 20 shares, the nightly job finishes only 3040 units over 200 periods — not even half of demand; when the run ends, 3052 units are still in the queue, and wait per unit climbs to 122.72 periods. At this limit the nightly job never finishes, it carries over to the next day. The neighbor, by contrast, is untouched: verifier wait 0.00, machine clipping 0. A tight limit builds isolation in full and bills its own container.

At the loose end, the cost changes location. At a limit of 60, the nightly job already finishes all of its work: 6092. Raising the limit to 80, then 100, does not increase completed work at all — 6092 in all three rows. The only thing that changes is wait: 10.54 → 5.48 → 2.49. The gain from loosening is only in latency, and the neighbor pays for it. The verifier’s completed work drops from 9012 to 8306 (706 units short), and its wait climbs from 0.12 to 10.38 periods. Machine clipping climbs from 0 to 4215 units.

This is the most misunderstood side of a resource limit, and the table lays it bare: a limit is a ceiling, not a guarantee. The limits of 50 and 100 sum to 150, and the machine can only give 100. Once the total is exceeded, what decides who gets how much is not the container’s limit but the distributor — and it sits outside the container. This is where isolation is punctured: the namespace separates processes, the limit sets a ceiling, but the machine itself continues to be shared, and under over-commitment the neighbor’s delay can climb to 86 times its baseline.

The Difference That Stays in the Environment

The second setting item is environment variables. The image pulled the filesystem difference inside itself: installed packages, library versions, file paths were all frozen. What does not freeze is the values that let the same image behave differently in different places — the data endpoint, the password, the tariff table, the time zone. These sit outside the image because, if they went into it, the image would have to be separate for every place.

• RT6 — The four services’ environment variable inventory is fictional; the names and required classes are model data. The missing-definition check really runs.

// runtime/environment-variables.mjs — counting the difference that cannot be pulled into the image.
// MODEL: the four services' environment variable inventory is fictional; the missing-definition check really runs.
const SERVICE = {                                                       // RT6: fictional inventory
  collector: { required: ["METER_NETWORK_ENDPOINT", "READING_QUEUE", "COLLECTION_INTERVAL"],
    defaulted: { LOG_LEVEL: "info", CONCURRENCY: "4" } },
  verifier: { required: ["READING_QUEUE", "RULE_VERSION", "DATA_ENDPOINT", "DATA_PASSWORD"],
    defaulted: { DEVIATION_THRESHOLD: "0.15", LOG_LEVEL: "info" } },
  billing: { required: ["DATA_ENDPOINT", "DATA_PASSWORD", "TARIFF_TABLE", "TIME_ZONE"],
    defaulted: { ROUNDING: "2", LOG_LEVEL: "info" } },
  workOrder: { required: ["DATA_ENDPOINT", "DATA_PASSWORD", "FIELD_ENDPOINT"],
    defaulted: { RETRY_COUNT: "3" } },
};

function start(name, env) {                        // service does not come up if a required definition is missing
  const s = SERVICE[name], missing = s.required.filter((k) => env[k] === undefined);
  if (missing.length) throw new Error(`${name}: undefined ${missing.join(", ")}`);
  return { ...s.defaulted, ...Object.fromEntries(s.required.map((k) => [k, env[k]])) };
}

const names = Object.keys(SERVICE);
const required = new Set(names.flatMap((a) => SERVICE[a].required));
const defaulted = new Set(names.flatMap((a) => Object.keys(SERVICE[a].defaulted)));
const all = new Set([...required, ...defaulted]);
const defs = names.reduce((t, a) => t + SERVICE[a].required.length + Object.keys(SERVICE[a].defaulted).length, 0);
const defaultOnly = [...defaulted].filter((k) => !required.has(k)).length;
console.log(`model: ${names.length} services, ${all.size} distinct variable names, ${defs} definition lines; ` +
  `required ${required.size} names, default-only ${defaultOnly} names`);

const filled = Object.fromEntries([...all].map((k) => [k, "value"]));
const countFailing = (env) => names.filter((a) => { try { start(a, env); return false; } catch { return true; } });
console.log(`services failing to start with empty env: ${countFailing({}).length}/${names.length}; ` +
  `failing to start with full env: ${countFailing(filled).length}/${names.length}`);
try { start("billing", { DATA_ENDPOINT: "x", DATA_PASSWORD: "y" }); } catch (e) { console.log(`example: ${e.message}`); }

console.log(`\n${"variable".padEnd(24)}${"in services".padStart(14)}${"drops when removed".padStart(19)}${"class".padStart(15)}`);
let dropped = 0, fragile = 0;
for (const k of [...all].sort()) {
  const count = names.filter((a) => SERVICE[a].required.includes(k) || k in SERVICE[a].defaulted).length;
  const d = countFailing(Object.fromEntries(Object.entries(filled).filter(([x]) => x !== k))).length;
  dropped += d;
  if (d > 0) fragile += 1;
  console.log(`${k.padEnd(24)}${String(count).padStart(14)}${String(d).padStart(19)}${(required.has(k) ? "required" : "defaulted").padStart(15)}`);
}
console.log(`\ntotal: ${fragile} of the ${all.size} variables individually bring down at least one service; ` +
  `the sum of one-at-a-time removals is ${dropped} service-drops. The output did not change, ${all.size} names stayed in the environment.`);

model: 4 services, 14 distinct variable names, 21 definition lines; required 9 names, default-only 5 names
services failing to start with empty env: 4/4; failing to start with full env: 0/4
example: billing: undefined TARIFF_TABLE, TIME_ZONE

variable                   in services drops when removed          class
COLLECTION_INTERVAL                  1                  1       required
CONCURRENCY                          1                  0      defaulted
DATA_ENDPOINT                        3                  3       required
DATA_PASSWORD                        3                  3       required
DEVIATION_THRESHOLD                  1                  0      defaulted
FIELD_ENDPOINT                       1                  1       required
LOG_LEVEL                            3                  0      defaulted
METER_NETWORK_ENDPOINT               1                  1       required
READING_QUEUE                        2                  2       required
RETRY_COUNT                          1                  0      defaulted
ROUNDING                             1                  0      defaulted
RULE_VERSION                         1                  1       required
TARIFF_TABLE                         1                  1       required
TIME_ZONE                            1                  1       required

total: 9 of the 14 variables individually bring down at least one service; the sum of one-at-a-time removals is 14 service-drops. The output did not change, 14 names stayed in the environment.

The number of the difference that cannot be pulled out of the environment reads here: 14 variable names, 21 definition lines. File paths and library versions were pulled into the output; these 14 names were not. Nine are required, and the service never comes up if they are left undefined — with an empty environment, all four services go down. Five have a default, and removing them drops no service; they carry the difference but produce no fragility.

The right-hand column shows where the fragility is concentrated. DATA_ENDPOINT and DATA_PASSWORD are required in three services at once; forgetting a single definition keeps three containers from coming up. These two are 14% of the 14 names, but alone they produce 6 of the 14 service-drops in the one-at-a-time removal sum — 43% of it. TIME_ZONE is another class: the image pulled in the time zone data but could not settle which time zone to operate in, because that decision belongs to the run, not the image. For billing, this one variable determines where a day boundary falls.

Here the cost of isolation is measured in writing: 21 definition lines, once per run site. What it buys is the same image running with four different behaviors in four different places.

Summary

• One of three decisions is made when a resource limit is exceeded, and all three distribute the same 6092 units of work to different places: throttle keeps loss at 0 but loads 10.54 periods of delay per unit; kill brings the delay down to 2.15 periods but loses 1968 units and spends 15 periods down; degrade brings delay down to 0.33 periods in exchange for a loss of 1106 units. Kill’s short queue is not a gain: the queue got short because the work inside it was deleted and the container restarted 3 times.
• The degrade decision’s result shows up in two containers: while the nightly job loses 1106 units, the verifier — which never once hit its own limit — also loses 121 units, because what clips it is the machine, not the container’s own limit.
• A tight limit bills its own container: at a limit of 20 shares, the nightly job finishes only 3040 units over 200 periods, 3052 units stay in the queue, and wait climbs to 122.72 periods; the neighbor’s wait stays at 0.00.
• A loose limit bills the neighbor: raising the limit from 60 to 100 does not increase the 6092 units the nightly job finishes at all, it only brings its wait down to 2.49; in exchange, the verifier finishes 706 units short and its wait climbs from 0.12 to 10.38 periods.
• Where isolation is punctured: a limit is a ceiling, not a guarantee. When the sum of the limits exceeds the machine’s capacity, the distribution decision is made outside the container; machine clipping climbs from 0 to 4215 units.
• The image pulled the filesystem difference inside itself; it could not pull in 14 variable names. Nine of them are required, and with an empty environment all four of the four services fail to come up; two of them, being required in three services at once, produce 6 of the 14 service-drops in the one-at-a-time removal sum.

Next Step

Both the resource limit and the environment variables were settings given from outside the container; both were known before the container started, and neither changed while the container ran. But a running container does not sit idle: it writes logs, fills a cache, saves the state of an unfinished batch job somewhere, puts a downloaded file in a directory. This data was not in the image, and it did not come from the environment either — the container itself produced it. So where does it get written, and what happens to it when the container stops? The next lesson builds the writable layer with real directories and counts this: how many bytes accumulate over the course of a run, how many records go away when the container is deleted, why a one-byte change does not take up one byte of space, and which data class this loss is acceptable in.