Container Runtime

Four lessons measured four mechanism families one by one: namespaces narrowed visibility, the cgroup narrowed consumption, capability dropping narrowed privilege, the mandatory label narrowed access. Each left something open on its own axis, and a narrowing path was written next to each. On a real system, though, these mechanisms are not turned on one by one. A program reads a definition file, applies every mechanism written in the file in the correct order, and starts the process. That program’s name is the container runtime.

This lesson treats the runtime as a bundle of mechanisms and asks a single question: how much does adding a mechanism narrow the surface. Container images, the layer model, placement, scaling, and orchestration were measured in the DevOps curriculum’s Containers and Container Orchestration courses; they are not repeated. The subject here is only the parts the kernel provides and how a runtime brings them together.

What the Runtime Does

The runtime takes as input a bundle made of a directory and a definition file. The directory is the process’s root filesystem tree; the definition file writes which mechanisms open with which settings. The runtime interprets this file, translates it into kernel calls, and starts the process. Its job is exactly that: it does not invent a new isolation type, it applies the existing ones in order.

# example runtime definition (abbreviated, field names are examples)
# — not executed
process:
  command: /usr/local/bin/metrics-collector
  uid: 10001
  capability-ceiling: [CAP_NET_BIND_SERVICE]
  no-new-privileges: yes
  access-label: system_u:system_r:metrics_t:s0
root:
  path: root-tree
  read-only: yes
mounts:
  - target: /data/metrics
    options: [rw, nosuid, nodev, noexec]
namespaces: [process, mount, network, machine, user, ipc, cgroup]
id-mapping:
  - inside: 0
    outside: 100000
    size: 65536
cgroup:
  path: /metrics.slice/metrics-collector
  cpu-quota: 20000/100000
  memory-ceiling: 536870912

Every line in the definition corresponds to one of the previous four lessons. The field names vary by specification, and the dump has not been executed; no numeric claim comes from it.

Order is binding, and the wrong order leaves a silent gap. The user namespace opens first, because it is the only type that lets an unprivileged user open the other types, and the identity mapping must be written before it closes. The cgroup is prepared before the process is started, or the first seconds pass unbounded. The root tree is changed after the process moves into its own mount namespace; changing it first pollutes the machine’s mount list. Capability dropping and the no-new-privileges flag are left for last, right before the exec; done early, the preparation steps fail, done late, the dropping is never applied.

# example dump — not executed
$ runtime create metrics-01 --bundle ./bundle
$ runtime start  metrics-01
$ runtime state  metrics-01
$ runtime kill   metrics-01 TERM
$ runtime delete metrics-01

The lifecycle has five steps, and create and start are separate. Create sets up every mechanism and holds the process before it starts; start only ends the hold. The reason for the split is operational: it allows the configuration to be verified after all isolation is set up, while the process has not yet run a single command.

The runtime itself is also a process and needs privilege to set up the mechanisms. There are two operating modes. In privileged mode, the runtime runs as a privileged service, opens every mechanism directly, and stays outside the isolation it builds. In rootless mode, the chain starts with a user namespace: the runtime first opens a user namespace, writes the identity mapping, and sets up the other mechanisms from inside that namespace. The second mode closes no additional fact in this course’s measurement, but it narrows the blast radius in the event of a flaw — because the runtime’s own capabilities are also bounded by the mapping.

A boundary that stays outside the bundle must also be written down openly. All of these mechanisms share a single kernel; the runtime separates processes from one another, it does not duplicate the kernel. If a separate kernel is wanted, what is needed is a different technology class, and it is not the subject of this course. In the measurement’s last section, why the kernel axis never closes comes from exactly this sharing.

Two Bundles

• IM31 — The common bundle is six mechanisms: the process, mount, network, and UTS namespaces, the cgroup, capability dropping. The user namespace and the mandatory label are not in this bundle.
• IM32 — The strict bundle is eight mechanisms: the user namespace and the mandatory label are added to the common bundle.
• IM33 — The bundles differ only in their mechanism list; the fact set, the axes, and the facts left open are the same as in the shared definition.
• IM34 — The model counts eight mechanisms. The system call filter a real runtime applies, and the step of masking some paths of the process file system, are not counted as mechanisms in the model; both are written below as narrowing paths.

"""Shared definition: isolation is a filter. A container runtime is a
BUNDLE of mechanisms. What is measured: REMAINING EXPOSED SURFACE."""

FACTS = [
    ("other-processes-list", "process"), ("own-process-number", "process"),
    ("ancestor-process-chain", "process"),
    ("root-filesystem-tree", "mount"), ("other-mounts", "mount"),
    ("shared-temp-dir", "mount"),
    ("machine-interfaces", "network"), ("machine-routing-table", "network"),
    ("listening-ports", "network"),
    ("user-id-mapping", "user"), ("file-ownership", "user"),
    ("machine-name", "machine"),
    ("cpu-share", "resource"), ("memory-limit", "resource"),
    ("used-memory", "resource"), ("cpu-count", "resource"),
    ("kernel-version", "kernel"), ("kernel-settings", "kernel"),
    ("system-load", "kernel"), ("clock", "kernel"),
    ("capability-set", "privilege"), ("file-permissions", "privilege"),
    ("access-label", "label"), ("policy-rules", "label"),
]
MECHANISMS = {
    "pid-namespace":     {"axis": "process", "left_open": ["own-process-number"]},
    "mount-namespace":   {"axis": "mount", "left_open": ["root-filesystem-tree"]},
    "network-namespace": {"axis": "network", "left_open": []},
    "user-namespace":    {"axis": "user", "left_open": ["file-ownership"]},
    "uts-namespace":     {"axis": "machine", "left_open": []},
    "cgroup":            {"axis": "resource", "left_open": ["cpu-count"]},
    "capability-dropping": {"axis": "privilege", "left_open": ["file-permissions"]},
    "mandatory-label":   {"axis": "label", "left_open": ["access-label"]},
}
COMMON = ["pid-namespace", "mount-namespace", "network-namespace",
          "uts-namespace", "cgroup", "capability-dropping"]
STRICT = COMMON + ["user-namespace", "mandatory-label"]


def visible(mechanisms, facts=None):
    facts = FACTS if facts is None else facts
    closed = set()
    for m in mechanisms:
        v = MECHANISMS[m]
        closed |= {o for o, e in facts if e == v["axis"]}
        closed -= set(v["left_open"])
    return [o for o, _ in facts if o not in closed]


def open_axes(mechanisms, facts=None):
    facts = FACTS if facts is None else facts
    g = set(visible(mechanisms, facts))
    return sorted({e for o, e in facts if o in g})


print("ORACLE  fact:", len(FACTS), "| axis:", len({e for _, e in FACTS}))
print()
print("bundle    mechanism  closed  visible  open axis")
for name, d in (("bare", []), ("common", COMMON), ("strict", STRICT)):
    g = visible(d)
    print(f"  {name:8s} {len(d):7d}  {len(FACTS) - len(g):7d}"
          f"  {len(g):7d}  {len(open_axes(d)):10d}")
print()
print("difference between the two bundles:", [d for d in STRICT if d not in COMMON])
print("facts it makes a difference to     :",
      sorted(set(visible(COMMON)) - set(visible(STRICT))))
print("the two mechanisms narrow the surface by",
      len(visible(COMMON)) - len(visible(STRICT)), "facts")

ORACLE  fact: 24 | axis: 9

bundle    mechanism  closed  visible  open axis
  bare           0        0       24           9
  common         6       12       12           7
  strict         8       14       10           7

difference between the two bundles: ['user-namespace', 'mandatory-label']
facts it makes a difference to     : ['policy-rules', 'user-id-mapping']
the two mechanisms narrow the surface by 2 facts

Three numbers stand side by side. Oracle: 24 facts, 9 axes. Isolation: the common bundle closes 12 facts, the strict bundle 14 facts. Remaining exposed surface: 12 facts in the common bundle, 10 facts in the strict bundle; 7 axes in both.

This is the strongest payment on the course’s second claim. Going from the common bundle to the strict bundle grows the mechanism count by a third, adds two separate mechanisms, and noticeably increases configuration and operational load — and it narrows the surface by only 2 facts. The facts that narrow are user-id-mapping and policy-rules. The open axis count does not change at all.

The result does not mean the two mechanisms are worthless, and it should not be read that way. The user namespace narrows the scope of privilege by closing identity mapping; the mandatory label, as counted in the previous lesson, binds 28 of 36 access pairs to policy. The measurement does not count these; it only counts visibility. What it says is narrow and precise: on the visibility side, the two mechanisms’ gain is two facts.

The measurement’s practical use also comes from this. When deciding whether to tighten a bundle, the question to ask should not be “how many mechanisms did I turn on” but “which fact is still visible.” The first question rewards the length of the configuration; the second shows where the gain stops. Written down, the difference between the two bundles is only two lines, and those two lines can be weighed against the cost of tightening.

As Mechanisms Are Added

Comparing the bundles end to end does not show where the gain accumulates. When the same strict bundle is built mechanism by mechanism, the curve becomes readable.

• IM35 — The order of addition is the order in the strict bundle’s list.
• IM36 — The fact added in the sweep, kernel-build-options, belongs to the kernel axis; the removed clock is on the same axis too.

# --- This block builds on the previous block: FACTS, MECHANISMS, COMMON, STRICT,
# visible and open_axes come from there.
print("as mechanisms are added (in strict-bundle order)")
for i in range(len(STRICT) + 1):
    added = STRICT[i - 1] if i else "-"
    print(f"  {i} mechanism ({added:20s}) visible {len(visible(STRICT[:i])):2d}"
          f"  open axis {len(open_axes(STRICT[:i]))}")
print()
g = set(visible(STRICT))
print("left open after the strict bundle:", len(g), "facts (by axis)")
for e in open_axes(STRICT):
    o = [x for x, ax in FACTS if ax == e and x in g]
    print(f"  {e:10s} {len(o)}  {', '.join(o)}")
print()
ADDED = FACTS + [("kernel-build-options", "kernel")]
MISSING = [(o, e) for o, e in FACTS if o != "clock"]
print("fact set sweep (strict bundle applied)")
print("set                             fact  axis  visible  open axis")
for name, facts in (("base", FACTS), ("+kernel-build-options", ADDED),
                     ("-clock", MISSING)):
    print(f"  {name:29s} {len(facts):4d}  {len({x for _, x in facts}):5d}"
          f"  {len(visible(STRICT, facts)):7d}  {len(open_axes(STRICT, facts)):10d}")

as mechanisms are added (in strict-bundle order)
  0 mechanism (-                   ) visible 24  open axis 9
  1 mechanism (pid-namespace       ) visible 22  open axis 9
  2 mechanism (mount-namespace     ) visible 20  open axis 9
  3 mechanism (network-namespace   ) visible 17  open axis 8
  4 mechanism (uts-namespace       ) visible 16  open axis 7
  5 mechanism (cgroup              ) visible 13  open axis 7
  6 mechanism (capability-dropping ) visible 12  open axis 7
  7 mechanism (user-namespace      ) visible 11  open axis 7
  8 mechanism (mandatory-label     ) visible 10  open axis 7

left open after the strict bundle: 10 facts (by axis)
  kernel     4  kernel-version, kernel-settings, system-load, clock
  label      1  access-label
  mount      1  root-filesystem-tree
  privilege  1  file-permissions
  process    1  own-process-number
  resource   1  cpu-count
  user       1  file-ownership

fact set sweep (strict bundle applied)
set                             fact  axis  visible  open axis
  base                            24      9       10           7
  +kernel-build-options           25      9       11           7
  -clock                          23      9        9           7

The shape of the curve is a result on its own. The first four mechanisms drop the visible fact count from 24 to 16: eight facts. The last four mechanisms drop it from 16 to 10: six facts. The open axis count, though, never changes again after the fourth mechanism — four mechanisms were added from the fifth through the eighth, and the axis count stayed at 7.

The reason is written in the last table. Seven of the 10 facts left open after the strict bundle are one apiece on seven separate axes. Because every mechanism leaves one fact open on its own axis, that axis never drops off the list. Something is always left open on seven axes, and this does not close by adding mechanisms. This is exactly the course’s second claim: adding a mechanism reduces facts but not axes.

The sweep confirms the same structure and gives a different result from the previous lessons. When the twenty-fifth fact is added to the kernel axis, the surface rises from 10 to 11: the addition maps directly onto the surface. When a fact is removed, the surface drops to 9, but the open axis count stays at 7. Where does the difference come from — in the previous lessons, the added fact fell onto a closed axis and the mechanism closed it too; here, the axis was never closed.

The Axis That Never Closes: Kernel

Four of the 10 open facts are gathered on a single axis, and that axis’s name is kernel: kernel version, kernel settings, system load, and clock. These are the single largest block of the exposed surface in the strict bundle, and the reason is not a gap in the model — there is no mechanism that closes the kernel axis. A namespace does not produce a second kernel; the machine has a single kernel, and every process talks to it. This is isolation’s most fundamental boundary, and it is not overcome by configuration.

By contrast, the axis can be narrowed, and there are three ways. First, masking or making read-only the paths that publish kernel information: in the process’s root tree, these paths are either never visible or not writable. This is what a runtime does by default, and it is one of the steps not counted as a mechanism in the model. Second, giving separately the kernel settings that can be split by namespace; not all settings can be split, but the ones that can stop showing the machine’s value. Third, narrowing the set of calls a process can make to the kernel with a system call filter: the filter does not make a fact invisible, but it reduces the operations that correspond to the visible fact. None of the three zeroes out the surface; the honest reading of the measurement is: something on this axis will always be seen, and the design is made accordingly.

For the remaining six facts, the narrowing paths were written in the previous lessons: shrinking the root tree and mounting it read-only, reading the quota and binding pool sizing to it, clearing ownership flags and mounting the file system with nosuid, normalizing ownership to the mapped range, naming the access label so it carries no hidden information. All five follow the same pattern: the fact left open is narrowed not on the process, but in the process’s world.

Summary

• A container runtime does not invent a new isolation type; it is a program that applies the mechanisms in a definition file in the correct order. Order is binding, and the wrong order leaves a silent gap.
• The common six-mechanism bundle leaves 12 facts open, the strict eight-mechanism bundle 10 facts: adding two mechanisms narrows the surface by only 2 facts, and the facts that narrow are user-id-mapping and policy-rules.
• The open axis count stands at 7 after the fourth mechanism and does not change at all when four more mechanisms are added; adding a mechanism reduces facts, not axes.
• Of the 10 facts left open after the strict bundle, seven are one apiece on seven separate axes; the remaining four are on the kernel axis, and no mechanism closes that axis.
• The kernel axis cannot be closed, but it can be narrowed: masking the paths that publish information, giving namespace-splittable settings separately, and a system call filter. Every number in the model is a full count over the fiction, not a measurement.

Next Step

This topic counted what isolation can and cannot close; the strict bundle closed 14 of 24 facts and left 10 open. Whether the 14 closed facts have genuinely closed has not been tested yet. This course’s second topic asks exactly that question, and its answer points in the opposite direction from the first topic: tools that read a process’s conversation with the kernel, sample CPU usage, capture events inside the kernel, and inspect a crashed process’s core afterward bring back some of the facts isolation closed. The next lesson measures the first of these, system call tracing: it counts how many facts come back and the cost it adds to the run. This course’s two topics are each other’s opposite, and it is written openly — observation punctures isolation.