Control Groups

The previous lesson wrote that the five namespaces narrowed visibility but never limited consumption at all. The resource axis’s four facts — CPU share, memory limit, used memory, and CPU count — stood exactly as they were even after the five namespaces were opened. A namespace is a visibility filter, and visibility does not stop consumption: a process that sees its own process table can still keep the machine’s entire CPU busy.

This lesson takes up the mechanism that closes that axis and pays the second half of the debt left by the System Administration course. In the service manager lesson, a unit was an entry in the manager’s ledger, and the kernel side of resource accounting was explicitly left to this course. The question is the same again: does the cgroup close the resource axis, or does it leave something behind.

Tree, Controller, and Two Jobs

A cgroup is a process group, and groups form a tree. Every node of the tree is a directory; when a process’s number is written to a node’s member file, the process moves into that group, and its children inherit the group. A process is in exactly one group at a time; changing groups is a write operation and can be done while the process is running.

Controllers operate over the tree: each resource type — CPU, memory, I/O, process count — is a separate controller. A controller is enabled at one node and applies to the entire subtree beneath that node. A controller not enabled at an upper node cannot be enabled below it either; this rule is what makes delegating a branch of the tree to another manager safe.

Every controller does two separate jobs, and mixing them up produces the wrong expectation. Limiting sets a ceiling or a weight: the ceiling cannot be exceeded, and the weight decides the share under contention. Accounting, on the other hand, limits nothing, it only counts — memory used, CPU time spent, time spent waiting. A group can have accounting turned on with no limit set at all; in that case the group is a unit of measurement, not a constraint.

The path in the cgroup tree can be read by the process itself. There is a separate namespace type that hides this path, and it was named and passed over in the previous lesson: the cgroup namespace shifts the root of the path a process sees to its own group. Hiding the path does not mean hiding the limit; the limit files are still readable.

# example dump — not executed; paths and values are fiction
$ cat /sys/fs/cgroup/metrics.slice/cgroup.controllers
cpu io memory pids
$ cat /sys/fs/cgroup/metrics.slice/cpu.max
20000 100000
$ cat /sys/fs/cgroup/metrics.slice/memory.max
536870912
$ cat /sys/fs/cgroup/metrics.slice/memory.current
331350016
$ nproc
8

The first number pair in the dump is a quota: twenty thousand units in a hundred-thousand-unit window, that is, 0.2 CPU. The last line gives the machine’s CPU count. The two lines are both true, on the same machine, for the same process, and they do not contradict each other; they give two separate answers to the same question. What this lesson measures is exactly the gap between these two lines. The dump has not been executed, and no numeric claim comes from it.

Its Difference from the Per-Process Constraint

The System Administration course measured per-process resource limits: soft and hard values, inheritance on fork, and a running process’s limit not being changeable from the shell. That lesson is not repeated here, but the cgroup cannot be understood without writing down the difference between the two mechanisms.

The per-process limit is a ceiling on a single process. A hundred processes with the same limit can consume the machine a hundred times over; the machine runs out without any of them exceeding their limit. A cgroup, by contrast, looks at the group’s total: whatever the number of processes in the group, total consumption is compared against the ceiling. The second difference is timing. The per-process limit freezes at fork time and cannot later be changed from outside; a cgroup’s limit is written to a file and takes effect while the process is running. The third difference is accounting: the per-process limit has no counter, it produces a failure when exceeded; a cgroup counts even when it is not exceeded.

The two mechanisms do not replace each other, they stack. The same process is subject to both its own ceiling and its group’s ceiling, and which one it hits first depends on the value of both. In diagnosis, this stacking is a trap: a process’s own limit reading does not show the group’s ceiling, and the group’s counter does not show a single process’s own ceiling. The two readings are in separate places, and neither substitutes for the other.

• IM10 — The cgroup is the only mechanism that closes the resource axis; the resource axis has 4 facts.
• IM11 — The cgroup closes 3 facts of the resource axis and leaves open the cpu-count fact.
• IM12 — Closing does not mean the fact vanishes; it means the value the process sees stops being the machine’s value and becomes the group’s value.
• IM13 — The measurement does not model per-process limits; the layer measured in the System Administration course is not carried over here.

"""Shared definition, the part this lesson uses: 24 facts, 9 axes, and
the cgroup that closes the resource axis. What is measured: REMAINING EXPOSED SURFACE."""

FACTS = [
    ("other-processes-list", "process"), ("own-process-number", "process"),
    ("ancestor-process-chain", "process"),
    ("root-filesystem-tree", "mount"), ("other-mounts", "mount"),
    ("shared-temp-dir", "mount"),
    ("machine-interfaces", "network"), ("machine-routing-table", "network"),
    ("listening-ports", "network"),
    ("user-id-mapping", "user"), ("file-ownership", "user"),
    ("machine-name", "machine"),
    ("cpu-share", "resource"), ("memory-limit", "resource"),
    ("used-memory", "resource"), ("cpu-count", "resource"),
    ("kernel-version", "kernel"), ("kernel-settings", "kernel"),
    ("system-load", "kernel"), ("clock", "kernel"),
    ("capability-set", "privilege"), ("file-permissions", "privilege"),
    ("access-label", "label"), ("policy-rules", "label"),
]
MECHANISMS = {
    "pid-namespace":     {"axis": "process", "left_open": ["own-process-number"]},
    "mount-namespace":   {"axis": "mount", "left_open": ["root-filesystem-tree"]},
    "network-namespace": {"axis": "network", "left_open": []},
    "user-namespace":    {"axis": "user", "left_open": ["file-ownership"]},
    "uts-namespace":     {"axis": "machine", "left_open": []},
    "cgroup":            {"axis": "resource", "left_open": ["cpu-count"]},
}
FIVE = ["pid-namespace", "mount-namespace", "network-namespace",
        "user-namespace", "uts-namespace"]


def visible(mechanisms, facts=None):
    facts = FACTS if facts is None else facts
    closed = set()
    for m in mechanisms:
        v = MECHANISMS[m]
        closed |= {o for o, e in facts if e == v["axis"]}
        closed -= set(v["left_open"])
    return [o for o, _ in facts if o not in closed]


def open_axes(mechanisms, facts=None):
    facts = FACTS if facts is None else facts
    g = set(visible(mechanisms, facts))
    return sorted({e for o, e in facts if o in g})


resource = [o for o, e in FACTS if e == "resource"]
open_facts = set(visible(["cgroup"]))
print("ORACLE  fact:", len(FACTS), "| axis:", len({e for _, e in FACTS}))
print("resource axis has", len(resource), "facts")
print()
print("fact                 cgroup")
for o in resource:
    print(f"  {o:20s} {'LEAVES OPEN' if o in open_facts else 'closes'}")
print()
g = visible(["cgroup"])
print("cgroup alone: closed", len(FACTS) - len(g),
      "| left open 1 | visible", len(g),
      "| open axis", len(open_axes(["cgroup"])))

ORACLE  fact: 24 | axis: 9
resource axis has 4 facts

fact                 cgroup
  cpu-share            closes
  memory-limit         closes
  used-memory          closes
  cpu-count            LEAVES OPEN

cgroup alone: closed 3 | left open 1 | visible 21 | open axis 9

Three numbers stand side by side. Oracle: 24 facts, 9 axes; 4 facts on the resource axis. Isolation: the cgroup alone closes 3 facts. Remaining exposed surface: 21 facts and 9 axes — the single mechanism drops no axis from the list, because one fact stays standing on the resource axis.

The Fact Left Open: CPU Count

The three facts that close are replaced by the group’s own measures. When the process asks for its CPU share, it sees the group’s quota; when it asks for the memory limit, it sees the group’s ceiling; when it asks for used memory, it sees the group’s counter. There is no such substitution for the fourth: the machine’s CPU count shows through unchanged. A process with a quota of 0.2 CPU gets the answer eight on a machine with eight CPUs.

The cost of this is not abstract. Most runtimes choose their thread pool, the number of concurrently running workers, the number of garbage-collector threads, and the connection pool size once, at startup, by looking at the machine’s CPU count. A process that opens a pool forty times the size of its quota does not run faster; the scheduler still gives it a 0.2-CPU share, and the extra threads are paid for in waiting, context switching, and memory. The failure is silent too: no limit is exceeded, no error message appears, only latency grows.

The narrowing path has three steps, and all three correspond directly to this lesson’s table. First, change the source of the sizing decision: read the group’s quota instead of the machine’s CPU count; the quota file is already somewhere the process can see. Second, pass the quota to the process as an explicit value — if the runtime can read a setting, the number of concurrently running workers is given from configuration and not left to a guess. Third, place a CPU set constraint alongside the quota: when the CPUs a process can run on are narrowed, the visible count narrows too, and the two answers move closer together. None of the three fully closes the fact; they reduce the open surface from a fact to a miscalculation.

Who Writes the Limit

Writing to cgroup files by hand is rarely the right path, because another party manages the tree. The service manager opens a node for every unit, groups units into slices, and translates the resource settings in a unit file into these nodes’ files. A value written by hand is lost when the manager rebuilds the tree; the durable place is the unit definition.

# example unit file fragment — not executed
[Service]
CPUQuota=20%
AllowedCPUs=2-3
MemoryMax=512M
MemoryAccounting=yes
TasksMax=64

The first two lines in this fragment touch directly on the gap this lesson measures: the first writes the quota, the second narrows the CPUs the process can run on. The fourth line only turns on accounting and limits nothing. The settings’ names and syntax vary by service manager; what is shown here is not the form but the distinction: the line that sets a limit and the line that only counts are separate lines.

A branch of the tree can be delegated to another manager. Delegating means granting the authority to open nodes and move processes under that branch; the delegate divides its own subtree however it likes but cannot rise above the ceiling above it. This is exactly how a container runtime works, and it is measured in the fifth lesson. Delegation’s boundary must also be written down: if the number of nodes opened in the delegated branch and the number of processes moved into those nodes are not bounded from above, the tree keeps growing while the ceiling holds.

Accounting itself is a surface that faces this course’s second topic. In place of the three facts it closes, the cgroup substitutes the group’s own measures, and these measures are readable files: the CPU time the group has spent, the memory it currently holds, how many times it has hit the ceiling. This is not a flaw in isolation, it is a requirement of operation — whether a limit holds can only be known by counting. But the same numbers are also read by the process inside, and they carry information about the group’s load. This course’s second topic takes up reading paths of this kind as an observation channel and counts how much each one punctures isolation.

The decision made when a memory ceiling is hit also stays inside the group: the kernel picks one of the group’s own processes that exceeded the ceiling. This is its difference from the machine-wide memory exhaustion measured in the System Administration course, and it matters in diagnosis — a death inside a group does not mean the machine’s memory is exhausted. Because the sum of group limits can be set larger than the machine’s memory, the two situations can be seen separately on the same machine.

The Growing Bundle and Fact Set Sweep

Throughout the course, mechanisms are added onto the same process, and how the surface narrows is counted. The previous lesson ended with five namespaces; this lesson adds the sixth mechanism. The sweep is again done on the fact set, because the model has no randomness and no second seed.

• IM14 — The added fact io-share belongs to the resource axis and is not among what the cgroup leaves open.
• IM15 — The removed cpu-count is the resource axis’s single fact left open.
• IM16 — Bundle order is binding: five namespaces, then the cgroup.

# --- This block builds on the previous block: FACTS, MECHANISMS, FIVE,
# visible and open_axes come from there.
BUNDLE = FIVE + ["cgroup"]
print("the bundle growing through the course")
print("stage                      mechanism  visible  open axis")
for name, d in (("five namespaces", FIVE), ("+cgroup", BUNDLE)):
    print(f"  {name:24s} {len(d):7d}  {len(visible(d)):7d}  {len(open_axes(d)):10d}")
print()
ADDED = FACTS + [("io-share", "resource")]
MISSING = [(o, e) for o, e in FACTS if o != "cpu-count"]
print("fact set sweep (six mechanisms applied)")
print("set                        fact  axis  visible  open axis  resource open")
for name, facts in (("base", FACTS), ("+io-share", ADDED),
                     ("-cpu-count", MISSING)):
    g = visible(BUNDLE, facts)
    e = open_axes(BUNDLE, facts)
    ro = sum(1 for o, ax in facts if ax == "resource" and o in set(g))
    print(f"  {name:24s} {len(facts):4d}  {len({x for _, x in facts}):5d}"
          f"  {len(g):7d}  {len(e):10d}  {ro:11d}")

the bundle growing through the course
stage                      mechanism  visible  open axis
  five namespaces                5       15           7
  +cgroup                        6       12           7

fact set sweep (six mechanisms applied)
set                        fact  axis  visible  open axis  resource open
  base                       24      9       12           7            1
  +io-share                  25      9       12           7            1
  -cpu-count                 23      9       11           6            0

The first table is the first payment on the course’s second claim. The sixth mechanism drops the visible fact count from 15 to 12 — a gain of three facts — but the open axis count stays at 7. The resource axis did not close, it only thinned. Adding a mechanism reduces the fact count, not the axis count.

The sweep confirms the same pattern. When the twenty-fifth fact is added to the resource axis, the cgroup closes it too, and the exposed surface stays at 12: adding a fact to a closed axis does not grow the surface. By contrast, when the single fact left open is removed from the list, the axis closes completely, the visible fact count drops to 11, and the open axis count to 6. Where the measurement is sensitive is the same again: what determines the exposed surface is not the fact count on the axis, but which fact is left open.

Every number in the model is a full count over the fiction; it is not a measurement taken on a machine. How many facts the resource axis carries on a real system varies with which controllers are enabled and how the process file system is mounted.

Summary

• A cgroup is a process group; groups form a tree, controllers operate over them, and every controller does two separate jobs: limiting and accounting.
• It differs from the per-process resource limit on three points: it looks at the group’s total, it can be changed while the process is running, and it counts even when not exceeded.
• The resource axis has 4 facts; the cgroup closes 3 of them, leaves open the cpu-count fact, and on its own leaves 21 of the 24 facts visible.
• The fact left open produces a concrete miscalculation: a process with a quota of a fifth of a CPU sees the machine’s CPU count and sizes its pools accordingly. The narrowing path is reading the quota, giving the concurrent worker count from configuration, and narrowing the CPU set.
• In the six-mechanism bundle, the visible fact count drops from 15 to 12, and the open axis count stays at 7; in the sweep, removing the single fact left open closes the axis and the surface drops to 11.

Next Step

The resource axis thinned, but what a process can do has still not been measured. Narrowing visibility and limiting consumption does not determine which privileged operation a process can request from the kernel: a process inside the same group can still attempt to change network configuration, open a mount point, or touch another user’s file. The next lesson takes up the superuser privilege’s division into separately grantable pieces, measures capability sets and dropping, and counts why only one of the privilege axis’s two facts closes. The fact left open is file permissions, and the reason is unexpected: a capability dropped from a process does not change what is written on the file.