The Service Manager Model

The process management topic looked at individual processes: which one is really heavy, which one responds to a termination request, which one hits an imposed resource limit. Most of the processes on a server, though, are not started by an operator. The machine boots, dozens of programs come up with no one watching, one crashes and another takes its place, and by morning everything looks in order. What does this is a service manager, and what it manages is not the process itself but the process’s definition.

This lesson builds that definition and the graph between definitions. Its question is: how much of the system’s real structure does the manager’s printed list show? The list prints six lines, and all six lines are correct. What is measured is how many times the diagnosis drawn from those six correct lines is wrong.

Unit, Target, and Two Kinds of Edge

The object the service manager manages is called a unit (service unit). A unit is a text file: it says which program runs as which user, what happens if it fails, and which other units it needs. The manager reads these files, turns the relationship between them into a graph, and starts processes by following the graph. This distinction carries the rest of the course: the unit file sits on disk, the process runs in memory. Changing the file does not change the running process, killing the process does not delete the file, and a unit can be “loaded” without ever having run.

There are unit types besides services, and all of them sit in the same graph: a unit representing a mount point, a unit that listens on a network socket and wakes a service on an incoming connection, a unit carrying a scheduled task, a unit that watches a file path. This topic measures only service units; the other types are named and passed over.

When units need to be referred to as a group rather than one by one, a target is used. A target does no work itself and starts no process; it is a name representing the units grouped under it. Stages like “network ready,” “multi-user system ready,” and “shutdown” are named with targets. Moving to a target means enabling the units under it. That a target does no work raises the question of how much information an “arrived at target” notification carries; that question is measured in the boot process lesson.

Edges in the graph are of two kinds, and confusing them is the error this lesson measures. A requires edge says one unit is useless without another: if the required unit stops, the requiring unit stops too. An ordering edge only says timing: this unit should start after that one. An ordering edge does not create a dependency. Two units can have ordering without requiring, or requiring without ordering. Command names and unit-file syntax vary by service manager and distribution; the model here describes common behavior.

Requiring also has a weak form, called wanting: the wanted unit is attempted, and if it cannot be started, the wanting unit keeps running anyway. The weak edge is commonly preferred because it stops crash propagation; in exchange it silences the breakage of an edge in the graph. The mock server’s graph carries no weak edge, because the confusion this lesson measures sits between ordering and requiring. Printing all three edge types in the same tree, in the same form, is the subject of the next section.

Process isolation from one another, how the kernel side of resource accounting is kept, and control-group and namespace mechanisms are not covered in this course; they are left to the M03/K05 Kernel Interfaces and Isolation course. Here, a unit is a record in the manager’s bookkeeping ledger.

The manager’s need for a graph is not only bookkeeping. Without a graph, units would have to be started one after another, each waiting on the last; with a graph, units with no edge between them can start at the same time. This distinction determines boot time and is counted layer by layer in lesson 04.

The Mock Server’s Unit Graph

The same mock server is used throughout the course. It has six units: ingest receives data from outside, queue buffers the data it receives, process reads from the queue, report produces output from the processed data, metrics collects the ingestion pipeline’s counters, and backup runs without depending on anyone.

SV1 — a unit is a definition, not the running process itself. SV2 — the requires edge is transitive: if report requires process and process requires queue, then report is also indirectly tied to queue. SV3 — the ordering edge is not counted as transitive and creates no dependency. SV4 — report starts only after backup but does not require it; SV5 — metrics starts only after queue but does not require it. SV6 — the graph is acyclic. SV7 — stopping a unit stops every unit that requires it. SV8 — the graph carries no weak wanting edge; only requiring and ordering exist. SV9 — every unit belongs to the same manager and sits on a single machine; no edge reaches out to a remote node. SV10 — the graph does not change during the measurement; no unit file is added, removed, or reread.

The last of these assumptions does not always hold on a real system, and what happens when it does not is covered separately later. The rest of the measurement rests on a fixed graph: the oracle is this graph.

The manager’s unit list does not show the graph; it prints units in alphabetical order, one per line. The dump below has not been run on this machine; it is written as an example to show the format:

UNIT               LOAD    ACTIVE   SUB      DESCRIPTION
backup.service     loaded  active   running  Backup job
ingest.service     loaded  active   running  Data ingestion unit
metrics.service    loaded  active   running  Metrics collector
process.service    loaded  active   running  Processing unit
queue.service      loaded  active   running  Queue unit
report.service     loaded  active   running  Report generator

Six lines, six units, zero edges. The only information available to an operator looking at this list is which units exist. The answer to “what happens if I stop this unit” is not in the list and cannot be extracted from it.

The Command That Prints the Graph, and Which Way It Prints

Service managers offer a command that prints dependencies as a tree. This command has two directions, and they answer different questions. The default direction is downward: it shows what a unit needs in order to run. The reverse direction is upward: it shows who needs that unit. The answer to an operator’s question “what breaks if I stop this” lies only in the reverse direction, while the parameterless call prints the downward direction. The two dumps below have not been run; they are written to show what each direction of the command prints:

$ systemctl list-dependencies report.service
report.service
* +-process.service
*   +-queue.service
*     +-ingest.service

$ systemctl list-dependencies --reverse ingest.service
ingest.service
* +-queue.service
* | +-process.service
* |   +-report.service
* +-metrics.service

The second dump lists four units, and that is the correct count. But the tree prints the requires edge, the ordering edge, and the weak wanting edge in the same form; which line comes from which kind of edge cannot be read from the tree itself. Whether a unit shown in the tree will really stop when stopped, or only start later, requires looking at the unit files. What the next section measures is exactly the cost of this ambiguity.

Four Readings, Four Diagnoses

The same graph can be read at four levels of detail. The flat list gives only the units. The direct reading counts, for each unit, the units that directly require it. The transitive graph follows the chain to its end. The with ordering reading counts ordering edges as requiring too. The oracle is the transitive result computed over requires edges; the metric is how many units each reading diverges from that result on.

UNIT = {
    "ingest":  {"requires": [], "restart": "always"},
    "queue":   {"requires": ["ingest"], "restart": "always"},
    "process": {"requires": ["queue"], "restart": "on-failure"},
    "report":  {"requires": ["process"], "restart": "no"},
    "metrics": {"requires": ["ingest"], "restart": "always"},
    "backup":  {"requires": [], "restart": "no"},
}
AFTER = {"report": ["backup"], "metrics": ["queue"]}      # ordering only


def dependents_of(unit, units=None):
    units = UNIT if units is None else units
    direct = [a for a, v in units.items() if unit in v["requires"]]
    all_ = set(direct)
    for d in direct:
        all_ |= dependents_of(d, units)
    return all_


def with_ordering(unit):
    """Reading that counts the ordering edge as a dependency too."""
    direct = [a for a, v in UNIT.items()
              if unit in v["requires"] or unit in AFTER.get(a, [])]
    all_ = set(direct)
    for d in direct:
        all_ |= with_ordering(d)
    return all_


oracle = {b: len(dependents_of(b)) for b in UNIT}
diagnoses = {
    "flat list": {b: 0 for b in UNIT},
    "direct": {b: sum(1 for v in UNIT.values() if b in v["requires"])
               for b in UNIT},
    "transitive graph": dict(oracle),
    "with ordering": {b: len(with_ordering(b)) for b in UNIT},
}
print("unit       oracle  direct  flat list")
for b in UNIT:
    print(f"  {b:9s} {oracle[b]:5d}  {diagnoses['direct'][b]:6d}"
          f"  {diagnoses['flat list'][b]:9d}")
print("  oracle total:", sum(oracle.values()))
print()
print("reading           output lines  wrong diagnosis")
for ad, t in diagnoses.items():
    wrong = sum(1 for b in oracle if t[b] != oracle[b])
    print(f"  {ad:16s} {sum(t.values()):12d}  {wrong:15d}")

unit       oracle  direct  flat list
  ingest        4       2          0
  queue         2       1          0
  process       1       1          0
  report        0       0          0
  metrics       0       0          0
  backup        0       0          0
  oracle total: 7

reading           output lines  wrong diagnosis
  flat list                   0                3
  direct                      4                2
  transitive graph            7                0
  with ordering               9                2

Three Numbers and the Second Claim

Three numbers sit side by side. Oracle: the graph’s real state is 7 dependency relationships across six units; if ingest stops, four units stop with it. Tool output: the flat list prints 0 edge lines, the direct reading 4, the transitive graph 7, the with-ordering reading 9. Wrong diagnosis: 3 for the flat list, 2 for the direct reading, 0 for the transitive graph, and 2 again for the with-ordering reading.

The with-ordering reading prints the most lines, and its diagnosis is worse than the transitive graph’s. The two extra lines say that stopping queue also stops metrics, and that stopping backup also stops report; both are wrong. This is the first payment of the course’s second claim: more output does not mean a better diagnosis. What leads to the right count is not how many lines are printed, but which kind of edge is counted.

The flat list’s 3 wrong diagnoses come from the same place. Because the list shows no edge at all, it produces the answer “stopping this stops only this” for every unit; that answer is wrong for ingest, queue, and process, and happens to be right for the remaining three. The word “happens” is not used loosely here: the reason the answer holds for report, metrics, and backup is not that the reading is good, it is that those units sit at the graph’s edge. The same reading errs more often on a graph with fewer edges at the tip. Output whose diagnosis is not tested counts as unmeasured; the list is correct, the conclusion drawn from it is not.

The direct reading’s 2 wrong diagnoses come from a different flaw. This reading sees edges but cuts the chain after one step: 2 for ingest, real value 4; 1 for queue, real value 2. The direction of error here is always the same, always undercounting. The with-ordering reading’s error direction is always overcounting. Because the two error types land in the same column, the counts can come out equal, even though their operational consequences are opposite: the operator who undercounts is caught unprepared, the operator who overcounts does unnecessary work.

The working habit that follows can be written in one sentence: before stopping a unit, the reverse dependency tree is read, and every line in that tree is verified against the unit files as either requiring or ordering. When either step is skipped, the direction of the error is known. If the tree is not read, units are undercounted and unexpected units stop; if the tree is read but the edge type is not verified, units are overcounted and an unnecessary maintenance window is opened. Both land in the same “wrong diagnosis” column in the measurement.

The Graph Is a Snapshot

Unit files do not sit in a single directory on disk. The definition a package installs is in one directory, a locally written definition is in another, a definition generated at runtime is in a third, and a file with the same name shadows another across these directories. Alongside shadowing there are also drop-in files: these do not replace the whole file, they only add a few lines or change a single setting. As a result, a unit’s effective definition sits not in one file but in the combination of several; only the command that prints the merged definition says which line came from where. Opening and reading a unit’s file does not mean seeing which settings it actually runs with.

There is a second fact that catches people out even more: the manager does not reread these files on every command. It reads them at startup, builds the graph in memory, and works with that copy. Changing the file on disk does not change the graph in memory; the change only enters the graph once the manager is told to reread. So the observation “I fixed the file but the behavior did not change” is not a malfunction, it is the expected result. The status output does not distinguish between the two states either: a unit shows as “active” even while the file on disk and the definition in memory differ. The gap between the oracle and the tool output opens here not in edge count but in time; the graph is a snapshot of the disk, and when the copy was taken does not appear in the output.

Second Seed

The numbers might be tied to a single graph. When the same measurement is repeated with two seeds over twenty random six-unit graphs built with a seeded generator, total wrong diagnoses come out to 60 and 69 for the flat list, 16 and 21 for the direct reading, 0 and 0 for the transitive graph, and 52 and 36 for the with-ordering reading. The order of magnitude holds: the transitive graph is error-free in both seeds, and the reading that counts ordering as dependency produces an error close to the flat list’s in both seeds too. The individual counts depend on the mock data; ordering is not requiring is not among them.

One point deserves underlining. The transitive-graph reading gives zero wrong diagnoses in both seeds, and this is not a property of the mock data, it is a consequence of the definition: the oracle was already defined as the transitive closure. The measurement’s meaning is not in the correct reading coming out error-free, it lies in the size of the wrong readings. Across the forty graphs’ total, the cost of only confusing edge types is eighty-eight wrong diagnoses; the cost of not reading edges at all, in the same forty graphs, is one hundred twenty-nine. That the two numbers sit this close says counting the ordering edge as a dependency is not much better than not reading the graph at all.

Summary

• The service manager’s object is the unit; a unit is not the running process but the process’s definition. A target is a name that groups units and does no work itself.
• The graph has two kinds of edge: the requires edge creates a dependency and is transitive, the ordering edge only says timing and creates no dependency.
• The mock server has six units and 7 dependency relationships; if ingest stops, four more units stop.
• The flat unit list shows 0 edges and produces a wrong diagnosis on 3 of six units; the direct-dependency column brings this down to 2, the transitive graph to 0.
• The reading that counts ordering edges as dependencies prints 9 lines and still gives 2 wrong diagnoses: the reading with the most output does not give the best diagnosis.
• Namespaces and control groups are not covered in this course; they are left to the M03/K05 Kernel Interfaces and Isolation course.

Next Step

This lesson built the graph and counted how it does not show up in the list. The next question is even more unsettling: how reliable is the status word, the one thing that does show up in the list? A unit showing “active” against its name means the unit is running; it does not mean it is healthy. The next lesson measures the status word of a unit stuck in a failure loop on the same mock server and counts how many of sixty crashes are seen at which polling interval.