The Boot Process

The previous lesson measured a single unit: one that crashes on its own and, depending on its policy, revives or does not revive on its own. In that measurement the unit was alone. In the mock server, units are the nodes of the graph built in the first lesson, and that graph is still there. Restarting one unit does not concern only that unit.

This lesson looks at how the graph is run over time. When the machine boots, this graph runs from end to end once; when a unit is restarted by hand, part of the graph runs again. In both cases, the number of lines the manager prints and the work actually done differ from one another. What is measured is how many times the diagnosis drawn from the restart command’s output is wrong.

From Firmware to the First Process

A machine booting is not a single event, it is a handoff chain. Each link loads the next one into memory, hands off control, and steps off the stage. None of the links can inspect the one before it; this is why an error at the chain’s start cannot be seen with the tools at its end.

The first link is firmware. It sits in persistent memory on the motherboard, runs when power is applied, counts and tests the hardware, then looks at boot devices in a predetermined order. It loads the code on the first suitable device it finds into memory and runs it. The firmware’s device order is a configuration value, and this value is not visible from inside the operating system.

The second link is the bootloader. Its job is to choose a kernel, read it from disk into memory, bring along an initial ramdisk, and prepare the parameters to be passed to the kernel. The bootloader itself can consist of more than one part; the first part is very small and only large enough to read the second part. The bootloader’s details are the subject of the next lesson.

The third link is the kernel. It is loaded into memory, unpacks itself, recognizes devices, and mounts the real root file system using the initial ramdisk brought along with it. The initial ramdisk is necessary because the drivers needed to mount the real root often sit inside that root itself; this intermediate layer is what solves the ordering problem. Once the root is mounted, the kernel does its last job and runs the first process. The first process is the lone process at the root of the process tree; it lives until the system’s end, and when it dies the system stops.

The fourth link is the first process, and on most systems this process is the service manager. The manager reads unit files, builds the graph, and enables the units needed to reach the default target. The handoff chain ends here, and the unit graph begins here.

The chain’s property relevant to diagnosis is that each link writes its own evidence somewhere else. The firmware’s findings stay on screen and land in no file. An error the bootloader reports is visible only at that moment. The kernel’s boot messages are written to a ring buffer in memory and cannot be saved anywhere until the root file system is mounted; if the buffer fills, the oldest messages are overwritten. Persistent logging only begins once the root is mounted and the logging unit is running. So a failure in the chain’s first three links leaves no trace in any file as long as the system does not boot. How logs are collected and rotated is the next topic’s question; the point here is only this: evidence existing requires it to have been recorded.

Targets and Layers

Booting is the unit graph being run once from the start. The manager takes the default target, finds the units it requires and wants, finds what those require, and unfolds the graph backward. The result is a layer structure: units with no edge between them sit in the same layer and can be started at the same time.

The layer count sets the lower bound on boot time. If six units started one after another, that would be a six-unit wait; split into layers, the wait is as many as the layer count. This was the answer, in the first lesson, to why the manager needs a graph.

Service managers offer an analysis command that prints how long booting took and which unit held things up by how much. This command’s output is often misread, and the source of the misreading is the graph again: the duration shown against a unit is not that unit’s own work time, it is the time between its start request being given and it being counted ready. Time spent waiting on required units is included in this duration too. The unit shown at the top of the ranking is often not the slowest unit but the unit in the deepest layer. An operator who tries to optimize a unit based on this output optimizes the wrong unit. Boot time’s real numbers depend on hardware and configuration and are not written here; what can be written is what the ranking is ranking.

SV32 — the default target wants all six of the six units. SV33 — a unit does not start until every unit it requires has started. SV34 — units in the same layer start at the same time and do not wait on one another. SV35 — ordering edges do not enter the layer calculation; only requiring edges do. SV36 — restarting a unit stops and restarts every unit that requires it. SV37 — the restart command prints a single line and names only the unit given. SV38 — a unit that fails at boot causes the units that require it to never start at all. SV39 — the boot summary gives the count of failed units only from those that were actually tried and failed; units never tried do not enter this count. SV40 — the graph does not change during boot.

The reason these assumptions are written before the measurement is that none of them show up in the output. The manager knows the layers and it knows the skipped units; neither appears in the lines it prints.

The Line the Command Prints

When the restart command succeeds it prints nothing; on failure it prints a single line of error. The command that asks about the whole system after boot also answers with a single word. The dumps below have not been run; they are written to show what the output carries:

$ systemctl restart ingest.service
$ systemctl is-system-running
degraded
$ systemctl --failed
  UNIT               LOAD    ACTIVE  SUB     DESCRIPTION
* ingest.service     loaded  failed  failed  Data ingestion unit

1 loaded units listed.

The first command returned silently, and this silence is read as “only what was asked for happened.” The second command gives a single word about the whole system, and that word does not say which unit is broken. The third command lists the failed units; its last line prints a count. None of the three outputs say that the four units depending on this one unit have also stopped. The dependent units are not “failed,” they are in “inactive” state, and they do not enter this list. The gap is a status-word gap, and it flips the diagnosis.

UNIT = {
    "ingest":  {"requires": [], "restart": "always"},
    "queue":   {"requires": ["ingest"], "restart": "always"},
    "process": {"requires": ["queue"], "restart": "on-failure"},
    "report":  {"requires": ["process"], "restart": "no"},
    "metrics": {"requires": ["ingest"], "restart": "always"},
    "backup":  {"requires": [], "restart": "no"},
}


def dependents_of(unit, units=None):
    units = UNIT if units is None else units
    direct = [a for a, v in units.items() if unit in v["requires"]]
    all_ = set(direct)
    for d in direct:
        all_ |= dependents_of(d, units)
    return all_


def restart_chain(unit):
    """How many more units restarting one unit stops and starts."""
    affected = dependents_of(unit)
    return {"unit": unit, "affected": len(affected),
            "affected_names": sorted(affected)}


def layer(unit):
    """Which wave the unit can start in during boot."""
    g = UNIT[unit]["requires"]
    return 0 if not g else 1 + max(layer(x) for x in g)


def boot(failed):
    """If a unit fails at boot, which units never start at all."""
    skipped = dependents_of(failed)
    started = [b for b in UNIT if b != failed and b not in skipped]
    return {"started": len(started), "skipped": len(skipped), "reported": 1}


print("unit       layer  tool output  oracle (stopped and started)  correct diagnosis")
wrong = 0
for b in UNIT:
    z = restart_chain(b)
    oracle = 1 + z["affected"]
    correct = oracle == 1
    wrong += 0 if correct else 1
    print(f"  {b:9s} {layer(b):5d} {'1 unit':>12s}  {oracle:29d}"
          f"  {'yes' if correct else 'NO'}")
print("  wrong diagnosis:", wrong, "/ 6 | units hidden:",
      sum(len(dependents_of(b)) for b in UNIT))
print("  ingest chain:", restart_chain("ingest"))
print()
print("failed at boot      started  never started  reported failure")
for b in UNIT:
    o = boot(b)
    print(f"  {b:18s} {o['started']:8d}  {o['skipped']:14d}  {o['reported']:15d}")

unit       layer  tool output  oracle (stopped and started)  correct diagnosis
  ingest        0       1 unit                              5  NO
  queue         1       1 unit                              3  NO
  process       2       1 unit                              2  NO
  report        3       1 unit                              1  yes
  metrics       1       1 unit                              1  yes
  backup        0       1 unit                              1  yes
  wrong diagnosis: 3 / 6 | units hidden: 7
  ingest chain: {'unit': 'ingest', 'affected': 4, 'affected_names': ['metrics', 'process', 'queue', 'report']}

failed at boot      started  never started  reported failure
  ingest                    1               4                1
  queue                     3               2                1
  process                   4               1                1
  report                    5               0                1
  metrics                   5               0                1
  backup                    5               0                1

Three Numbers

Oracle: restarting the ingest unit stops and starts 4 more units — process, queue, metrics, and report. The number of units seeing the interruption together with itself is 5. Across six units, a total of 7 units are affected in the shadows. Tool output: the command prints a single line per unit, and that line names only 1 unit. Wrong diagnosis: on 3 of six units, the “only this unit was affected” diagnosis is wrong; for ingest the count is off by a factor of five, for queue by three, for process by two.

This is where the common definition’s third reading sits: restarting a unit stops four more. And the chain does not show in the output. The direction of the wrong diagnosis is always the same, always undercounting. This direction corresponds to a specific operational error: for a short maintenance window, ingest is restarted, report generation is cut at the same time, the reason for the cut does not appear in the report’s own log, and the fault is searched for in the wrong place.

The layer column says a second thing. ingest and backup are in layer zero, queue and metrics in layer one, process in layer two, report in layer three. The graph’s depth is four. The number of units a restart affects and the layer depth are not the same measure: metrics is in layer one and yet affects no unit, because no one requires it. Depth and impact are two independent measures, and confusing them leads to treating units at the bottom of the graph as needlessly dangerous.

A Unit That Fails at Boot

The second table reads the same graph in the boot direction. If ingest fails to start at boot, only 1 unit starts, 4 units are never tried, and the boot summary reports 1 failed unit. The reported count is correct — one unit really did fail — but it does not describe the system’s state. The four units did not fail; they were never tried, and because they were never tried, no record was produced about them.

This is this lesson’s form of the course’s third claim: evidence vanishes on its own. If a unit had run and errored, a failure record would remain. A unit that never started writes nothing at all. An operator looking after boot has one failure notice in hand and four silent units; the silence looks the same as health.

The correct reading is not looking at the number in the boot summary, it is looking at the difference between the expected unit set and the started unit set. This difference is computed from the graph, and no single command prints it on its own. The table’s last three rows confirm this too: when report, metrics, or backup fails, no unit is skipped, and the reported number is correct for once. Same notice, same shape, different meaning.

Shutdown Is the Same Graph, Reverse Direction

If boot runs the graph forward, shutdown runs the same graph backward. The topmost units are stopped first, then what they require. If the order were not reversed, a unit writing data would try to shut down while the storage unit underneath it has already been torn down.

Shutdown has its own time limit, and this limit is a source of diagnosis too. The manager sends a stop request to a unit, waits a set time, and if the unit does not stop, sends a forcing signal. The distinction built in the process management topic holds here too: request and force are different things, and the window between them is the only time a unit has to write its data to disk. When the limit is kept short, shutdown speeds up and produces half-written files; when kept long, a single stuck unit holds up the whole machine’s shutdown for minutes. This is the shutdown-side counterpart of the staying-up-versus-visibility trade-off.

Rescue Targets and the Next Link

The default target is not the only option. There are two smaller targets for system administrators: one only mounts the root file system and gives a single shell, the other does even less than that and starts almost no units. There are two ways to reach these targets. While the system is running, the manager can be told to switch targets; if the system does not boot at all, the only way is to set the target from the start with a boot parameter.

The second path shows this lesson’s limit. Everything covered up to here was for after the first process comes up. If the first process never comes up, there is neither a unit list nor a status query; all that is left in hand is the line the bootloader offers for editing. The tools at the chain’s end cannot see the failure at the chain’s start.

Second Seed

This lesson’s numbers come not from a seeded generator but from a fixed unit graph. To see whether the measurement depends on the graph, the same computation was repeated with two seeds over twenty random six-unit graphs built with a seeded generator. Total hidden units come out to 124 with 20260218 and 132 with 20260219; wrong-diagnosis count comes out to 64 and 68 over 120 units. The order of magnitude holds, and the ratio in both seeds sits a little above half: on a random graph, restarting one unit affecting others is the rule, not the exception. Individual counts depend on the mock data; the command’s output not carrying the affected-unit count does not.

Random graphs’ ratio comes out higher than the fixed unit graph’s: the mock server has wrong diagnosis on three of six units, random graphs have it on sixty-four of a hundred twenty and sixty-eight of a hundred twenty. The difference comes from the mock server’s graph being a thin chain; as edge density rises, so does the number of affected units. On a real server the unit count is measured in the hundreds, and the graph there is far denser than what is drawn here. The measurement’s real-system counterpart is therefore expected to be worse, not better.

Summary

• Boot is a handoff chain: firmware, bootloader, kernel and initial ramdisk, then the first process. Each link loads the next and steps off; the next link cannot inspect the one before.
• The first process is the service manager on most systems; it unfolds the graph backward from the default target and starts units layer by layer.
• On the mock server, the graph’s depth is four layers; depth and impact are different measures.
• When ingest is restarted, 4 more units stop and start, the affected count together with itself is 5, the command output says 1 unit; the diagnosis is wrong on 3 of six units.
• If ingest fails at boot, 4 units are never tried and the summary reports only 1 failure; a unit never tried leaves no record.
• The correct reading looks not at the number in the summary but at the difference between the expected set and the started set.

Next Step

Three of the handoff chain’s four links were tied to the unit graph in this lesson. What remains is the second link: the bootloader. It decides which kernel loads, which initial ramdisk comes with it, and which parameters are passed to the kernel. The next lesson covers how to read bootloader configuration, how to test whether a parameter change is really effective, and why operations that break the configuration are irreversible. In the same lesson, how many of twenty-four parameter changes actually do their job is counted; the producing command’s answer is all twenty-four.