Bootloaders

The previous lesson built the handoff chain’s four links and said the failure at the chain’s start cannot be seen with the tools at its end. This lesson returns to the second link, the bootloader. It decides which kernel loads, which initial ramdisk comes with it, and which parameters are passed to the kernel.

Changes here have one property that shapes the whole topic: their effect is seen only at the next boot. The command that writes the configuration returns successfully, the system keeps running, nothing looks changed. Whether the change was correct is understood hours or weeks later, when the machine reboots. What is measured is how many times the “change is effective” diagnosis drawn from the “configuration written” output is wrong.

What a Boot Entry Consists Of

The bootloader keeps a list of entries to choose from. Each entry has four parts: a name, a kernel image, the initial ramdisk to be loaded alongside that kernel, and the parameter string passed to the kernel. Having more than one entry on the same machine is normal; when a new kernel is installed, the old one is not deleted, a new entry is added to the list. This is the cheapest way back: if the new kernel does not boot, the old entry is chosen from the menu.

Firmware comes in two styles, and they determine where entries sit. In one style, the bootloader’s first part is written to a single sector at the very start of the disk; the rest, which does not fit in that sector, sits elsewhere on disk and the first part finds it by address. In the other style, the firmware can read a file system itself; entries sit as plain files in a separate boot partition, and the choice can be made from the firmware’s own menu. The two styles can coexist on the same machine, and that raises the question of which configuration is actually being read.

The dump below shows what an entry looks like. It has not been run; it is written to show the layout of the parts, and the file format varies by bootloader:

title    Mock server
linux    /vmlinuz-mock
initrd   /initrd-mock.img
options  root=UUID=<root-device-id> ro quiet

The linux line gives the kernel image, the initrd line the initial ramdisk, the options line the parameter string. The root device here is written not by device name but by a persistent identity; device names can change from boot to boot, identity does not. An entry relying on a device name can stop booting once a second disk is attached to the machine, and the reason will not appear in the configuration.

What Parameters Do

The parameter string is passed to the kernel, and the kernel does not use parameters it does not recognize itself — it passes them on to the first process. This behavior produces the topic’s most important trap: a misspelled parameter gives no error. The kernel does not recognize it, the first process does not recognize it either, and both silently move on. The string shows up in the running system’s kernel command line, and because it shows up, it is assumed “applied.”

Parameters’ functions fall into a few groups. Some say the root device and the mount mode. Some set how much of the boot messages get printed to the screen. Some disable a hardware driver. Some tell the first process which target to go to; this is the way to reach the rescue targets named in the previous lesson. The same thing can be said with two different parameters, and when they conflict, which one wins depends on order — this too is a source of silent wrong behavior.

Which parameters a running system booted with is read from a file the kernel offers. The dump below has not been run; it is written to show what the read returns:

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-mock root=UUID=<root-device-id> ro quiet unused=value

The unused=value at the end of the line is a made-up parameter; the kernel does not recognize it, the first process does not recognize it, neither complains, and the string just sits there as is. An operator using this line as verification counts the made-up parameter as applied too. The command-line reading method in the measurement is exactly this reading, and the reason it goes wrong is exactly this: the file shows the change was delivered, not that it did its job.

Menu and Timeout

The bootloader’s fallback value also depends on it showing its menu, as much as on the entry list it holds. The menu comes with a timeout: if no key is pressed within that time, the default entry boots. When the timeout is set to zero, the menu never shows and boot shortens by a few seconds. What that saved handful of seconds costs is the closing off of the one fix path left at the moment of failure: if the menu is not shown, the old entry cannot be chosen, no one-shot edit can be made either.

This trade-off sharpens once more on a remote server. On a machine with no console access, the menu is unusable anyway; the way back there is not the menu, it is an entry marked to be valid for just the next boot. Such an entry is tried once: if the boot succeeds it is made permanent, if it fails the machine reverts to the old entry on its own. This mechanism is the real counterpart of the “one-shot” row in the measurement.

The Irreversible Line

Some operations in this lesson are irreversible, and their commands are not written here in runnable, complete form. The line drawn is this: if an operation’s result cannot be fixed without the system booting again, the command is not written in full.

Three operations sit inside this line. First, writing the bootloader’s first part to the disk’s starting sector: given the wrong device name, that device’s partition table area is overwritten and every partition on the disk loses its address. Second, regenerating bootloader configuration with a generator: the generator writes over the existing file, changes made by hand are lost, and the old file does not come back. Third, deleting a kernel entry assumed unused: if the deleted entry was the only working one, the way back disappears.

What the three share is that the fault is not visible at that moment. The command returns successfully, the system keeps running, the failure surfaces at the next boot. At that moment there is no status query, no unit list, no log; there is only an error message sitting on the screen.

The safe way to test these operations has three parts. A copy of the configuration file is taken and kept somewhere it can be restored from on the same machine. The change is tried one-shot in the bootloader menu before being permanently written; an edit made in the menu is not written to disk, it is valid only for that boot. A new entry is added, the old entry is not deleted, and the new entry is made permanent only after it has booted once. Taking a snapshot on a virtual machine and testing there is a fourth path that satisfies all three at once.

Measurement: Twenty-Four Changes

Twenty-four changes are made to the bootloader configuration. Each change’s real outcome is one of four types: valid (does the intended thing), typo (not recognized, silently ignored), conflicting (recognized but clashes with another parameter and produces different behavior), and unbootable (the system never boots again). Three verification methods are compared: looking at the generator’s output, reading the kernel command line after boot, and actually testing the behavior the parameter is supposed to change.

SV41 — the generator command writes all twenty-four changes successfully and gives no warning on any of them. SV42 — the change’s type is chosen by a seeded generator. SV43 — a mistyped parameter shows up on the kernel command line but has no effect at all. SV44 — a conflicting parameter also shows up and produces an effect other than the intended one. SV45 — a change that leads to an unbootable state never shows up on the command line at all, because the system never boots. SV46 — command-line reading looks only at whether the string is present. SV47 — the behavior test measures the parameter’s real effect and is never wrong. SV48 — an unbootable boot writes no line at all to the system log. SV49 — in persistent writing, every attempt that fails to boot requires recovery with external media. SV50 — in a one-shot attempt, a failed boot requires no recovery; a reboot returns to the old entry.

SEED = 20260218
TYPE = ("valid", "typo", "conflicting", "unbootable")


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


def edits(seed=SEED, count=24):
    """Changes made to the bootloader configuration and their REAL outcomes."""
    r = generator(seed)
    items = []
    for i in range(count):
        a = r(100)
        kind = (TYPE[0] if a < 50 else TYPE[1] if a < 70
                else TYPE[2] if a < 88 else TYPE[3])
        items.append({"no": i + 1, "kind": kind, "boots": kind != "unbootable",
                      "effective": kind == "valid",
                      "visible_on_line": kind != "unbootable"})
    return items


def assess(d, method):
    """The 'is the change effective' answer of three verification methods."""
    if method == "generator output":
        return True
    if method == "command line reading":
        return d["visible_on_line"]
    return d["effective"]


def recovery(items, persistent):
    """In persistent writes, every unbootable attempt needs external-media recovery."""
    unbootable = [d for d in items if not d["boots"]]
    return {"unbootable": len(unbootable),
            "recovery": len(unbootable) if persistent else 0, "log_lines": 0}


D = edits()
print("oracle:", len(D), "edits |",
      {t: sum(1 for d in D if d["kind"] == t) for t in TYPE})
print()
print("verification method    says effective  really effective  wrong diagnosis")
for y in ("generator output", "command line reading", "behavior test"):
    said = sum(1 for d in D if assess(d, y))
    wt = sum(1 for d in D if assess(d, y) != d["effective"])
    print(f"  {y:22s} {said:10d}  {sum(1 for d in D if d['effective']):15d}  {wt:11d}")
print()
print("write mode    unbootable  recovery events  remaining log lines")
for persistent, name in ((True, "persistent"), (False, "one-shot")):
    k = recovery(D, persistent)
    print(f"  {name:12s} {k['unbootable']:9d}  {k['recovery']:14d}"
          f"  {k['log_lines']:25d}")

oracle: 24 edits | {'valid': 11, 'typo': 8, 'conflicting': 2, 'unbootable': 3}

verification method    says effective  really effective  wrong diagnosis
  generator output               24               11           13
  command line reading           21               11           10
  behavior test                  11               11            0

write mode    unbootable  recovery events  remaining log lines
  persistent           3               3                          0
  one-shot             3               0                          0

Three Numbers

Oracle: 11 of the twenty-four changes actually do their job; 8 are typos, 2 conflict, 3 leave the system unbootable. Tool output: the generator command reports all 24 changes as written; reading the kernel command line shows 21 changes as “applied”; the behavior test says 11. Wrong diagnosis: trusting the generator’s output is wrong 13 times, trusting command-line reading 10 times; the behavior test’s wrong diagnosis is 0.

The middle row is this lesson’s form of the course’s second claim. Command-line reading is more detailed than the generator’s output: it really looks at the system, really sees the string, really produces twenty-one lines. Its wrong-diagnosis count drops by only a third, and all of that drop comes from the three unbootable attempts dropping out on their own. The eight typos and two conflicts are still counted as effective by this method, because both show up on the command line. Only the third method closes the gap between showing up and doing the job.

It is true that the third method is expensive, and that is why it gets skipped. Really testing that a parameter does its job requires measuring, one by one, the behavior that parameter is supposed to change: seeing that a disabled driver’s device really does not show up, that message volume really dropped once the level was lowered, that changing the target really went to that target. Twenty-four changes mean twenty-four separate tests. Against that, reading the generator’s output is one line. What the measurement says is that the cheap method leaves thirteen wrong diagnoses, and ten of those thirteen would never be noticed without a behavior test; the remaining three are noticed in the harshest way possible, at the next boot. Output whose diagnosis is not tested counts as unmeasured.

This distribution also says how the work order should be set. The three changes that lead to an unbootable state are the most expensive, and what catches them is not the behavior test, it is the one-shot attempt: a machine that boots once and reverts exposes all three for free. The ten changes that quietly do nothing are caught only by the behavior test, and this test can be done after boot, while the system is up. The two methods do not substitute for one another; one tests the boot, the other tests the effect. An operator who does only one has done half the measurement.

Where the Evidence Ends Up

The second table pays the course’s third claim. In persistent writing, three attempts leave the system unbootable, and all three require recovery with external media. In one-shot attempts, the same three attempts still fail to boot, but recovery events are 0: a reboot returns to the old entry.

The two rows’ shared column is even more striking. In both cases, remaining log lines are 0. An unbootable boot writes nothing to the system log, because the unit that keeps the log has not run yet, and in most configurations the file system the log would be written to is not even mounted yet. Evidence stays only on the screen; evidence left on the screen is erased at the next boot. Looking late sees little; here, looking late sees nothing at all.

The right place for a bootloader test is therefore somewhere evidence can be recorded: a virtual machine whose console is recorded, a remotely accessed console, or a machine with someone standing in front of it. A test that produces no evidence counts as unmeasured, whatever its outcome.

The recovery column alone is enough to tell the difference. In persistent writing, each of the three recovery events means physical or console-level access to the machine, booting from external media, and a manual fix; on a remote server this means a person going to where the machine is. In one-shot attempts, the same three failures cost no more than a single reboot. The change’s content is the same in both, the oracle is the same in both; the only thing that differs is how the attempt was written. The cost of a wrong diagnosis depends less on the diagnosis itself than on the method that produced it.

Second Seed

Because change types are seeded, the distribution depends on the seed. With the second seed (20260219), twenty-four changes come out 12 valid, 5 typos, 5 conflicting, and 2 unbootable. Wrong-diagnosis counts come out at 13 and 12 for generator output, 10 and 10 for command-line reading, 0 and 0 for the behavior test. The order of magnitude holds and the ranking of methods is the same in both seeds. The count of unbootable attempts is 3 and 2; because this number is small, it is mock-dependent and should not be generalized from a single run. What can be generalized is that an unbootable attempt’s log line count is zero in every run.

Summary

• A boot entry has four parts: name, kernel image, initial ramdisk, and parameter string. A new kernel does not delete the old entry; the old entry in the menu is the cheapest way back.
• The root device is written with a persistent identity; device names can change from boot to boot.
• A misspelled parameter gives no error, shows up on the kernel command line, and is assumed applied because it shows up.
• 11 of twenty-four changes do their job; trusting generator output produces 13 wrong diagnoses, command-line reading 10, the behavior test 0. More detailed reading fixes the diagnosis by only a third.
• An unbootable boot leaves 0 lines in the system log; evidence stays on screen and is erased at the next boot.
• Persistent writing produces 3 recovery events on three unbootable attempts, one-shot testing produces 0; the change is tried in the menu before being permanently written, and the old entry is not deleted.

Next Step

This topic built the unit model, counted what the status word hides, measured what the policy conceals, worked out the boot chain, and showed how to test changes at the chain’s start. The five lessons’ shared conclusion fits in one sentence: the line the service manager prints is a projection of the system’s state, and a projection always carries loss. The last lesson showed loss in its harshest form: evidence never being written at all. The next topic starts exactly here and examines the evidence that does get written — how the system log and service logs are collected, how they are queried, and how they vanish on their own through rotation.