Service Status and Control

The previous lesson built the unit graph and counted how it does not show up in the unit list. The list carried one piece of information: the status word next to each line. Six of six lines read “active,” and this was a reassuring sight for whoever was looking at the list. This lesson measures what that word says and what it does not.

The short answer is this: the status word says the manager currently sees the unit’s process standing. This is true and misleading. It is true because the manager really is seeing a process. It is misleading because the process it sees may have been born ten seconds ago, and the corpse of the one before it may already have been cleared away. What is measured is how many times the healthy diagnosis drawn from the word “active” is wrong.

Enabling and Starting Are Separate Axes

The two most commonly confused actions of a service manager are starting and enabling, and the difference between them is an axis difference. Starting changes the present: the unit’s process is brought up immediately, and no trace of this decision remains after the system reboots. Enabling changes the future: the unit is linked under a target and started every time that target is reached, but enabling itself starts no process. Stopping and disabling split the same way.

The two axes produce four combinations, and all four are seen in practice. A unit can be enabled and running: the expected state. It can be enabled but not running: it will start at boot, it is stopped right now. It can be not enabled but running: started by hand, it will not come back after a reboot. It can be not enabled and not running: off. The third combination is the most expensive one, because a server that looks fine today loses that unit at the first reboot, and the cause of the loss is a decision made months earlier.

The command that shows both axes on one line prints the unit’s status in several separate fields. The dump below has not been run; it is written to show what the fields say:

$ systemctl status ingest.service
* ingest.service - Data ingestion unit
     Loaded: loaded (/etc/systemd/system/ingest.service; enabled)
     Active: active (running)
   Main PID: <process number>
      Tasks: 3

The Loaded field says the unit file was found and read, and the word in parentheses carries the enabling axis. The Active field says the current state; the second word in parentheses is a finer sub-state and comes from the service’s own life cycle. These two fields are independent of one another. The value of one does not determine the other, though sitting side by side on the same output line suggests they determine each other.

How enabling works explains this independence. What the manager does when a unit is enabled is link the unit file into a target’s “wanted by” directory. No new process exists, nothing running changes; only a link forms on disk. This is why the enable command does not stop a running unit, and the disable command does not kill it either. An operator wanting to do both at once has to give two separate commands, and forgetting one of the two is the most common way the third combination gets produced.

The list of control actions does not end with these two. Restarting stops and starts a unit, and its process identity changes. Reloading asks the process to reread its configuration without being killed; whether this is possible depends on the unit supporting it, and on a unit that does not, the command can silently turn into a restart or do nothing at all. Masking is the harshest of all: the unit’s name is linked to an empty definition, and the unit can no longer be started either by hand or through a dependency. Another unit that tries to start a masked unit fails, and the error message names not the mask but “unit could not be started.” A forgotten mask leaves behind a fault that is understood only weeks later.

The Command Returning Is Not the Same as Being Ready

The sentence built for the termination signal in the process management topic holds here too: the command returning does not mean the job is done. What the manager waits for once a start command is given depends on the unit’s type. In the plainest unit type, the manager runs the program, and the moment the run call returns successfully, it counts the unit as “started.” At that moment the process may not yet have read its configuration, opened its listening socket, or established its database connection. The command returns successfully, the status becomes “active,” and the service still cannot answer requests for a few more seconds.

The way to close this gap is for the unit to report its own readiness to the manager; in unit types that support notification, the manager waits for the notification before saying “started,” and the next unit in the dependency chain is not started until then either. Without the notification, the chain moves too early: queue is started while ingest is not yet ready, and fails to connect on its first attempt. Both units show “active” in the status table. This gap is not measured in this lesson; what is measured is the invisibility of the failure itself. The two share the same root: all the manager knows is that the process exists.

Measurement: A Unit in a Failure Loop

All six of the mock server’s units are put into a failing state: each crashes at regular intervals. The only thing that differs is the units’ restart policy. The policy for ingest, queue, and metrics is “always”; for process it is “on-failure”; for report and backup it is “no.” The observation window is 600 seconds, the crash interval is 10 seconds.

SV11 — the observation window is 600 seconds and every duration in the model is in this unit. SV12 — all six of the six units are failing in this window; the oracle knows this. SV13 — restarting is instant and never fails. SV14 — a unit with the “no” policy stops after its first crash and its state becomes “failed.” SV15 — a restarted unit’s state stays “active.” SV16 — the diagnosis drawn from the status word takes only two values: healthy if “active” is seen, failed if “failed” is seen. SV17 — a unit is down for 2 seconds on every crash, and the crash moment within each period is chosen by a seeded generator. SV18 — polling is instantaneous and sees the state exactly at the polled second. SV19 — polling has no cost; it does not slow the system down or change what it measures. SV20 — a status query has no lag; it returns the real state at the asked second, the tool itself is never wrong.

The last two assumptions deliberately favor the tool. On a real system, polling has its own cost, and a status query can return from a previous scan’s cache. The model zeroes these out; so the wrong-diagnosis counts below are the lower bound the tool produces at its best.

SEED = 20260218
PERIOD = 600
UNIT = {
    "ingest":  {"requires": [], "restart": "always"},
    "queue":   {"requires": ["ingest"], "restart": "always"},
    "process": {"requires": ["queue"], "restart": "on-failure"},
    "report":  {"requires": ["process"], "restart": "no"},
    "metrics": {"requires": ["ingest"], "restart": "always"},
    "backup":  {"requires": [], "restart": "no"},
}


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


def failure_cycle(unit, duration=PERIOD, interval=10, units=None):
    """The 'always' policy hides the failure: the service always shows 'active'."""
    units = UNIT if units is None else units
    p = units[unit]["restart"]
    if p == "no":
        return {"policy": p, "restarts": 0, "seen_state": "failed",
                "hidden_time": 0}
    if p == "on-failure":
        return {"policy": p, "restarts": duration // interval,
                "seen_state": "active", "hidden_time": duration}
    return {"policy": p, "restarts": duration // interval,
            "seen_state": "active", "hidden_time": duration}


def dead_intervals(seed=SEED, duration=PERIOD, period=10, dead=2):
    """The unit crashes once every period and is down for `dead` seconds."""
    r = generator(seed)
    return [(t + r(period - dead + 1), dead) for t in range(0, duration, period)]


def poll(intervals, step, duration=PERIOD):
    """An operator asking for status once every `step` seconds."""
    looks = seen = 0
    for t in range(0, duration, step):
        looks += 1
        if any(b <= t < b + u for b, u in intervals):
            seen += 1
    return {"looks": looks, "seen": seen, "wrong": looks - seen}


print("unit       policy      restarts  seen state  hidden time  diagnosis")
wrong_diagnosis = 0
for b in UNIT:
    a = failure_cycle(b)
    correct = a["seen_state"] == "failed"
    wrong_diagnosis += 0 if correct else 1
    print(f"  {b:9s} {a['policy']:10s} {a['restarts']:8d}"
          f"  {a['seen_state']:10s}  {a['hidden_time']:11d}"
          f"  {'correct' if correct else 'WRONG'}")
print("  oracle: 6 of 6 units failed | wrong diagnosis:", wrong_diagnosis)
print()
A = dead_intervals()
print("real crashes:", len(A), "| total dead seconds:", sum(u for _, u in A))
print("poll interval  looks  saw failed  wrong diagnosis")
for step in (1, 5, 15, 30, 60):
    y = poll(A, step)
    print(f"  {step:13d}  {y['looks']:5d}  {y['seen']:11d}  {y['wrong']:15d}")

unit       policy      restarts  seen state  hidden time  diagnosis
  ingest    always           60  active              600  WRONG
  queue     always           60  active              600  WRONG
  process   on-failure       60  active              600  WRONG
  report    no                0  failed                0  correct
  metrics   always           60  active              600  WRONG
  backup    no                0  failed                0  correct
  oracle: 6 of 6 units failed | wrong diagnosis: 4

real crashes: 60 | total dead seconds: 120
poll interval  looks  saw failed  wrong diagnosis
              1    600          120              480
              5    120           21               99
             15     40            7               33
             30     20            2               18
             60     10            0               10

Three Numbers

Oracle: all six units are failing; a unit with the “always” policy restarts 60 times in 600 seconds and is down a total of 120 seconds. Tool output: four units read “active,” two read “failed”; even at the tightest polling, the failed state is seen in 120 looks, at the loosest, in 0. Wrong diagnosis: the health diagnosis drawn from the status word is wrong on 4 of six units; an operator polling once a second gets it wrong 480 times, one polling once every sixty seconds, 10 times.

This table holds the common definition’s fourth reading: the restart policy hides the failure. A unit with the “always” policy dies and comes back sixty times and its state never changes; the failure stays invisible for 600 seconds. A unit with the “no” policy, in contrast, becomes “failed” on its first crash and gives a correct diagnosis. The result should be read the other way around: a unit configured to be more resilient produces less visible failure, because here resilience is bought at the price of visibility.

The table’s most uncomfortable row is process’s row. Its policy is not “always” but “on-failure” — supposedly a more measured choice. The result is the same: 60 restarts, “active” state, 600 seconds of invisibility. The difference between the two policies never shows up in this run, because the unit always exits with an error. Where the difference does show up, and which failure type disappears under which policy, is what the next lesson measures.

Looking More Often Is Not Enough

The polling sweep pays the second claim. An operator polling once a second makes 600 looks and catches the failed state in 120 of them; one polling every five seconds catches 21 of 120 looks, every fifteen seconds 7 of 40, every thirty seconds 2 of 20, every sixty seconds 0 of 10. As the look count grows sixty-fold, the number of caught failures grows too — but its ratio falls. The caught share is one in five at one-second polling, zero at sixty-second polling.

The reason the number behaves this way is that the failure sits in a narrow window. The unit crashes once every ten seconds and is down for only 2 seconds; for the remaining 8 seconds the manager really does see a running process. Polling has to land inside that 2-second window. An operator who polls rarely does not just “see less” — in most runs they see nothing at all, and every piece of evidence they hold says “active.” Looking late sees little.

The conclusion this leads to is not shortening the polling interval. Even polling once a second leaves 480 wrong diagnoses. However many times it is asked, the status word does not carry the restart count; the field that carries it is elsewhere. How many times a unit has restarted and when it last started sit in the manager’s own separate fields, and that is the right question: not “is it active,” but “how long has it been active.” An uptime shorter than sixty seconds is the only visible trace of a unit that has restarted sixty times.

The same command’s full output already prints these fields; the problem is not that the fields are missing, it is that the eye fixes on the first line. The dump below has not been run; it is written to show where the evidence sits in the output of a unit stuck in a failure loop:

* queue.service - Queue unit
     Loaded: loaded (/etc/systemd/system/queue.service; enabled)
     Active: active (running) since <a moment ago>
   Main PID: <process number>
     Status: "ready"
   <timestamp> host systemd[1]: queue.service: Main process exited
   <timestamp> host systemd[1]: queue.service: Scheduled restart job

The first line says “active,” the third line says how long that run has lasted, and the last two lines say the process exited and a restart was scheduled. All three are in the same output, all three are correct. The wrong diagnosis comes not from missing output but from which line gets read. Reading a unit’s health by uptime instead of the word “active” eliminates every one of this lesson’s 480 wrong diagnoses; a unit that has restarted sixty times never has an uptime longer than ten seconds at any look.

This is the reverse face of the course’s second claim, and it fits in two sentences. More output does not fix the diagnosis, because line count is not what distinguishes it. Reading fewer but the right field fixes the diagnosis, because the metric itself is what distinguishes it.

Second Seed

Because crash moments are seeded, the table could depend on the seed. When the measurement is repeated with a second seed (20260219), the caught-failure counts come out as follows: 120 in both seeds at one-second polling; 21 in both at five seconds; 7 and 8 at fifteen seconds; 2 and 3 at thirty seconds; 0 and 3 at sixty seconds. Frequent polling agrees across both seeds; sparse polling is mock-dependent and should be written up that way. Sixty-second polling coming out at zero is specific to this run, and coming out at three is too. What is seed-independent is the direction of the ratio: as polling gets sparser, the caught share falls and never exceeds 120 in any seed.

Summary

• Enabling changes the future, starting changes the present; the two axes produce four combinations, and “not enabled but running” is lost at the first reboot.
• The loaded, enabled, and active fields in the status output are independent of one another; sitting side by side suggests they determine each other.
• While all six units in the failure loop are failing, 4 show “active”: the restart policy hides the failure.
• A unit with the “always” policy restarts 60 times in 600 seconds, is down a total of 120 seconds, and the failure stays invisible for 600 seconds.
• As polling gets more frequent, caught failures rise but the caught ratio falls: 120/600 at 1 second, 0/10 at 60 seconds. Even one-second polling leaves 480 wrong diagnoses.
• The right question is not “is it active” but “how long has it been active”; the restart count sits not in the status word but in separate fields.

Next Step

This lesson counted what the policy hides, without asking who set the policy. The policy is one line in the unit file, and whoever wrote that line usually thought “keep the service up no matter what.” The next lesson rewrites the unit file from scratch and counts, across nine combinations, which failure type each of the three policies — “no,” “on-failure,” “always” — hides. In the same lesson, the mechanism that breaks the concealment, the start limit, is measured: the table comes out showing how far the 600-second invisible time can be brought down.