Defining a New Service

The previous lesson counted that the restart policy hides failure: four of six units showed “active” while all six were failing. One row of the table was left open. The process unit’s policy was “on-failure” — a supposedly more measured choice than “always” — but the result was the same: sixty restarts, six hundred seconds of invisibility. The difference between the two policies never showed up in that run.

This lesson brings the difference out. First it looks at where the policy is written, the unit file; then it multiplies three policies by three exit types and counts which failure disappears in which of the nine combinations. Finally, the mechanism that breaks the concealment is measured: the start limit.

The Unit File’s Sections

A unit file is plain text divided into sections, and a service unit has three of them. The first section places the unit in the graph: its description, what it requires, what it starts after are written here. The second section says how the unit is run: which command, which user, which working directory, which environment variables, unit type, and restart policy. The third section says what enabling means: which target this unit links under. A unit file that skips the third section can be built and started by hand, but cannot be enabled; this is one way to produce the previous lesson’s “not enabled but running” combination.

The file below defines the mock server’s ingest unit. It has not been run; it is written to show the fields’ layout, and field names vary by service manager:

[Unit]
Description=Data ingestion unit
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/ingest --config /etc/ingest.conf
User=app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Two fields are directly relevant to this lesson. The Type field determines when the manager counts the unit as “started”; the notification-waiting type closes the readiness gap covered in the previous lesson. The Restart field is the restart policy, and it is what this lesson measures. RestartSec gives the wait before restarting; a value close to zero can revive a crashing unit several times a second and keep the processor busy.

If the manager is not told to reread after the file is written, nothing changes; the in-memory copy of the graph stays old. This is the practical counterpart of the point established in lesson one. A syntax error in the file also does not show up at write time but at reread or start time; the manager skips the bad line and loads the unit with an incomplete definition. After writing, seeing what got loaded through the command that prints the unit’s merged definition is more informative than reading the file one more time.

Unit Type Determines When “Started” Is Said

The Type field looks small and affects the entire chain. In the plainest type, the manager runs the command, and the instant the run returns successfully, it counts the unit as started. In the forking type, the process backgrounds itself, the first process exits, and the manager finds the real process from a PID file or the process tree; this type is the most exposed to the manager and the program lying to each other. In the one-shot type, the unit does a job and exits; exiting is not a failure, and this type is the legitimate owner of the “exited” state seen in the previous lesson. In the notifying type, the program explicitly tells the manager when it is ready, and the chain advances only then.

Choosing the wrong type produces wrong behavior with no error message at all. Defining a program that backgrounds itself with the plainest type lets the manager watch the short-lived process that existed before backgrounding and count the unit as crashed once that process exits. If the restart policy is “always,” the result is exactly the loop measured in the previous lesson: the unit restarts once a second, the state always reads “active.”

A Command That Works by Hand Does Not Work as a Unit

The most common fault unit-file writers run into is a command working in the shell but not as a unit. The reason fits in one sentence: the manager is not a shell. The configuration files read at login are not read, the search path differs from the operator’s, the working directory is the root directory if not specified, environment variables are empty, and if no user is given, the process runs with the same privilege as the manager. There is no shell expansion either: wildcards, quotes, and pipes do not work as expected, because the command line is not handed to a shell.

The consequence is that the fault is looked for in the wrong place. The program exits because it cannot find a file, the error message says “no such file,” and the operator searches for the file; the file is right where it should be, what is missing is the working directory. The right habit is writing paths as absolute in the unit file, stating the working directory and user explicitly, and having required environment variables read from a separate file. Where a unit’s output goes and how it is found there is not this topic’s question, it is the logs topic’s.

Three Policies, Three Exit Types

There is more than one way for a process to stop running. Clean exit: the process ends on its own with a success exit code. Error exit: it ends with a nonzero code. Termination by signal: it is killed by an external signal or by kernel intervention. Restart policies split these three types, and where they split differs from one another.

Policy “no” never restarts, under any circumstance. Policy “on-failure” restarts on error exit and termination by signal, not on clean exit. Policy “always” restarts under all three. The difference only shows up when a clean exit occurs, and that is exactly where it is dangerous: a service expected to be long-lived exiting with code zero is a failure, but the “on-failure” policy does not count it as one.

SV21 — the unit is designed to be long-lived; every exit, whatever its code, is a failure. SV22 — the unit is genuinely failing in all nine of the nine combinations; the oracle knows this. SV23 — the crash interval is seeded between 8 and 12 seconds. SV24 — restarting is instant and never fails. SV25 — a unit not restarted on clean exit has state “exited,” not failed. SV26 — a unit not restarted on error exit or termination by signal has state “failed.” SV27 — a restarted unit’s state is “active.” SV28 — the operator’s diagnosis is drawn from three status words: healthy if “active,” exited normally if “exited,” failed if “failed.”

SEED = 20260218
PERIOD = 600
POLICY = ("no", "on-failure", "always")
EXIT = ("clean", "error", "signal")


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


def will_restart(policy, exit_):
    if policy == "always":
        return True
    if policy == "on-failure":
        return exit_ in ("error", "signal")
    return False


def crash_moments(seed=SEED, duration=PERIOD, at_least=8, at_most=12):
    """The unit crashes at an irregular interval; the gap is seeded."""
    r = generator(seed)
    t, items = 0, []
    while True:
        t += at_least + r(at_most - at_least + 1)
        if t >= duration:
            return items
        items.append(t)


def run(policy, exit_, moments, limit=0, window=100, duration=PERIOD):
    """The manager gives up if `limit` restarts are exceeded within `window` seconds."""
    if not will_restart(policy, exit_):
        return {"restarts": 0, "state": "exited" if exit_ == "clean" else "failed",
                "hidden": duration if exit_ == "clean" else 0}
    done = []
    for t in moments:
        if limit and sum(1 for x in done if x >= t - window) >= limit:
            return {"restarts": len(done), "state": "failed", "hidden": t}
        done.append(t)
    return {"restarts": len(done), "state": "active", "hidden": duration}


A = crash_moments()
print("oracle: unit crashes", len(A), "times in", PERIOD, "seconds")
print()
print("policy      exit    restarts  seen state  operator's diagnosis  correct")
wrong = 0
for p in POLICY:
    for c in EXIT:
        k = run(p, c, A)
        diagnosis = {"active": "healthy", "exited": "exited normally",
                     "failed": "failed"}[k["state"]]
        correct = diagnosis == "failed"
        wrong += 0 if correct else 1
        print(f"  {p:10s} {c:7s} {k['restarts']:7d}  {k['state']:13s}"
              f"  {diagnosis:17s}  {'yes' if correct else 'NO'}")
print("  wrong diagnosis:", wrong, "/ 9")
print()
print("start limit  restarts  seen state  hidden time  wrong diagnosis")
for limit in (0, 20, 10, 5, 3):
    k = run("always", "error", A, limit=limit)
    wt = len([t for t in range(0, PERIOD, 10) if t < k["hidden"]])
    print(f"  {'none' if limit == 0 else str(limit):11s} {k['restarts']:7d}"
          f"  {k['state']:13s}  {k['hidden']:13d}  {wt:11d}")

oracle: unit crashes 60 times in 600 seconds

policy      exit    restarts  seen state  operator's diagnosis  correct
  no         clean         0  exited         exited normally    NO
  no         error         0  failed         failed             yes
  no         signal        0  failed         failed             yes
  on-failure clean         0  exited         exited normally    NO
  on-failure error        60  active         healthy            NO
  on-failure signal       60  active         healthy            NO
  always     clean        60  active         healthy            NO
  always     error        60  active         healthy            NO
  always     signal       60  active         healthy            NO
  wrong diagnosis: 7 / 9

start limit  restarts  seen state  hidden time  wrong diagnosis
  none             60  active                   600           60
  20               60  active                   600           60
  10               10  failed                   106           11
  5                 5  failed                    62            7
  3                 3  failed                    41            5

What Each Policy Hides

7 of nine combinations give the operator a wrong diagnosis. What gets hidden sits in different places depending on the policy.

Policy “no” hides only 1 of the three exit types: clean exit. On error exit and termination by signal, the unit shows “failed” and the diagnosis comes out right. It is the policy that hides the least, because it offers no resilience at all. Policy “on-failure” hides all 3 of the three types, but in two different forms: on error and signal it revives the unit and shows “active”; on clean exit it does not revive it and shows “exited.” The second form is sneakier, because “exited” is a normal result for one-shot jobs and goes unnoticed. Policy “always” hides all 3 types the same way: all of them “active.”

The result ranks in reverse. As resilience rises, visibility falls, and no setting balances the two on its own. “Always” is the most direct way to keep a server up, and at the same time the option that best hides failure.

The table’s second reading says policy choice cannot be made without knowing the exit-type contract. The “on-failure” policy relies on the assumption that the program reports failure with a nonzero code. This assumption does not hold for every program: one that cannot read its configuration can print an error and still exit with code zero, an error in the shutdown path can be swallowed, a wrapper script can return its own exit code and hide the real program’s. The exit-code contract was established in the M03/K02 Shell Programming course; here, the cost of breaking that contract shows up as “exited normally” in two of nine rows.

The selection rule that follows has three parts. For a long-lived service, “always” is chosen and a ceiling is always placed next to it. For a one-shot job, “no” is chosen; that unit exiting really is normal. “On-failure” is chosen only once the program’s exit-code contract has been verified, and verifying it requires either reading the code or deliberately breaking the program and reading its exit code.

The Start Limit Breaks the Concealment

Service managers carry a second mechanism to close this trap: the start limit. If a unit restarts more times than a set count within a set window, the manager gives up trying and marks the unit failed. The limit is not resilience being turned off, it is resilience having a ceiling.

The sweep table shows the ceiling’s effect. With no limit in a hundred-second window, the unit restarts 60 times over 600 seconds and never shows failed: an operator polling every ten seconds gets a wrong diagnosis 60 times. With the limit at 20, the table does not change, because the crash rate in the mock data never reaches twenty within a hundred-second window: a limit that is set does the same as no limit. At 10, the unit is given up on at the 106th second and wrong diagnosis drops to 11. At 5, hidden time is 62 seconds, wrong diagnosis 7; at 3, 41 seconds and 5.

The 20 row carries a warning of its own. A limit that is in the configuration, that shows in the output, that is written with correct syntax, does nothing here at all. Whether a setting is effective cannot be read from the configuration unless it is compared against the oracle; a setting existing and a setting working are not the same thing. Whether the limit is effective depends on the crash rate exceeding the limit within the window, and the crash rate does not appear in the configuration.

The gain in between is large: hidden time drops from 600 seconds to 41, wrong diagnosis from 60 to 5. The cost is just as clear: a unit that hits the limit no longer comes back on its own. At limit 3, a forty-second wobble stops the service permanently. The right value for the limit is where the trade-off between visible failure and the service staying up is placed, and that trade-off varies by environment; there is no single correct number.

SV29 — the start-limit window is 100 seconds. SV30 — a unit that hits the limit stays failed until started by hand. SV31 — the operator polls every ten seconds; wrong diagnosis is the number of polls before the moment of giving up.

Three Numbers and the Second Seed

Oracle: the unit crashes 60 times in 600 seconds and is failing in all nine of nine combinations. Tool output: across nine combinations, five times “active,” twice “exited,” twice “failed”; in the start-limit sweep, restart counts of 60, 60, 10, 5, 3. Wrong diagnosis: 7 in the policy table; 60 with no limit, 11 at limit 10, 7 at limit 5, 5 at limit 3.

When crash moments are regenerated with a second seed (20260219), the crash count still comes out 60, and hidden times come out as: 600 and 600 with no limit, 106 and 107 at limit 10, 62 and 56 at limit 5, 41 and 36 at limit 3. The order of magnitude holds across both seeds. Individual seconds depend on the mock data; that hidden time drops by an order of magnitude once a limit is set does not. The nine-combination table is not touched by the seed at all, because what it counts there is not crash moments but the policy’s decision; that table depends only on the model itself.

Summary

• A unit file has three sections: placing in the graph, running, and enabling. A unit without the third section can be started by hand but not enabled.
• There are three ways for a process to stop — clean exit, error exit, termination by signal — and policies split these types in different places.
• In 7 of nine combinations, the diagnosis drawn from the status word is wrong: “no” hides one type, “on-failure” and “always” both hide all three.
• The “on-failure” policy, which does not count clean exit as failure, shows a long-lived service’s silent stop as exited normally.
• The start limit breaks the concealment: hidden time drops from 600 seconds to 41 seconds at limit 3, wrong diagnosis from 60 to 5.
• The limit’s cost is that a unit that hits it does not come back on its own; the right value is a trade-off set according to the environment.

Next Step

So far a single unit has been measured: one that crashes on its own and revives on its own. In the mock server, units are not alone; the graph built in the first lesson is still there, and restarting one unit does not concern only that unit. The next lesson builds the chain that begins with the machine powering on — from firmware to bootloader, from kernel to first process and targets — and counts how many units a restart stops and starts. The command prints one line; the oracle’s number is five.