--- title: 'Defining a New Service' source: 'https://academia.sh/en/courses/system-administration/defining-a-new-service' course: 'System Administration' language: en updated: '2026-08-17T18:10:05+00:00' license: 'CC BY-SA 4.0' --- # Defining a New Service The unit file's sections and fields are established; what three restart policies hide across three exit types is counted over nine combinations, and how far the start limit brings the hidden time down is swept. The previous lesson counted that the restart policy hides failure: four of six units showed "active" while all six were failing. One row of the table was left open. The `process` unit's policy was "on-failure" — a supposedly more measured choice than "always" — but the result was the same: sixty restarts, six hundred seconds of invisibility. The difference between the two policies never showed up in that run. This lesson brings the difference out. First it looks at where the policy is written, the **unit file**; then it multiplies three policies by three exit types and counts which failure disappears in which of the nine combinations. Finally, the mechanism that breaks the concealment is measured: the **start limit**. ## The Unit File's Sections A unit file is plain text divided into sections, and a service unit has three of them. The first section places the unit in the graph: its description, what it requires, what it starts after are written here. The second section says how the unit is run: which command, which user, which working directory, which environment variables, unit type, and restart policy. The third section says what enabling means: which target this unit links under. A unit file that skips the third section can be built and started by hand, but cannot be enabled; this is one way to produce the previous lesson's "not enabled but running" combination. The file below defines the mock server's `ingest` unit. It has not been run; it is written to show the fields' layout, and field names vary by service manager: ```text [Unit] Description=Data ingestion unit After=network.target [Service] Type=notify ExecStart=/usr/local/bin/ingest --config /etc/ingest.conf User=app Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` Two fields are directly relevant to this lesson. The `Type` field determines when the manager counts the unit as "started"; the notification-waiting type closes the readiness gap covered in the previous lesson. The `Restart` field is the restart policy, and it is what this lesson measures. `RestartSec` gives the wait before restarting; a value close to zero can revive a crashing unit several times a second and keep the processor busy. If the manager is not told to reread after the file is written, nothing changes; the in-memory copy of the graph stays old. This is the practical counterpart of the point established in lesson one. A syntax error in the file also does not show up at write time but at reread or start time; the manager skips the bad line and loads the unit with an incomplete definition. After writing, seeing what got loaded through the command that prints the unit's **merged definition** is more informative than reading the file one more time. ## Unit Type Determines When "Started" Is Said The `Type` field looks small and affects the entire chain. In the plainest type, the manager runs the command, and the instant the run returns successfully, it counts the unit as started. In the forking type, the process backgrounds itself, the first process exits, and the manager finds the real process from a PID file or the process tree; this type is the most exposed to the manager and the program lying to each other. In the one-shot type, the unit does a job and exits; exiting is not a failure, and this type is the legitimate owner of the "exited" state seen in the previous lesson. In the notifying type, the program explicitly tells the manager when it is ready, and the chain advances only then. Choosing the wrong type produces wrong behavior with no error message at all. Defining a program that backgrounds itself with the plainest type lets the manager watch the short-lived process that existed before backgrounding and count the unit as crashed once that process exits. If the restart policy is "always," the result is exactly the loop measured in the previous lesson: the unit restarts once a second, the state always reads "active." ## A Command That Works by Hand Does Not Work as a Unit The most common fault unit-file writers run into is a command working in the shell but not as a unit. The reason fits in one sentence: **the manager is not a shell.** The configuration files read at login are not read, the search path differs from the operator's, the working directory is the root directory if not specified, environment variables are empty, and if no user is given, the process runs with the same privilege as the manager. There is no shell expansion either: wildcards, quotes, and pipes do not work as expected, because the command line is not handed to a shell. The consequence is that the fault is **looked for in the wrong place**. The program exits because it cannot find a file, the error message says "no such file," and the operator searches for the file; the file is right where it should be, what is missing is the working directory. The right habit is writing paths as absolute in the unit file, stating the working directory and user explicitly, and having required environment variables read from a separate file. Where a unit's output goes and how it is found there is not this topic's question, it is the logs topic's. ## Three Policies, Three Exit Types There is more than one way for a process to stop running. **Clean exit**: the process ends on its own with a success exit code. **Error exit**: it ends with a nonzero code. **Termination by signal**: it is killed by an external signal or by kernel intervention. Restart policies split these three types, and where they split differs from one another. Policy **"no"** never restarts, under any circumstance. Policy **"on-failure"** restarts on error exit and termination by signal, not on clean exit. Policy **"always"** restarts under all three. The difference only shows up when a clean exit occurs, and that is exactly where it is dangerous: a service expected to be long-lived exiting with code zero is a failure, but the "on-failure" policy does not count it as one. **SV21 — the unit is designed to be long-lived**; every exit, whatever its code, is a failure. **SV22 — the unit is genuinely failing in all nine of the nine combinations**; the oracle knows this. **SV23 — the crash interval is seeded between 8 and 12 seconds.** **SV24 — restarting is instant** and never fails. **SV25 — a unit not restarted on clean exit has state "exited,"** not failed. **SV26 — a unit not restarted on error exit or termination by signal has state "failed."** **SV27 — a restarted unit's state is "active."** **SV28 — the operator's diagnosis is drawn from three status words:** healthy if "active," exited normally if "exited," failed if "failed." ```python SEED = 20260218 PERIOD = 600 POLICY = ("no", "on-failure", "always") EXIT = ("clean", "error", "signal") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def will_restart(policy, exit_): if policy == "always": return True if policy == "on-failure": return exit_ in ("error", "signal") return False def crash_moments(seed=SEED, duration=PERIOD, at_least=8, at_most=12): """The unit crashes at an irregular interval; the gap is seeded.""" r = generator(seed) t, items = 0, [] while True: t += at_least + r(at_most - at_least + 1) if t >= duration: return items items.append(t) def run(policy, exit_, moments, limit=0, window=100, duration=PERIOD): """The manager gives up if `limit` restarts are exceeded within `window` seconds.""" if not will_restart(policy, exit_): return {"restarts": 0, "state": "exited" if exit_ == "clean" else "failed", "hidden": duration if exit_ == "clean" else 0} done = [] for t in moments: if limit and sum(1 for x in done if x >= t - window) >= limit: return {"restarts": len(done), "state": "failed", "hidden": t} done.append(t) return {"restarts": len(done), "state": "active", "hidden": duration} A = crash_moments() print("oracle: unit crashes", len(A), "times in", PERIOD, "seconds") print() print("policy exit restarts seen state operator's diagnosis correct") wrong = 0 for p in POLICY: for c in EXIT: k = run(p, c, A) diagnosis = {"active": "healthy", "exited": "exited normally", "failed": "failed"}[k["state"]] correct = diagnosis == "failed" wrong += 0 if correct else 1 print(f" {p:10s} {c:7s} {k['restarts']:7d} {k['state']:13s}" f" {diagnosis:17s} {'yes' if correct else 'NO'}") print(" wrong diagnosis:", wrong, "/ 9") print() print("start limit restarts seen state hidden time wrong diagnosis") for limit in (0, 20, 10, 5, 3): k = run("always", "error", A, limit=limit) wt = len([t for t in range(0, PERIOD, 10) if t < k["hidden"]]) print(f" {'none' if limit == 0 else str(limit):11s} {k['restarts']:7d}" f" {k['state']:13s} {k['hidden']:13d} {wt:11d}") ``` ``` oracle: unit crashes 60 times in 600 seconds policy exit restarts seen state operator's diagnosis correct no clean 0 exited exited normally NO no error 0 failed failed yes no signal 0 failed failed yes on-failure clean 0 exited exited normally NO on-failure error 60 active healthy NO on-failure signal 60 active healthy NO always clean 60 active healthy NO always error 60 active healthy NO always signal 60 active healthy NO wrong diagnosis: 7 / 9 start limit restarts seen state hidden time wrong diagnosis none 60 active 600 60 20 60 active 600 60 10 10 failed 106 11 5 5 failed 62 7 3 3 failed 41 5 ``` ## What Each Policy Hides **7** of nine combinations give the operator a wrong diagnosis. What gets hidden sits in different places depending on the policy. Policy **"no"** hides only **1** of the three exit types: clean exit. On error exit and termination by signal, the unit shows "failed" and the diagnosis comes out right. It is the policy that hides the least, because it offers no resilience at all. Policy **"on-failure"** hides all **3** of the three types, but in two different forms: on error and signal it revives the unit and shows "active"; on clean exit it does not revive it and shows "exited." The second form is sneakier, because "exited" is a normal result for one-shot jobs and goes unnoticed. Policy **"always"** hides all **3** types the same way: all of them "active." The result ranks in reverse. As resilience rises, visibility falls, and no setting balances the two on its own. "Always" is the most direct way to keep a server up, and at the same time the option that best hides failure. The table's second reading says policy choice cannot be made **without knowing the exit-type contract**. The "on-failure" policy relies on the assumption that the program reports failure with a nonzero code. This assumption does not hold for every program: one that cannot read its configuration can print an error and still exit with code zero, an error in the shutdown path can be swallowed, a wrapper script can return its own exit code and hide the real program's. The exit-code contract was established in the M03/K02 Shell Programming course; here, the cost of breaking that contract shows up as "exited normally" in two of nine rows. The selection rule that follows has three parts. For a long-lived service, "always" is chosen and a ceiling is always placed next to it. For a one-shot job, "no" is chosen; that unit exiting really is normal. "On-failure" is chosen only once the program's exit-code contract has been verified, and verifying it requires either reading the code or deliberately breaking the program and reading its exit code. ## The Start Limit Breaks the Concealment Service managers carry a second mechanism to close this trap: the **start limit**. If a unit restarts more times than a set count within a set window, the manager gives up trying and marks the unit failed. The limit is not resilience being turned off, it is resilience having a **ceiling**. The sweep table shows the ceiling's effect. With **no** limit in a hundred-second window, the unit restarts **60** times over 600 seconds and never shows failed: an operator polling every ten seconds gets a wrong diagnosis **60** times. With the limit at **20**, the table does not change, because the crash rate in the mock data never reaches twenty within a hundred-second window: a limit that is set does the same as no limit. At **10**, the unit is given up on at the 106th second and wrong diagnosis drops to **11**. At **5**, hidden time is **62** seconds, wrong diagnosis **7**; at **3**, **41** seconds and **5**. The **20** row carries a warning of its own. A limit that is in the configuration, that shows in the output, that is written with correct syntax, does nothing here at all. Whether a setting is effective cannot be read from the configuration unless it is compared against the oracle; **a setting existing and a setting working are not the same thing.** Whether the limit is effective depends on the crash rate exceeding the limit within the window, and the crash rate does not appear in the configuration. The gain in between is large: hidden time drops from 600 seconds to 41, wrong diagnosis from 60 to 5. The cost is just as clear: a unit that hits the limit no longer comes back on its own. At limit 3, a forty-second wobble stops the service permanently. **The right value for the limit is where the trade-off between visible failure and the service staying up is placed**, and that trade-off varies by environment; there is no single correct number. **SV29 — the start-limit window is 100 seconds.** **SV30 — a unit that hits the limit stays failed until started by hand.** **SV31 — the operator polls every ten seconds;** wrong diagnosis is the number of polls before the moment of giving up. ## Three Numbers and the Second Seed **Oracle:** the unit crashes **60** times in 600 seconds and is failing in all nine of nine combinations. **Tool output:** across nine combinations, five times "active," twice "exited," twice "failed"; in the start-limit sweep, restart counts of 60, 60, 10, 5, 3. **Wrong diagnosis:** **7** in the policy table; **60** with no limit, **11** at limit 10, **7** at limit 5, **5** at limit 3. When crash moments are regenerated with a second seed (`20260219`), the crash count still comes out **60**, and hidden times come out as: **600** and **600** with no limit, **106** and **107** at limit 10, **62** and **56** at limit 5, **41** and **36** at limit 3. The order of magnitude holds across both seeds. Individual seconds depend on the mock data; that hidden time drops by an order of magnitude once a limit is set does not. The nine-combination table is not touched by the seed at all, because what it counts there is not crash moments but the policy's decision; that table depends only on the model itself. ## Summary - A unit file has three sections: placing in the graph, running, and enabling. A unit without the third section can be started by hand but not enabled. - There are three ways for a process to stop — clean exit, error exit, termination by signal — and policies split these types in different places. - In **7** of nine combinations, the diagnosis drawn from the status word is wrong: "no" hides one type, "on-failure" and "always" both hide all three. - The "on-failure" policy, which does not count clean exit as failure, shows a long-lived service's silent stop as **exited normally**. - The **start limit** breaks the concealment: hidden time drops from 600 seconds to **41** seconds at limit 3, wrong diagnosis from **60** to **5**. - The limit's cost is that a unit that hits it does not come back on its own; the right value is a trade-off set according to the environment. ## Next Step So far a single unit has been measured: one that crashes on its own and revives on its own. In the mock server, units are not alone; the graph built in the first lesson is still there, and restarting one unit does not concern only that unit. The next lesson builds the chain that begins with the machine powering on — from firmware to bootloader, from kernel to first process and targets — and counts how many units a restart stops and starts. The command prints one line; the oracle's number is five.