--- title: 'System Administration' source: 'https://academia.sh/en/courses/system-administration' language: en updated: '2026-08-17T04:41:57+00:00' license: 'CC BY-SA 4.0' lessons: 22 outcomes: - 'Monitors, prioritizes and manages processes with signals while counting what the tool cannot see' - 'Defines services and measures which failure the restart policy hides' - 'Changes disk, file system and mount configuration by separating two distinct fullness limits' --- # System Administration Comparing the diagnosis drawn from the output of process monitoring, signals, service units, logs and storage tools against the real state of the system. ## Process Management The sampling-interval dependence of listing and searching processes, job control, signals turning into requests, the visible effect of the priority value, and the cause that resource limits hide. ### Listing and Searching Processes Source: https://academia.sh/en/courses/system-administration/listing-and-searching-processes On a synthetic server of twenty-four processes, 14 are actually heavy: once the sampling interval rises to 15 seconds the tool misses 3 of them, and at 30 and 60 seconds it misses 5; a single glance gives 5–7 false diagnoses, the lifetime average gives 7, and selecting by name touches 9 processes wrongly in the best case. The Shell Programming course turned a script into a tool: it gained an argument interface, stopped on errors, cleaned up after itself, and was handed to a scheduler. Writing a script is *making something happen*. System administration begins with a different task — first, **seeing what is happening**. And seeing has its own errors. This course counts those errors. The Operating System Concepts course measured the process, the scheduler, and context switching **as mechanism**; what was measured there was the cost of the abstraction. Here the mechanism is not rebuilt: what is measured is **what the operator can see of that mechanism.** #### The Course's Measure In every lesson, three numbers stand side by side. The first is the **oracle**: the system's real state. Because we generate the fiction ourselves, we know which process is actually heavy. The second is the **tool's output**: what the tool shows. The third is the **false diagnosis** count: how many times the conclusion drawn from the output contradicts the oracle. The rule itself is one sentence: **a command's value is not the number of lines it shows, but how many times the diagnosis drawn from those lines is wrong.** An output whose diagnosis is not tested is counted as unmeasured. - **PM1.** The observation window is **600 seconds**. All durations are seconds in the model; no real time is measured. - **PM2.** The synthetic server has **24 processes**. Each process's real CPU usage is not a single number but a **time series**. - **PM3.** There are three patterns: **flat** (continuously low), **spike** (a five-second jump once every sixty seconds), **heavy** (continuously high). - **PM4.** A process is counted as **heavy** if its usage exceeds **50** even once during the window. The oracle applies this definition by looking at the full 600 seconds; the tool looks only at the instants it observes. - **PM5.** The monitoring tool looks once every `interval` seconds and keeps the **highest value it saw** per process. It knows nothing of what happened between glances. - **PM6.** The false diagnosis count is the sum of missed heavy processes and light processes mistaken for heavy. - **PM7.** The second seed is **20260219**. If the resulting diagnosis counts do not stay in the same order of magnitude, the result depends on the fiction, and is written as such. #### Reading the Process Table The command that gives the process list is `ps`; the command that gives a continuously refreshed view is `top` or one of its interactive derivatives. `ps` is a **snapshot**; the list it gives is the process table at the instant the command ran. The transcript below is **illustrative**, produced for this page, and has not been run: ```text $ ps -eo pid,user,ni,pcpu,pmem,etime,comm --sort=-pcpu PID USER NI %CPU %MEM ELAPSED COMMAND 1128 app 0 71.2 3.1 00:10:00 queue-worker 1043 monitor 5 68.4 1.2 00:10:00 metric-collector 1014 app -3 64.9 2.8 00:10:00 report-generator 1092 root 0 6.1 0.4 00:10:00 backup-job 1071 backup 10 3.0 0.2 00:10:00 cache-cleaner ``` Most columns explain themselves; two do not. `NI` is the **priority value** that will be measured in the fourth lesson. `%CPU` is the most misleading column: `ps` divides this value by the time elapsed **since the process's birth**, that is, it is a **lifetime average**. A process that has run for ten hours appears low in this column even if it consumed the core by itself for the last five minutes. `top` computes the same column differently: it represents the short interval between two refreshes. When the two commands are run on the same machine at the same time, they give different numbers for the same process, and **both are correct**; they are measuring different time spans. This distinction is exactly the source of what this lesson measures. #### The Synthetic Server The measurement rests not on a real command but on the course's common definition. The output of a command run on a real server cannot be reproduced: the machine differs, the load differs, the moment differs. Because we generate the synthetic server ourselves, we know the oracle. ```python """M03/K03 common definition: a synthetic server, the text the tools produce, and the diagnosis drawn from that text compared against the ORACLE. No real command is ever called. """ SEED = 20260218 SECOND_SEED = 20260219 PERIOD = 600 # observation window: 600 seconds def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def processes(seed=SEED, count=24): """Each process's real CPU usage is a TIME SERIES, not a single number.""" r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result def usage(process, second): """Real CPU usage, second by second. The oracle knows this; the tool does not.""" p = process["pattern"] if p == "flat": return 3 + (process["pid"] + second) % 4 if p == "heavy": return 60 + (process["pid"] + second) % 25 phase = (second + process["pid"]) % 60 # spike: a 5-second jump every 60 seconds return 88 + phase if phase < 5 else 2 + phase % 3 S = processes() print("synthetic server:", len(S), "processes |", {d: sum(1 for s in S if s["pattern"] == d) for d in ("flat", "spike", "heavy")}) print("processes actually crossing the threshold:", sum(1 for s in S if max(usage(s, t) for t in range(PERIOD)) >= 50)) ``` ``` synthetic server: 24 processes | {'flat': 10, 'spike': 7, 'heavy': 7} processes actually crossing the threshold: 14 ``` **The oracle is 14.** Seven processes are continuously heavy, and seven spike once every sixty seconds, comfortably exceeding the threshold when they do. Of the 24 processes in total, 14 cross the threshold somewhere in the window. Every number from here on is read against this 14. #### The Sampling Interval Determines the Diagnosis The monitoring tool has one setting: how often it looks. Enlarging the interval shrinks the output and reduces the load. What this does to the diagnosis can be measured. ```python # On top of the first block: S, usage, PERIOD, SECOND_SEED, processes come from there. def sample(processes_, interval, window=PERIOD): """A monitoring tool that looks once every `interval` seconds. Returns: highest seen per process.""" seen = {s["pid"]: 0 for s in processes_} glances = 0 for t in range(0, window, interval): glances += 1 for s in processes_: seen[s["pid"]] = max(seen[s["pid"]], usage(s, t)) return {"glances": glances, "seen": seen} def true_peak(processes_, window=PERIOD): return {s["pid"]: max(usage(s, t) for t in range(window)) for s in processes_} def diagnose_heavy(measurement, threshold=50): """The diagnosis drawn from the output: which processes are heavy.""" return {p for p, v in measurement.items() if v >= threshold} INTERVALS = (1, 5, 15, 30, 60) def sweep(processes_): truth = diagnose_heavy(true_peak(processes_)) rows = [] for a in INTERVALS: o = sample(processes_, a) t = diagnose_heavy(o["seen"]) rows.append((a, o["glances"], len(t), len(truth - t), len(truth - t) + len(t - truth))) return len(truth), rows oracle, rows = sweep(S) print("interval glances lines produced seen heavy missed false diagnosis") for a, glances, seen, missed, wrong in rows: print(f"{a:8d} {glances:7d} {glances * len(S):14d} {seen:10d}" f" {missed:6d} {wrong:15d}") print("oracle (actually heavy):", oracle, "| total processes:", len(S)) print() oracle2, rows2 = sweep(processes(SECOND_SEED)) print("second seed", SECOND_SEED, "- oracle:", oracle2) print(" false diagnosis:", {a: y for a, _, _, _, y in rows2}) print(" first seed :", {a: y for a, _, _, _, y in rows}) ``` ``` interval glances lines produced seen heavy missed false diagnosis 1 600 14400 14 0 0 5 120 2880 14 0 0 15 40 960 11 3 3 30 20 480 9 5 5 60 10 240 9 5 5 oracle (actually heavy): 14 | total processes: 24 second seed 20260219 - oracle: 12 false diagnosis: {1: 0, 5: 0, 15: 5, 30: 6, 60: 7} first seed : {1: 0, 5: 0, 15: 3, 30: 5, 60: 5} ``` A tool looking once every one or five seconds sees all 14 of the 14 heavy processes: **zero false diagnoses.** Once the interval rises to 15 seconds, **3** are missed; at 30 and 60 seconds, **5** are missed. The ones missed are always the same class — those that rise once every sixty seconds, for five seconds. A tool that looks once every thirty seconds is not obligated to catch a five-second spike; most of the time it does not. **The tool is not lying.** At the moment it looked, that process really was light. What is wrong is not the output but the diagnosis drawn from it: "it is not on the list, so it is not a problem." The second column carries the second claim. At interval 1, the tool produces **14,400 lines**; at interval 60, **240**. As the line count rises sixtyfold, the false diagnosis count drops from 5 to 0 — but the relationship between these two numbers is not linear: **2,880 lines give a diagnosis just as good as 14,400 lines.** Every line added past five seconds adds nothing to the diagnosis. With the second seed, the oracle drops to 12 and the false diagnoses come out as 5, 6, 7. The numbers are not the same; **the direction and order of magnitude are** — starting from zero, growing within a single-digit band as the interval grows. The result that stands is this: **enlarging the interval produces missed processes.** How many will be missed depends on the fiction. #### Single Glance vs. Lifetime Average The sampling interval is a problem of many glances. What the operator does most often, however, is a **single glance**: running `ps` once. The cost of this, and of `ps`'s own column average, is measured separately. ```python # On top of the previous blocks: S, usage, PERIOD, and diagnose_heavy come from there. TRUTH = diagnose_heavy({s["pid"]: max(usage(s, t) for t in range(PERIOD)) for s in S}) def glance(processes_, t): """The value read on a single glance: what it was using at that instant.""" return {s["pid"]: usage(s, t) for s in processes_} def lifetime_average(processes_, window=PERIOD): """The process's average since its birth. Dilutes the spike.""" return {s["pid"]: sum(usage(s, t) for t in range(window)) // window for s in processes_} readings = [(f"glance t={t:<3d} ", glance(S, t)) for t in (0, 3, 30, 300)] readings.append(("lifetime average ", lifetime_average(S))) print("reading form seen heavy missed false diagnosis") for name, measurement in readings: d = diagnose_heavy(measurement) print(f" {name}{len(d):10d} {len(TRUTH - d):9d}" f" {len(d - TRUTH) + len(TRUTH - d):11d}") print("oracle:", len(TRUTH), "heavy processes |", len(S), "processes total") ``` ``` reading form seen heavy missed false diagnosis glance t=0 9 5 5 glance t=3 8 6 6 glance t=30 7 7 7 glance t=300 9 5 5 lifetime average 7 7 7 oracle: 14 heavy processes | 24 processes total ``` A single glance gives **between 5 and 7** false diagnoses, depending on the moment it looks. Same machine, same 600 seconds, same command: the answer depends on which second it ran. This is the measure of the sentence "I ran `ps` to see the problem, and there was nothing there." The real result is in the last row. **The lifetime average is as wrong as the worst single glance:** 7 false diagnoses. Averaging does not erase the spike, it dilutes it — a five-second spike, divided over 600 seconds, drops far below the threshold. Looking at a longer window does not give a better diagnosis here; **which measure is taken comes before how much data was collected.** #### The Cost of Searching by Name There are two ways to search for a process. Filtering the list as text — narrowing `ps` output with `grep` — also catches the searching process itself, and brings back unrelated lines when the pattern is written wrong. The second way is `pgrep` and its counterpart `pkill`, which query the process table directly: ```text $ pgrep -a -u app queue-worker 1128 /opt/queue/queue-worker --shard 3 1135 /opt/queue/queue-worker --shard 4 ``` This transcript, too, is **illustrative and has not been run.** `pgrep` is more precise than text filtering: it does not list itself, and it can be narrowed by user and session. But what is precise is the **match**, not the diagnosis. Selecting by name silently accepts the assumption "the processes carrying this name are the problem." ```python # On top of the previous blocks: S and TRUTH come from there. name_table = {} for s in S: name_table.setdefault(s["command"], [0, 0]) name_table[s["command"]][0] += 1 name_table[s["command"]][1] += 1 if s["pid"] in TRUTH else 0 print("command name matched actually heavy wrong touches by name") for k in sorted(name_table): total, heavy = name_table[k] print(f" {k:22s} {total:6d} {heavy:14d} {min(heavy, total - heavy):21d}") print("total wrong touches (best case):", sum(min(a, t - a) for t, a in name_table.values())) print("how many heavy processes a single name catches at most:", max(a for _, a in name_table.values()), "/", len(TRUTH)) ``` ``` command name matched actually heavy wrong touches by name backup-job 5 3 2 cache-cleaner 2 1 1 data-receiver 1 1 0 metric-collector 5 2 2 queue-worker 6 3 3 report-generator 5 4 1 total wrong touches (best case): 9 how many heavy processes a single name catches at most: 4 / 14 ``` No command name coincides exactly with the heavy processes. Of the six processes carrying the name `queue-worker`, three are heavy and three are not; an operator selecting by name either sweeps in the three innocent processes too or leaves out the three heavy ones. Even in the best case, **9 processes are touched wrongly**, and a single name catches at most **4** of the heavy processes — the oracle is 14. The reason for this is not the fiction but the structure itself: **a command name is not an identity, it is a label.** The same program runs as multiple instances, and the instances' loads are independent of each other. Selecting by name is fast, and this speed turns into a cost when sending signals, in the third lesson of this topic. #### Summary - The course's measure is three numbers: the oracle, the tool's output, and the false diagnosis count. A command's value is not the number of lines it shows, but how many times the diagnosis drawn from those lines is wrong. - **14** of the 24 processes on the synthetic server are actually heavy; this is the oracle, and every number in the lesson is read against it. - When the sampling interval rises to 15 seconds, the tool misses **3** heavy processes; at 30 and 60 seconds, **5**. The ones missed are those that spike once every sixty seconds; the tool is correct at the instant it looks. - At a 1-second interval, **14,400** lines are produced; at 60 seconds, **240**. But 2,880 lines give a diagnosis just as good as 14,400. Past a certain point, more lines add nothing to the diagnosis. - A single glance gives between **5 and 7** false diagnoses depending on the moment it looks; the lifetime average taken since the process's birth is, with **7** false diagnoses, as bad as the worst single glance. - Searching by name touches **9 processes** wrongly in the best case, and a single name catches at most **4** of the heavy processes; a command name is not an identity but a label. #### Next Step This lesson looked at processes from outside: list, column, search. Processes also have a bond with the operator — which terminal they are attached to, which session they belong to, what happens when the session closes. The next lesson measures this bond: which processes survive when a script is backgrounded and the session is closed may not overlap at all with what `jobs` output shows. ### Foreground and Background Jobs Source: https://academia.sh/en/courses/system-administration/foreground-and-background-jobs When a session closes, all 13 of the shell's `&`-backgrounded jobs die, 9 of the disowned ones die, while nohup and a separate session keep all 13 alive; `jobs` shows 13 lines while the number of jobs actually progressing is 8. The previous lesson looked at processes from outside the machine: which exist, which are heavy. Processes also have a bond with the operator. When a script is started from a terminal, more than a single process is born: that process belongs to a **process group**, the group to a **session**, the session to a **controlling terminal**. When the terminal closes, events travel along this entire chain. The Shell Programming course established job control at the level of usage: `&`, `wait`, the exit code. This lesson does not repeat that side. What it measures is a single question: **which processes survive when the session closes**, and how well `jobs` output predicts this in advance. #### Session, Group, and Controlling Terminal The commands in a pipeline are placed into a single **process group**; the shell sends signals per group, not to individual processes. A shell opened from a terminal and all its children share the same **session**; the shell stands at the head of the session, and that terminal is the session's **controlling terminal**. These three identities can be listed: ```text $ ps -eo pid,ppid,pgid,sid,tty,stat,comm PID PPID PGID SID TTY STAT COMMAND 1000 998 1000 1000 pts/4 Ss bash 1014 1000 1014 1000 pts/4 R report-generator 1021 1000 1014 1000 pts/4 R metric-collector 1043 1000 1043 1000 pts/4 T backup-job 1092 1 1092 1092 ? Ss queue-worker ``` This transcript is **illustrative and has not been run.** There are three things to read here. The first two processes carry the same `PGID` value: the two ends of a pipeline. The last process's `SID` value equals its own process number, and its `TTY` column is empty — that process is in **its own separate session** and is attached to no terminal. The third process's state is `T`: **stopped.** When the terminal closes, two separate events travel, and either one can kill. The first is the **hangup signal**: the kernel delivers it to the head of the session, and the shell propagates it to its own jobs. The second is quieter — because the terminal no longer exists, a process trying to write to it gets an **input/output error**, and most programs stop on this error. The second event leaves no signal record at all. - **PM8.** Part of the synthetic server's 24 processes has been started from the operator's shell; the rest belong to other sessions. - **PM9.** Every process has two extra properties: whether its output still goes to the terminal (`writes`) and whether it waits for input from the terminal (`reads`). - **PM10.** Five detachment methods are measured: **foreground**, **background** (`&`), **disown**, **nohup**, and **separate session**. - **PM11.** A method determines three things: whether the process is in the session, whether the shell sends it the hangup signal, whether the process ignores that signal. Methods that redirect also turn off the `writes` property. - **PM12.** The survival rule: no event reaches processes outside the session; those inside it are eliminated first by the signal, then by the write error. - **PM13.** The common expectation is this: the foreground job dies, and all four of the other methods keep the job alive. The false diagnosis is the number of processes where this expectation and reality diverge. - **PM14.** The second seed is **20260219**. ```python # --- common definition, in the form used in the first lesson: synthetic server and oracle SEED, SECOND_SEED = 20260218, 20260219 USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def processes(seed=SEED, count=24): r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result # --- this lesson's layer: the session bond METHOD = ("foreground", "background", "disown", "nohup", "separate session") def distribute_sessions(processes_, seed=SEED): """The shell's own jobs are separated from the rest of the machine. `writes`: its output still goes to the terminal. `reads`: it waits for input from the terminal.""" r = generator(seed) for s in processes_: s["shell_job"] = r(10) < 5 s["writes"] = r(10) < 6 s["reads"] = r(10) < 3 return processes_ def apply(process, method): """A method determines three properties of the process. The rest comes from the process itself.""" in_session = method != "separate session" shell_sends = method in ("foreground", "background") protected = method in ("nohup", "separate session") return {"in_session": in_session, "shell_sends": shell_sends, "ignores": protected or process["ignores_signal"], "writes": process["writes"] and not protected} def is_alive(d): """The terminal closing produces TWO separate events, and either one can kill.""" if not d["in_session"]: return True # outside the session: neither event reaches it if d["shell_sends"] and not d["ignores"]: return False # first event: the hangup signal if d["writes"]: return False # second event: write error once the terminal is gone return True S = distribute_sessions(processes()) SHELL_JOBS = [s for s in S if s["shell_job"]] print("total processes:", len(S), "| shell's own jobs:", len(SHELL_JOBS), "| belong to another session:", len(S) - len(SHELL_JOBS)) print("shell jobs: ignore the signal", sum(1 for s in SHELL_JOBS if s["ignores_signal"]), "| write to the terminal", sum(1 for s in SHELL_JOBS if s["writes"]), "| read from the terminal", sum(1 for s in SHELL_JOBS if s["reads"])) ``` ``` total processes: 24 | shell's own jobs: 13 | belong to another session: 11 shell jobs: ignore the signal 4 | write to the terminal 9 | read from the terminal 5 ``` #### Backgrounding a Job Job control's commands are the shell's own commands, and they do not appear in the process table. `&` starts a job in the background, `Ctrl+Z` **stops** the foreground job, `bg` resumes a stopped job in the background, `fg` brings it back to the foreground. `jobs` lists the shell's job table: ```text $ jobs -l [1]- 1014 Running ./report-generator --day 7 & [2]+ 1043 Stopped ./backup-job --full [3] 1021 Running ./metric-collector | ./summarize & ``` This transcript, too, is **illustrative and has not been run.** Jobs are referred to with markers like `%1`, `%2`; `fg %2` brings the second job to the foreground. The number in square brackets is not a process number — it is the shell's own counter and has no meaning in another shell. The `jobs` table has two limits, and both produce exactly what this lesson measures. First, the table carries **only this shell's jobs**: no process started from another session, or by the service manager, appears here. Second, the table shows **the bond as it is now**, not the future outcome. The practical consequence of these two limits is this: `jobs` is not the way to verify that a job has genuinely detached from the session. Verification is done by **opening a second session** and searching for the process there; in `ps` output, an empty `TTY` column and a `PPID` of `1` are the proof of detachment. `jobs` output can never answer this question, because what is being asked lies exactly outside `jobs` output's scope. The exit code contract established in the Shell Programming course holds here too: `wait %1` waits for the job to end and returns its exit code. When a job is terminated by a signal, the code becomes **128 plus the signal number**; whether a background job died silently is read from this code. If `wait` is not called and the code read, the information is lost — the job table goes away when the shell closes. There are three ways to detach a process from the session. `disown` removes the job from the shell's table; the shell no longer sends it the hangup signal. The `nohup` command starts the process so that it ignores the hangup signal, and redirects its output to a file. `setsid` places the process in a **new session**; that session has no controlling terminal, so there is no terminal to close either. The three methods do not do the same job, and the difference can be measured. #### Five Methods, One Hangup ```python # On top of the previous block: SHELL_JOBS, METHOD, apply, is_alive come from there. # Common expectation: the foreground job dies, all the others survive. EXPECTATION = {"foreground": False, "background": True, "disown": True, "nohup": True, "separate session": True} print("method applied expected actual died false diagnosis") for y in METHOD: actual = {s["pid"] for s in SHELL_JOBS if is_alive(apply(s, y))} expected = {s["pid"] for s in SHELL_JOBS} if EXPECTATION[y] else set() print(f" {y:16s} {len(SHELL_JOBS):7d} {len(expected):8d} {len(actual):6d}" f" {len(SHELL_JOBS) - len(actual):4d} {len(expected ^ actual):15d}") print("oracle: the `actual` column on every row, since we generated the fiction") ``` ``` method applied expected actual died false diagnosis foreground 13 0 0 13 0 background 13 13 0 13 13 disown 13 13 4 9 9 nohup 13 13 13 0 0 separate session 13 13 13 0 0 oracle: the `actual` column on every row, since we generated the fiction ``` The second row is this lesson's main result. **`&` keeps nothing alive:** all 13 of the 13 backgrounded jobs die, false diagnosis **13**. The ampersand puts a job in the background; it does not take it out of the session. When the shell propagates the hangup signal to its own jobs, it includes the backgrounded ones too. The sentence "I put `&` after it, so it survives logout" is wrong for every job in the fiction. The third row gives a finer result. `disown` genuinely blocks the hangup signal — but of 13 jobs, only **4** survive, **9** die. The ones that die are victims of the second event: their output was still going to the terminal, and once the terminal vanished, they stopped on their first write attempt. `disown` **cuts the signal, not the stream.** This is the cost of trying to meet two separate events with a single precaution. In the last two rows, the false diagnosis is zero. `nohup` meets both events at once: it makes the signal ignored **and** redirects the output to a file. A separate session removes the problem entirely; there is no bond left to close. This is the right arrangement for lasting jobs — either a separate session, or not starting the job from the shell at all and turning it into a service unit instead. The second is the subject of this course's next topic. The detail can vary between shells; in some shells, whether the hangup signal propagates to jobs is controlled by an option. What is measured is the **common behavior**, and in the common behavior, `&` alone is not a protection. #### A Job Sitting in the List, Not Progressing The detachment problem has a counterpart. If a background process tries to **read from the terminal**, the kernel stops it; the process does not die, it waits. A stopped process appears with state `T` in `ps` output, and as "Stopped" in `jobs` output. Both confirm that the process exists; neither says that there is no progress. ```python # On top of the previous blocks: SHELL_JOBS, apply, is_alive, distribute_sessions, processes. stalled = [s for s in SHELL_JOBS if s["reads"]] print("backgrounded jobs trying to read from the terminal:", len(stalled)) print(" these get STOPPED: `jobs` lists them, `ps` shows them, progress is zero") print(" `jobs` rows:", len(SHELL_JOBS), "| actually progressing:", len(SHELL_JOBS) - len(stalled), "| false diagnosis:", len(stalled)) print() for seed in (SEED, SECOND_SEED): S2 = distribute_sessions(processes(seed), seed) K2 = [s for s in S2 if s["shell_job"]] d = {y: len({s["pid"] for s in K2} ^ {s["pid"] for s in K2 if is_alive(apply(s, y))}) for y in ("background", "disown", "nohup")} print(f"seed {seed}: shell jobs {len(K2):2d} | false diagnosis {d}" f" | stalled {sum(1 for s in K2 if s['reads'])}") ``` ``` backgrounded jobs trying to read from the terminal: 5 these get STOPPED: `jobs` lists them, `ps` shows them, progress is zero `jobs` rows: 13 | actually progressing: 8 | false diagnosis: 5 seed 20260218: shell jobs 13 | false diagnosis {'background': 13, 'disown': 9, 'nohup': 0} | stalled 5 seed 20260219: shell jobs 11 | false diagnosis {'background': 11, 'disown': 6, 'nohup': 0} | stalled 2 ``` `jobs` shows **13 lines**, the jobs actually progressing are **8**. Five jobs are waiting on a password or a confirmation, and nobody can write to them; they are in the list, not in the progress. False diagnosis **5**. The practical consequence of this is that every job to be backgrounded needs its input closed too — redirecting output alone is not enough. A process whose input is connected to an empty file is not stopped; a read attempt returns end-of-file immediately, and the process proceeds down its own error path. Stopping itself is not a visible event either: it is written nowhere except the status field in the job table. With the second seed, the shell's jobs drop to 11, stalled jobs to 2; the false diagnosis for `&` is again **all the jobs**, for `disown` **6** instead of 9, for `nohup` again **0**. The numbers depend on the fiction; **the order and direction do not change** — `&` is worst, `disown` is half protection, `nohup` and a separate session are full protection. #### Orphan and Zombie Processes When a session closes, the parent process of any surviving processes has died. The kernel does not leave them **orphaned**: it reassigns their parent process number, and the `PPID` column in `ps` output becomes `1`. Being orphaned is not a malfunction; three of the detachment methods work exactly this way. The situation that creates real confusion is the opposite. When a process ends, its exit code is kept in the process table until its parent **reaps** it. If the parent does not do this reaping, the child stays in the **zombie** state: it runs no code, holds no memory, but its table entry remains. In `ps` output its state is `Z`, and its command name is in square brackets. Sending a signal to a zombie has no effect at all — a signal is delivered to a process that runs code, and a zombie runs no code. What `wait` reaps in the Shell Programming course is exactly this; when a parent process is written that does not reap, the result is a zombie buildup. The fix is not to terminate the zombies, but to fix or terminate the **parent process**; once the parent is gone, the zombies are reassigned and reaped there. The cost of the buildup is a countable resource: every zombie holds one process table entry and one process number. Once enough have built up, a new process cannot be created. Which error message this limit shows up as is the subject of this topic's fifth lesson. For lasting jobs, `nohup` and `setsid` are the right tools, but not the only option. Tools in the **terminal multiplexer** class carry the session themselves: the job does not die when the connection drops, because the terminal the job is attached to is the multiplexer's, not the operator's. The difference here is not one of measurement but of ownership — when the multiplexer itself closes, the same two events travel again. #### Summary - A process is bound to three identities: process group, session, controlling terminal. The shell sends signals to the process group; in `ps` output, the `PGID`, `SID`, and `TTY` columns show this bond. - When the terminal closes, two separate events travel: the hangup signal, and the input/output error a process gets when it tries to write to the terminal. The second event leaves no signal record. - `&` does not detach from the session: all **13 of 13 backgrounded jobs** die, false diagnosis **13**. - `disown` cuts the signal but not the stream; of 13 jobs, **4** remain, **9** die because their output was going to the terminal. `nohup` and a separate session meet both events at once, false diagnosis **0**. - `jobs` shows **13 lines** while the jobs actually progressing number **8**; 5 jobs trying to read from the terminal have been stopped and appear to be running in the list. #### Next Step This lesson saw, from the side, how a signal reaches processes: the hangup signal propagated from the shell to its jobs, and some of them ignored it. Ignoring itself has not yet been measured. The next lesson takes up signals directly and counts a single sentence: a termination request is a **request**, and a command returning without error does not mean the target is gone. ### Signals Source: https://academia.sh/en/courses/system-administration/signals When a termination request is sent to seven heavy processes, 2 accept it and 5 ignore it; with the forced signal, 7/7 finish, but the cleanup of 2 processes is cut short. Because the command returns 0 in every case, the diagnosis given without waiting is wrong for all seven targets. The previous lesson passed by the edge of a signal: when the terminal closed, the hangup signal was delivered to the head of the session, the shell propagated it to its own jobs, and some of the jobs **ignored** it. Ignoring itself was not measured. This lesson takes up signals directly. What it measures is a single sentence: **a termination request is a request.** A command returning without error does not mean the target is gone, and the difference between the two can be counted. #### What a Signal Is, and Is Not A signal is an **interrupt notification** the kernel delivers to a process. It carries no data inside it; it has only an identity. When a process receives a signal, one of three things happens: the **handler** the process has defined for that signal runs, the signal is **ignored**, or the signal's **default action** is applied. The default action varies from signal to signal — terminate, stop, continue, or do nothing at all. `trap` in the Shell Programming course sets up exactly the first option: it defines the script's signal handler. The difference here is the direction of view — there, the **receiving** side was written; here, the **sending** side is measured. | Signal | Default action | Catchable | For what | |---|---|---|---| | `SIGHUP` | terminate | yes | the controlling terminal is gone; many services treat this as a reload request | | `SIGINT` | terminate | yes | interrupt from the terminal (`Ctrl+C`) | | `SIGQUIT` | terminate and dump core | yes | quit request from the terminal | | `SIGTERM` | terminate | yes | the **termination request** — the polite form, the default | | `SIGKILL` | terminate | **no** | the **forced signal** — the process has no say | | `SIGSTOP` | stop | **no** | freezes the process | | `SIGCONT` | continue | yes | resumes a stopped process | | `SIGUSR1`, `SIGUSR2` | terminate | yes | its meaning is the program's own contract | | `SIGPIPE` | terminate | yes | writing to a pipe whose reading end has closed | | `SIGCHLD` | ignore | yes | a child process ended or stopped | Two rows stand apart from the others. `SIGKILL` and `SIGSTOP` **cannot be caught, ignored, or blocked**; the kernel makes the decision, and the process is not even informed. This is exactly the cost: a process killed by the forced signal can **never run its own shutdown procedure**. Signals also have numbers, but the numbers are not the same on every architecture. Portable code is written **by name**: `kill -TERM` is written, not the number. #### Sending a Signal The command that sends a signal is named `kill`, and the name is misleading: the command does not kill a process, it delivers a signal. Which signal is delivered depends on the option; if no option is given, the termination request is sent. ```text $ kill -TERM 1128 $ echo $? 0 $ kill -0 1128 # no signal sent, only check whether it is reachable $ echo $? 0 $ pkill -TERM -u app queue-worker $ kill -TERM -1014 # a negative number: the signal goes to the PROCESS GROUP ``` This transcript is **illustrative and has not been run.** It carries four details. `kill -0` sends no signal at all; it only tests whether the target exists and whether there is authority to send it a signal — this is the tool of verification. `pkill` selects the target by name and user; the measure from the first lesson applies here too, a name is a label, not an identity. A negative number turns the target into a **process group**: it sends to the whole of a pipeline, not to a single member. `killall` also works by name, and because its behavior can vary between systems, `pkill` is preferred in portable scripts. The most critical detail is in the second line. `kill`'s exit code reports **whether the signal was delivered**, not anything else. How the target met the signal, what its handler did, whether the process finished, none of this is **ever reflected** in this code. An operator seeing a code of 0 and saying "the job is done" is an unmeasured diagnosis. - **PM15.** The target set is the processes on the synthetic server with the **continuously heavy** pattern. - **PM16.** Some of the processes **ignore** the termination request; the `ignores_signal` field in the common definition carries this. The forced signal cannot be ignored. - **PM17.** A process that accepts the request does not finish **immediately**: it does its own cleanup first. The cleanup time is between **0 and 39 seconds** per process. - **PM18.** `kill`'s exit code is **0** on every attempt; it means the signal was delivered. - **PM19.** The diagnosis given without waiting is this: "the command returned, the targets are gone." The false diagnosis count is the number of targets that, at that moment, have not actually finished. - **PM20.** The forced signal finishes every target; the cleanup of processes that accepted the request and were still cleaning up gets **cut short**. - **PM21.** The second seed is **20260219**. ```python # --- common definition, in the form used in the first lesson SEED, SECOND_SEED = 20260218, 20260219 USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def processes(seed=SEED, count=24): r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result # --- this lesson's layer: signal and cleanup def send_signal(processes_, signal, targets): """Common definition: `terminate` can be ignored, `force` cannot.""" finished, resisting = [], [] for s in processes_: if s["pid"] not in targets: continue if signal == "force" or not s["ignores_signal"]: finished.append(s["pid"]) else: resisting.append(s["pid"]) return {"finished": len(finished), "resisting": len(resisting), "resisting_pid": sorted(resisting)} S = processes() TARGET = {s["pid"] for s in S if s["pattern"] == "heavy"} print("target:", len(TARGET), "heavy processes") print("terminate:", send_signal(S, "terminate", TARGET)) print("force :", send_signal(S, "force", TARGET)) ``` ``` target: 7 heavy processes terminate: {'finished': 2, 'resisting': 5, 'resisting_pid': [1014, 1043, 1092, 1108, 1128]} force : {'finished': 7, 'resisting': 0, 'resisting_pid': []} ``` Only **2** of the seven heavy processes accept the termination request, **5** resist. With the forced signal, **7 of the 7** finish. The list of resisting processes does not appear in real output — the `kill` command does not give this information, and cannot. This is the second binding reading of the course's common definition. #### The Measure of Waiting The two rows above are the signal's final result. The operator, however, sees not the result but **the state at one instant**. A process that accepts the termination request does not finish immediately; it closes its open files, flushes its buffer, releases its lock. During this, the process still sits in the list. ```python # On top of the previous block: S, TARGET, generator, SEED come from there. def assign_cleanup(processes_, seed=SEED): """A process that accepts the termination request does not finish immediately: it cleans up first.""" r = generator(seed) for s in processes_: s["cleanup"] = r(40) # seconds return processes_ def termination_attempt(processes_, targets, wait): """The request was sent, checked again after `wait` seconds. `kill`'s exit code is always 0: it means the signal was DELIVERED, not that the target FINISHED.""" finished, cleaning_up, ignoring = [], [], [] for s in processes_: if s["pid"] not in targets: continue if s["ignores_signal"]: ignoring.append(s["pid"]) elif s["cleanup"] <= wait: finished.append(s["pid"]) else: cleaning_up.append(s["pid"]) return len(finished), len(cleaning_up), len(ignoring) assign_cleanup(S) print("target:", len(TARGET), "processes | `kill` exit code is 0 on every line") print("wait finished cleaning up ignoring alive false diagnosis") for b in (0, 1, 5, 10, 30, 60): finished, cln, ign = termination_attempt(S, TARGET, b) print(f"{b:7d} {finished:5d} {cln:12d} {ign:9d} {cln + ign:7d}" f" {len(TARGET) - finished:11d}") ``` ``` target: 7 processes | `kill` exit code is 0 on every line wait finished cleaning up ignoring alive false diagnosis 0 0 2 5 7 7 1 0 2 5 7 7 5 0 2 5 7 7 10 1 1 5 6 6 30 2 0 5 5 5 60 2 0 5 5 5 ``` The first row is the lesson's harshest number. At the instant the command returns, **none** of the targets have finished; false diagnosis **7/7**. Five processes ignored the request, and the other two have not finished their cleanup yet. The exit code is still 0 — the command has done its job, the signal was delivered. The second reading is in the wait column. Waiting five seconds changes nothing; at the tenth second **1** process finishes, at the thirtieth **2**. After thirty, nothing changes: the remaining **5** processes ignore the request and **will not finish no matter how long the wait**. Waiting completes the cleanup, it does not resolve resistance. This distinction determines the correct procedure. The correct order is four steps: **send the request**, **wait**, **verify** with `kill -0`, and **force** only what is still standing. Service managers do this too, and the wait time is a value set in the unit definition; that side is the subject of this course's next topic. The choice of wait time is made by looking at this table, and it has a cost in both directions. If chosen short, processes still cleaning up are forced needlessly; forcing at five seconds in the table means cutting off two processes that would have finished on their own at the tenth and thirtieth seconds. If chosen long, resisting processes are waited for pointlessly: no column in the table changes after the thirtieth second, and waiting sixty seconds only prolongs the malfunction. Without a measured threshold, the chosen duration remains a guess, and that guess produces false diagnoses in both directions. #### The Signal That Does Not Arrive, and the One That Waits Resistance is not the only reason a signal fails to work. There are three more situations, and all three produce the same picture: the command returns 0, the target stands. The first is a **pending signal**. A termination request sent to a stopped process is not processed immediately; because the process runs no code, the signal **waits**. The stopped jobs from the previous lesson are in exactly this state: the request is delivered, written to the queue, and stays there until the process is resumed with `SIGCONT`. The forced signal is an exception to this rule; the kernel applies it and does not wait for the process to run code. The portable way to truly stop a job is therefore to resume it first and then send the request. The second is a **blocked signal**. A process can temporarily block certain signals while entering a critical section; the signal waits until the block is lifted. An operator looking from outside cannot tell a blocked signal apart from an ignored one — in both cases, nothing happens. The third is **the wrong target**. Process numbers are reused: after a process ends, the same number can be assigned to a different process. Scripts that read a number from a file and send to it should therefore never use the number without verifying it; testing existence with `kill -0` is not enough, what needs to be tested is **whom** the number belongs to. The common conclusion of these three situations is this: sending a signal is not an event, it is a **request**, and the result of the request is learned only by looking at the target. Whether a shutdown procedure is written correctly is also the sending side's concern — this is why `SIGHUP` is commonly used to reconfigure services: the service rereads its configuration without restarting, and does not drop open connections. In another program, the default action of the same signal is to terminate; the meaning belongs not to the signal, but to **the program's contract**. #### The Cost of Forcing Skipping the wait step and sending the forced signal directly is an attractive shortcut: the result is certain, all the targets finish. Certainty has a cost, and it too can be counted. ```python # On top of the previous blocks: S, TARGET, termination_attempt, send_signal, # assign_cleanup, processes, SECOND_SEED. forced = send_signal(S, "force", TARGET) interrupted = [s["pid"] for s in S if s["pid"] in TARGET and not s["ignores_signal"] and s["cleanup"] > 0] print("direct forced signal:", forced["finished"], "/", len(TARGET), "finished") print(" cleanup interrupted midway:", len(interrupted), "processes") print("correct order (request, wait, verify, force):") finished, cln, ign = termination_attempt(S, TARGET, 30) print(" finished on its own after 30 seconds:", finished, "| needing to be forced:", cln + ign) print() for seed in (SEED, SECOND_SEED): S2 = assign_cleanup(processes(seed), seed) H2 = {s["pid"] for s in S2 if s["pattern"] == "heavy"} g = send_signal(S2, "terminate", H2) b0 = termination_attempt(S2, H2, 0)[0] b30 = termination_attempt(S2, H2, 30)[0] print(f"seed {seed}: target {len(H2)} | accepted the request {g['finished']}" f" | finished after 0 s {b0} | finished after 30 s {b30}" f" | false diagnosis {len(H2) - b0}") ``` ``` direct forced signal: 7 / 7 finished cleanup interrupted midway: 2 processes correct order (request, wait, verify, force): finished on its own after 30 seconds: 2 | needing to be forced: 5 seed 20260218: target 7 | accepted the request 2 | finished after 0 s 0 | finished after 30 s 2 | false diagnosis 7 seed 20260219: target 5 | accepted the request 5 | finished after 0 s 0 | finished after 30 s 4 | false diagnosis 5 ``` Forcing directly gives a **7/7** result, and **the cleanup of 2 processes is cut short**. The interrupted cleanup is not an abstract loss: a process that has not flushed its buffer loses data it thought it had written, a process that has not released its lock leaves a lock file behind, a process that has not deleted its temporary directory leaves it on disk. None of these three outcomes appears in `kill`'s output, and all three surface as a different malfunction on the next run. In the sequence done by waiting, the same two processes finish on their own, and only **5** targets need to be forced. The difference is not two processes; it is **two clean shutdowns.** With the second seed, the target count drops to 5, and the number accepting the request becomes **5** instead of 2 — the composition changes completely. The one thing that does not change is the first column: **in both seeds, the number of targets finished at the instant the command returns is 0**, and the diagnosis given without waiting is wrong for every target. How many processes will resist depends on the fiction; that the command returning says nothing is not. A limit outside the model should also be noted: the forced signal finishes every target here, but on a real system, a process waiting in uninterruptible sleep inside a kernel call does not respond even to that. In that case the process appears in state `D`, and the only remedy is removing the cause of the wait. #### Summary - A signal is an interrupt notification that carries no data; a process can catch it, ignore it, or leave it to the default action. `SIGKILL` and `SIGSTOP` cannot be caught or blocked. - `kill`'s exit code reports that the signal was **delivered**, not that the target has finished. The tool of verification is `kill -0`. - Of seven heavy targets, only **2** accept the termination request, **5** ignore it; with the forced signal, **7/7** finish. - The number of targets finished at the instant the command returns is **0**: false diagnosis **7/7**. At the tenth second 1 process finishes, at the thirtieth, 2; after thirty, nothing changes. - Forcing directly **cuts short the cleanup of 2 processes**; in the wait-based sequence, the same two processes shut down cleanly and only 5 targets are forced. The correct order is request, wait, verify, force. #### Next Step This lesson measured stopping a process. Stopping is not always the right answer: a heavy process may be doing necessary work, and what is wanted may not be to end it but to **slow it down**. The next lesson takes up the priority value and counts a single question: how much of a genuinely visible effect does changing a process's priority have, and is a changed column the same thing as changed behavior. ### Process Priorities Source: https://academia.sh/en/courses/system-administration/process-priorities When the priority value of the process doing the most work is pulled from the lowest to the highest, its share drops from 41,878 units to 0, but total work goes from 237,906 to 238,244 and load goes from 3.97 to 3.97: 94 percent of the freed capacity goes to other heavy processes, 0 percent to light ones. The previous lesson measured stopping a process. Stopping is not always the right answer: a heavy process may be doing necessary work, and what is wanted may not be to end it but to **slow it down**. There is a control designed for exactly this, and it sits in a column of the process table. This lesson measures that control. The question is: how much of a **visible** effect does changing a process's priority have, and is a changed column the same thing as changed behavior. #### The Priority Value The `NI` column in the process table carries the **nice value**. Its scale is inverted: **the process's priority drops as the value grows.** The range is `-20` to `19`; `-20` is the most aggressive process, `19` the most withdrawn. The default is `0`. The `PRI` column next to it is a different thing. `NI` is the operator's **input**; `PRI` is the value the scheduler **computes** from that input and the process's past behavior. The operator cannot write the `PRI` column directly. Confusing the two is confusing the input with the result. ```text $ nice -n 10 ./backup-job --full & $ renice -n 5 -p 1043 1043 (process ID) old priority 0, new priority 5 $ renice -n 5 -u backup $ ps -eo pid,ni,pri,comm --sort=ni PID NI PRI COMMAND 1014 -5 24 queue-worker 1128 0 19 report-generator 1043 5 14 metric-collector 1071 10 9 backup-job ``` This transcript is **illustrative and has not been run.** `nice` gives the value while the process **starts**, `renice` changes a **running** process; the `-u` option covers all of a user's processes. A rule is hidden here: an unprivileged user can only **increase** the priority value, meaning they can only withdraw their own process. Ordinarily there is no authority to reverse it — a process pulled to 10 cannot be returned to 5 by an unprivileged user; the sole exception is a process carrying a resource limit set aside for the priority value. This means a careless `renice` call can be permanent. The Operating System Concepts course compared scheduling algorithms against each other; that comparison is not repeated here. The question here is not which algorithm it is, but **how much of the operator's one control shows up in the output.** - **PM22.** The synthetic server has **4 cores**; per-second capacity is **400 units**. - **PM23.** The processes' total demand exceeds capacity; the system is saturated. - **PM24.** The priority value is converted to a **weight**: every unit changes the weight by roughly a quarter. The weight of value `0` is **1024**. - **PM25.** Capacity is divided according to weight, but **no process gets more than it asked for**. The surplus of satisfied processes is redistributed among the rest. - **PM26.** The oracle is the **total work actually done** over the window. Because capacity is fixed, the upper bound on this number is fixed too. - **PM27.** The resolution is **1 percent** of total work. A difference below this is counted as unmeasured. - **PM28.** The diagnosis is this: "lowering the priority of the process doing the most work reduces the system's load." The false diagnosis is the number of trials in which this sentence does not hold up. - **PM29.** The second seed is **20260219**. ```python # --- common definition, in the form used in the first lesson SEED, SECOND_SEED, PERIOD = 20260218, 20260219, 600 USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def processes(seed=SEED, count=24): r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result def usage(process, second): p = process["pattern"] if p == "flat": return 3 + (process["pid"] + second) % 4 if p == "heavy": return 60 + (process["pid"] + second) % 25 phase = (second + process["pid"]) % 60 return 88 + phase if phase < 5 else 2 + phase % 3 # --- this lesson's layer: how the priority value divides up capacity CORES, CAPACITY = 4, 400 def weight(priority): """The share SHRINKS as the priority value GROWS. Roughly a quarter per unit.""" w = 1024 for _ in range(abs(priority)): w = w * 5 // 4 if priority < 0 else w * 4 // 5 return w def distribute_share(processes_, second, capacity=CAPACITY): """If demand exceeds capacity, the share is divided by WEIGHT; nobody gets more than they asked for. The scheduler itself was measured in another course.""" demand = {s["pid"]: usage(s, second) for s in processes_} share = {p: 0 for p in demand} waiting, remaining = list(processes_), capacity while waiting and remaining > 0: total = sum(weight(s["priority"]) for s in waiting) satisfied = [s for s in waiting if remaining * weight(s["priority"]) // total >= demand[s["pid"]]] if not satisfied: for s in waiting: share[s["pid"]] = remaining * weight(s["priority"]) // total break for s in satisfied: share[s["pid"]] = demand[s["pid"]] remaining -= demand[s["pid"]] waiting.remove(s) return share def window_share(processes_, window=PERIOD): total = {s["pid"]: 0 for s in processes_} for t in range(window): for p, v in distribute_share(processes_, t).items(): total[p] += v return total S = processes() print("cores:", CORES, "| capacity:", CAPACITY, "| t=0 demand:", sum(usage(s, 0) for s in S)) print("priority ", " ".join(f"{o:4d}" for o in (-5, -3, 0, 3, 5, 10, 19))) print("weight ", " ".join(f"{weight(o):4d}" for o in (-5, -3, 0, 3, 5, 10, 19))) HEAVIEST = max(S, key=lambda s: sum(usage(s, t) for t in range(PERIOD))) print("heaviest process:", HEAVIEST["pid"], HEAVIEST["command"], "| priority value", HEAVIEST["priority"]) ``` ``` cores: 4 | capacity: 400 | t=0 demand: 736 priority -5 -3 0 3 5 10 19 weight 3125 2000 1024 524 335 108 12 heaviest process: 1014 queue-worker | priority value -5 ``` The weight table shows how steep the scale is: between `-5` and `19`, weight drops from **3125** to **12**, a 260-fold difference. Demand is 736, capacity 400 — the system is saturated, and the priority value has meaning only under this condition. In an unsaturated system, everyone gets what they ask for, and weights have no effect at all. #### The Visible Effect of Lowering Priority The process doing the most work is `1014`, and its priority value is currently `-5`, at the most aggressive end. The operator's habitual move is to withdraw it. ```python # On top of the previous block: S, HEAVIEST, window_share, PERIOD come from there. import copy def renice(processes_, pid, new_value): result = copy.deepcopy(processes_) for s in result: if s["pid"] == pid: s["priority"] = new_value return result TARGET = HEAVIEST["pid"] BASELINE = window_share(S) BASELINE_TOTAL = sum(BASELINE.values()) print("new priority target's share change others' share total work load") for v in (-5, -4, -3, 0, 5, 10, 19): share = window_share(renice(S, TARGET, v)) total = sum(share.values()) print(f"{v:13d} {share[TARGET]:14d} {share[TARGET] - BASELINE[TARGET]:6d}" f" {total - share[TARGET]:14d} {total:10d}" f" {total / PERIOD / 100:4.2f}") print("oracle: capacity", CAPACITY * PERIOD, "units | baseline usage", BASELINE_TOTAL) ``` ``` new priority target's share change others' share total work load -5 41878 0 196028 237906 3.97 -4 37372 -4506 200460 237832 3.96 -3 31462 -10416 206434 237896 3.96 0 17668 -24210 220318 237986 3.97 5 6120 -35758 231974 238094 3.97 10 1830 -40048 236342 238172 3.97 19 0 -41878 238244 238244 3.97 oracle: capacity 240000 units | baseline usage 237906 ``` The second column convinces the operator. A single-step change — from `-5` to `-4` — drops the target's share by **4,506 units**; pulled all the way to the end, the share drops from **41,878 to 0**. This column genuinely changes in the tool's output: the `NI` column shows the new value on every trial, and the target drops to the bottom of the list in the `%CPU` column. This is the only place where the work done is visible. The last two columns break the diagnosis. **Total work** goes from 237,906 to 238,244 — a difference of **338 units**, that is, 0.14 percent. Because the resolution is 1 percent, this difference **counts as unmeasured**. The **load** column reads **3.97** in six of the seven rows, 3.96 in one: it has not changed at all. The system was saturated and stayed saturated. The reason is a single-sentence observation: **the priority value does not reduce work, it divides it up.** As many units as the capacity allows, that much work gets done; who does it changes, how much gets done does not. And what an operator complaining about load is looking for with `renice` is exactly "how much gets done." #### Where the Freed Capacity Goes The 41,878 units the target gave up did not vanish; they went to other processes. Which ones tells us who the `renice` move genuinely helps. ```python # On top of the previous blocks: S, TARGET, BASELINE, BASELINE_TOTAL, renice. final = window_share(renice(S, TARGET, 19)) freed = BASELINE[TARGET] - final[TARGET] gain = {d: 0 for d in ("flat", "spike", "heavy")} for s in S: if s["pid"] != TARGET: gain[s["pattern"]] += final[s["pid"]] - BASELINE[s["pid"]] print("work freed from target:", freed, "units | where it went:") for d in ("flat", "spike", "heavy"): print(f" {d:6s} pattern processes: {gain[d]:6d}" f" ({round(100 * gain[d] / freed):3d} percent)") print("total work change:", sum(final.values()) - BASELINE_TOTAL, "units |", f"{100 * (sum(final.values()) - BASELINE_TOTAL) / BASELINE_TOTAL:.2f} percent") print("`NI` column changed in how many attempts: 6 / 6 | load dropped in how many: 0 / 6") print() for seed in (SEED, SECOND_SEED): S2 = processes(seed) h = max(S2, key=lambda s: sum(usage(s, t) for t in range(PERIOD)))["pid"] before, after = window_share(S2), window_share(renice(S2, h, 19)) print(f"seed {seed}: target's share {before[h]:6d} -> {after[h]:5d}" f" | total work {sum(before.values()):6d} -> {sum(after.values()):6d}" f" | load {sum(before.values()) / PERIOD / 100:.2f} ->" f" {sum(after.values()) / PERIOD / 100:.2f}") ``` ``` work freed from target: 41878 units | where it went: flat pattern processes: 0 ( 0 percent) spike pattern processes: 2774 ( 7 percent) heavy pattern processes: 39442 ( 94 percent) total work change: 338 units | 0.14 percent `NI` column changed in how many attempts: 6 / 6 | load dropped in how many: 0 / 6 seed 20260218: target's share 41878 -> 0 | total work 237906 -> 238244 | load 3.97 -> 3.97 seed 20260219: target's share 43200 -> 12786 | total work 239078 -> 239450 | load 3.98 -> 3.99 ``` **94 percent of the freed capacity goes to other heavy processes**. The share going to light-pattern processes is **0 percent** — exactly zero units. The reason for this is clear, and it gives the real rule of priority: light processes were already satisfied. A process that asks for little gets what it asks for even in a saturated system; there is nothing extra to give it. This result determines when the `renice` move actually helps. **The priority value changes something only among those competing.** A shell that opens slowly, a connection that lags, or an editor that stutters are usually low-demand processes, and their problem is not priority but some other resource. Withdrawing a heavy batch job, on the other hand, genuinely helps — but the winner is not the process the operator cares about, it is **the other heavy processes**. The false diagnosis count sits in the last lines: the `NI` column changed in **6 of 6 trials**, load dropped in **0 of 6**. The sentence "I lowered the priority, and load dropped" is wrong in **all six** of the six trials. With the second seed, the target's share drops from 43,200 to 12,786, total work rises from 239,078 to 239,450, and load goes from 3.98 to 3.99: the share changes a great deal, **load again does not change.** How much the share will drop depends on the fiction; that the load does not change does not. There is one more case where CPU priority does nothing at all, and it falls outside this measurement: if the bottleneck is not the CPU, `renice` touches nothing. A process waiting on disk or network is not asking for a core in the first place. There is a separate tool for input/output priority, `ionice`; strict scheduling classes are set with `chrt`, and used incorrectly they can cause a process to lock up the system entirely. Which the bottleneck actually is is measured in this course's logs topic. #### Inheritance and Correct Use The priority value has a property that is not measured here but is decisive: **child processes inherit it.** Every forked process is born with its parent's value. The practical consequence of this is that an entire tree can be withdrawn with a single call — when a wrapper script is started with `nice`, every step beneath it runs with the same value. The same property works in the other direction too: the value given by a scheduler or a service manager sticks to everything that starts from there, and the behavior measured in an interactive session is not reproduced there. Three rules of use follow from the measurement. **Give the value at the start.** `renice` does not bring back a running process's share up to that moment; it changes only what comes after. A long batch job should be withdrawn at the start, not after it has entered contention. Starting a job already withdrawn is a single line with `nice`, and thanks to inheritance it covers the sub-steps too. **Verify that the intended beneficiary is actually competing.** This was the measurement's sharpest number: the share going to light processes was **zero**. If a process's slowness comes from priority, that process must be queued for a core. Giving priority to a process that is not queued changes no number at all. **Do not try to solve load with priority.** Load is the total of work done; priority divides up that total. What genuinely reduces load is something else: reducing the work, spreading it over time, adding capacity, or stopping it. The first three are outside this course's scope; the fourth was the subject of the previous lesson, and its cost was measured in the table there. A warning also comes from the opposite end. Pulling the priority value negative is an operation only a privileged user can do, and its effect is this table's mirror image: the favored process gains share, and the loser is **everything else**. If the system administration tools themselves run with withdrawn values, the operator's own shell slows down at the moment of a malfunction too. #### Summary - The `NI` column carries the priority value, the operator's input; the `PRI` column carries the value the scheduler computes from it. The scale is inverted, and an unprivileged user can only increase the value. - The priority value is converted to a weight: the weight of `-5` is **3125**, of `19` is **12**. Weights have meaning only in a **saturated** system. - When the heaviest process's value is pulled to the end, its share drops from **41,878 to 0**; this is the only change visible in the tool's output. - Total work goes from **237,906 to 238,244** (0.14 percent, below the resolution) and load stays at **3.97** across all seven trials. Priority does not reduce work, it divides it up. - Of the freed 41,878 units, **94 percent goes to other heavy processes**, **0 percent** to light ones; light processes were already satisfied. - The diagnosis "I lowered the priority, load dropped" is wrong in **all six** of the six trials: the `NI` column changes 6/6, load drops 0/6. #### Next Step Priority affects **how fast** a process runs, it does not limit **how much it can consume**. Limiting has its own mechanism: resource limits set per process. The next lesson measures these limits and asks a single question: when a limit is exceeded, does the error message that appears say which limit it was — or do two separate causes show up as the same sentence. ### Resource Limits Source: https://academia.sh/en/courses/system-administration/resource-limits Eight distinct failure causes collapse into three error messages and one silence: 13 of 60 events produce no message at all, a diagnosis based on the message alone is wrong 21 times, one based on the limit reading alone is wrong 29 times, and the one combining both is wrong 13 times. Priority affects **how fast** a process runs; it does not limit **how much it can consume.** The process pulled back in the previous lesson's table could still open as many files as it liked, spawn as many children as it liked, request as much memory as it liked. Limiting consumption has its own mechanism. This lesson measures that mechanism and the failure that appears when a limit is exceeded. The question is this: does the error message say which limit it was — or do distinct causes show up under the same sentence. #### The Per-Process Constraint Every process has a set of resource limits, and every limit has **two** values. The **soft limit** is the value in effect; a process can raise it itself, up to the hard limit. The **hard limit** is the ceiling and can only be raised by a privileged process. Limits are read and written with the shell's `ulimit` builtin: ```text $ ulimit -a # the limits of the server in the fiction, sample transcript core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 1024 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited $ ulimit -Hn # hard limit 4096 $ ulimit -Sn 2048 # raise the soft limit up to the hard limit $ cat /proc/1014/limits Limit Soft Limit Hard Limit Units Max open files 2048 4096 files Max processes 4096 8192 processes ``` This transcript is **illustrative and has not been run.** Two reading paths appear here: `ulimit` gives the limits of **the calling shell**, while the file under `/proc` gives the limits of **a specific process**. The two are not the same thing, and the difference is the source of part of this lesson's diagnostic errors. The critical rule is **inheritance (at fork)**: limits are inherited from the parent process at fork time and cannot be changed afterward with the shell's builtin. Three consequences follow. First, a limit raised with `ulimit` in a shell covers only **that shell's children from that point on.** Second, a process started by the service manager inherits nothing from the operator's shell; its limits are written in the unit definition. Third, raising a running process's limit later cannot be done with a shell builtin — the process must be restarted. This is the measure behind "I raised the limit but the service still gives the same error." The per-process constraint is not the only kind of constraint. Constraints placed on the **total** of a group of processes are a separate mechanism; namespaces and control groups are not covered in this course and are left to the **Kernel Interfaces and Isolation** course. Every number here is the limit of **a single process.** - **PM30.** **60 failure events** are seen in the observation window. - **PM31.** The failures have **eight distinct causes**, and the causes are not equally frequent; their relative frequencies are fixed in the model. - **PM32.** Eight causes produce **three error messages** and **one silence**. Each message corresponds to **two distinct causes**. - **PM33.** The file size and CPU time limits produce no message at all: the process is killed by signal and only an exit code remains. - **PM34.** Six of the causes are **per-process** limits; two (the system-wide descriptor limit and free memory) are not per-process and do not appear in the process's own limit reading. - **PM35.** The first diagnostic path is **the message alone**. The best diagnosis that can be drawn from the message is the most frequent of the causes that produce it. - **PM36.** The second diagnostic path is **the limit reading**: whether the counter is still sitting at its limit is checked. - **PM37.** Part of the events are **transient**: by the time it is looked at, the counter has dropped back and the reading shows nothing. The evidence has vanished on its own. - **PM38.** The second seed is **20260219**. ```python # --- common definition, in the form used in the first lesson SEED, SECOND_SEED = 20260218, 20260219 USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def processes(seed=SEED, count=24): r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result # --- this lesson's layer: eight causes, three messages, one silence # (name, message, is it a per-process limit) CAUSE = ( ("process count limit", "resource temporarily unavailable", True), ("thread count limit", "resource temporarily unavailable", True), ("per-process descriptor limit", "too many open files", True), ("system-wide descriptor limit", "too many open files", False), ("address space limit", "cannot allocate memory", True), ("no free memory in the system", "cannot allocate memory", False), ("file size limit", "", True), ("CPU time limit", "", True), ) FREQUENCY = (5, 2, 8, 1, 4, 1, 2, 3) # relative frequency of causes POOL = [i for i, w in enumerate(FREQUENCY) for _ in range(w)] def failures(processes_, seed=SEED, count=60): """Failure events seen over the window. Oracle: the REAL cause of each event.""" r = generator(seed) event = [] for _ in range(count): s = processes_[r(len(processes_))] name, message, per_process = CAUSE[POOL[r(len(POOL))]] event.append({"pid": s["pid"], "cause": name, "message": message, "per_process": per_process, "transient": r(10) < 4}) # counter has dropped back by the time it is checked return event S = processes() EVENT = failures(S) print("failure events:", len(EVENT), "| distinct causes:", len({o["cause"] for o in EVENT})) counts = {} for o in EVENT: counts.setdefault(o["message"] or "(no message, killed by signal)", []).append(o["cause"]) print("message events distinct causes") for i, n in sorted(counts.items()): print(f" {i:38s} {len(n):4d} {len(set(n)):16d}") ``` ``` failure events: 60 | distinct causes: 8 message events distinct causes (no message, killed by signal) 13 2 cannot allocate memory 12 2 resource temporarily unavailable 16 2 too many open files 19 2 ``` The table gives this lesson's structure in one glance. Eight causes collapse onto four rows; **each row carries two distinct causes.** The top row has no message at all: **13 events** produced no text whatsoever. The process that exceeds the file size limit gets `SIGXFSZ`, the one that exceeds the CPU time limit gets `SIGXCPU`; if no handler is defined the process dies silently and all that remains is an exit code in the form of 128 plus the signal number. **The failure has no text, only a number.** #### The Cause the Message Hides The message looks as if it shows a cause, but what it actually shows is the **pair** the cause belongs to. The "resource temporarily unavailable" message can come from either the process count limit or the thread count limit; both mean that forking failed, and the kernel produces no text that distinguishes between them. ```python # Building on the previous block: EVENT, CAUSE come from there. # BEST diagnosis derivable from the message: the most frequent cause of each message. MESSAGE_DIAGNOSIS = {"resource temporarily unavailable": "process count limit", "too many open files": "per-process descriptor limit", "cannot allocate memory": "address space limit", "": "unknown"} def diagnose_by_message(events): return sum(1 for o in events if MESSAGE_DIAGNOSIS[o["message"]] != o["cause"]) def diagnose_by_limit(events): """Message + `ulimit` reading: which counter is sitting at its limit. If the counter is still at the limit the cause is read; if the event was transient the counter has dropped back.""" return sum(1 for o in events if o["transient"]) def diagnose_combined(events): """The limit is read first; if the counter has dropped back, fall back to the message.""" return sum(1 for o in events if o["transient"] and MESSAGE_DIAGNOSIS[o["message"]] != o["cause"]) print("diagnostic path wrong diagnoses /", len(EVENT)) print(f" message alone {diagnose_by_message(EVENT):9d}") print(f" limit reading alone {diagnose_by_limit(EVENT):9d}") print(f" limit reading, else message {diagnose_combined(EVENT):9d}") print(" transient events among these ", f"{sum(1 for o in EVENT if o['transient']):9d}") print("oracle: the real cause of each event is known, we generated the fiction") print() print("cause events correct from message?") for name, message, _ in CAUSE: n = sum(1 for o in EVENT if o["cause"] == name) print(f" {name:32s} {n:3d} {'yes' if MESSAGE_DIAGNOSIS[message] == name else 'no'}") ``` ``` diagnostic path wrong diagnoses / 60 message alone 21 limit reading alone 29 limit reading, else message 13 transient events among these 29 oracle: the real cause of each event is known, we generated the fiction cause events correct from message? process count limit 13 yes thread count limit 3 no per-process descriptor limit 16 yes system-wide descriptor limit 3 no address space limit 10 yes no free memory in the system 2 no file size limit 3 no CPU time limit 10 no ``` The lower table shows which causes can be read from the message: **three of the eight causes** can be correctly inferred, five cannot. The ones that cannot fall into two classes — the rarer member of a pair, and the causes that produce no message at all. **The CPU time limit alone accounts for 10 of the 60 events**, and not a single word is written about it. A diagnosis based on the message alone is wrong **21 times**. This is the best that a message-based diagnosis can realistically be: we picked the most frequent cause for each message. An operator who does not know the distribution picks worse. #### Evidence Vanishes on Its Own The correct response is to not trust the message and to read the limit instead: the process's counters are checked, and it is seen which one has hit the ceiling. This path has its own weakness, and it sits in the table. **A diagnosis based on the limit reading alone is wrong 29 times** — **worse** than the message-based diagnosis. The reason can be said in one word: **transience**. A failed attempt to fork does not raise the counter; the counter drops back right after the attempt. When the operator looks after the event, every counter sits below its limit and the reading says "everything is fine." **29 of the 60 events** are in this state. This is this lesson's version of the course's second claim: **a richer diagnostic path does not, on its own, give a better diagnosis.** The limit reading is far richer evidence than the message — it gives a number, a ceiling, a ratio — but the evidence lives only at the moment of the event and is gone afterward. The diagnosis that combines both paths is wrong **13 times**: it uses the limit reading when it works, and falls back to the message when the counter has dropped back. The remaining 13 events are the ones where both the evidence is lost and the message is misleading, and this number **cannot be reduced** — no reading taken afterward can bring back that moment's counter. The only fix is to catch the event **at the moment it happens**: watching the approach to the limit from a threshold, not waiting for the failure and looking afterward. ```python # Building on the previous blocks: failures, processes, and the three diagnostic paths. for seed in (SEED, SECOND_SEED): O2 = failures(processes(seed), seed) transient = sum(1 for o in O2 if o["transient"]) print(f"seed {seed}: events {len(O2)} | message {diagnose_by_message(O2):2d}" f" | limit {diagnose_by_limit(O2):2d} | combined {diagnose_combined(O2):2d}" f" | transient {transient:2d}" f" | silent {sum(1 for o in O2 if not o['message']):2d}") ``` ``` seed 20260218: events 60 | message 21 | limit 29 | combined 13 | transient 29 | silent 13 seed 20260219: events 60 | message 15 | limit 25 | combined 5 | transient 25 | silent 7 ``` With the second seed all the numbers drop: message goes from 21 to **15**, limit from 29 to **25**, combined from 13 to **5**. The numbers themselves depend on the fiction. What holds up is the **ranking**: in both seeds **the combined path is best, the limit reading alone is worst**, and the message-based diagnosis stays between the two. The richness of a diagnostic path does not, on its own, determine its place in the ranking. The practical management of limits follows from this measurement. Limits are not set so a failure occurs; they are set so **a process cannot consume the machine** — a good limit contains the failure but hides the cause. That is why setting a limit goes together with **watching the approach to it**. A per-process limit is also a blunt tool: a process count limit covering one user's processes can stop a faulty script while also blocking that user from logging in. #### Death Outside the Limit All eight causes measured here were a limit being exceeded, and in every one of them the process that died or failed was **the very process that exceeded the limit**. There is one more form of death that falls outside the limits, and it breaks the diagnosis from an entirely different direction. When the system's memory is truly exhausted, the kernel picks a **victim** and kills it with a forcing signal. The choice is not based on which process consumed the memory, but on a score in which current memory footprint weighs heavily. The result is often that **the process holding the most memory** dies — and that does not have to be the process that leaked it. A small leaking process can push the system under pressure and get a large, innocent process killed instead. The dying process leaves no trace in its own output; the only record of the event is in the system log. A second silencer is the dump itself. In the example transcript above, the core file size limit is **zero**, and this is a common default configuration. When it is zero, a process that dies by signal leaves nothing behind that can be examined. For the **13 silent events** in this lesson's first table, all that is left is the exit code; as long as the dump limit is zero, that code is left standing alone. Both observations lead to the same place. Finding a failure's cause **after** the event depends on the evidence still existing: the counter must stay at the limit, the dump must be written, the death must be recorded somewhere. All three default to **not** keeping the evidence. What belongs alongside a limit, then, is not a number but a recording path — a hook that writes the state at the moment of failure, or threshold monitoring. Where the death gets recorded, and how long that record lives, is measured in this course's logs topic. #### Summary - Every limit has two values, soft and hard; limits are inherited at the moment of forking, and a running process's limit cannot be changed from the shell afterward. - Eight failure causes collapse into **three error messages and one silence**; each message corresponds to **two distinct causes**, and **13 events** produce no text at all. - Only **three causes** can be correctly inferred from the message; the CPU time limit alone accounts for 10 of the 60 events and has no text about it at all. - The false-diagnosis count of the three diagnostic paths: message alone **21**, limit reading alone **29**, the combination of both **13**. Richer evidence does not, on its own, give a better diagnosis. - The limit reading's weakness is transience: in **29 of the 60 events** the counter has dropped back by the time it is checked. The remaining 13 false diagnoses cannot be reduced by any after-the-fact reading. - With the second seed the numbers drop to 15, 25, and 5; what holds up is the ranking — the combined path is best, the limit reading alone is worst. #### Next Step Throughout this topic, processes were seen from the operator's window: individual processes listed, searched, backgrounded, signaled, reprioritized, and limited. All of them shared one weakness — they were running while the operator was there to look. A server's real work, though, continues while no one is watching. The course's next topic covers the layer that binds that work to a durable object: the service manager, units, and the dependencies between them. The first thing measured there is what it means for a unit to appear "running." ## Services and Boot The unit and dependency model, the gap between apparent status and being healthy, the failure a restart policy hides, the boot chain, and bootloader parameters. ### The Service Manager Model Source: https://academia.sh/en/courses/system-administration/service-manager-model The unit, target, and dependency concepts are established; the diagnosis given by the unit list is compared against the diagnosis given by the dependency graph at four levels of detail, and wrong diagnoses are counted. The process management topic looked at individual processes: which one is really heavy, which one responds to a termination request, which one hits an imposed resource limit. Most of the processes on a server, though, are not started by an operator. The machine boots, dozens of programs come up with no one watching, one crashes and another takes its place, and by morning everything looks in order. What does this is a **service manager**, and what it manages is not the process itself but the process's **definition**. This lesson builds that definition and the graph between definitions. Its question is: how much of the system's real structure does the manager's printed list show? The list prints six lines, and all six lines are correct. What is measured is how many times the diagnosis drawn from those six correct lines is wrong. #### Unit, Target, and Two Kinds of Edge The object the service manager manages is called a **unit** (service unit). A unit is a text file: it says which program runs as which user, what happens if it fails, and which other units it needs. The manager reads these files, turns the relationship between them into a graph, and starts processes by following the graph. This distinction carries the rest of the course: **the unit file sits on disk, the process runs in memory.** Changing the file does not change the running process, killing the process does not delete the file, and a unit can be "loaded" without ever having run. There are unit types besides services, and all of them sit in the same graph: a unit representing a mount point, a unit that listens on a network socket and wakes a service on an incoming connection, a unit carrying a scheduled task, a unit that watches a file path. This topic measures only service units; the other types are named and passed over. When units need to be referred to as a group rather than one by one, a **target** is used. A target does no work itself and starts no process; it is a name representing the units grouped under it. Stages like "network ready," "multi-user system ready," and "shutdown" are named with targets. Moving to a target means enabling the units under it. That a target does no work raises the question of how much information an "arrived at target" notification carries; that question is measured in the boot process lesson. Edges in the graph are of two kinds, and confusing them is the error this lesson measures. A **requires edge** says one unit is useless without another: if the required unit stops, the requiring unit stops too. An **ordering edge** only says timing: this unit should start after that one. An ordering edge does not create a dependency. Two units can have ordering without requiring, or requiring without ordering. Command names and unit-file syntax vary by service manager and distribution; the model here describes common behavior. Requiring also has a weak form, called **wanting**: the wanted unit is attempted, and if it cannot be started, the wanting unit keeps running anyway. The weak edge is commonly preferred because it stops crash propagation; in exchange it silences the breakage of an edge in the graph. The mock server's graph carries no weak edge, because the confusion this lesson measures sits between ordering and requiring. Printing all three edge types in the same tree, in the same form, is the subject of the next section. Process isolation from one another, how the kernel side of resource accounting is kept, and control-group and namespace mechanisms are **not covered in this course**; they are left to the M03/K05 Kernel Interfaces and Isolation course. Here, a unit is a record in the manager's bookkeeping ledger. The manager's need for a graph is not only bookkeeping. Without a graph, units would have to be started one after another, each waiting on the last; with a graph, units with no edge between them can start at the same time. This distinction determines boot time and is counted layer by layer in lesson `04`. #### The Mock Server's Unit Graph The same mock server is used throughout the course. It has six units: `ingest` receives data from outside, `queue` buffers the data it receives, `process` reads from the queue, `report` produces output from the processed data, `metrics` collects the ingestion pipeline's counters, and `backup` runs without depending on anyone. **SV1 — a unit is a definition**, not the running process itself. **SV2 — the requires edge is transitive:** if `report` requires `process` and `process` requires `queue`, then `report` is also indirectly tied to `queue`. **SV3 — the ordering edge is not counted as transitive** and creates no dependency. **SV4 — `report` starts only after `backup`** but does not require it; **SV5 — `metrics` starts only after `queue`** but does not require it. **SV6 — the graph is acyclic.** **SV7 — stopping a unit stops every unit that requires it.** **SV8 — the graph carries no weak wanting edge;** only requiring and ordering exist. **SV9 — every unit belongs to the same manager** and sits on a single machine; no edge reaches out to a remote node. **SV10 — the graph does not change during the measurement;** no unit file is added, removed, or reread. The last of these assumptions does not always hold on a real system, and what happens when it does not is covered separately later. The rest of the measurement rests on a fixed graph: the oracle **is** this graph. The manager's unit list does not show the graph; it prints units in alphabetical order, one per line. The dump below has not been run on this machine; it is written as an example to show the format: ```text UNIT LOAD ACTIVE SUB DESCRIPTION backup.service loaded active running Backup job ingest.service loaded active running Data ingestion unit metrics.service loaded active running Metrics collector process.service loaded active running Processing unit queue.service loaded active running Queue unit report.service loaded active running Report generator ``` Six lines, six units, zero edges. The only information available to an operator looking at this list is which units exist. The answer to "what happens if I stop this unit" is not in the list and cannot be extracted from it. #### The Command That Prints the Graph, and Which Way It Prints Service managers offer a command that prints dependencies as a tree. This command has two directions, and they answer different questions. The default direction is **downward**: it shows what a unit needs in order to run. The reverse direction is **upward**: it shows who needs that unit. The answer to an operator's question "what breaks if I stop this" lies only in the reverse direction, while the parameterless call prints the downward direction. The two dumps below have not been run; they are written to show what each direction of the command prints: ```text $ systemctl list-dependencies report.service report.service * +-process.service * +-queue.service * +-ingest.service $ systemctl list-dependencies --reverse ingest.service ingest.service * +-queue.service * | +-process.service * | +-report.service * +-metrics.service ``` The second dump lists four units, and that is the correct count. But the tree prints the requires edge, the ordering edge, and the weak wanting edge **in the same form**; which line comes from which kind of edge cannot be read from the tree itself. Whether a unit shown in the tree will really stop when stopped, or only start later, requires looking at the unit files. What the next section measures is exactly the cost of this ambiguity. #### Four Readings, Four Diagnoses The same graph can be read at four levels of detail. The **flat list** gives only the units. The **direct** reading counts, for each unit, the units that directly require it. The **transitive graph** follows the chain to its end. The **with ordering** reading counts ordering edges as requiring too. The oracle is the transitive result computed over requires edges; the metric is how many units each reading diverges from that result on. ```python UNIT = { "ingest": {"requires": [], "restart": "always"}, "queue": {"requires": ["ingest"], "restart": "always"}, "process": {"requires": ["queue"], "restart": "on-failure"}, "report": {"requires": ["process"], "restart": "no"}, "metrics": {"requires": ["ingest"], "restart": "always"}, "backup": {"requires": [], "restart": "no"}, } AFTER = {"report": ["backup"], "metrics": ["queue"]} # ordering only def dependents_of(unit, units=None): units = UNIT if units is None else units direct = [a for a, v in units.items() if unit in v["requires"]] all_ = set(direct) for d in direct: all_ |= dependents_of(d, units) return all_ def with_ordering(unit): """Reading that counts the ordering edge as a dependency too.""" direct = [a for a, v in UNIT.items() if unit in v["requires"] or unit in AFTER.get(a, [])] all_ = set(direct) for d in direct: all_ |= with_ordering(d) return all_ oracle = {b: len(dependents_of(b)) for b in UNIT} diagnoses = { "flat list": {b: 0 for b in UNIT}, "direct": {b: sum(1 for v in UNIT.values() if b in v["requires"]) for b in UNIT}, "transitive graph": dict(oracle), "with ordering": {b: len(with_ordering(b)) for b in UNIT}, } print("unit oracle direct flat list") for b in UNIT: print(f" {b:9s} {oracle[b]:5d} {diagnoses['direct'][b]:6d}" f" {diagnoses['flat list'][b]:9d}") print(" oracle total:", sum(oracle.values())) print() print("reading output lines wrong diagnosis") for ad, t in diagnoses.items(): wrong = sum(1 for b in oracle if t[b] != oracle[b]) print(f" {ad:16s} {sum(t.values()):12d} {wrong:15d}") ``` ``` unit oracle direct flat list ingest 4 2 0 queue 2 1 0 process 1 1 0 report 0 0 0 metrics 0 0 0 backup 0 0 0 oracle total: 7 reading output lines wrong diagnosis flat list 0 3 direct 4 2 transitive graph 7 0 with ordering 9 2 ``` #### Three Numbers and the Second Claim Three numbers sit side by side. **Oracle:** the graph's real state is **7** dependency relationships across six units; if `ingest` stops, four units stop with it. **Tool output:** the flat list prints **0** edge lines, the direct reading **4**, the transitive graph **7**, the with-ordering reading **9**. **Wrong diagnosis:** **3** for the flat list, **2** for the direct reading, **0** for the transitive graph, and **2** again for the with-ordering reading. The with-ordering reading prints the most lines, and its diagnosis is worse than the transitive graph's. The two extra lines say that stopping `queue` also stops `metrics`, and that stopping `backup` also stops `report`; both are wrong. This is the first payment of the course's second claim: **more output does not mean a better diagnosis.** What leads to the right count is not how many lines are printed, but which kind of edge is counted. The flat list's **3** wrong diagnoses come from the same place. Because the list shows no edge at all, it produces the answer "stopping this stops only this" for every unit; that answer is wrong for `ingest`, `queue`, and `process`, and happens to be right for the remaining three. The word "happens" is not used loosely here: the reason the answer holds for `report`, `metrics`, and `backup` is not that the reading is good, it is that those units sit at the graph's edge. The same reading errs more often on a graph with fewer edges at the tip. **Output whose diagnosis is not tested counts as unmeasured**; the list is correct, the conclusion drawn from it is not. The direct reading's **2** wrong diagnoses come from a different flaw. This reading sees edges but cuts the chain after one step: **2** for `ingest`, real value **4**; **1** for `queue`, real value **2**. The direction of error here is always the same, always **undercounting**. The with-ordering reading's error direction is always **overcounting**. Because the two error types land in the same column, the counts can come out equal, even though their operational consequences are opposite: the operator who undercounts is caught unprepared, the operator who overcounts does unnecessary work. The working habit that follows can be written in one sentence: before stopping a unit, the **reverse dependency tree** is read, and every line in that tree is verified against the unit files as either requiring or ordering. When either step is skipped, the direction of the error is known. If the tree is not read, units are undercounted and unexpected units stop; if the tree is read but the edge type is not verified, units are overcounted and an unnecessary maintenance window is opened. Both land in the same "wrong diagnosis" column in the measurement. #### The Graph Is a Snapshot Unit files do not sit in a single directory on disk. The definition a package installs is in one directory, a locally written definition is in another, a definition generated at runtime is in a third, and a file with the same name **shadows** another across these directories. Alongside shadowing there are also **drop-in files**: these do not replace the whole file, they only add a few lines or change a single setting. As a result, a unit's effective definition sits not in one file but in the combination of several; only the command that prints the **merged definition** says which line came from where. Opening and reading a unit's file does not mean seeing which settings it actually runs with. There is a second fact that catches people out even more: the manager does not reread these files on every command. It reads them at startup, builds the graph in memory, and works with that copy. Changing the file on disk does not change the graph in memory; the change only enters the graph once the manager is told to reread. So the observation "I fixed the file but the behavior did not change" is not a malfunction, it is the expected result. The status output does not distinguish between the two states either: a unit shows as "active" even while the file on disk and the definition in memory differ. The gap between the oracle and the tool output opens here not in edge count but in **time**; the graph is a snapshot of the disk, and when the copy was taken does not appear in the output. #### Second Seed The numbers might be tied to a single graph. When the same measurement is repeated with two seeds over twenty random six-unit graphs built with a seeded generator, total wrong diagnoses come out to **60** and **69** for the flat list, **16** and **21** for the direct reading, **0** and **0** for the transitive graph, and **52** and **36** for the with-ordering reading. The order of magnitude holds: the transitive graph is error-free in both seeds, and the reading that counts ordering as dependency produces an error close to the flat list's in both seeds too. The individual counts depend on the mock data; **ordering is not requiring** is not among them. One point deserves underlining. The transitive-graph reading gives **zero** wrong diagnoses in both seeds, and this is not a property of the mock data, it is a consequence of the definition: the oracle was already defined as the transitive closure. The measurement's meaning is not in the correct reading coming out error-free, it lies in **the size of the wrong readings**. Across the forty graphs' total, the cost of only confusing edge types is eighty-eight wrong diagnoses; the cost of not reading edges at all, in the same forty graphs, is one hundred twenty-nine. That the two numbers sit this close says counting the ordering edge as a dependency is not much better than not reading the graph at all. #### Summary - The service manager's object is the **unit**; a unit is not the running process but the process's definition. A **target** is a name that groups units and does no work itself. - The graph has two kinds of edge: the **requires** edge creates a dependency and is transitive, the **ordering** edge only says timing and creates no dependency. - The mock server has six units and **7** dependency relationships; if `ingest` stops, four more units stop. - The flat unit list shows **0** edges and produces a wrong diagnosis on **3** of six units; the direct-dependency column brings this down to **2**, the transitive graph to **0**. - The reading that counts ordering edges as dependencies prints **9** lines and still gives **2** wrong diagnoses: the reading with the most output does not give the best diagnosis. - Namespaces and control groups are not covered in this course; they are left to the M03/K05 Kernel Interfaces and Isolation course. #### Next Step This lesson built the graph and counted how it does not show up in the list. The next question is even more unsettling: how reliable is the **status word**, the one thing that does show up in the list? A unit showing "active" against its name means the unit is running; it does not mean it is healthy. The next lesson measures the status word of a unit stuck in a failure loop on the same mock server and counts how many of sixty crashes are seen at which polling interval. ### Service Status and Control Source: https://academia.sh/en/courses/system-administration/service-status-and-control Starting, stopping, and enabling are established as separate axes; the status word of a unit stuck in a failure loop is compared against its real state, and how many of sixty crashes are seen at which polling interval is counted. The previous lesson built the unit graph and counted how it does not show up in the unit list. The list carried one piece of information: the **status word** next to each line. Six of six lines read "active," and this was a reassuring sight for whoever was looking at the list. This lesson measures what that word says and what it does not. The short answer is this: the status word says the manager currently sees the unit's process standing. This is true and misleading. It is true because the manager really is seeing a process. It is misleading because the process it sees may have been born ten seconds ago, and the corpse of the one before it may already have been cleared away. What is measured is how many times the **healthy** diagnosis drawn from the word "active" is wrong. #### Enabling and Starting Are Separate Axes The two most commonly confused actions of a service manager are **starting** and **enabling**, and the difference between them is an axis difference. Starting changes the present: the unit's process is brought up immediately, and no trace of this decision remains after the system reboots. Enabling changes the future: the unit is linked under a target and started every time that target is reached, but enabling itself starts no process. Stopping and disabling split the same way. The two axes produce four combinations, and all four are seen in practice. A unit can be **enabled and running**: the expected state. It can be **enabled but not running**: it will start at boot, it is stopped right now. It can be **not enabled but running**: started by hand, it will not come back after a reboot. It can be **not enabled and not running**: off. The third combination is the most expensive one, because a server that looks fine today loses that unit at the first reboot, and the cause of the loss is a decision made months earlier. The command that shows both axes on one line prints the unit's status in several separate fields. The dump below has not been run; it is written to show what the fields say: ```text $ systemctl status ingest.service * ingest.service - Data ingestion unit Loaded: loaded (/etc/systemd/system/ingest.service; enabled) Active: active (running) Main PID: Tasks: 3 ``` The `Loaded` field says the unit file was found and read, and the word in parentheses carries the enabling axis. The `Active` field says the current state; the second word in parentheses is a finer **sub-state** and comes from the service's own life cycle. These two fields are independent of one another. The value of one does not determine the other, though sitting side by side on the same output line suggests they determine each other. How enabling works explains this independence. What the manager does when a unit is enabled is link the unit file into a target's "wanted by" directory. No new process exists, nothing running changes; only a link forms on disk. This is why the enable command does not stop a running unit, and the disable command does not kill it either. An operator wanting to do both at once has to give two separate commands, and forgetting one of the two is the most common way the third combination gets produced. The list of control actions does not end with these two. **Restarting** stops and starts a unit, and its process identity changes. **Reloading** asks the process to reread its configuration without being killed; whether this is possible depends on the unit supporting it, and on a unit that does not, the command can silently turn into a restart or do nothing at all. **Masking** is the harshest of all: the unit's name is linked to an empty definition, and the unit can no longer be started either by hand or through a dependency. Another unit that tries to start a masked unit fails, and the error message names not the mask but "unit could not be started." A forgotten mask leaves behind a fault that is understood only weeks later. #### The Command Returning Is Not the Same as Being Ready The sentence built for the termination signal in the process management topic holds here too: **the command returning does not mean the job is done.** What the manager waits for once a start command is given depends on the unit's type. In the plainest unit type, the manager counts the unit as "started" the moment the main process is forked off; at that instant the program has not even begun running its own code. At that moment the process may not yet have read its configuration, opened its listening socket, or established its database connection. The command returns successfully, the status becomes "active," and the service still cannot answer requests for a few more seconds. The way to close this gap is for the unit to report its own readiness to the manager; in unit types that support notification, the manager waits for the notification before saying "started," and the next unit in the dependency chain is not started until then either. Without the notification, the chain moves too early: `queue` is started while `ingest` is not yet ready, and fails to connect on its first attempt. Both units show "active" in the status table. This gap is not measured in this lesson; what is measured is the invisibility of the failure itself. The two share the same root: **all the manager knows is that the process exists.** #### Measurement: A Unit in a Failure Loop All six of the mock server's units are put into a failing state: each crashes at regular intervals. The only thing that differs is the units' **restart policy**. The policy for `ingest`, `queue`, and `metrics` is "always"; for `process` it is "on-failure"; for `report` and `backup` it is "no." The observation window is 600 seconds, the crash interval is 10 seconds. **SV11 — the observation window is 600 seconds** and every duration in the model is in this unit. **SV12 — all six of the six units are failing** in this window; the oracle knows this. **SV13 — restarting is instant** and never fails. **SV14 — a unit with the "no" policy stops after its first crash** and its state becomes "failed." **SV15 — a restarted unit's state stays "active."** **SV16 — the diagnosis drawn from the status word takes only two values:** healthy if "active" is seen, failed if "failed" is seen. **SV17 — a unit is down for 2 seconds on every crash**, and the crash moment within each period is chosen by a seeded generator. **SV18 — polling is instantaneous** and sees the state exactly at the polled second. **SV19 — polling has no cost;** it does not slow the system down or change what it measures. **SV20 — a status query has no lag;** it returns the real state at the asked second, the tool itself is never wrong. The last two assumptions deliberately favor the tool. On a real system, polling has its own cost, and a status query can return from a previous scan's cache. The model zeroes these out; so the wrong-diagnosis counts below are the lower bound the tool produces **at its best.** ```python SEED = 20260218 PERIOD = 600 UNIT = { "ingest": {"requires": [], "restart": "always"}, "queue": {"requires": ["ingest"], "restart": "always"}, "process": {"requires": ["queue"], "restart": "on-failure"}, "report": {"requires": ["process"], "restart": "no"}, "metrics": {"requires": ["ingest"], "restart": "always"}, "backup": {"requires": [], "restart": "no"}, } def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def failure_cycle(unit, duration=PERIOD, interval=10, units=None): """The 'always' policy hides the failure: the service always shows 'active'.""" units = UNIT if units is None else units p = units[unit]["restart"] if p == "no": return {"policy": p, "restarts": 0, "seen_state": "failed", "hidden_time": 0} if p == "on-failure": return {"policy": p, "restarts": duration // interval, "seen_state": "active", "hidden_time": duration} return {"policy": p, "restarts": duration // interval, "seen_state": "active", "hidden_time": duration} def dead_intervals(seed=SEED, duration=PERIOD, period=10, dead=2): """The unit crashes once every period and is down for `dead` seconds.""" r = generator(seed) return [(t + r(period - dead + 1), dead) for t in range(0, duration, period)] def poll(intervals, step, duration=PERIOD): """An operator asking for status once every `step` seconds.""" looks = seen = 0 for t in range(0, duration, step): looks += 1 if any(b <= t < b + u for b, u in intervals): seen += 1 return {"looks": looks, "seen": seen, "wrong": looks - seen} print("unit policy restarts seen state hidden time diagnosis") wrong_diagnosis = 0 for b in UNIT: a = failure_cycle(b) correct = a["seen_state"] == "failed" wrong_diagnosis += 0 if correct else 1 print(f" {b:9s} {a['policy']:10s} {a['restarts']:8d}" f" {a['seen_state']:10s} {a['hidden_time']:11d}" f" {'correct' if correct else 'WRONG'}") print(" oracle: 6 of 6 units failed | wrong diagnosis:", wrong_diagnosis) print() A = dead_intervals() print("real crashes:", len(A), "| total dead seconds:", sum(u for _, u in A)) print("poll interval looks saw failed wrong diagnosis") for step in (1, 5, 15, 30, 60): y = poll(A, step) print(f" {step:13d} {y['looks']:5d} {y['seen']:11d} {y['wrong']:15d}") ``` ``` unit policy restarts seen state hidden time diagnosis ingest always 60 active 600 WRONG queue always 60 active 600 WRONG process on-failure 60 active 600 WRONG report no 0 failed 0 correct metrics always 60 active 600 WRONG backup no 0 failed 0 correct oracle: 6 of 6 units failed | wrong diagnosis: 4 real crashes: 60 | total dead seconds: 120 poll interval looks saw failed wrong diagnosis 1 600 120 480 5 120 21 99 15 40 7 33 30 20 2 18 60 10 0 10 ``` #### Three Numbers **Oracle:** all six units are failing; a unit with the "always" policy restarts **60 times** in 600 seconds and is down a total of **120 seconds**. **Tool output:** four units read **"active,"** two read "failed"; even at the tightest polling, the failed state is seen in **120** looks, at the loosest, in **0**. **Wrong diagnosis:** the health diagnosis drawn from the status word is wrong on **4** of six units; an operator polling once a second gets it wrong **480** times, one polling once every sixty seconds, **10** times. This table holds the common definition's fourth reading: **the restart policy hides the failure.** A unit with the "always" policy dies and comes back sixty times and its state never changes; the failure stays **invisible for 600 seconds.** A unit with the "no" policy, in contrast, becomes "failed" on its first crash and gives a correct diagnosis. The result should be read the other way around: **a unit configured to be more resilient produces less visible failure**, because here resilience is bought at the price of visibility. The table's most uncomfortable row is `process`'s row. Its policy is not "always" but "on-failure" — supposedly a more measured choice. The result is the same: **60** restarts, **"active"** state, **600** seconds of invisibility. The difference between the two policies never shows up in this run, because the unit always exits with an error. Where the difference does show up, and which failure type disappears under which policy, is what the next lesson measures. #### Looking More Often Is Not Enough The polling sweep pays the second claim. An operator polling once a second makes 600 looks and catches the failed state in **120** of them; one polling every five seconds catches **21** of 120 looks, every fifteen seconds **7** of 40, every thirty seconds **2** of 20, every sixty seconds **0** of 10. As the look count grows sixty-fold, the number of caught failures grows too — but its ratio falls. The caught share is one in five at one-second polling, zero at sixty-second polling. The reason the number behaves this way is that the failure sits in a **narrow window**. The unit crashes once every ten seconds and is down for only 2 seconds; for the remaining 8 seconds the manager really does see a running process. Polling has to land inside that 2-second window. An operator who polls rarely does not just "see less" — in most runs they **see nothing at all**, and every piece of evidence they hold says "active." **Looking late sees little.** The conclusion this leads to is not shortening the polling interval. Even polling once a second leaves 480 wrong diagnoses. However many times it is asked, the status word does not carry the **restart count**; the field that carries it is elsewhere. How many times a unit has restarted and when it last started sit in the manager's own separate fields, and that is the right question: not "is it active," but **"how long has it been active."** An uptime shorter than sixty seconds is the only visible trace of a unit that has restarted sixty times. The same command's full output already prints these fields; the problem is not that the fields are missing, it is that the eye fixes on the first line. The dump below has not been run; it is written to show where the evidence sits in the output of a unit stuck in a failure loop: ```text * queue.service - Queue unit Loaded: loaded (/etc/systemd/system/queue.service; enabled) Active: active (running) since Main PID: Status: "ready" host systemd[1]: queue.service: Main process exited host systemd[1]: queue.service: Scheduled restart job ``` The third line says both "active" and **how long** that run has lasted, and the last two lines say the process exited and a restart was scheduled. All three are in the same output, all three are correct. The wrong diagnosis comes not from missing output but from which line gets read. Reading a unit's health by **uptime** instead of the word "active" eliminates every one of this lesson's 480 wrong diagnoses; a unit that has restarted sixty times never has an uptime longer than ten seconds at any look. This is the reverse face of the course's second claim, and it fits in two sentences. More output does not fix the diagnosis, because line count is not what distinguishes it. Reading fewer but the **right field** fixes the diagnosis, because the metric itself is what distinguishes it. #### Second Seed Because crash moments are seeded, the table could depend on the seed. When the measurement is repeated with a second seed (`20260219`), the caught-failure counts come out as follows: **120** in both seeds at one-second polling; **21** in both at five seconds; **7** and **8** at fifteen seconds; **2** and **3** at thirty seconds; **0** and **3** at sixty seconds. Frequent polling agrees across both seeds; sparse polling is **mock-dependent** and should be written up that way. Sixty-second polling coming out at zero is specific to this run, and coming out at three is too. What is seed-independent is the direction of the ratio: as polling gets sparser, the caught share falls and never exceeds 120 in any seed. #### Summary - **Enabling** changes the future, **starting** changes the present; the two axes produce four combinations, and "not enabled but running" is lost at the first reboot. - The loaded, enabled, and active fields in the status output are independent of one another; sitting side by side suggests they determine each other. - While all six units in the failure loop are failing, **4** show "active": the restart policy hides the failure. - A unit with the "always" policy restarts **60 times** in 600 seconds, is down a total of **120 seconds**, and the failure stays **invisible for 600 seconds**. - As polling gets more frequent, caught failures rise but the caught **ratio** falls: 120/600 at 1 second, 0/10 at 60 seconds. Even one-second polling leaves **480** wrong diagnoses. - The right question is not "is it active" but **"how long has it been active"**; the restart count sits not in the status word but in separate fields. #### Next Step This lesson counted what the policy hides, without asking who set the policy. The policy is one line in the unit file, and whoever wrote that line usually thought "keep the service up no matter what." The next lesson rewrites the unit file from scratch and counts, across nine combinations, which failure type each of the three policies — "no," "on-failure," "always" — hides. In the same lesson, the mechanism that breaks the concealment, the **start limit**, is measured: the table comes out showing how far the 600-second invisible time can be brought down. ### Defining a New Service Source: https://academia.sh/en/courses/system-administration/defining-a-new-service The unit file's sections and fields are established; what three restart policies hide across three exit types is counted over nine combinations, and how far the start limit brings the hidden time down is swept. The previous lesson counted that the restart policy hides failure: four of six units showed "active" while all six were failing. One row of the table was left open. The `process` unit's policy was "on-failure" — a supposedly more measured choice than "always" — but the result was the same: sixty restarts, six hundred seconds of invisibility. The difference between the two policies never showed up in that run. This lesson brings the difference out. First it looks at where the policy is written, the **unit file**; then it multiplies three policies by three exit types and counts which failure disappears in which of the nine combinations. Finally, the mechanism that breaks the concealment is measured: the **start limit**. #### The Unit File's Sections A unit file is plain text divided into sections, and a service unit has three of them. The first section places the unit in the graph: its description, what it requires, what it starts after are written here. The second section says how the unit is run: which command, which user, which working directory, which environment variables, unit type, and restart policy. The third section says what enabling means: which target this unit links under. A unit file that skips the third section can be built and started by hand, but cannot be enabled; this is one way to produce the previous lesson's "not enabled but running" combination. The file below defines the mock server's `ingest` unit. It has not been run; it is written to show the fields' layout, and field names vary by service manager: ```text [Unit] Description=Data ingestion unit After=network.target [Service] Type=notify ExecStart=/usr/local/bin/ingest --config /etc/ingest.conf User=app Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` Two fields are directly relevant to this lesson. The `Type` field determines when the manager counts the unit as "started"; the notification-waiting type closes the readiness gap covered in the previous lesson. The `Restart` field is the restart policy, and it is what this lesson measures. `RestartSec` gives the wait before restarting; a value close to zero can revive a crashing unit several times a second and keep the processor busy. If the manager is not told to reread after the file is written, nothing changes; the in-memory copy of the graph stays old. This is the practical counterpart of the point established in lesson one. A syntax error in the file also does not show up at write time but at reread or start time; the manager skips the bad line and loads the unit with an incomplete definition. After writing, seeing what got loaded through the command that prints the unit's **merged definition** is more informative than reading the file one more time. #### Unit Type Determines When "Started" Is Said The `Type` field looks small and affects the entire chain. In the plainest type, the manager counts the unit as started the instant the main process is forked off; that instant comes even before the program begins running its own code. In the forking type, the process backgrounds itself, the first process exits, and the manager finds the real process from a PID file or the process tree; this type is the most exposed to the manager and the program lying to each other. In the one-shot type, the unit does a job and exits; exiting is not a failure, and this type is the legitimate owner of the "exited" state defined just below. In the notifying type, the program explicitly tells the manager when it is ready, and the chain advances only then. Choosing the wrong type produces wrong behavior with no error message at all. Defining a program that backgrounds itself with the plainest type lets the manager watch the short-lived process that existed before backgrounding and count the unit as crashed once that process exits. If the restart policy is "always," the result is exactly the loop measured in the previous lesson: the unit restarts once a second, the state always reads "active." #### A Command That Works by Hand Does Not Work as a Unit The most common fault unit-file writers run into is a command working in the shell but not as a unit. The reason fits in one sentence: **the manager is not a shell.** The configuration files read at login are not read, the search path differs from the operator's, the working directory is the root directory if not specified, environment variables are empty, and if no user is given, the process runs with the same privilege as the manager. There is no shell expansion either: wildcards, quotes, and pipes do not work as expected, because the command line is not handed to a shell. The consequence is that the fault is **looked for in the wrong place**. The program exits because it cannot find a file, the error message says "no such file," and the operator searches for the file; the file is right where it should be, what is missing is the working directory. The right habit is writing paths as absolute in the unit file, stating the working directory and user explicitly, and having required environment variables read from a separate file. Where a unit's output goes and how it is found there is not this topic's question, it is the logs topic's. #### Three Policies, Three Exit Types There is more than one way for a process to stop running. **Clean exit**: the process ends on its own with a success exit code. **Error exit**: it ends with a nonzero code. **Termination by signal**: it is killed by an external signal or by kernel intervention. Restart policies split these three types, and where they split differs from one another. Policy **"no"** never restarts, under any circumstance. Policy **"on-failure"** restarts on error exit and termination by signal, not on clean exit. Policy **"always"** restarts under all three. The difference only shows up when a clean exit occurs, and that is exactly where it is dangerous: a service expected to be long-lived exiting with code zero is a failure, but the "on-failure" policy does not count it as one. **SV21 — the unit is designed to be long-lived**; every exit, whatever its code, is a failure. **SV22 — the unit is genuinely failing in all nine of the nine combinations**; the oracle knows this. **SV23 — the crash interval is seeded between 8 and 12 seconds.** **SV24 — restarting is instant** and never fails. **SV25 — a unit not restarted on clean exit has state "exited,"** not failed. **SV26 — a unit not restarted on error exit or termination by signal has state "failed."** **SV27 — a restarted unit's state is "active."** **SV28 — the operator's diagnosis is drawn from three status words:** healthy if "active," exited normally if "exited," failed if "failed." ```python SEED = 20260218 PERIOD = 600 POLICY = ("no", "on-failure", "always") EXIT = ("clean", "error", "signal") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def will_restart(policy, exit_): if policy == "always": return True if policy == "on-failure": return exit_ in ("error", "signal") return False def crash_moments(seed=SEED, duration=PERIOD, at_least=8, at_most=12): """The unit crashes at an irregular interval; the gap is seeded.""" r = generator(seed) t, items = 0, [] while True: t += at_least + r(at_most - at_least + 1) if t >= duration: return items items.append(t) def run(policy, exit_, moments, limit=0, window=100, duration=PERIOD): """The manager gives up if `limit` restarts are exceeded within `window` seconds.""" if not will_restart(policy, exit_): return {"restarts": 0, "state": "exited" if exit_ == "clean" else "failed", "hidden": duration if exit_ == "clean" else 0} done = [] for t in moments: if limit and sum(1 for x in done if x >= t - window) >= limit: return {"restarts": len(done), "state": "failed", "hidden": t} done.append(t) return {"restarts": len(done), "state": "active", "hidden": duration} A = crash_moments() print("oracle: unit crashes", len(A), "times in", PERIOD, "seconds") print() print("policy exit restarts seen state operator's diagnosis correct") wrong = 0 for p in POLICY: for c in EXIT: k = run(p, c, A) diagnosis = {"active": "healthy", "exited": "exited normally", "failed": "failed"}[k["state"]] correct = diagnosis == "failed" wrong += 0 if correct else 1 print(f" {p:10s} {c:7s} {k['restarts']:7d} {k['state']:13s}" f" {diagnosis:17s} {'yes' if correct else 'NO'}") print(" wrong diagnosis:", wrong, "/ 9") print() print("start limit restarts seen state hidden time wrong diagnosis") for limit in (0, 20, 10, 5, 3): k = run("always", "error", A, limit=limit) wt = len([t for t in range(0, PERIOD, 10) if t < k["hidden"]]) print(f" {'none' if limit == 0 else str(limit):11s} {k['restarts']:7d}" f" {k['state']:13s} {k['hidden']:13d} {wt:11d}") ``` ``` oracle: unit crashes 60 times in 600 seconds policy exit restarts seen state operator's diagnosis correct no clean 0 exited exited normally NO no error 0 failed failed yes no signal 0 failed failed yes on-failure clean 0 exited exited normally NO on-failure error 60 active healthy NO on-failure signal 60 active healthy NO always clean 60 active healthy NO always error 60 active healthy NO always signal 60 active healthy NO wrong diagnosis: 7 / 9 start limit restarts seen state hidden time wrong diagnosis none 60 active 600 60 20 60 active 600 60 10 10 failed 106 11 5 5 failed 62 7 3 3 failed 41 5 ``` #### What Each Policy Hides **7** of nine combinations give the operator a wrong diagnosis. What gets hidden sits in different places depending on the policy. Policy **"no"** hides only **1** of the three exit types: clean exit. On error exit and termination by signal, the unit shows "failed" and the diagnosis comes out right. It is the policy that hides the least, because it offers no resilience at all. Policy **"on-failure"** hides all **3** of the three types, but in two different forms: on error and signal it revives the unit and shows "active"; on clean exit it does not revive it and shows "exited." The second form is sneakier, because "exited" is a normal result for one-shot jobs and goes unnoticed. Policy **"always"** hides all **3** types the same way: all of them "active." The result ranks in reverse. As resilience rises, visibility falls, and no setting balances the two on its own. "Always" is the most direct way to keep a server up, and at the same time the option that best hides failure. The table's second reading says policy choice cannot be made **without knowing the exit-type contract**. The "on-failure" policy relies on the assumption that the program reports failure with a nonzero code. This assumption does not hold for every program: one that cannot read its configuration can print an error and still exit with code zero, an error in the shutdown path can be swallowed, a wrapper script can return its own exit code and hide the real program's. The exit-code contract was established in the M03/K02 Shell Programming course; here, the cost of breaking that contract shows up as "exited normally" in two of nine rows. The selection rule that follows has three parts. For a long-lived service, "always" is chosen and a ceiling is always placed next to it. For a one-shot job, "no" is chosen; that unit exiting really is normal. "On-failure" is chosen only once the program's exit-code contract has been verified, and verifying it requires either reading the code or deliberately breaking the program and reading its exit code. #### The Start Limit Breaks the Concealment Service managers carry a second mechanism to close this trap: the **start limit**. If a unit restarts more times than a set count within a set window, the manager gives up trying and marks the unit failed. The limit is not resilience being turned off, it is resilience having a **ceiling**. The sweep table shows the ceiling's effect. With **no** limit in a hundred-second window, the unit restarts **60** times over 600 seconds and never shows failed: an operator polling every ten seconds gets a wrong diagnosis **60** times. With the limit at **20**, the table does not change, because the crash rate in the mock data never reaches twenty within a hundred-second window: a limit that is set does the same as no limit. At **10**, the unit is given up on at the 106th second and wrong diagnosis drops to **11**. At **5**, hidden time is **62** seconds, wrong diagnosis **7**; at **3**, **41** seconds and **5**. The **20** row carries a warning of its own. A limit that is in the configuration, that shows in the output, that is written with correct syntax, does nothing here at all. Whether a setting is effective cannot be read from the configuration unless it is compared against the oracle; **a setting existing and a setting working are not the same thing.** Whether the limit is effective depends on the crash rate exceeding the limit within the window, and the crash rate does not appear in the configuration. The gain in between is large: hidden time drops from 600 seconds to 41, wrong diagnosis from 60 to 5. The cost is just as clear: a unit that hits the limit no longer comes back on its own. At limit 3, a forty-second wobble stops the service permanently. **The right value for the limit is where the trade-off between visible failure and the service staying up is placed**, and that trade-off varies by environment; there is no single correct number. **SV29 — the start-limit window is 100 seconds.** **SV30 — a unit that hits the limit stays failed until started by hand.** **SV31 — the operator polls every ten seconds;** wrong diagnosis is the number of polls before the moment of giving up. #### Three Numbers and the Second Seed **Oracle:** the unit crashes **60** times in 600 seconds and is failing in all nine of nine combinations. **Tool output:** across nine combinations, five times "active," twice "exited," twice "failed"; in the start-limit sweep, restart counts of 60, 60, 10, 5, 3. **Wrong diagnosis:** **7** in the policy table; **60** with no limit, **11** at limit 10, **7** at limit 5, **5** at limit 3. When crash moments are regenerated with a second seed (`20260219`), the crash count still comes out **60**, and hidden times come out as: **600** and **600** with no limit, **106** and **107** at limit 10, **62** and **56** at limit 5, **41** and **36** at limit 3. The order of magnitude holds across both seeds. Individual seconds depend on the mock data; that hidden time drops by an order of magnitude once a limit is set does not. The nine-combination table is not touched by the seed at all, because what it counts there is not crash moments but the policy's decision; that table depends only on the model itself. #### Summary - A unit file has three sections: placing in the graph, running, and enabling. A unit without the third section can be started by hand but not enabled. - There are three ways for a process to stop — clean exit, error exit, termination by signal — and policies split these types in different places. - In **7** of nine combinations, the diagnosis drawn from the status word is wrong: "no" hides one type, "on-failure" and "always" both hide all three. - The "on-failure" policy, which does not count clean exit as failure, shows a long-lived service's silent stop as **exited normally**. - The **start limit** breaks the concealment: hidden time drops from 600 seconds to **41** seconds at limit 3, wrong diagnosis from **60** to **5**. - The limit's cost is that a unit that hits it does not come back on its own; the right value is a trade-off set according to the environment. #### Next Step So far a single unit has been measured: one that crashes on its own and revives on its own. In the mock server, units are not alone; the graph built in the first lesson is still there, and restarting one unit does not concern only that unit. The next lesson builds the chain that begins with the machine powering on — from firmware to bootloader, from kernel to first process and targets — and counts how many units a restart stops and starts. The command prints one line; the oracle's number is five. ### The Boot Process Source: https://academia.sh/en/courses/system-administration/boot-process The handoff chain from firmware to the first process and targets is established; how many units restarting one unit stops and starts, and how many units a unit failing at boot never starts at all, are counted. The previous lesson measured a single unit: one that crashes on its own and, depending on its policy, revives or does not revive on its own. In that measurement the unit was alone. In the mock server, units are the nodes of the graph built in the first lesson, and that graph is still there. Restarting one unit does not concern only that unit. This lesson looks at how the graph is run **over time**. When the machine boots, this graph runs from end to end once; when a unit is restarted by hand, part of the graph runs again. In both cases, the number of lines the manager prints and the work actually done differ from one another. What is measured is how many times the diagnosis drawn from the restart command's output is wrong. #### From Firmware to the First Process A machine booting is not a single event, it is a **handoff chain**. Each link loads the next one into memory, hands off control, and steps off the stage. None of the links can inspect the one before it; this is why an error at the chain's start cannot be seen with the tools at its end. The first link is **firmware**. It sits in persistent memory on the motherboard, runs when power is applied, counts and tests the hardware, then looks at boot devices in a predetermined order. It loads the code on the first suitable device it finds into memory and runs it. The firmware's device order is a configuration value, and this value is not visible from inside the operating system. The second link is the **bootloader**. Its job is to choose a kernel, read it from disk into memory, bring along an initial ramdisk, and prepare the parameters to be passed to the kernel. The bootloader itself can consist of more than one part; the first part is very small and only large enough to read the second part. The bootloader's details are the subject of the next lesson. The third link is the **kernel**. It is loaded into memory, unpacks itself, recognizes devices, and mounts the real root file system using the initial ramdisk brought along with it. The initial ramdisk is necessary because the drivers needed to mount the real root often sit inside that root itself; this intermediate layer is what solves the ordering problem. Once the root is mounted, the kernel does its last job and runs the **first process**. The first process is the lone process at the root of the process tree; it lives until the system's end, and when it dies the system stops. The fourth link is the **first process**, and on most systems this process is the service manager. The manager reads unit files, builds the graph, and enables the units needed to reach the **default target**. The handoff chain ends here, and the unit graph begins here. The chain's property relevant to diagnosis is that each link writes its own evidence **somewhere else**. The firmware's findings stay on screen and land in no file. An error the bootloader reports is visible only at that moment. The kernel's boot messages are written to a ring buffer in memory and cannot be saved anywhere until the root file system is mounted; if the buffer fills, the oldest messages are overwritten. Persistent logging only begins once the root is mounted and the logging unit is running. So a failure in the chain's first three links leaves **no trace in any file** as long as the system does not boot. How logs are collected and rotated is the next topic's question; the point here is only this: evidence existing requires it to have been recorded. #### Targets and Layers Booting is the unit graph being run once from the start. The manager takes the default target, finds the units it requires and wants, finds what those require, and unfolds the graph backward. The result is a **layer** structure: units with no edge between them sit in the same layer and can be started at the same time. The layer count sets the lower bound on boot time. If six units started one after another, that would be a six-unit wait; split into layers, the wait is as many as the layer count. This was the answer, in the first lesson, to why the manager needs a graph. Service managers offer an analysis command that prints how long booting took and which unit held things up by how much. This command's output is often misread, and the source of the misreading is the graph again: the duration shown against a unit is not that unit's **own** work time, it is the time between its start request being given and it being counted ready. Time spent waiting on required units is included in this duration too. The unit shown at the top of the ranking is often not the slowest unit but the unit in the **deepest layer**. An operator who tries to optimize a unit based on this output optimizes the wrong unit. Boot time's real numbers depend on hardware and configuration and are not written here; what can be written is **what** the ranking is ranking. **SV32 — the default target wants all six of the six units.** **SV33 — a unit does not start until every unit it requires has started.** **SV34 — units in the same layer start at the same time** and do not wait on one another. **SV35 — ordering edges do not enter the layer calculation;** only requiring edges do. **SV36 — restarting a unit stops and restarts every unit that requires it.** **SV37 — the restart command prints a single line** and names only the unit given. **SV38 — a unit that fails at boot causes the units that require it to never start at all.** **SV39 — the boot summary gives the count of failed units only from those that were actually tried and failed;** units never tried do not enter this count. **SV40 — the graph does not change during boot.** The reason these assumptions are written before the measurement is that none of them show up in the output. The manager knows the layers and it knows the skipped units; neither appears in the lines it prints. #### The Line the Command Prints When the restart command succeeds it prints nothing; on failure it prints a single line of error. The command that asks about the whole system after boot also answers with a single word. The dumps below have not been run; they are written to show what the output carries: ```text $ systemctl restart ingest.service $ systemctl is-system-running degraded $ systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION * ingest.service loaded failed failed Data ingestion unit 1 loaded units listed. ``` The first command returned silently, and this silence is read as "only what was asked for happened." The second command gives a single word about the whole system, and that word does not say which unit is broken. The third command lists the failed units; its last line prints a count. None of the three outputs say that the four units depending on this one unit have also stopped. The dependent units are not "failed," they are in "inactive" state, and they do not enter this list. The gap is a status-word gap, and it flips the diagnosis. ```python UNIT = { "ingest": {"requires": [], "restart": "always"}, "queue": {"requires": ["ingest"], "restart": "always"}, "process": {"requires": ["queue"], "restart": "on-failure"}, "report": {"requires": ["process"], "restart": "no"}, "metrics": {"requires": ["ingest"], "restart": "always"}, "backup": {"requires": [], "restart": "no"}, } def dependents_of(unit, units=None): units = UNIT if units is None else units direct = [a for a, v in units.items() if unit in v["requires"]] all_ = set(direct) for d in direct: all_ |= dependents_of(d, units) return all_ def restart_chain(unit): """How many more units restarting one unit stops and starts.""" affected = dependents_of(unit) return {"unit": unit, "affected": len(affected), "affected_names": sorted(affected)} def layer(unit): """Which wave the unit can start in during boot.""" g = UNIT[unit]["requires"] return 0 if not g else 1 + max(layer(x) for x in g) def boot(failed): """If a unit fails at boot, which units never start at all.""" skipped = dependents_of(failed) started = [b for b in UNIT if b != failed and b not in skipped] return {"started": len(started), "skipped": len(skipped), "reported": 1} print("unit layer tool output oracle (stopped and started) correct diagnosis") wrong = 0 for b in UNIT: z = restart_chain(b) oracle = 1 + z["affected"] correct = oracle == 1 wrong += 0 if correct else 1 print(f" {b:9s} {layer(b):5d} {'1 unit':>12s} {oracle:29d}" f" {'yes' if correct else 'NO'}") print(" wrong diagnosis:", wrong, "/ 6 | units hidden:", sum(len(dependents_of(b)) for b in UNIT)) print(" ingest chain:", restart_chain("ingest")) print() print("failed at boot started never started reported failure") for b in UNIT: o = boot(b) print(f" {b:18s} {o['started']:8d} {o['skipped']:14d} {o['reported']:15d}") ``` ``` unit layer tool output oracle (stopped and started) correct diagnosis ingest 0 1 unit 5 NO queue 1 1 unit 3 NO process 2 1 unit 2 NO report 3 1 unit 1 yes metrics 1 1 unit 1 yes backup 0 1 unit 1 yes wrong diagnosis: 3 / 6 | units hidden: 7 ingest chain: {'unit': 'ingest', 'affected': 4, 'affected_names': ['metrics', 'process', 'queue', 'report']} failed at boot started never started reported failure ingest 1 4 1 queue 3 2 1 process 4 1 1 report 5 0 1 metrics 5 0 1 backup 5 0 1 ``` #### Three Numbers **Oracle:** restarting the `ingest` unit stops and starts **4** more units — `process`, `queue`, `metrics`, and `report`. The number of units seeing the interruption together with itself is **5**. Across six units, a total of **7** units are affected in the shadows. **Tool output:** the command prints a single line per unit, and that line names only **1 unit**. **Wrong diagnosis:** on **3** of six units, the "only this unit was affected" diagnosis is wrong; for `ingest` the count is off by a factor of five, for `queue` by three, for `process` by two. This is where the common definition's third reading sits: **restarting a unit stops four more.** And the chain does not show in the output. The direction of the wrong diagnosis is always the same, always **undercounting**. This direction corresponds to a specific operational error: for a short maintenance window, `ingest` is restarted, report generation is cut at the same time, the reason for the cut does not appear in the report's own log, and the fault is searched for in the wrong place. The layer column says a second thing. `ingest` and `backup` are in layer zero, `queue` and `metrics` in layer one, `process` in layer two, `report` in layer three. The graph's depth is four. The number of units a restart affects and the layer depth are not the same measure: `metrics` is in layer one and yet affects no unit, because no one requires it. **Depth and impact are two independent measures**, and confusing them leads to treating units at the bottom of the graph as needlessly dangerous. #### A Unit That Fails at Boot The second table reads the same graph in the boot direction. If `ingest` fails to start at boot, only **1** unit starts, **4** units are never tried, and the boot summary reports **1** failed unit. The reported count is correct — one unit really did fail — but it does not describe the system's state. The four units did not fail; they were **never tried**, and because they were never tried, no record was produced about them. This is this lesson's form of the course's third claim: **evidence vanishes on its own.** If a unit had run and errored, a failure record would remain. A unit that never started writes nothing at all. An operator looking after boot has one failure notice in hand and four silent units; the silence looks the same as health. The correct reading is not looking at the number in the boot summary, it is looking at **the difference between the expected unit set and the started unit set.** This difference is computed from the graph, and no single command prints it on its own. The table's last three rows confirm this too: when `report`, `metrics`, or `backup` fails, no unit is skipped, and the reported number is correct for once. Same notice, same shape, different meaning. #### Shutdown Is the Same Graph, Reverse Direction If boot runs the graph forward, shutdown runs the same graph backward. The topmost units are stopped first, then what they require. If the order were not reversed, a unit writing data would try to shut down while the storage unit underneath it has already been torn down. Shutdown has its own time limit, and this limit is a source of diagnosis too. The manager sends a stop request to a unit, waits a set time, and if the unit does not stop, sends a forcing signal. The distinction built in the process management topic holds here too: **request and force are different things**, and the window between them is the only time a unit has to write its data to disk. When the limit is kept short, shutdown speeds up and produces half-written files; when kept long, a single stuck unit holds up the whole machine's shutdown for minutes. This is the shutdown-side counterpart of the staying-up-versus-visibility trade-off. #### Rescue Targets and the Next Link The default target is not the only option. There are two smaller targets for system administrators: one mounts all file systems and gives a single shell with only the most basic services running, the other mounts no file systems at all and starts almost no units. There are two ways to reach these targets. While the system is running, the manager can be told to switch targets; if the system does not boot at all, the only way is to set the target from the start with a **boot parameter**. The second path shows this lesson's limit. Everything covered up to here was for after the first process comes up. If the first process never comes up, there is neither a unit list nor a status query; all that is left in hand is the line the bootloader offers for editing. The tools at the chain's end cannot see the failure at the chain's start. #### Second Seed This lesson's numbers come not from a seeded generator but from a fixed unit graph. To see whether the measurement depends on the graph, the same computation was repeated with two seeds over twenty random six-unit graphs built with a seeded generator. Total hidden units come out to **124** with `20260218` and **132** with `20260219`; wrong-diagnosis count comes out to **64** and **68** over 120 units. The order of magnitude holds, and the ratio in both seeds sits a little above half: on a random graph, restarting one unit affecting others is the rule, not the exception. Individual counts depend on the mock data; **the command's output not carrying the affected-unit count** does not. Random graphs' ratio comes out higher than the fixed unit graph's: the mock server has wrong diagnosis on three of six units, random graphs have it on sixty-four of a hundred twenty and sixty-eight of a hundred twenty. The difference comes from the mock server's graph being a thin chain; as edge density rises, so does the number of affected units. On a real server the unit count is measured in the hundreds, and the graph there is far denser than what is drawn here. The measurement's real-system counterpart is therefore expected to be **worse**, not better. #### Summary - Boot is a **handoff chain**: firmware, bootloader, kernel and initial ramdisk, then the first process. Each link loads the next and steps off; the next link cannot inspect the one before. - The first process is the service manager on most systems; it unfolds the graph backward from the default target and starts units **layer by layer**. - On the mock server, the graph's depth is four layers; depth and impact are different measures. - When `ingest` is restarted, **4** more units stop and start, the affected count together with itself is **5**, the command output says **1 unit**; the diagnosis is wrong on **3** of six units. - If `ingest` fails at boot, **4** units are never tried and the summary reports only **1** failure; a unit never tried leaves no record. - The correct reading looks not at the number in the summary but at **the difference between the expected set and the started set.** #### Next Step Three of the handoff chain's four links were tied to the unit graph in this lesson. What remains is the second link: the **bootloader**. It decides which kernel loads, which initial ramdisk comes with it, and which parameters are passed to the kernel. The next lesson covers how to read bootloader configuration, how to test whether a parameter change is really effective, and why operations that break the configuration are irreversible. In the same lesson, how many of twenty-four parameter changes actually do their job is counted; the producing command's answer is all twenty-four. ### Bootloaders Source: https://academia.sh/en/courses/system-administration/bootloaders The parts of bootloader configuration, kernel selection, and boot parameters are established; how many of twenty-four parameter changes actually do their job is counted with three verification methods, and the safe way to test irreversible operations is shown. The previous lesson built the handoff chain's four links and said the failure at the chain's start cannot be seen with the tools at its end. This lesson returns to the second link, the **bootloader**. It decides which kernel loads, which initial ramdisk comes with it, and which parameters are passed to the kernel. Changes here have one property that shapes the whole topic: **their effect is seen only at the next boot.** The command that writes the configuration returns successfully, the system keeps running, nothing looks changed. Whether the change was correct is understood hours or weeks later, when the machine reboots. What is measured is how many times the "change is effective" diagnosis drawn from the "configuration written" output is wrong. #### What a Boot Entry Consists Of The bootloader keeps a list of **entries** to choose from. Each entry has four parts: a name, a kernel image, the initial ramdisk to be loaded alongside that kernel, and the parameter string passed to the kernel. Having more than one entry on the same machine is normal; when a new kernel is installed, the old one is not deleted, a new entry is added to the list. This is the cheapest way back: if the new kernel does not boot, the old entry is chosen from the menu. Firmware comes in two styles, and they determine where entries sit. In one style, the bootloader's first part is written to a single sector at the very start of the disk; the rest, which does not fit in that sector, sits elsewhere on disk and the first part finds it by address. In the other style, the firmware can read a file system itself; entries sit as plain files in a separate boot partition, and the choice can be made from the firmware's own menu. The two styles can coexist on the same machine, and that raises the question of which configuration is actually being read. The dump below shows what an entry looks like. It has not been run; it is written to show the layout of the parts, and the file format varies by bootloader: ```text title Mock server linux /vmlinuz-mock initrd /initrd-mock.img options root=UUID= ro quiet ``` The `linux` line gives the kernel image, the `initrd` line the initial ramdisk, the `options` line the parameter string. The root device here is written not by device name but by a **persistent identity**; device names can change from boot to boot, identity does not. An entry relying on a device name can stop booting once a second disk is attached to the machine, and the reason will not appear in the configuration. #### What Parameters Do The parameter string is passed to the kernel, and the kernel does not use parameters it does not recognize itself — it passes them on to the first process. This behavior produces the topic's most important trap: **a misspelled parameter gives no error.** The kernel does not recognize it, the first process does not recognize it either, and both silently move on. The string shows up in the running system's kernel command line, and because it shows up, it is assumed "applied." Parameters' functions fall into a few groups. Some say the root device and the mount mode. Some set how much of the boot messages get printed to the screen. Some disable a hardware driver. Some tell the first process which target to go to; this is the way to reach the rescue targets named in the previous lesson. The same thing can be said with two different parameters, and when they conflict, which one wins depends on order — this too is a source of silent wrong behavior. Which parameters a running system booted with is read from a file the kernel offers. The dump below has not been run; it is written to show what the read returns: ```text $ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-mock root=UUID= ro quiet unused=value ``` The `unused=value` at the end of the line is a made-up parameter; the kernel does not recognize it, the first process does not recognize it, neither complains, and the string just sits there as is. An operator using this line as verification counts the made-up parameter as applied too. The **command-line reading** method in the measurement is exactly this reading, and the reason it goes wrong is exactly this: the file shows the change was **delivered**, not that it **did its job**. #### Menu and Timeout The bootloader's fallback value also depends on it showing its menu, as much as on the entry list it holds. The menu comes with a timeout: if no key is pressed within that time, the default entry boots. When the timeout is set to zero, the menu never shows and boot shortens by a few seconds. What that saved handful of seconds costs is the closing off of the one fix path left at the moment of failure: if the menu is not shown, the old entry cannot be chosen, no one-shot edit can be made either. This trade-off sharpens once more on a remote server. On a machine with no console access, the menu is unusable anyway; the way back there is not the menu, it is an entry marked to be valid for **just the next boot**. Such an entry is tried once: if the boot succeeds it is made permanent, if it fails the machine reverts to the old entry on its own. This mechanism is the real counterpart of the "one-shot" row in the measurement. #### The Irreversible Line Some operations in this lesson are **irreversible**, and their commands are not written here in runnable, complete form. The line drawn is this: if an operation's result cannot be fixed without the system booting again, the command is not written in full. Three operations sit inside this line. First, **writing** the bootloader's first part to the disk's starting sector: given the wrong device name, that device's partition table area is overwritten and every partition on the disk loses its address. Second, **regenerating** bootloader configuration with a generator: the generator writes over the existing file, changes made by hand are lost, and the old file does not come back. Third, **deleting** a kernel entry assumed unused: if the deleted entry was the only working one, the way back disappears. What the three share is that the fault **is not visible at that moment**. The command returns successfully, the system keeps running, the failure surfaces at the next boot. At that moment there is no status query, no unit list, no log; there is only an error message sitting on the screen. The safe way to test these operations has three parts. **A copy of the configuration file is taken** and kept somewhere it can be restored from on the same machine. **The change is tried one-shot in the bootloader menu before being permanently written**; an edit made in the menu is not written to disk, it is valid only for that boot. **A new entry is added, the old entry is not deleted**, and the new entry is made permanent only after it has booted once. Taking a snapshot on a virtual machine and testing there is a fourth path that satisfies all three at once. #### Measurement: Twenty-Four Changes Twenty-four changes are made to the bootloader configuration. Each change's real outcome is one of four types: **valid** (does the intended thing), **typo** (not recognized, silently ignored), **conflicting** (recognized but clashes with another parameter and produces different behavior), and **unbootable** (the system never boots again). Three verification methods are compared: looking at the generator's output, reading the kernel command line after boot, and actually testing the behavior the parameter is supposed to change. **SV41 — the generator command writes all twenty-four changes successfully** and gives no warning on any of them. **SV42 — the change's type is chosen by a seeded generator.** **SV43 — a mistyped parameter shows up on the kernel command line** but has no effect at all. **SV44 — a conflicting parameter also shows up** and produces an effect other than the intended one. **SV45 — a change that leads to an unbootable state never shows up on the command line at all**, because the system never boots. **SV46 — command-line reading looks only at whether the string is present.** **SV47 — the behavior test measures the parameter's real effect** and is never wrong. **SV48 — an unbootable boot writes no line at all to the system log.** **SV49 — in persistent writing, every attempt that fails to boot requires recovery with external media.** **SV50 — in a one-shot attempt, a failed boot requires no recovery;** a reboot returns to the old entry. ```python SEED = 20260218 TYPE = ("valid", "typo", "conflicting", "unbootable") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def edits(seed=SEED, count=24): """Changes made to the bootloader configuration and their REAL outcomes.""" r = generator(seed) items = [] for i in range(count): a = r(100) kind = (TYPE[0] if a < 50 else TYPE[1] if a < 70 else TYPE[2] if a < 88 else TYPE[3]) items.append({"no": i + 1, "kind": kind, "boots": kind != "unbootable", "effective": kind == "valid", "visible_on_line": kind != "unbootable"}) return items def assess(d, method): """The 'is the change effective' answer of three verification methods.""" if method == "generator output": return True if method == "command line reading": return d["visible_on_line"] return d["effective"] def recovery(items, persistent): """In persistent writes, every unbootable attempt needs external-media recovery.""" unbootable = [d for d in items if not d["boots"]] return {"unbootable": len(unbootable), "recovery": len(unbootable) if persistent else 0, "log_lines": 0} D = edits() print("oracle:", len(D), "edits |", {t: sum(1 for d in D if d["kind"] == t) for t in TYPE}) print() print("verification method says effective really effective wrong diagnosis") for y in ("generator output", "command line reading", "behavior test"): said = sum(1 for d in D if assess(d, y)) wt = sum(1 for d in D if assess(d, y) != d["effective"]) print(f" {y:22s} {said:10d} {sum(1 for d in D if d['effective']):15d} {wt:11d}") print() print("write mode unbootable recovery events remaining log lines") for persistent, name in ((True, "persistent"), (False, "one-shot")): k = recovery(D, persistent) print(f" {name:12s} {k['unbootable']:9d} {k['recovery']:14d}" f" {k['log_lines']:25d}") ``` ``` oracle: 24 edits | {'valid': 11, 'typo': 8, 'conflicting': 2, 'unbootable': 3} verification method says effective really effective wrong diagnosis generator output 24 11 13 command line reading 21 11 10 behavior test 11 11 0 write mode unbootable recovery events remaining log lines persistent 3 3 0 one-shot 3 0 0 ``` #### Three Numbers **Oracle:** **11** of the twenty-four changes actually do their job; **8** are typos, **2** conflict, **3** leave the system unbootable. **Tool output:** the generator command reports all **24** changes as written; reading the kernel command line shows **21** changes as "applied"; the behavior test says **11**. **Wrong diagnosis:** trusting the generator's output is wrong **13** times, trusting command-line reading **10** times; the behavior test's wrong diagnosis is **0**. The middle row is this lesson's form of the course's second claim. Command-line reading is **more detailed** than the generator's output: it really looks at the system, really sees the string, really produces twenty-one lines. Its wrong-diagnosis count drops by only a third, and all of that drop comes from the three unbootable attempts dropping out on their own. The eight typos and two conflicts are still counted as effective by this method, because both show up on the command line. Only the third method closes the gap between showing up and doing the job. It is true that the third method is expensive, and that is why it gets skipped. Really testing that a parameter does its job requires measuring, one by one, the behavior that parameter is supposed to change: seeing that a disabled driver's device really does not show up, that message volume really dropped once the level was lowered, that changing the target really went to that target. Twenty-four changes mean twenty-four separate tests. Against that, reading the generator's output is one line. What the measurement says is that the cheap method leaves **thirteen** wrong diagnoses, and **ten** of those thirteen would never be noticed without a behavior test; the remaining **three** are noticed in the harshest way possible, at the next boot. **Output whose diagnosis is not tested counts as unmeasured.** This distribution also says how the work order should be set. The three changes that lead to an unbootable state are the most expensive, and what catches them is not the behavior test, it is the **one-shot attempt**: a machine that boots once and reverts exposes all three for free. The ten changes that quietly do nothing are caught only by the behavior test, and this test can be done after boot, while the system is up. The two methods do not substitute for one another; one tests the boot, the other tests the effect. An operator who does only one has done half the measurement. #### Where the Evidence Ends Up The second table pays the course's third claim. In persistent writing, three attempts leave the system unbootable, and all three require recovery with external media. In one-shot attempts, the same three attempts still fail to boot, but recovery events are **0**: a reboot returns to the old entry. The two rows' shared column is even more striking. In both cases, remaining log lines are **0**. An unbootable boot writes nothing to the system log, because the unit that keeps the log has not run yet, and in most configurations the file system the log would be written to is not even mounted yet. Evidence stays only on the screen; evidence left on the screen is erased at the next boot. **Looking late sees little;** here, looking late sees nothing at all. The right place for a bootloader test is therefore somewhere evidence can be recorded: a virtual machine whose console is recorded, a remotely accessed console, or a machine with someone standing in front of it. A test that produces no evidence counts as **unmeasured**, whatever its outcome. The recovery column alone is enough to tell the difference. In persistent writing, each of the three recovery events means physical or console-level access to the machine, booting from external media, and a manual fix; on a remote server this means a person going to where the machine is. In one-shot attempts, the same three failures cost no more than a single reboot. The change's content is the same in both, the oracle is the same in both; the only thing that differs is **how the attempt was written**. The cost of a wrong diagnosis depends less on the diagnosis itself than on the method that produced it. #### Second Seed Because change types are seeded, the distribution depends on the seed. With the second seed (`20260219`), twenty-four changes come out **12** valid, **5** typos, **5** conflicting, and **2** unbootable. Wrong-diagnosis counts come out at **13** and **12** for generator output, **10** and **10** for command-line reading, **0** and **0** for the behavior test. The order of magnitude holds and the ranking of methods is the same in both seeds. The count of unbootable attempts is **3** and **2**; because this number is small, it is mock-dependent and should not be generalized from a single run. What can be generalized is that **an unbootable attempt's log line count is zero in every run.** #### Summary - A boot entry has four parts: name, kernel image, initial ramdisk, and parameter string. A new kernel does not delete the old entry; the old entry in the menu is the cheapest way back. - The root device is written with a persistent identity; device names can change from boot to boot. - A misspelled parameter gives no error, shows up on the kernel command line, and is assumed applied because it shows up. - **11** of twenty-four changes do their job; trusting generator output produces **13** wrong diagnoses, command-line reading **10**, the behavior test **0**. More detailed reading fixes the diagnosis by only a third. - An unbootable boot leaves **0** lines in the system log; evidence stays on screen and is erased at the next boot. - Persistent writing produces **3** recovery events on three unbootable attempts, one-shot testing produces **0**; the change is tried in the menu before being permanently written, and the old entry is not deleted. #### Next Step This topic built the unit model, counted what the status word hides, measured what the policy conceals, worked out the boot chain, and showed how to test changes at the chain's start. The five lessons' shared conclusion fits in one sentence: **the line the service manager prints is a projection of the system's state, and a projection always carries loss.** The last lesson showed loss in its harshest form: evidence never being written at all. The next topic starts exactly here and examines the evidence that does get written — how the system log and service logs are collected, how they are queried, and how they vanish on their own through rotation. ## Logs and Diagnostics System and service logs, the authentication trail, the evidence rotation deletes, load average's lag, and separating bottleneck types. ### Log Architecture Source: https://academia.sh/en/courses/system-administration/log-architecture The distinction between the system log and service logs: where a record comes from, what level it is stamped with, and how many times a diagnosis looking at a single unit file turns out wrong. The previous topic followed the machine's boot from start to finish: firmware called the bootloader, the bootloader called the kernel, the kernel called the first process, and the service manager brought units up in dependency order. Every step of that chain did a job, but none of them did it in front of the operator's eyes. Once the machine is up, the only witness left is the written trail. This topic takes up that trail. The first question is architectural: how does a line enter the **log**, who stamps it, which address is it written to, and what can the person who opens that address not see. The last part of the question ties back to the course's rule: a log's value is not how many lines it holds but how many times the diagnosis drawn from those lines turns out wrong. The scope of this topic is a single machine. Pipelines that collect multiple machines' records in one place, stores that keep metrics, and alert chains that notify when a threshold is crossed were measured in the Observability and Operations and the Security Operations and Incident Response courses; they are not repeated here. The question here comes earlier: is the record sitting on the machine's own disk and in its own memory enough to say what happened on that machine. #### The Log's Three Sources On a single machine, log lines are born from three separate places, and the three do not take the same path. The first is the **kernel**. The kernel writes its own messages to a ring buffer in memory; when the buffer fills, the oldest line drops. Device detection, block device errors, and memory pressure warnings come from here. This buffer is not disk: a reboot empties it, so the previous boot's kernel messages are lost unless a persistent copy was taken. The second is the **service manager**. The standard output and standard error the manager gives a unit when it starts are each a pipe; whatever the unit writes to those two streams, the manager captures, stamps with the unit's name, and puts into the **system log (journal)**. A program needs no special library to write logs; printing to the screen is enough. The standard-output-versus-standard-error distinction built in the Shell Programming course finds its operational counterpart here: data comes from the first stream, diagnostic messages from the second, and the manager can stamp the two with different levels. The third is the **application itself**. A program can write to a file of its own choosing, in its own format. This file is outside the service manager's field of view: it survives even if the unit is stopped, it is not stamped with the unit's name, and its levels do not have to match the manager's levels. It is ordinary for the same event to sit in two different forms — once in the system log, once in the application's own file. What the three sources have in common is this: none knows what the others write. The practical consequence of the split is that there is **no guarantee an event's trail will be found at a single address**. #### Persistence: In Memory or on Disk Where the system log is written is a configuration decision, and the default behavior can vary by distribution. There are three options. The log can be kept **in memory only**; in that case a reboot erases the entire history, and once the machine recovers, no trace of the failure remains. The log can be written **to disk**; past boots become queryable, but the file does not grow without limit, and it drops the oldest entries once it reaches a size or duration limit. The third option is a mix of the two: to disk if the directory exists, to memory otherwise. This decision directly determines the diagnosis. In a log kept in memory, the failure record of a service that recovers by restarting itself **is erased along with its own recovery**. The situation measured in the Services topic comes up here a second time: a unit with an "always" policy restarted sixty times in six hundred seconds and looked "running"; if that unit has a failure that also reboots the machine, neither the state nor the record survives. For a log written to disk, the limit is size. Discarding the old part of a file that has reached the limit is called **rotation**, and this topic's third lesson is entirely devoted to counting what it deletes. For now, one sentence is enough: the evidence's lifetime can be shorter than the lifetime of the event that produced it. A fourth distinction is access. Log files are ordinary files too and follow the permission model built in the Introduction to Linux course; files holding identity events are mostly closed to unprivileged users. A record existing does not mean you can read it. #### Timestamp and Sequence The most trusted field of a log line is time, and yet it is also its most fragile field. There are two separate clocks: a **monotonic clock** counting time elapsed since the machine booted, and a **wall clock** showing calendar time. The first never runs backward. The second can: synchronizing with a time server, manual correction, and daylight-saving transitions can jump the wall clock forward or backward. The consequence is this: in a log sorted by wall clock, an event that actually happened earlier can appear **below** an event that happened later. Causality is the main tool of a failure investigation, and two reversed lines flip the diagnosis directly. This is why structured records keep, alongside the wall clock, the monotonic clock and a monotonically increasing **sequence number**; when sorting, trusting the counter is sturdier than trusting the wall clock. In this lesson's measurement and in the lessons that follow, time is the shared definition's **model second**: the observation window is six hundred seconds, and every duration is counted within that window. A real timestamp is written nowhere, because a real timestamp is machine-dependent and cannot be reproduced. #### Level, Field, and Format Every record has a **level**. The common ordering is eight steps, ranging from debug messages to emergency messages; three steps are enough for this topic: `info`, `warning`, `error`. The level is the writer's own declaration, not an independent measurement. If a program writes a fatal condition at the `info` level, every tool that filters by level drops that line. The level's second problem is the threshold. A system log mostly works with a floor: lines below the set level are not written. Keeping the debug level off shrinks the log, but the detail that would be most useful at the moment of failure sits exactly there. Opening the threshold after the failure does not bring the past back; the threshold determines what gets written while the event is happening. This is the plainest form of the course's third claim: **looking late sees little.** The difference between a structured record and plain text also shows up here. A plain text line is a single string; reading it requires parsing. A structured record consists of fields: unit name, level, process ID, time. Because fields are queryable, filtering rests on field comparison, not text matching. This distinction determines what will be measured in the next lesson: a text-matching filter and a field-aware filter give different answers to the same question. #### The Same Event, Two Addresses The system log keeps every unit's records in a single stream, in time order. The dump below is an example and has not been run; the number in brackets is the number of seconds elapsed since boot. ```text # example dump , not run [ 461.250] process[1043]: error: no response from queue [ 461.400] process[1043]: info: retry 1/5 [ 463.100] metrics[1092]: warning: sampling window exceeded [ 466.050] ingest[1014]: error: stream closed ``` If the same events are stored in files split per unit, the picture changes. The `process` unit's own file looks like this; this dump is also an example and has not been run. ```text # example dump , not run [ 461.250] error: no response from queue [ 461.400] info: retry 1/5 [ 527.500] error: no response from queue ``` The second dump has no `metrics` or `ingest` lines. What is lost is not a line but **the adjacency between lines**: the fact that `ingest`'s stream closed five seconds after `process`'s error sits only in the combined stream. The second difference between the two dumps is the stamp. In the combined stream, every line has the unit name in front of it; in the unit file, it does not, because the file's name already states the unit. This looks like savings while the file is small, but once two files are merged, which line came from where is lost. Merging dumps from different sources onto a single time axis is an ordinary step in a failure investigation; unstamped lines stay unidentified at that step. The third difference is that the service manager places the stamp **itself**. The unit name, process ID, and level come not from the line's content but from context the manager already knows. When a program writes to its own file, this context does not exist: the same event's counterpart in the application's own file carries no unit name, the process ID is only whatever the program chose to write, and the level is marked on the program's own scale. The same event is represented with two different levels of accuracy at two addresses. #### What Is Measured: A Diagnosis That Looks at One File The measurement is built on the shared definition's log generator. The setup's assumptions are these: the observation window is six hundred seconds, the log has 4000 lines, and level weights come from the shared definition (**GN1**); records are spread across six units (**GN2**); the adjacency window is five seconds (**GN3**); the operator decides by looking at the single file they opened (**GN4**). The oracle is how many errors the six units actually wrote; it is known because we built the setup. The tool output is the **single** unit file the operator opened. The diagnosis is the "this unit is the source of the error" decision made by looking at that file. ```python SEED = 20260218 PERIOD = 600 # observation window: 600 seconds def generator(seed): d = seed def next_val(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_val UNIT = { "ingest": {"requires": [], "restart": "always"}, "queue": {"requires": ["ingest"], "restart": "always"}, "process": {"requires": ["queue"], "restart": "on-failure"}, "report": {"requires": ["process"], "restart": "no"}, "metrics": {"requires": ["ingest"], "restart": "always"}, "backup": {"requires": [], "restart": "no"}, } def generate_log(seed=SEED, lines=4000): r = generator(seed) records = [] for i in range(lines): weight = r(100) level = "error" if weight < 3 else ("warning" if weight < 12 else "info") records.append({"seq": i, "second": i * (PERIOD / lines), "level": level, "unit": list(UNIT)[r(len(UNIT))]}) return records def unit_errors(records): d = {b: 0 for b in UNIT} for k in records: if k["level"] == "error": d[k["unit"]] += 1 return d def single_log_diagnosis(records): """GN4: operator looking at only ONE unit file: 'the source is this unit'.""" d = unit_errors(records) culprit = max(d, key=lambda b: d[b]) return {"actual_source": culprit, "silent_files": sum(1 for b in d if d[b] == 0), "tried": len(d), "wrong_diagnosis": sum(1 for b in d if b != culprit)} def neighboring_error(records, window=5.0): """GN3: errors that have an error from ANOTHER unit within five seconds.""" h = [k for k in records if k["level"] == "error"] return sum(1 for i, a in enumerate(h) if any(abs(b["second"] - a["second"]) <= window and b["unit"] != a["unit"] for j, b in enumerate(h) if i != j)) for seed in (SEED, 20260219): G = generate_log(seed) d = unit_errors(G) print("seed", seed, "| lines", len(G), "| errors", sum(d.values()), "| per unit", list(d.values())) print(" ", single_log_diagnosis(G), "| neighboring errors", neighboring_error(G)) ``` ``` seed 20260218 | lines 4000 | errors 86 | per unit [28, 0, 31, 0, 27, 0] {'actual_source': 'process', 'silent_files': 3, 'tried': 6, 'wrong_diagnosis': 5} | neighboring errors 50 seed 20260219 | lines 4000 | errors 163 | per unit [0, 50, 0, 51, 0, 62] {'actual_source': 'backup', 'silent_files': 3, 'tried': 6, 'wrong_diagnosis': 5} | neighboring errors 135 ``` #### What the Output Says Versus the System's Truth Three numbers sit side by side. **The oracle:** the 4000-line window has 86 errors, and 31 of them belong to the `process` unit; that is the real source. **The tool output:** the six unit files hold, in order, 28, 0, 31, 0, 27, and 0 errors. **Wrong diagnosis: 5.** The operator opens one of six files and says "this unit is the source of the error"; they are wrong on five of six tries. Three of these five misses are not the same kind. Three files are **completely silent**: they hold no error line at all. The operator who opens that file concludes "there is no problem here," and that conclusion, even if true for their own unit, is **wrong about the system** — the system has 86 errors. The remaining two files show errors but not the most; the operator there looks at the problem and blames the wrong unit. Silence and innocence are not the same thing. The second number measures adjacency: **50** of the 86 errors have a neighboring error from another unit within five seconds. None of these 50 lines appears next to its neighbor in a single unit file. Files split per unit do not lose lines; they lose **order**, and order is itself evidence. This yields an architectural decision: a per-unit file is useful when examining a single unit in depth; finding where a failure started requires a **combined, time-ordered** stream. Choosing one of the two addresses is not mandatory, but it must be known that which one is looked at determines the diagnosis. The limit of the measurement must also stand clearly. Here, the oracle is the setup itself; on a real machine, no party knows which unit is truly the source, and the only thing that tests the diagnosis is whether the repair made afterward actually works. What the setup provides is being able to see where the "five of six" number comes from: the number arises not from a flaw in the tool but from **the choice of address looked at**. If the same 4000 lines had been read in the combined stream, the error distribution would have been visible with a single query and the wrong diagnosis would have dropped to zero. The tool is not lying; what is missing is where the question was asked. The operational conclusion has two parts. First, a failure investigation starts not from the unit file but from the **combined stream**; the unit file is opened for detail once the source unit has been determined. Second, a silent file does not count as evidence: that unit writing no record at all shows only that it **did not write**, not that the unit is healthy. A failure in which a unit goes quiet and a window in which it has no problem at all are represented identically by the log. #### The Second Seed With the second seed, the setup changes: the error count rises to 163, the errors spread across different units, and the real source becomes `backup`. The wrong-diagnosis count stays **5 again**, the silent-file count stays **3 again**; in both seeds, only three units write errors. The neighboring-error ratio does change: in the first seed, 50 of 86 errors are adjacent; in the second, 135 of 163. The ratio itself **depends on the setup**; what does not depend on the setup is that a diagnosis looking at a single file is wrong on five of six, and that adjacency never appears in a single file. #### Summary - Log lines are born from three separate sources: the kernel's ring buffer, the standard output and standard error the service manager captures, and the application's own file. - Level is the writer's own declaration, not an independent measurement. Every tool that filters by level drops a line stamped wrong. - The oracle knows the real source of the 86 errors; a diagnosis looking at a single unit file is wrong on **five of six** tries, and three files are completely silent. - **50** of the 86 errors are adjacent to an error from another unit within five seconds; this adjacency is visible only in a combined, time-ordered stream. - Files split per unit do not lose lines, they lose order; order is evidence too. #### Next Step Every record looked at in this lesson was a message services produced themselves. A machine also keeps a separate set of events recording who logged in when, which account requested which privilege, and which request was denied. Those records fall into the same system log but are read differently: the question asked is not "which unit broke" but "who did it." The next lesson takes up authentication logs and counts how many lines a text-matching filter's answer to that question turns out wrong. ### Authentication Logs Source: https://academia.sh/en/courses/system-administration/authentication-logs Tracing login, privilege request, and session-close events: the distinction between authentication and authorization, how many lines text-matching filters get wrong, and why an attack's most important line is the successful one, not a failed one. The previous lesson established where a record comes from and which address it is written to; what it measured was a diagnosis looking at a single unit file being wrong on five of six tries. The question asked there was "which unit broke." A machine has a second set of events that falls into the same system log but serves an entirely different question: who logged in, what did they request, was it granted. This lesson takes up that set. The records themselves are ordinary; what is hard is the diagnosis drawn from them. In a failure investigation, a log line's accuracy is not in question, but in a security investigation, **which event** a line represents is in question, and two different events mostly produce two lines that look very much alike. One reason for the difference is the position of the party writing the record. A program writing service logs knows its own internal state; it can accurately describe what it did. The party writing identity events, by contrast, **does not know** who they are facing: who stands behind the incoming request is exactly what it is trying to measure. This is why an identity log is not an observation record but a **decision record**: it writes the decision the system made, not whether that decision was correct. #### Two Separate Questions, One File **Authentication** answers a single question: does this request genuinely belong to the account it claims. The answer is given with a password, a key, or some other proof, and its result is binary. **Authorization** comes after that: does the account, once its identity is proven, have the right to do the operation it is requesting. The distinction matters, because the two kinds of failure describe entirely different things. An authentication failure means "this person could not prove who they are"; a user who mistypes a password and an attacker trying passwords both produce this line. An authorization failure means "this person proved who they are, but they cannot do this operation"; here, who the account belongs to is not in doubt, the requested work is too much. The first line shows identity being forced from outside; the second shows a missing internal configuration, or an attempted privilege overreach. There is a third kind of event: **session** open and close. A session is the context that begins once authentication is accepted and lasts until it closes. A missing close record is ordinary: if the connection drops, the process is force-killed, or the machine reboots, there is an open record but no close record. A diagnosis that derives the open-session count from the log therefore counts **more** than the number of sessions actually open. Protecting these records against tampering, signing them, and turning them into an independent audit trail is a separate topic, covered in the Governance, Risk and Compliance course; it is not repeated here. The question here is narrower: what diagnosis follows from the lines sitting in the machine's own identity log. #### The Trail Privilege Elevation Leaves For an account to do work beyond its own privilege, an elevation step comes in between: a request is made, the rule set is checked, and an accept or deny record is written. The fields the record carries are the account making the request, the target identity, and the command to be run. What this line does not say is **why**. An accept record does not write which rule clause accepted the request; a deny record does not write which clause blocked it either. Two different configuration errors — the rule not existing at all, and the rule existing but being shadowed by a narrower one — produce the same deny line. A diagnosis that wants to tell these two apart has to look at the configuration, not the log. The log says an event **happened**; it does not say **why**. The second gap is the command itself. The record writes the command requested to run, but not what the command does. A one-line command that invokes a shell is recorded; what is done in that shell is not, because it happens inside a separate process. The subprocess model built in the Shell Programming course turns into a logging boundary here: the elevation record shows a **doorway**, not what happens behind it. The third gap is time. Elevated privilege mostly stays valid for a stretch without being asked for again. Second and third requests made within that stretch do not produce a new authentication record. The log can answer "how many times did the account prove its identity"; it cannot answer "how many times did the account do work with elevated privilege." Answering both questions with the same number is exactly the kind of wrong diagnosis this lesson measures. #### What the Lines Look Like Identity events are mostly collected in a separate file; the file's name varies by distribution, and the same lines can also be found in the system log. The dump below is an example and has not been run. ```text # example dump , not run [ 301.020] sshd[1207]: login failed: account backup, source remote [ 301.900] sshd[1207]: login failed: account backup, source remote [ 318.400] sudo[1214]: privilege check failed: account app, command backup-task [ 342.000] sshd[1207]: login succeeded: account backup, source remote ``` Three of the four lines carry the word "failed," but the three do not describe the same event. The first two are authentication failures from a remote source; the third is a local account requesting a command it has no privilege for. A text-matching filter throws all three into the same bucket. ```text # example dump , not run $ grep failed /var/log/auth.log [ 301.020] sshd[1207]: login failed: account backup, source remote [ 318.400] sudo[1214]: privilege check failed: account app, command backup-task ``` A field-aware query, by contrast, can restrict the event type, the account, and the source separately; this is the practical counterpart of the structured-record-versus-plain-text distinction built in the previous lesson. The text filter has one more fragility. The filter depends on the word the program writing the record happened to choose; a program that writes the same event with a different word drops silently out of the filter. If two programs on the same machine write the same event with two different phrasings, an operator searching with a single pattern never sees one of the two programs' records at all, and no warning arises to reveal that gap. A query made over fields does not remove this dependency, but it moves the dependency from the body of the text to **the field name**, and a field name changes far more slowly than message text. #### What Is Measured: Three Filters and One Field The measurement is made with an identity log built on the shared definition's generator. The setup's assumptions: the machine has five accounts (**GN5**); 560 background events are generated and event-type weights are fixed (**GN6**); the attack starts at second three hundred and contains forty failed remote logins followed two seconds later by one successful remote login (**GN7**); the field-aware filter's threshold is ten failed logins, its window sixty seconds (**GN8**). The oracle is the attack placed into the setup: forty failed lines and **one successful remote login** at the end of the sequence — 41 lines in total. The tool output is the line set each of four separate filters returns. Wrong diagnosis is the sum of lines read for nothing and attack lines never read. ```python SEED = 20260218 PERIOD = 600 # observation window: 600 seconds def generator(seed): d = seed def next_val(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_val ACCOUNT = ("root", "app", "backup", "monitor", "operator") # GN5 def auth_log(seed=SEED, background=560, attempts=40, target="backup", start=300): """GN6 background events + GN7 ORACLE: password-attempt sequence starting at second `start` and a SUCCESSFUL login at the end. Attack lines are marked.""" r = generator(seed) records = [] for _ in range(background): t = r(100) h = ACCOUNT[r(len(ACCOUNT))] remote = r(10) < 4 if t < 6: kind, result = "login", "failed" # password mistyped elif t < 18: kind, result = "privilege", "failed" # identity correct , no privilege elif t < 26: kind, result = "session", "closed" elif t < 34: kind, result = "privilege", "succeeded" else: kind, result = "login", "succeeded" records.append({"second": r(PERIOD), "account": h, "kind": kind, "result": result, "source": "remote" if remote else "local", "attack": False}) for i in range(attempts): records.append({"second": start + i, "account": target, "kind": "login", "result": "failed", "source": "remote", "attack": True}) records.append({"second": start + attempts + 2, "account": target, "kind": "login", "result": "succeeded", "source": "remote", "attack": True}) records.sort(key=lambda k: k["second"]) return records def account_window(records, threshold=10, window=60): """GN8 field-aware filter: an account's `threshold`-or-more failed remote logins within the window, plus any successful remote login following that cluster.""" selected = set() for h in ACCOUNT: d = [i for i, k in enumerate(records) if k["account"] == h and k["kind"] == "login" and k["result"] == "failed" and k["source"] == "remote"] cluster = set() for i in d: within = [j for j in d if 0 <= records[j]["second"] - records[i]["second"] <= window] if len(within) >= threshold: cluster.update(within) if not cluster: continue last = max(records[j]["second"] for j in cluster) cluster.update(j for j, k in enumerate(records) if k["account"] == h and k["kind"] == "login" and k["result"] == "succeeded" and k["source"] == "remote" and 0 <= k["second"] - last <= window) selected |= cluster return [records[j] for j in sorted(selected)] def measure(records, matched): """Wrong diagnosis = lines read for nothing + attack lines never read.""" actual = sum(1 for k in records if k["attack"]) correct = sum(1 for k in matched if k["attack"]) success = sum(1 for k in matched if k["attack"] and k["result"] == "succeeded") return {"matched": len(matched), "correct": correct, "success_found": success, "wrong_diagnosis": (len(matched) - correct) + (actual - correct)} FILTER = ( ("failed", lambda k: k["result"] == "failed"), ("failed login", lambda k: k["kind"] == "login" and k["result"] == "failed"), ("failed remote login", lambda k: k["kind"] == "login" and k["result"] == "failed" and k["source"] == "remote"), ) for seed in (SEED, 20260219): K = auth_log(seed) print("seed", seed, "| lines", len(K), "| attack lines", sum(1 for k in K if k["attack"])) for name, f in FILTER: print(f" {name:22s}", measure(K, [k for k in K if f(k)])) print(f" {'account window':22s}", measure(K, account_window(K))) ``` ``` seed 20260218 | lines 601 | attack lines 41 failed {'matched': 153, 'correct': 40, 'success_found': 0, 'wrong_diagnosis': 114} failed login {'matched': 67, 'correct': 40, 'success_found': 0, 'wrong_diagnosis': 28} failed remote login {'matched': 51, 'correct': 40, 'success_found': 0, 'wrong_diagnosis': 12} account window {'matched': 43, 'correct': 41, 'success_found': 1, 'wrong_diagnosis': 2} seed 20260219 | lines 601 | attack lines 41 failed {'matched': 156, 'correct': 40, 'success_found': 0, 'wrong_diagnosis': 117} failed login {'matched': 79, 'correct': 40, 'success_found': 0, 'wrong_diagnosis': 40} failed remote login {'matched': 60, 'correct': 40, 'success_found': 0, 'wrong_diagnosis': 21} account window {'matched': 47, 'correct': 41, 'success_found': 1, 'wrong_diagnosis': 6} ``` #### What the Output Says Versus the System's Truth Three numbers sit side by side. **The oracle:** the 601-line log has **41** lines belonging to the attack. **The tool output:** the crudest filter returns **153** lines, the narrowest text filter **51**, the field-aware filter **43**. **Wrong diagnosis:** **114**, **28**, **12**, and **2**, respectively. The numbers improve as the filter narrows, but the source of the improvement is not the line count. The filter searching for the word "failed" brings back 153 lines; only 40 of them belong to the attack, the remaining 113 are authorization denials and scattered password errors. The second filter, which restricts event type, brings wrong diagnosis down to 28; the third, which also restricts source, to 12. All three have something in common: **none finds more than 40 of the 41 lines.** The field-aware filter returns 43 lines and includes **all 41** of the attack's 41 lines; the two extra lines are the same account's other records within the window. Wrong diagnosis drops to 2. What produces the gain is not reading more lines but **which field** the lines are grouped by: not individual lines but the same account's density within a sixty-second window is queried. The two kinds of miss do not carry the same weight either. A line read for nothing wastes time; an attack line never read misdirects the diagnosis. In the measurement, the two are summed into one number, but their operational counterparts are separate. 113 of the crude filter's 114 wrong diagnoses are of the first kind: the result points the right way, it is just expensive. The three text filters' shared **1** shortfall is of the second kind, and it does not just cost more — it changes the result. Narrowing the filter has a limit too. The third filter, which restricts the source to "remote," would have found nothing if the attack had come from a local session; narrowing does not always improve — it only improves in this setup. A filter's accuracy comes not from its own logic but from where the thing it is looking for actually sits. #### The Most Important Line of Evidence The last column of the output carries the real point. The three text filters' `success_found` value is **0**, the field-aware filter's is **1**. That single **successful** login line following the forty failed attempts is the line that determines the event's outcome: if the attempts stayed failed, the event is noise; if the last one succeeded, the event is a breach. No filter searching for failure brings it back, because that line is not a failure. This is the form the course's rule takes in identity logs. However correct the answer to "how many failed logins are there" might be, it is not the question that determines the diagnosis. The correct question is: **after this attempt sequence, could that account log in.** The same distinction works in session records too: instead of counting sessions left open, which open has no matching close is asked. The shape of the question determines the shape of the filter too. A filter that searches for failure produces a **counter**; a counter is compared against a threshold, and the threshold is always debatable. A filter that searches for a sequence produces a **pattern**: the same account, the same source, a narrow window, and a result that changes at the end. The measurement's `account_window` function does exactly this; what brings wrong diagnosis down from 12 to 2 is not a smarter threshold but turning the question from a counter into a pattern. The pattern has a cost. The counter is computed in a single pass; the pattern requires a window scan per account and gets more expensive as the log grows. If the records sit in field-separated form, this scan can be done on the query side; if there is only plain text, parsing has to happen first. The cost of the format distinction built in the previous lesson is paid here: **plain text forces the question to be asked as a counter.** #### The Second Seed With the second seed, the background noise changes; the attack stays at 41 lines because it is placed into the setup that way. The crude filter brings back 156 lines and produces **117** wrong diagnoses, the event-type filter **40**, the source filter **21**, the field-aware filter **6**. The numbers grow, the ranking does not change, and the `success_found` column is again **1** only for the field-aware filter. The exact value of the wrong-diagnosis counts **depends on the setup**; what does not depend on the setup is that text matching structurally misses the successful line. #### Summary - Authentication failure and authorization failure are two separate events; a text-matching filter throws both into one bucket. - A missing session-close record is ordinary; a diagnosis deriving the open-session count from the log counts more than are actually open. - The oracle holds 41 attack lines; the crude filter brings back 153 lines and produces **114** wrong diagnoses, the field-aware filter brings back 43 lines and produces **2**. - None of the three text filters finds the attack's **successful** login line; the field-aware filter finds it, and that line is what separates the diagnosis between breach and noise. - The gain comes not from changing the line count but from grouping lines by account and time window. #### Next Step This lesson's filters worked with the entire log in hand: all 601 lines could be read. On a real machine this assumption does not always hold, because log files do not grow without limit. Once a file reaches a certain size, the old part is discarded, and the discarded part does not come back. The next lesson takes up filtering and rotation together and counts how far the 86 errors in a 4000-line log drop as the rotation size shrinks. ### Log Filtering and Rotation Source: https://academia.sh/en/courses/system-administration/log-filtering-and-rotation Measuring level, unit, and time filters together with the rotation policy: while 4000 lines have 86 errors, a 1000-line file keeps 19 and a 200-line file keeps 5, and how many of six investigation questions go unanswered is counted. The previous two lessons worked with the entire log in hand: both the 4000 lines and the 601 lines were complete, and the filters ran over that whole. This assumption holds on a machine only for a limited time. Log files do not grow without limit; disk is finite, and a full disk is a bigger failure than the log itself. This lesson takes up two mechanisms together. The first is **filtering**: selecting a subset from the lines you have. The second is **rotation**: permanently discarding part of the lines you have. The two look similar on the surface — fewer lines remain after either — but one is reversible and the other is not. What is measured is the damage each does to the diagnosis. #### The Context Filtering Drops There are three kinds of filter, and each cuts along a different axis. A **level filter** keeps only what is above a certain severity. A **unit filter** keeps only one service's records; this was what the previous lesson measured, and it produces adjacency loss. A **time filter** discards everything outside an interval. The level filter is the most commonly used and the most deceptive. A 4000-line log has 86 errors; a view restricted to errors reduces 4000 lines to 86 and makes them readable. But **what came before** the error is inside the 3914 lines that drop. The warning a service writes before it fails can say more than the failure itself: retry count, timeout duration, queue fullness. The level filter makes these invisible. The time filter's trap is different. Once a failure is noticed, the investigation mostly starts with a narrow window going backward from the moment it was noticed. The failure itself can be in that window, but the event that **started** it can fall outside it. Widening the window increases the line count and makes reading harder; this is one face of the course's second claim: more lines do not mean a better diagnosis, but fewer lines do not always give a clearer one either. The defining property of filtering is that it is reversible. If the filter was chosen wrong, the command is run again. There is no loss, only time spent. How the filter is built also depends on format. In a structured record, the three axes can be given separately; in plain text, lines are selected by text matching, and the time restriction is parsed out of the text itself. The two dumps below are examples and have not been run. ```text # example dump , not run $ journalctl -u process -p warning --since -10min $ grep -E 'error|warning' /var/log/process/process.log | tail -n 40 ``` The two lines do not do the same job. The first restricts unit, level, and time interval **as fields**; the second searches for two words in the message text and takes the last forty lines instead of a time restriction. The second silently drops the lines of a program that does not use the word "warning," and it does not say how many seconds "the last forty lines" spans. #### Rotation: A Cut That Cannot Be Undone Rotation is the mechanism that limits a file's growth. Once the file reaches the set size or age, it is renamed, a new one is opened in its place, and once the number of old copies kept is exceeded, the oldest is **deleted**. Compression delays the moment of deletion; it does not remove it. A **retention policy** consists of these three parameters: threshold, number of copies kept, and compression. The system log's own limit works similarly: once the total size or duration limit is exceeded, the oldest records are dropped. What the mechanism is called does not matter; the result is the same. The dump below is an example and has not been run. ```text # example dump , not run $ ls -1 /var/log/process/ process.log process.log.1 process.log.2.gz ``` The question here is: what happens if the file limit is shorter than the failure's duration. The answer is the course's third claim: **evidence disappears on its own.** Nobody deletes it, nobody decides; the policy runs, and the evidence goes. The sneakiest part of this loss is its silence. No marker is left in place of the deleted lines; there is no line at the top of the file saying "everything before this was discarded." What the operator sees on opening the file is not the look of a missing record but of **a short history**. With filtering, the situation is different: because the operator wrote the filter themselves, they know what they left out. With rotation, the operator is not the one who makes the choice, and the choice was written to a configuration file long before the failure. #### What Is Measured: Surviving Evidence and Unanswered Questions The measurement has two parts. The first part is the surviving-evidence ratio: how many of the errors at the start of the window survive as the file size shrinks. The second part is how many of the answers to the **six questions** genuinely asked in a failure investigation turn out wrong: total error count, the first error's second, the first error's unit, the last error's second, the error count in the window's first half, and the number of units that write no error at all. The oracle is the answers read from the full log. The setup's assumptions: a rotated file always keeps the **last** lines, and the discarded part does not come back (**GN9**); the sizes tested are 4000, 2000, 1000, 500, and 200 lines (**GN10**); the review question set is the six questions above (**GN11**); the precursor-warning window is five seconds, and the precursor must come from the same unit (**GN12**); line rate is constant across the window (**GN13**). ```python SEED = 20260218 PERIOD = 600 # observation window: 600 seconds def generator(seed): d = seed def next_val(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_val UNIT = { "ingest": {"requires": [], "restart": "always"}, "queue": {"requires": ["ingest"], "restart": "always"}, "process": {"requires": ["queue"], "restart": "on-failure"}, "report": {"requires": ["process"], "restart": "no"}, "metrics": {"requires": ["ingest"], "restart": "always"}, "backup": {"requires": [], "restart": "no"}, } def generate_log(seed=SEED, lines=4000): r = generator(seed) records = [] for i in range(lines): weight = r(100) level = "error" if weight < 3 else ("warning" if weight < 12 else "info") records.append({"seq": i, "second": i * (PERIOD / lines), "level": level, "unit": list(UNIT)[r(len(UNIT))]}) return records def rotate(records, size): """GN9: the file fills at `size` lines and the old part is deleted. What's left.""" return records[-size:] if size < len(records) else list(records) def evidence_ratio(records, remaining, level="error"): total = [k for k in records if k["level"] == level] left = [k for k in remaining if k["level"] == level] return {"total": len(total), "remaining": len(left), "ratio": round(len(left) / len(total), 4) if total else 0.0} def preceded_errors(records, window=5.0): """GN12: errors that have a warning from the SAME unit in the five seconds before them.""" u = [k for k in records if k["level"] == "warning"] h = [k for k in records if k["level"] == "error"] return sum(1 for a in h if any(a["unit"] == b["unit"] and 0 <= a["second"] - b["second"] <= window for b in u)) def answers(records): """GN11: the review questions, answered from the lines sitting in the file.""" h = [k for k in records if k["level"] == "error"] if not h: return {"total": 0, "first": None, "first_unit": None, "last": None, "first_half": 0, "silent_units": len(UNIT)} d = {b: sum(1 for k in h if k["unit"] == b) for b in UNIT} return {"total": len(h), "first": round(h[0]["second"], 1), "first_unit": h[0]["unit"], "last": round(h[-1]["second"], 1), "first_half": sum(1 for k in h if k["second"] < PERIOD / 2), "silent_units": sum(1 for b in d if d[b] == 0)} for seed in (SEED, 20260219): G = generate_log(seed) K = answers(G) print("seed", seed, "| lines", len(G), "| warning", sum(1 for k in G if k["level"] == "warning"), "| error", K["total"]) print(" level filter: errors hidden by a same-unit preceding warning", preceded_errors(G), "/", K["total"]) print(" oracle:", K) print(" size remain error evidence ratio wrong answer first error") for size in (4000, 2000, 1000, 500, 200): # GN10 remaining = rotate(G, size) er = evidence_ratio(G, remaining) Y = answers(remaining) print(f" {size:5d} {len(remaining):5d} {er['remaining']:4d} {er['ratio']:11.4f}" f" {sum(1 for a in K if Y[a] != K[a]):12d} {Y['first']}") ``` ``` seed 20260218 | lines 4000 | warning 408 | error 86 level filter: errors hidden by a same-unit preceding warning 63 / 86 oracle: {'total': 86, 'first': 6.1, 'first_unit': 'process', 'last': 598.0, 'first_half': 44, 'silent_units': 3} size remain error evidence ratio wrong answer first error 4000 4000 86 1.0000 0 6.1 2000 2000 42 0.4884 3 314.2 1000 1000 19 0.2209 4 461.2 500 500 12 0.1395 4 527.5 200 200 5 0.0581 4 573.4 seed 20260219 | lines 4000 | warning 314 | error 163 level filter: errors hidden by a same-unit preceding warning 99 / 163 oracle: {'total': 163, 'first': 1.3, 'first_unit': 'report', 'last': 594.0, 'first_half': 80, 'silent_units': 3} size remain error evidence ratio wrong answer first error 4000 4000 163 1.0000 0 1.3 2000 2000 83 0.5092 3 301.8 1000 1000 35 0.2147 3 451.5 500 500 12 0.0736 3 528.1 200 200 3 0.0184 5 578.2 ``` #### What the Output Says Versus the System's Truth Three numbers sit side by side. **The oracle:** the six-hundred-second window had **86** errors; the first at the sixth second, in the `process` unit. **The tool output:** the 1000-line file shows **19** errors, the 200-line file **5**. **Wrong diagnosis:** **4** of the six questions are answered wrong at 1000 lines, and **4** again at 200 lines. The surviving-evidence ratio falls together with rotation size: 0.4884 at 2000 lines, **0.2209** at 1000, 0.1395 at 500, **0.0581** at 200. Halving the file does not take half the evidence, it takes more than half; because records are not distributed evenly over time, the loss is not linear. The real damage is not in the number but in the number's **credibility**. The operator who opens the 200-line file sees five errors, and all five are genuine; none is fabricated. But the sentence they draw from it — "there were five errors in this window" — is wrong. The tool is not lying; the file is not lying. What is wrong is **drawing a complete conclusion from an incomplete file.** The table of six questions details this. In the full file, all six of the six answers are correct. At 2000 lines, three answers break: total error count, the first error's second, and the error count in the window's first half. From 1000 lines on, the first error's **unit** also comes out wrong: while the real first error is in the `process` unit, the first error remaining in the file belongs to another unit. The investigation answers "where did the failure start" by pointing to the wrong unit, and there is no sign anywhere in the file showing that answer is wrong. A notable detail is that the wrong-answer count does not grow past 1000: it stays at four. The reason is that the remaining two questions — the last error's second and the silent-unit count — can be answered from records near the end of the window. Rotation does not hit every question equally: **questions that look at the past die first, questions that look at the present survive.** Another detail is that the evidence ratio can preserve the error **rate** even while it does not preserve the error **count**. The 200-line file has five errors, and this, in terms of errors per line, is close to the full log's rate. A diagnosis that looks only at the rate sees nothing strange: the file looks like a healthy slice of a healthy system. What is lost is not the rate but the **duration**; how long the event lasted, when it started, and how it spread can only be read from the long window. The measure of filtering is in the same output too. **63** of the 86 errors have a warning from the same unit in the five seconds before them. A view showing only errors hides all 63 of these precursors; the lines sit in the file, but not where they are being looked at. The difference with filtering is this: relaxing the filter brings the 63 lines back; relaxing rotation does not bring back a line already deleted in the past. #### How Many Seconds of History a File Holds The rotation threshold is in lines, while what the investigation asks about is time. The conversion between the two is done with line rate, and in this setup the rate is taken as constant (**GN13**): 4000 lines in six hundred seconds, that is, 0.15 seconds per line. From this, the history each threshold holds follows directly. A 2000-line file holds 300 seconds, a 1000-line file 150 seconds, a 200-line file only 30 seconds of history. The output's last column confirms this calculation. In the 1000-line file, the first error seen is at second 461.2; the window the file covers starts at second 450. In the 200-line file, the first error is at second 573.4; the window started at 570. The real first error, however, is at second 6.1, and it is found in none of the small files. This conversion determines the question to ask when choosing a threshold. The question "how many lines should we keep" has no meaningful answer; the meaningful question is "how many seconds of history are needed," and if the line rate is known, the conversion between the two is a multiplication. Line rate is not constant either: services write more at the moment of failure, so the file fills exactly **faster** at the moment producing the most evidence, and rotates faster. The moment evidence concentrates is the moment evidence is deleted fastest. #### What the Policy Means This yields two decisions on the operations side. First, the rotation threshold looks like a storage decision but is actually an **investigation decision**: the threshold should not be shorter than the longest reasonable duration of a failure. The surviving-evidence ratio is the measure of this and can be computed in advance; if the log's line rate is known, how many seconds of history a threshold holds is a division. Second, "keep everything just in case" is not a solution. If the disk fills, writing fails, and that failure is a worse problem than the lost evidence. The Storage topic will show that this limit comes in two separate forms. The right decision is to decide in advance which question needs to stay answerable and to choose the threshold for that question. A third decision is **when** rotation happens. Time-based rotation runs at fixed intervals and rotates even if the file stays small; size-based rotation runs when the file fills and may never rotate on a quiet machine. When the two are used together, whichever triggers first wins. The result is this: how much history a threshold holds varies with how talkative the machine is, and on a busy day it is shorter than expected. The fourth decision is how rotation is done. If the file is renamed and a new one opened in its place, a process still holding the file open keeps writing to the old file, and the new file stays empty; this is why, after rotation, a signal is sent to the writer telling it to reopen the file. The signal model built in the Process Management topic is part of a logging mechanism here. If the signal is not sent, the failure is silent: files rotate, the size limit works, but records keep falling into a file nobody is looking at anymore. #### The Second Seed In the second seed, the log is noisier: 163 errors and 314 warnings come out. The evidence ratio falls similarly — 0.2147 at 1000 lines, 0.0184 at 200. The wrong-answer count is **3** at 2000, 1000, and 500 lines, **5** at 200. The exact values **depend on the setup**; what does not depend on the setup is that the surviving-evidence ratio falls rapidly with rotation size and that questions looking at the past go unanswered first. #### Summary - Filtering is reversible, rotation is not; both reduce lines, but only one produces no loss. - While 4000 lines have 86 errors, a 1000-line file keeps **19** (0.2209) and a 200-line file keeps **5** (0.0581); the loss is not linear. - **4** of six review questions are answered wrong at 1000 lines, and one of them is "where did the failure start"; there is no sign of the wrongness anywhere in the file. - Rotation does not hit questions equally: questions looking at the past die first, questions looking at the present survive. - **63** of the 86 errors have a preceding warning from the same unit; the level filter hides them, but relaxing the filter brings them back. #### Next Step The loss measured in this lesson belonged to the past: a deleted line did not come back. The second kind of data describing a machine's state is not lines but numbers — load, memory, disk fullness — and there, loss shows up in a different form. These numbers do not belong to a single moment; they are produced by averaging a past window, and the average smooths the spike. The next lesson takes up system health metrics and shows numerically that load average is a lagging indicator: while the instantaneous load is 7.36, the sixty-second average stays at 6.22. ### System Health Metrics Source: https://academia.sh/en/courses/system-administration/system-health-metrics What load average, memory, and disk indicators say and do not say: while the instantaneous load is 7.36, the sixty-second average stays at 6.22, and the averaged indicator never shows the real load that crosses the threshold for 54 seconds. The previous lesson measured a loss belonging to the past: in a rotated log, a deleted line did not come back. The second kind of data speaking about a machine's state is not lines but numbers. Load, memory, disk fullness; each is presented as a single number, and that number is the first place looked at during a failure. This lesson takes up how these numbers are produced. The problem that shows up belongs to the same family as rotation's, but comes from a different mechanism: no line is deleted, **the number is averaged**. An averaging indicator mixes the past into the present, and the spike dissolves inside the average. What is measured is how many seconds that dissolving keeps the diagnosis wrong for. A distinction has to be made from the start. A log line records an **event**: the trail of something that happened sits on disk and can be read later. A health indicator, by contrast, reads a **state**: it is produced at the moment it is read and leaves no trace for the moment it is not read. A log's absence is explained by rotation; an indicator's absence has no explanation, because a moment that was not recorded cannot be told apart from a moment that never happened. #### What Load Average Counts **Load average** is the average, over a time interval, of the number of processes that are ready to run or waiting uninterruptibly. Two points should be stated right away. First, this is not a percentage: it describes not processor occupancy but **the amount of work queued up**. Second, a load average always comes with a window; there is no such number as "the load" on its own. The common presentation places three windows side by side, and the three describe the same moment with three different histories. Interpreting the number depends on core count. The same number can mean saturation on a single-core machine and comfort on a multi-core one. This is why there is no absolute threshold; the threshold is set per machine, and setting it uses the form **divided by core count**. The scope of what enters the number can also mislead. Load average counts not only processes waiting on the processor but also processes sitting in uninterruptible I/O wait. The result is this: on a machine whose processor is nearly idle but whose disk is jammed, load average reads high and produces a "the processor is not enough" diagnosis. The three bottleneck types the next lesson tries to separate are all collected inside this one number. The third point, and the one this lesson measures, is this: load average is a **lagging indicator.** Its value gives not the state at the moment being looked at but the average of the window up to that moment. When a spike begins, the indicator rises slowly; when the spike ends, it falls slowly. The gap in between is the truth the average misses. The dump below is an example and has not been run; the three numbers are, in order, the one-, five-, and fifteen-minute windows. ```text # example dump , not run $ uptime load average: 6.22, 6.19, 6.08 ``` #### What Is Measured: The Spike the Average Suppresses The measurement is built on the shared definition's process setup. The setup's assumptions: load is the sum of twenty-four processes' second-by-second usage, presented divided by a hundred (**GN14**); measurement is done only from the sixtieth second onward, once the window has filled (**GN15**); the threshold is the value separating the busiest ten percent of the real load (**GN16**); the averaging windows tested are 15, 60, and 300 seconds (**GN17**); memory per process is constant across the window (**GN18**). The oracle is the **real** load at every second; it is known second by second because we built the setup. The tool output is the same load averaged over a given window. The diagnosis is the decision "the system is at its busiest right now." Wrong diagnosis is the number of seconds the indicator misses this decision. ```python SEED = 20260218 PERIOD = 600 # observation window: 600 seconds def generator(seed): d = seed def next_val(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_val ACCOUNT = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-task", "measurement-collector", "queue-worker", "cache-cleaner") def processes(seed=SEED, count=24): """Every process: real CPU usage is a TIME SERIES, not a single number.""" r = generator(seed) result = [] for i in range(count): kind = r(10) if kind < 5: pattern = "flat" # steady low usage elif kind < 8: pattern = "burst" # short , high spikes else: pattern = "heavy" # steady high usage result.append({ "pid": 1000 + i * 7 + r(5), "account": ACCOUNT[r(len(ACCOUNT))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2, # some processes ignore termination }) return result def usage(process, second): """Real CPU usage, second by second. The oracle knows this; the tool does not.""" d = process["pattern"] if d == "flat": return 3 + (process["pid"] + second) % 4 if d == "heavy": return 60 + (process["pid"] + second) % 25 # burst: a 5-second spike every 60 seconds phase = (second + process["pid"]) % 60 return 88 + phase if phase < 5 else 2 + phase % 3 def actual_load(processes_, second): return sum(usage(s, second) for s in processes_) def load_average(processes_, second, window=60): """Load average is a LAGGING indicator: it averages the past window.""" start = max(0, second - window) span = range(start, second + 1) return round(sum(actual_load(processes_, t) for t in span) / len(list(span)) / 100, 2) def memory_share(processes_): """GN18: the tool lists the 'top five memory-using processes'. How much does this explain.""" values = sorted((s["memory"] for s in processes_), reverse=True) cumulative, count = 0, 0 for v in values: cumulative, count = cumulative + v, count + 1 if cumulative / sum(values) >= 0.20: break return {"total": sum(values), "largest_share": round(values[0] / sum(values), 4), "top_five_share": round(sum(values[:5]) / sum(values), 4), "for_20_percent": count} for seed in (SEED, 20260219): S = processes(seed) span = range(60, PERIOD) # GN15: after the window fills instant = {t: actual_load(S, t) / 100 for t in span} ordered = sorted(instant.values()) threshold = ordered[int(len(ordered) * 0.9)] # GN16 oracle: busiest ten percent print("seed", seed, f"| instant load {min(instant.values()):.2f}-{max(instant.values()):.2f}" f" | threshold {threshold:.2f} | seconds over threshold" f" {sum(1 for v in instant.values() if v >= threshold)}") print(" second 0/60/300 instant:", [round(instant.get(t, actual_load(S, t) / 100), 2) for t in (0, 60, 300)], "| average(60):", [load_average(S, t) for t in (0, 60, 300)]) print(" window lowest highest over threshold missed false alarm") for p in (15, 60, 300): # GN17 series = {t: load_average(S, t, p) for t in span} missed = sum(1 for t in span if instant[t] >= threshold and series[t] < threshold) extra = sum(1 for t in span if series[t] >= threshold and instant[t] < threshold) print(f" {p:6d} {min(series.values()):6.2f} {max(series.values()):7.2f}" f" {sum(1 for v in series.values() if v >= threshold):14d}" f" {missed:6d} {extra:11d}") print(" memory:", memory_share(S)) ``` ``` seed 20260218 | instant load 5.36-7.80 | threshold 7.15 | seconds over threshold 54 second 0/60/300 instant: [7.36, 7.31, 7.36] | average(60): [7.36, 6.25, 6.22] window lowest highest over threshold missed false alarm 15 5.62 6.80 0 54 0 60 6.18 6.25 0 54 0 300 6.13 6.29 0 54 0 memory: {'total': 4876, 'largest_share': 0.0779, 'top_five_share': 0.3659, 'for_20_percent': 3} seed 20260219 | instant load 4.04-6.29 | threshold 6.04 | seconds over threshold 54 second 0/60/300 instant: [4.33, 4.33, 4.33] | average(60): [4.33, 4.85, 4.85] window lowest highest over threshold missed false alarm 15 4.52 5.23 0 54 0 60 4.83 4.90 0 54 0 300 4.80 4.88 0 54 0 memory: {'total': 5444, 'largest_share': 0.0755, 'top_five_share': 0.3611, 'for_20_percent': 3} ``` #### What the Output Says Versus the System's Truth Three numbers sit side by side. **The oracle:** the real load fluctuates between 5.36 and 7.80 and stays above the 7.15 threshold for **54 seconds**. **The tool output:** the sixty-second average sits between 6.18 and 6.25; it **never** crosses the threshold. **Wrong diagnosis: 54.** In every second the system is at its busiest, the indicator says "normal." The false-alarm count is zero, and this makes the indicator's silence even more dangerous. If the indicator were noisy, the operator would learn not to trust it; this indicator is not wrong, it just **never speaks.** An averaged series produces no false alarms, because it produces no spikes in the first place. Where the threshold comes from should also stand clearly. Here, the threshold is not a number chosen from outside; it is the value separating the busiest ten percent of the real load, so by definition 54 seconds exceed it. What is measured is not whether the threshold was chosen correctly but **whether it is possible to see the same event from an averaged series at all.** The answer is no, and lowering the threshold does not fix it: once the threshold is brought inside the average's band, the indicator crosses it constantly instead and cannot distinguish any second from another. The second line of the template shows the lag directly. At second zero, instantaneous load is 7.36 and the average is 7.36 too: because the window has not filled yet, the two are the same. At second sixty, instantaneous is 7.31 while the average has dropped to 6.25; at second three hundred, instantaneous is again **7.36** while the average is **6.22**. The gap between the two numbers at the same moment is not an error; the two measure different things. The problem is that the two are presented under the same name. The effect of window width is also read from the table. Instantaneous load has a range of 2.44 units (5.36 to 7.80). At the fifteen-second average, this range narrows to 1.18; at sixty seconds, to **0.07**. At the three-hundred-second window, it rises to 0.16, but it still never crosses the threshold; the reason for the rise is not window length but the first seconds during which the window has not filled. The reason the sixty-second window gives the flattest series is that the spikes in the setup repeat on a sixty-second cycle: when a window covers an exact multiple of the cycle, it dilutes the spike by the same amount every time. This is a situation encountered on real machines too. A recurring job — a scheduled task, a batch transfer, a cache cleanup — leaves no trace at all in the indicator if it runs at the same rhythm as the averaging window. The reason for the invisibility is not that the job is small, but that **the rhythms overlap.** The rule that follows is this: load average is not a **spike** indicator, it is a **trend** indicator. It does not answer "what is happening right now"; it answers "what was the general direction over the last minute." What was measured for the sampling interval in the Process Management topic is repeated here for the averaging window: the indicator is not lying, the question asked is a different one. #### Three Windows Side by Side There is a reason load average is presented with three windows: a single number gives level, three numbers give **direction**. If the short window is greater than the long one, load is rising; if smaller, it is falling; if the three are close to each other, the state is steady. This reading does not remove the average's lag, but it makes the sign of the lag visible. The direction itself determines the diagnosis too. In a rising trend, the real question is not "has the current value crossed the threshold" but "at this rate, when will it cross." In a falling trend, a high value is the residue of a past event, and reacting to it means chasing a failure that has already ended. This second situation is common in operations: the indicator reads high, an investigation begins, and by the time the investigation ends the indicator has come down on its own; nobody learns what happened. The limit of the three-window reading also sits in the measurement. The spikes in the setup last five seconds; the shortest window is fifteen seconds. A five-second event shrinks to a third in a fifteen-second average and leaves too little of a mark in any of the three windows to change direction. Reading direction requires the event to last **longer than the shortest window**. The way to see short-lived events is not to read the average but to read an unaveraged counter: runnable process count, context-switch count, and interrupt count all give an instantaneous value on every read. These counters are noisy and a single reading of them is misleading, but they carry no lag. The choice between a lag-free, noisy counter and a lagging, smooth average is this lesson's real decision; no indicator gives both at once. #### Memory and Disk Indicators The memory indicator's trap is different. A tool lists "the top five memory-using processes," and that list is the easiest output to read. The measurement counts how much of the picture this list explains: the largest process holds only **7.79 percent** of total memory, the top five together **36.59 percent**. The remaining nineteen processes hold 63 percent, and none of them is on the list. The decision "kill the largest process" recovers less than eight percent of the total; freeing twenty percent requires **three** processes. The list is not wrong; it just does not answer the question being asked. In this setup, the answer to "who is consuming the memory" is not a single name but a long queue. The second trap is that the operating system does not leave free memory idle. Files that have been read are held in cache and released when needed; this memory looks "in use" but produces no pressure. A diagnosis that looks only at percentage used mistakes a healthy machine for a full one. The distinguishing indicator is not the percentage used but the portion that is **not reclaimable**. The third trap is that memory reported per process is not summable. Shared libraries and shared pages are counted separately in every process's own line; a calculation that sums process lines finds more memory than exists on the machine. A total drawn from the process list should not be used without comparing it against the system-level total. There is a similar duality on the disk side too, and it goes beyond this topic's scope. Percentage full is a single number, whereas in a file system **two** independent limits get exhausted and both produce the same error message. The Storage topic will measure this distinction; for now, what needs to be known is that a low percentage full **does not guarantee** a write will succeed. Another property of the disk indicator is that it is insensitive to the moment of measurement. Load changes within seconds, fullness within hours. This does not make the disk indicator more reliable; it just gives a different kind of lag. What is dangerous in a filling disk is not the current percentage but **the rate of filling**; a rate cannot be derived from a single reading, and unless the difference between two readings is taken, the indicator does not say how many hours remain until failure. #### The Second Seed In the second seed, the setup's load level drops: instantaneous load fluctuates between 4.04 and 6.29 and the threshold falls to 6.04. Even though the numbers change, the structure stays the same — the real load again crosses the threshold for **54 seconds**, the averaged indicators again **never** cross it, and wrong diagnosis is again **54**. The sixty-second average's range stays at 0.07 units. The absolute value of the threshold and the load **depends on the setup**; what does not depend on the setup is that the average completely suppresses the spike and that the false-alarm count is zero. #### Summary - Load average is not a percentage; it is the average, over a window, of the amount of work queued up; interpreting it depends on core count. - While the real load crosses the threshold for **54 seconds**, the sixty-second average **never** crosses it; wrong diagnosis 54, false alarms 0. - Instantaneous load's 2.44-unit range narrows to **0.07** in the sixty-second average; widening the window calms the indicator, and a calmed indicator cannot show a spike. - The largest memory-using process holds only 7.79 percent of the total; the top five hold 36.59 percent, and freeing twenty percent requires three processes. - Load average is a trend indicator; it cannot answer "what is happening right now," and when forced to, it stays silent. #### Next Step This lesson's indicators reduced the entire system to a single number, and the reduction ate the spike. When a performance problem is investigated, the question asked is even more specific: is the slowdown caused by the processor, the disk, or memory. The three causes look very similar to each other in most tool output, and raising the level of detail does not reduce the similarity. The next lesson takes up separating the three bottleneck types and counts how a view producing four times as many lines never changes the wrong-diagnosis count at all, while a three-line view does. ### Tracing Performance Problems Source: https://academia.sh/en/courses/system-administration/tracing-performance-problems Separating CPU, I/O, and memory bottlenecks: a view producing four times as many lines is still wrong on 12 of 18 events, a three-line view drops to 2, and the reading order of the same three metrics removes the remaining two misses. The previous lesson took up indicators that reduce a machine's state to single numbers and measured that the reduction eats the spike. One gap remained there: load average collected processes waiting on the processor and processes waiting on disk **inside the same number**. When a machine slows down, the question asked requires exactly separating that sum: is the slowdown caused by the processor, the disk, or memory. This lesson takes up separating the three bottleneck types. The expected solution is more detailed output: more columns, more lines, finer detail. What is measured is whether this expectation is correct, and the course's second claim is paid here: **more output does not mean a better diagnosis.** Where the expectation comes from is understandable too. Facing a failure, the only control knob at hand is mostly the level of detail: a longer list, a finer breakdown, more frequent sampling. All of these grow the output, and growing output gives a feeling that effort is increasing. The measurement tests this feeling, and the result is clear: if the direction of the effort is wrong, its amount does not change the outcome. #### Three Bottlenecks and Their Shared Symptoms A machine slowing down can arise from three mechanisms, and the three look the same to the user. In a **CPU bottleneck**, the number of runnable processes exceeds the number of processors; processes queue up, each waiting for its turn. Processor busy percentage is high, and that high value corresponds to real work. In an **I/O bottleneck**, processes wait not on the processor but on disk. While waiting, they are in uninterruptible wait; they do not consume the processor but they are counted. What shows up on the processor side is the **wait** share: the processor is idle, but the reason it is idle is not that no work is left but that data has not arrived. A view that reports busyness by folding the wait share into it cannot tell this situation apart from a CPU bottleneck. In a **memory bottleneck**, physical memory runs short and pages are moved to swap. The trap here is two-fold. First, swap movement is itself a disk operation; heavy swapping **produces I/O wait**, and the memory bottleneck looks like a disk bottleneck. Second, memory indicators stay high not just during the pressure but after it has passed too: pages that went out to swap sit there until recalled. There is a fourth possibility, and it falls outside this trio: the bottleneck can be not in the machine itself but in a counterpart it is waiting on. A process waiting on a network request is slow too, but it leaves no trace in any of the three metrics; the processor is idle, the disk is idle, memory is comfortable. This lesson does not measure that situation, and it is the one scenario where the machine is slow even though all three metrics say "normal." Diagnosis on the network side is the subject of the next course. The Operating System Concepts course built the mechanism of paged memory and scheduling; it is not repeated here. The question here is not the mechanism but **which mechanism the operator can read from which output.** The same distinction was set up at the opening of this course: measuring an abstraction's cost and measuring whether the operator can see that abstraction are two separate jobs. #### Five Views The measurement compares five separate views, and each produces a different number of lines. **A — process list.** Twenty-four lines, one meaningful column: CPU percentage per process. This is the most common first look. **B — thread detail.** The same list, expanded to thread level: ninety-six lines. Four times as much output, **the same columns**. The level of detail has gone up; the metric set has not changed. **C — A plus a wait column.** Twenty-four lines again, but the per-process I/O wait share has been added to the list. The line count is the same as A, the metric set is one unit larger. **D — three system metrics.** Three lines: CPU busy, wait share, swap movement. No per-process detail at all. **E — the same three metrics, a different reading order.** The exact same three lines as D; the only difference is which metric is checked first. D looks at wait share first, E at swap movement. The dump below is an example and has not been run; it shows the three metrics view D carries. ```text # example dump , not run $ vmstat 1 1 r b swpd free ... si so ... us sy id wa 4 6 102400 81920 ... 38 44 ... 61 12 5 22 ``` #### Where These Three Metrics Are Read From Each of the three metrics comes from a separate counter, and which tool reads them is a secondary question; what matters is the counter itself. **CPU busy** is computed from the time buckets the kernel keeps: time spent in user mode, time spent in kernel mode, idle time, and time spent waiting. These buckets are **cumulative** counters increasing since boot; the instantaneous percentage comes from the difference between two readings. A single reading gives the average since boot, and that average says almost nothing during a failure. This is another form of the lag problem from the previous lesson: the counter itself does not lag, but a single reading **averages everything**. **Wait share** is the name of one of the same buckets, and it means exactly "the processor was idle, but at least one process was waiting on disk." Two warnings are needed. First, this is a duration measure, not a queue measure; it does not say how many processes are waiting. Second, if the processor is genuinely busy, wait share **reads low**, because the processor has no idle time. That is, when the processor and the disk hit a bottleneck at the same time, wait share understates the size of the problem. **Swap movement** is counted in two directions: pages read back into memory, and pages written to disk. The distinction matters. Writing to disk alone is not evidence of pressure; the operating system can move pages untouched for a long time out to disk even while things are comfortable. The evidence of pressure is pages being **read back**: a page in use was evicted and immediately requested again. A diagnosis that looks only at whether the swap area appears full searches the present for a pressure event that happened and ended in the past. #### What Is Measured: Line Count or Metric Set The setup's assumptions: eighteen events are generated, and each event has a single real cause (**GN19**); the three metrics' bands are derived from the cause (**GN20**); once swap movement rises above sixty, memory pressure also produces I/O wait (**GN21**); the views' lines-per-event counts are 24, 96, 24, 3, and 3 (**GN22**); the decision thresholds are 60 for CPU, 25 for wait, 20 for swap (**GN23**). The oracle is the real bottleneck cause of the eighteen events placed into the setup. The tool output is the line count each of the five views produces. The diagnosis is the cause each view derives from its own metrics; wrong diagnosis is the count of diagnoses that come out different from the real cause. ```python SEED = 20260218 def generator(seed): d = seed def next_val(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_val CAUSE = ("cpu", "io", "memory") def bottlenecks(seed=SEED, count=18): """GN19 ORACLE: every event's single real bottleneck. GN20: the three metrics are derived from the cause. GN21: swap over 60 also produces wait.""" r = generator(seed) events = [] for i in range(count): n = CAUSE[r(3)] if n == "cpu": cpu, swap, wait = 78 + r(18), r(3), 2 + r(4) elif n == "io": cpu, swap, wait = 74 + r(20), r(3), 46 + r(28) else: cpu, swap = 52 + r(30), 34 + r(40) wait = 30 + r(20) if swap >= 60 else 6 + r(9) events.append({"no": i + 1, "cause": n, "cpu": cpu, "wait": wait, "swap": swap}) return events def view_a(o): """GN23 thresholds. Process list , single metric: CPU percentage.""" return "cpu" if o["cpu"] >= 60 else "memory" def view_c(o): """Same list , one more column: wait.""" if o["wait"] >= 25: return "io" return "cpu" if o["cpu"] >= 60 else "memory" def view_d(o): """Three system metrics , wait read first.""" if o["wait"] >= 25: return "io" return "memory" if o["swap"] >= 20 else "cpu" def view_e(o): """Same three metrics , swap read first.""" if o["swap"] >= 20: return "memory" return "io" if o["wait"] >= 25 else "cpu" VIEW = (("A process list", view_a, 24), # GN22: lines/event ("B thread detail", view_a, 96), ("C A + wait column", view_c, 24), ("D three system metrics", view_d, 3), ("E same three metrics , swap first", view_e, 3)) for seed in (SEED, 20260219): O = bottlenecks(seed) print("seed", seed, "| events", len(O), "|", {n: sum(1 for o in O if o["cause"] == n) for n in CAUSE}) print(" view lines/event total lines wrong diagnosis") for name, f, lines in VIEW: y = sum(1 for o in O if f(o) != o["cause"]) print(f" {name:30s} {lines:10d} {lines * len(O):13d} {y:12d}/{len(O)}") ``` ``` seed 20260218 | events 18 | {'cpu': 4, 'io': 4, 'memory': 10} view lines/event total lines wrong diagnosis A process list 24 432 12/18 B thread detail 96 1728 12/18 C A + wait column 24 432 8/18 D three system metrics 3 54 2/18 E same three metrics , swap first 3 54 0/18 seed 20260219 | events 18 | {'cpu': 7, 'io': 6, 'memory': 5} view lines/event total lines wrong diagnosis A process list 24 432 11/18 B thread detail 96 1728 11/18 C A + wait column 24 432 5/18 D three system metrics 3 54 2/18 E same three metrics , swap first 3 54 0/18 ``` #### What the Output Says Versus the System's Truth Three numbers sit side by side. **The oracle:** four of the eighteen events are CPU-caused, four I/O-caused, ten memory-caused. **The tool output:** the views produce, in order, 432, 1728, 432, 54, and 54 lines. **Wrong diagnosis:** 12, 12, 8, 2, and 0. Comparing the first row against the second pays off the claim. View B produces exactly **four times** as many lines as A — 1728 instead of 432 — and the wrong-diagnosis count **does not change at all**: it stays at twelve. Raising the level of detail grew the text to be read; it corrected the diagnosis by not one bit. The reason is clear: the thread breakdown is the same metric sliced finer, and slicing a metric finer cannot see what that metric could not see in the first place. The third row shows what makes the difference. View C produces a quarter as many lines as B — the same as A, 432 — but wrong diagnosis drops from 12 to **8**. What produces the gain is not the line count but **a single column** added to the list: wait share. The correction of four wrong diagnoses comes from I/O bottlenecks becoming separable from CPU bottlenecks. The fourth row takes this to the extreme. View D produces **54 lines**; a thirty-second of B. Wrong diagnosis drops to **2**. Three lines carrying no per-process detail at all give six times better diagnosis than a view carrying ninety-six lines per process. This is the measurement's name: **what separates them is not the line count but which metric is looked at.** The gap between C and D also shows where scale fits in. C looks per process and can answer "which process"; D looks at the system as a whole and cannot answer that question. Yet D is wrong less often, because the question being asked is not "which process" but "which resource." The two questions are frequently mixed up: searching for the process consuming the most resource without first finding the source of the slowdown often ends up finding **the victim**. On a machine under memory pressure, the process waiting the most is not the process creating the pressure; it is the one most affected by it. Choosing a view also has a consequence for order. The correct sequence is to look at the system as a whole first, then the process: D says "memory," and then the process list is sorted by memory and the culprit is sought. The reverse sequence — process list first, system metric second — is represented in the measurement by row A, and it takes the wrong path on twelve of eighteen events. #### The Remaining Two Misses and Reading Order The fifth row pays off the remaining share. View E carries **the exact same three metrics** as D; the only difference is which one is asked first. D looks at wait share first and says "I/O" when it sees high wait. E looks at swap movement first and says "memory" when it sees high swap. Wrong diagnosis is 2 for D, **0** for E. The events producing the difference are memory bottlenecks whose swap rises above sixty. There, because swap movement itself creates disk work, wait share rises too; the two metrics are high at the same time, and which one is read first determines the diagnosis. Wait share is a **result**, swap movement is a **cause**; a reading order that reads the result first shadows the cause. This yields a rule for the operations side: if the causal direction between two metrics read high at the same time is not known, growing the metric set is not enough. Even an operator looking at just three numbers is wrong on two of eighteen events if they look in the wrong order. The metric set determines the diagnosis's **ceiling**; reading order determines whether that ceiling is reached. There is a general way to decide which metric a sequence should start with: the metric **highest in the chain** is read first. Swap movement is the direct result of memory pressure and arises from no other cause; wait share, on the other hand, can arise from disk slowness, from swap, or from an ordinary bulk read. A metric with multiple causes is read **after** a metric with a single cause. This rule is not specific to the setup; it works in every metric set where indicators feed into one another. The decision tree has a cost too. A procedure that reads the three metrics in a fixed order reduces a situation where two bottlenecks genuinely exist **at the same time** to a single cause; it stops at the first matching branch and never reports the second cause. This does not show up in the measurement, because in the setup every event has a single real cause. On a real machine, reducing the diagnosis to a single name is itself a kind of wrong diagnosis. The limit of the measurement has to be written clearly too. E's zero comes from the setup being perfectly separable; a real machine is not expected to have three metrics separate every event. What is meaningful is not the zero but **the sequence from 12 down to 0**: quadrupling the line count gained nothing, adding the right metric removed four misses, narrowing and correcting the metric set removed ten misses, and fixing the order removed the remaining two. #### The Second Seed In the second seed, the distribution of events changes: seven CPU, six I/O, five memory. Views A and B again give **the exact same** number — eleven wrong diagnoses each — and the line-count difference again changes nothing. C drops to **5**, D to **2**, E to **0**. The absolute numbers **depend on the setup**; what does not depend on the setup is that quadrupling the line count never changes wrong diagnosis, and that the three-line view outperforms the ninety-six-line view in both seeds. #### Summary - The three bottleneck types produce the same symptom; memory pressure also creates I/O wait through swap, and looks like a disk bottleneck. - Quadrupling the line count (from 432 to 1728) **never changes** wrong diagnosis: it stays the same in both seeds. - Adding a single correct column to the list brings wrong diagnosis from 12 to **8**; dropping process detail and reducing to three system metrics brings it to **2**. - Changing the reading order of the same three metrics removes the remaining two misses too; wait share is a result, swap movement is a cause. - What determines the diagnosis is not the volume of output but the metric set and the order that set is read in. #### Next Step What was measured throughout this topic was how much of the truth the text and numbers a machine produces about itself carry: a diagnosis looking at a single unit file was wrong on five of six, rotation deleted 81 of 86 errors, an averaged indicator never showed a fifty-four-second spike at all, and four times the output corrected not a single miss. The silent assumption behind all these measurements was that data could be written to disk. The next topic removes that assumption and descends into storage: block devices, partitions, file systems, and a file system's two independent limits that run out. There, the situation where a disk is one percent full and yet a write fails is shown numerically. ## Storage Block devices and partitions, file system families, inode exhaustion, mount options, logical volumes and snapshots, swap space, and disk expansion. ### Block Devices and Partitions Source: https://academia.sh/en/courses/system-administration/block-devices-and-partitions A partition list is not a map: while the device shows 3,702,784 free sectors, the largest contiguous gap is 2,881,536 sectors, and six of the twelve placement decisions drawn from the list turn out wrong. The Logs topic separated which indicator a bottleneck should be read from, and reduced I/O wait to a single number. Beneath that number sits a device, and this topic descends there: to the place where bytes are actually written. Storage is the layer in system administration where failure looks quietest; a process's stopping is visible, a service's not running is visible, but a disk filling up is usually visible only once a write fails. The first question is the lowest one. A file system is built not on an entire device but on a part of it; a small table says where that part begins and ends. This lesson reads that table and counts what the table **does not say**. #### Device, Sector, and Partition A **block device** is a storage unit read from and written to in fixed-size units. The smallest addressable unit is called a **sector**; a device is an array of sectors numbered starting from zero, and it has no structure beyond this array. All of the order inside it is read from a data structure sitting in the device's first sectors. A sector has two sizes, and the two need not be equal. The **logical sector** the device presents to the outside and the **physical sector** actually read and written in one operation internally are separate numbers; in a common setup, the logical sector is 512 bytes, the physical sector 4096 bytes. So the device accepts 512-byte addresses, but when 512 bytes are to be written, it reads a 4096-byte block, modifies it, and writes it back. This distinction is the source of the difference counted in the alignment section, and it sits nowhere in the partition table; it is the device's own declaration. That data structure is the **partition table**. For every **partition**, the table holds a start sector, an end sector, and a type label. This is all the table holds, and nothing more: the table does not know what is inside the partition, whether it is full or empty, or even whether a file system exists inside it at all. A partition is nothing but a range drawn on the device's body. Two table formats are common. The older format is limited to four primary partitions and carries more only by nesting entries inside one partition; the later format allows hundreds of entries, gives every partition an immutable identity, and keeps a copy of the table at the end of the device. The difference between them is not a ranking of superiority but a difference in capacity, and which one is in use is learned by looking at the device itself. Tools that list partitions only read, and are harmless: `lsblk` gives the device tree, `blkid` writes partitions' identities and type labels, `parted` dumps a table in units of sectors. The transcript below is a **sample transcript** showing the shape of `parted`'s output; it was not run, and no numeric claim in this lesson comes from it. ```text Model: fictional block device Disk /dev/sdb: 12582912s Sector size (logical/physical): 512B/4096B Partition Table: gpt Number Start End Size File system Name 1 2048s 1048575s 1046528s ext4 root 2 1048576s 3344383s 2295808s ext4 data 3 3753984s 4636671s 882688s ext4 log 4 4636672s 6383615s 1746944s linux-swap swap ``` What stands out in the transcript is that the gap between where the fourth row ends and where the next partition begins is **not written down**. The tool lists partitions; it does not list what remains in between. #### Why It Is Partitioned Building a single file system on a device is also an option, and partitioning is the option set against it. The reason for the distinction is not capacity but **limiting the spread of failure**. On the fictional server, the units that write logs and the units that produce measurement files share the same device; if both write to the same file system, the moment log rotation stalls once, the filled space floods not just the logs but the root file system too, and the system becomes unable to log in. A separate partition keeps the filling within its own boundary. The second reason is different mount options: one partition can be mounted read-only, another closed to running programs. The third is that a partition's file system type can be chosen independently of another's. There is also a cost, and it is measurable: every partition's free space serves only itself. In the layout above, while the log partition fills up, 1,320,960 sectors can sit empty in the reserved partition, and these two numbers cannot be transferred to each other. Partitioning gives up flexibility to buy isolation; one of the later lessons measures a layer that loosens this trade-off. #### A Partition List Is Not a Map The only way to compute free space from a partition list is subtraction: the sum of the partitions is taken away from the device's sector count. This operation gives a correct total and produces a wrong diagnosis, because free space is not a single piece. - **ST1** — The fictional device is **12,582,912 sectors**; every measure is in sectors, and no real device's size is written down. - **ST2** — Partitions are aligned to a **2048-sector** multiple. The first 2048 sectors are reserved for the partition table and cannot be used as a partition. - **ST3** — The layout is produced from the shared definition's generator with seed **20260218**; six partitions carry the fictional server's root, data, log, swap, backup, and reserved areas. - **ST4** — The tool's output is the **partition list**; the gap map is not in the output. The diagnosis is built from the list alone. - **ST5** — A placement decision consists of two questions: does the requested size **fit**, and **how many** of that size fit. ```python """M03/K03 storage: the fictional block device's partition layout and gap map.""" SEED = 20260218 ALIGN = 2048 # partitions start at this multiple of sectors DEVICE = 12582912 # the fictional device's sector count NAMES = ("root", "data", "log", "swap", "backup", "reserved") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def build_layout(seed=SEED): """Oracle: the partitions' real start and size, in sectors.""" r = generator(seed) parts, cursor = [], ALIGN for name in NAMES: size = (r(900) + 300) * ALIGN parts.append({"name": name, "start": cursor, "size": size}) cursor += size + r(4) * 100 * ALIGN return parts def gaps(parts, device=DEVICE): """Oracle: the CONTIGUOUS free ranges between and after the partitions.""" g, cursor = [], 0 for x in sorted(parts, key=lambda z: z["start"]): if x["start"] > cursor: g.append(x["start"] - cursor) cursor = x["start"] + x["size"] if cursor < device: g.append(device - cursor) return g def diagnose(parts, requests, device=DEVICE): """The tool's output is the partition LIST. The diagnosis drawn from the list: free space is the device minus the sum of the partitions.""" g = gaps(parts, device) total_free = device - sum(x["size"] for x in parts) wrong, rows = 0, [] for s in requests: tool_fits, actual_fits = s <= total_free, s <= max(g) tool_count, actual_count = total_free // s, sum(x // s for x in g) wrong += (tool_fits != actual_fits) + (tool_count != actual_count) rows.append((s, tool_fits, actual_fits, tool_count, actual_count)) return {"total_free": total_free, "largest": max(g), "gap_count": len(g), "rows": rows, "wrong": wrong} layout = build_layout() print("partition start size end") for x in layout: print(f" {x['name']:9s} {x['start']:9d} {x['size']:9d} {x['start'] + x['size']:11d}") print("used sectors:", sum(x["size"] for x in layout), "| free sectors:", DEVICE - sum(x["size"] for x in layout)) print("contiguous gap count:", len(gaps(layout)), "| gaps:", gaps(layout)) print() REQUESTS = (204800, 614400, 1228800, 2457600, 3072000, 3686400) print("requested tool: fits actual: fits tool: count actual: count") for s, a, ge, aa, ga in diagnose(layout, REQUESTS)["rows"]: print(f"{s:7d} {str(a):11s} {str(ge):13s} {aa:10d} {ga:12d}") print() for seed in (SEED, 20260219): t2 = diagnose(build_layout(seed), REQUESTS) print(f"seed {seed}: free {t2['total_free']:8d} largest contiguous {t2['largest']:8d}" f" gaps {t2['gap_count']:2d} wrong diagnosis {t2['wrong']:2d} / 12") ``` ``` partition start size end root 2048 1046528 1048576 data 1048576 2295808 3344384 log 3753984 882688 4636672 swap 4636672 1746944 6383616 backup 6793216 1587200 8380416 reserved 8380416 1320960 9701376 used sectors: 8880128 | free sectors: 3702784 contiguous gap count: 4 | gaps: [2048, 409600, 409600, 2881536] requested tool: fits actual: fits tool: count actual: count 204800 True True 18 18 614400 True True 6 4 1228800 True True 3 2 2457600 True True 1 1 3072000 True False 1 0 3686400 True False 1 0 seed 20260218: free 3702784 largest contiguous 2881536 gaps 4 wrong diagnosis 6 / 12 seed 20260219: free 3633152 largest contiguous 1787904 gaps 7 wrong diagnosis 6 / 12 ``` Three numbers side by side. **The oracle:** free space is four pieces — 2048, 409,600, 409,600, and 2,881,536 sectors; the largest contiguous gap is **2,881,536 sectors**. **The tool's output:** a partition list, six rows totaling 8,880,128 sectors; the free space drawn from this is **3,702,784 sectors**. **Wrong diagnosis:** **six** of twelve decisions. The distribution of the errors is instructive too. Two decisions say "it fits" when it does not: partitions of 3,072,000 and 3,686,400 sectors stay under the total free space but fit into no contiguous gap. Four decisions err in count: the tool says six 614,400-sector partitions fit; in reality four fit. The total is correct, the diagnosis is wrong. In the layout produced with the second seed, free space drops to 3,633,152 sectors, the largest contiguous gap to 1,787,904 sectors, and the gap count rises to seven; the wrong diagnosis is again **6**. The number staying the same is a coincidence; its staying in the same order of magnitude is not: as the gap count grows, the diagnosis drawn from the total keeps breaking down. #### Alignment Where a partition begins determines the performance of the file system built inside it. A file system works in 4096-byte blocks, and one block is eight sectors; the device's own physical block is also eight sectors, counted from zero. If a partition begins at a sector that is not a multiple of eight, every block of the file system spans two physical blocks. - **ST6** — In sequential writing, the number of physical blocks touched increases by only one under misalignment; the blocks are already consecutive. - **ST7** — In a random single-block write, a misaligned block touches two physical blocks, and because the unchanged portion of both must be preserved, they are read before being written: four operations. ```python """Alignment: does the file system block sit on the same boundary as the device block.""" SECTORS_PER_BLOCK = 8 # 4096-byte block, 8 sectors of 512 bytes BLOCK_COUNT = 512 # file system blocks written PARTITION_SIZE = 1048576 # same in all six layouts, sectors def cost(start, blocks=BLOCK_COUNT, s=SECTORS_PER_BLOCK): offset = start % s return {"aligned": offset == 0, "sequential": blocks + (1 if offset else 0), "random": blocks * (4 if offset else 1)} print("start partition size aligned sequential ops random ops") wrong = 0 for start in (63, 2048, 2049, 2052, 2056, 4096): m = cost(start) wrong += not m["aligned"] print(f"{start:9d} {PARTITION_SIZE:12d} {str(m['aligned']):6s}" f" {m['sequential']:12d} {m['random']:14d}") print("the partition size in the tool's output is the same across all six layouts:", PARTITION_SIZE) print("how many times the 'same layout' diagnosis drawn from that size is wrong:", wrong, "/ 6") ``` ``` start partition size aligned sequential ops random ops 63 1048576 False 513 2048 2048 1048576 True 512 512 2049 1048576 False 513 2048 2052 1048576 False 513 2048 2056 1048576 True 512 512 4096 1048576 True 512 512 the partition size in the tool's output is the same across all six layouts: 1048576 how many times the 'same layout' diagnosis drawn from that size is wrong: 3 / 6 ``` The six layouts' partition size is the same, and this size appears in the tool's output; the starting sector also appears but is not read, because whether a number divides by eight is not asked while looking at the list. Sequential writing's cost rises from 512 to 513 under misalignment, an immeasurable difference; random writing's rises from 512 to **2048**. The "no difference" diagnosis given by a sequential test is wrong on **three** of the six layouts. This is storage's counterpart to the course's second claim: what distinguishes is not multiplying the test, but testing the right load. The row for 2056 in the table shows that alignment is not specific to 2048: every start that is a multiple of eight is aligned. The reason partitioning tools default to 2048 sectors is not eight but staying aligned on devices that work with much larger internal units too; 2048 is a number divisible by all plausible internal unit sizes. Tools do not make this decision silently; they produce a warning too, and `lsblk` writes the alignment deviation in a separate column. The warning sits in the output; reading it is a matter of habit, and the difference that appears when it is not read shows up nowhere until the moment it is measured. #### Modifying the Partition Table Tools that read the table are harmless; ones that write it are irreversible. Deleting a partition or changing its boundary overwrites the start and end values in the table; the data inside the partition is not deleted, but the information about how to reach it is lost, and the content does not become reachable again until the table is rewritten. This is why commands that write the table are **not given in a runnable, complete form** in this lesson. If a change is unavoidable, three precautions provide measurable safety. The first is **taking a text dump** of the table; a reading command like `sfdisk --dump` writes the table to a file, and restoring that same file re-establishes the old layout. The second is a **dry run**: most tools that write the table carry an option that shows the result without applying the change, and the output from this option must be read before it is run for real. The third is doing the trial **on a separate device**; mounting a file as a **loop device** makes it possible to practice partitioning away from real data. There is also an ordering rule: when a command that changes the table returns, the kernel may still be holding the old layout in memory. In that case, `lsblk` shows the new table while mounted file systems keep working with the old boundaries. A command returning does not mean the change has taken effect; this is storage's counterpart to the sentence established for signals in the Process Management topic. #### Summary - A block device is an array of numbered sectors; all of the order inside it is read from the partition table sitting in the first sectors, and the table carries only start, end, and type information. - The free space drawn from a partition list is a total, not a map: the largest contiguous gap in 3,702,784 free sectors is 2,881,536 sectors, and the free space is split into four pieces. - Six of twelve placement decisions turn out wrong when only the list is consulted; two think something that does not fit does fit, four overcount how many fit. With the second seed, the gap count rises to seven and the wrong diagnosis is again 6. - A partition start being a multiple of eight sectors produces a difference of 513 operations instead of 512 in sequential writing, and 2048 instead of 512 in random writing; the "no difference" diagnosis drawn from output showing the same partition size is wrong on three of six layouts. - Commands that write the partition table are irreversible; three measurable precautions are taking a dump of the table, reading the dry-run option, and doing the trial on a separate device. #### Next Step This lesson never looked inside a partition: to the table, a partition was nothing but a range with a start and an end. Yet the moment a file system is built on that range, decisions are made that cannot be reversed without cost — how many files can be held, how much space metadata will take up, and whether corrupted data will be noticed when read. The next lesson measures how file system families make these decisions differently, and counts how many times a design that carries no checksum silently returns corrupted data. ### File Systems Source: https://academia.sh/en/courses/system-administration/file-systems A directory carrying the same files costs 449,985,000, 125,312, and 59,800 comparisons across three internal layouts; in a design carrying no checksum, all 54 of the reads that touch a corrupted block return without error. The previous lesson left the partition as a range drawn on the device's body: a piece with a start, an end, and a type label, saying nothing about what is inside. The moment an arrangement is built inside that range, several decisions are made, and most of these decisions cannot be changed afterward. This lesson's question is this: on the same partition, with the same files, at which points do different designs diverge. There are two axes to measure — the cost of a name lookup in a directory, and whether a corrupted block is noticed during a read. Both are countable, and neither shows up in the tool's output. #### What a File System Holds A **file system** keeps four separate ledgers inside the same partition. The first is the **name ledger**: directory entries, binding a name to an identity. The second is the **metadata ledger**: every file's size, permissions, owner, timestamps, and which blocks its content sits in. The third is the **space ledger**: which blocks are full, which are free. The fourth is the **consistency ledger**: the record that keeps the on-disk structures mutually consistent if an update is interrupted halfway. How these four ledgers are kept varies from design to design, and this is where the difference comes from. Families one might encounter include XFS, Btrfs, ZFS, and the extended file system family; this lesson measures not the names but the **axes** between them. There are four axes: whether the metadata ledger's size is fixed at format time or grows as needed; whether the name ledger is a sequential list or a searchable structure; whether the space ledger is kept block by block or as contiguous ranges; whether the block read is checked for corruption. Which family gets installed by default varies by distribution, and this lesson does not rank one family above another. What is measured is the **cost** of the choice, not a ranking. A design that is cheap on one axis is expensive on another, and which axis dominates depends on the workload itself; when the workload changes, the decision ages, but the file system keeps standing in place. Which design an installed partition carries is learned by reading. `lsblk -f` writes every partition's type label and identity, `blkid` gives the same information one at a time. The transcript below is a **sample transcript** showing the shape of the output; it was not run, and no numeric claim in this lesson comes from it. ```text NAME FSTYPE FSVER LABEL UUID MOUNTPOINT sdb ├─sdb1 ext4 1.0 root 11111111-1111-1111-1111-1 / ├─sdb2 xfs data 22222222-2222-2222-2222-2 /data ├─sdb3 ext4 1.0 log 33333333-3333-3333-3333-3 /log └─sdb4 swap 1 swap 44444444-4444-4444-4444-4 [SWAP] ``` The identity field in the transcript will be useful in later lessons: a partition's name can change, its identity cannot. #### Numbers Fixed at Format Time Building a file system on a partition means writing a header at the partition's start and allocating space for the ledgers. This operation fixes a series of numbers, and most of what gets fixed cannot be changed afterward, or can be changed only by emptying the file system. The first number fixed is the **block size**: the smallest unit the file system allocates. The second is the **metadata record count**; in designs that keep a fixed table, this number is set at format time and does not grow unless the file system is enlarged. The third is the **directory's internal layout**, measured below. The fourth is **which ledgers the checksum covers**. The fifth is the **consistency record's mode**: whether only metadata is journaled or data too. The file system abstraction lesson in the Operating System Concepts course measured the write multiplier of a journaled design; that measurement is not repeated here. There is also the space ledger's format. In a design that keeps a block list, one record is needed per block; in a design that keeps contiguous ranges, a hundred blocks running consecutively is a single record. The second produces far less metadata for large, unfragmented files, and its gain erodes as fragmentation grows. Which format gets chosen depends on the file-size distribution, and this distribution is not known at format time; the decision is made by guessing. The destructive-command boundary begins here: the command that formats a partition **irreversibly** erases everything on that partition, and if the wrong partition name is written, it can run without a warning. This lesson does not give the formatting command in a runnable, complete form. The safe way to test it is mounting a file as a loop device and doing the trial there. #### Same Directory, Three Internal Layouts On the fictional server's `/data` partition, the files produced by the measurement-collecting unit pile up in a single directory. A directory is a table, and that table's internal layout is a design decision. The file system abstraction lesson in the Operating System Concepts course counted reading a directory as **one step** and assumed the entry count did not change the step count. This lesson pays that assumption's cost. - **ST8** — The work measured is creating a new file. Every creation is first a **lookup**: does the same name already exist in the directory. What is counted are the comparisons in this lookup. - **ST9** — In the sequential-list layout, the lookup goes start to end. In the hashed layout, a name falls into a bucket and is compared only against the names in that bucket; the bucket count is **4096** and is fixed at format time. In the tree layout, a branch carries **200 names**, and the lookup finishes in as many steps as the tree's depth. - **ST10** — What is measured is the comparison count; wall-clock time is not measured. ```python """Directory lookup: same files, three different internal layouts.""" BUCKETS = 4096 # bucket count in the hashed layout BRANCH = 200 # names carried by a B-tree node def build_cost(n, layout): """Every new name is first LOOKED UP (does it exist). Total comparisons.""" if layout == "linear": return n * (n - 1) // 2 if layout == "hashed": return sum(i // BUCKETS + 1 for i in range(n)) depth, total = 1, 0 for i in range(1, n + 1): if i > BRANCH ** depth: depth += 1 total += depth return total LAYOUTS = ("linear", "hashed", "tree") SCALE = (1000, 5000, 10000, 30000, 100000) BASELINE = {d: build_cost(1000, d) for d in LAYOUTS} print("file count ", " ".join(f"{d:>12s}" for d in LAYOUTS)) for n in SCALE: print(f"{n:12d} ", " ".join(f"{build_cost(n, d):12d}" for d in LAYOUTS)) print() print("estimate scaled linearly from 1000 files, vs actual") print("file count layout estimate actual deviation ratio wrong diagnosis") wrong = 0 for n in SCALE: for d in LAYOUTS: estimate = BASELINE[d] * n // 1000 actual = build_cost(n, d) ratio = actual / estimate y = ratio > 2 wrong += y print(f"{n:12d} {d:10s} {estimate:12d} {actual:12d} {ratio:10.2f} {str(y):>11s}") print("wrong diagnosis:", wrong, "/", len(SCALE) * len(LAYOUTS)) ``` ``` file count linear hashed tree 1000 499500 1000 1800 5000 12497500 5904 9800 10000 49995000 17712 19800 30000 449985000 125312 59800 100000 4999950000 1271200 259800 estimate scaled linearly from 1000 files, vs actual file count layout estimate actual deviation ratio wrong diagnosis 1000 linear 499500 499500 1.00 False 1000 hashed 1000 1000 1.00 False 1000 tree 1800 1800 1.00 False 5000 linear 2497500 12497500 5.00 True 5000 hashed 5000 5904 1.18 False 5000 tree 9000 9800 1.09 False 10000 linear 4995000 49995000 10.01 True 10000 hashed 10000 17712 1.77 False 10000 tree 18000 19800 1.10 False 30000 linear 14985000 449985000 30.03 True 30000 hashed 30000 125312 4.18 True 30000 tree 54000 59800 1.11 False 100000 linear 49950000 4999950000 100.10 True 100000 hashed 100000 1271200 12.71 True 100000 tree 180000 259800 1.44 False wrong diagnosis: 6 / 15 ``` Three numbers side by side. **The oracle:** creating 30,000 files in a single directory is **449,985,000** comparisons in the sequential-list layout, **125,312** in the hashed layout, **59,800** in the tree layout. **The tool's output:** all three directories give the same 30,000 names with `ls`, and the directory entry count is equal across all three; the internal layout is not in the output. **Wrong diagnosis:** an estimate measured on a small directory and scaled linearly deviates by more than double in **six** of fifteen cases. There is no superiority in the table, only a difference in curves. At a thousand files, the tree layout is **more expensive** than the hashed layout (1800 against 1000); at a hundred thousand files, it is five times **cheaper**. The hashed layout also shows a limit of its own: because the bucket count is fixed at format time, the chains lengthen, and at 100,000 files the cost climbs to 12.71 times the linear estimate. Every number that gets fixed is a debt paid as the scale grows. #### Reading With and Without a Checksum The second axis is the correctness of a read. A block can corrupt itself on disk; even though it no longer holds what was written, the device returns it without error. If the design does not check for this, the corruption is **silent**: the application takes the wrong byte for the right one. - **ST11** — The shared definition's `/data` file system carries 30,000 files, and every file holds one block. The inode table is **2048 blocks**; 16 inodes fit in one block. - **ST12** — Reading a file touches two blocks: its own data block, and the table block where its inode sits. - **ST13** — **40 blocks** get corrupted; which ones is chosen by the generator. A design that carries a checksum catches the corruption at read time; one that does not, does not. ```python """Checksum: is a corrupted block caught during a read.""" SEED = 20260218 BLOCK = 4096 FILES = 30000 # the shared definition's /data file system INODES_PER_BLOCK = 16 # 16 inodes (256 bytes) fit in a 4096-byte block META_BLOCKS = 2048 # a 32768-inode table, in blocks CORRUPT = 40 def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def corrupted(seed=SEED, count=CORRUPT, files=FILES, meta=META_BLOCKS): """Oracle: which blocks really got corrupted. The first `files` are data blocks.""" r = generator(seed) return {r(files + meta) for _ in range(count)} def reads(design, corrupt, files=FILES): """Every file reads its own data block and its block in the inode table.""" data_checksum, meta_checksum = design caught = silent = 0 for i in range(files): data_bad = i in corrupt meta_bad = (files + i // INODES_PER_BLOCK) in corrupt if (data_bad and data_checksum) or (meta_bad and meta_checksum): caught += 1 elif data_bad or meta_bad: silent += 1 return {"caught": caught, "silent": silent} DESIGN = {"no checksum": (False, False), "meta checksum": (False, True), "data+meta checksum": (True, True)} corrupt = corrupted() print("corrupted blocks:", len(corrupt), "| data blocks:", sum(1 for x in corrupt if x < FILES), "| metadata blocks:", sum(1 for x in corrupt if x >= FILES)) print("oracle: reads touching a corrupted block:", reads((True, True), corrupt)["caught"], "/", FILES) print() print("design returns with error returns corrupted silently wrong diagnosis") for name, t in DESIGN.items(): o = reads(t, corrupt) print(f" {name:20s} {o['caught']:13d} {o['silent']:20d} {o['silent']:11d}") overhead = -(-(FILES + META_BLOCKS) * 4 // BLOCK) print("checksum storage:", overhead, "blocks |", round(overhead / (FILES + META_BLOCKS) * 100, 3), "percent") print() for seed in (SEED, 20260219): c2 = corrupted(seed) print(f"seed {seed}: no checksum silent {reads((False, False), c2)['silent']:4d}" f" meta checksum {reads((False, True), c2)['silent']:4d}" f" full checksum {reads((True, True), c2)['silent']:4d}") ``` ``` corrupted blocks: 40 | data blocks: 38 | metadata blocks: 2 oracle: reads touching a corrupted block: 54 / 30000 design returns with error returns corrupted silently wrong diagnosis no checksum 0 54 54 meta checksum 16 38 38 data+meta checksum 54 0 0 checksum storage: 32 blocks | 0.1 percent seed 20260218: no checksum silent 54 meta checksum 38 full checksum 0 seed 20260219: no checksum silent 70 meta checksum 38 full checksum 0 ``` Three numbers side by side. **The oracle:** 40 blocks got corrupted, and the number of reads touching them is **54** — 38 files are affected through their own data block, 16 files through two corrupted table blocks. **The tool's output:** in the design that carries no checksum, every read call returns **without error**; no tool reports anything. **Wrong diagnosis:** in that design, **54 reads** take a corrupted byte for a correct one; this drops to 38 once a metadata checksum is added, and to **zero** once data is checksummed too. The cost is in the table: the checksum for 32,048 blocks takes up 32 blocks, that is, 0.1 percent. The space cost is small; the processing cost is a computation added to every read, and this lesson does not count it. In the second seed, the corrupted-block distribution changes, the silent-read count comes out to 70 instead of 54; the ordering does not change. The number itself depends on the fiction, **the gap from zero does not**. The real conclusion here is this: in a design with no checksum, a "no error" output does not mean "the data is correct." Corruption only surfaces once a consistency check is run or once the data is used; both come after the read. The course's third claim holds here too — one who looks late sees little. A consistency check is not free either, and what it does must be watched. Tools of the `fsck` class walk the ledgers from end to end and look at whether they are mutually consistent; they leave **repairing** whatever inconsistency they find, and repair sometimes means moving a file into a directory where orphaned records are collected. This work cannot be done on a mounted file system: the file system must not change while the tool is running, or the tool invalidates the very ledger it is reading. This is why the repair command on a mounted file system is **not given in a runnable, complete form**; testing is done after the file system is unmounted, or with the report-only option. What the check can find also has a limit. Inconsistency between ledgers is found; a data block's content having changed is not found, because there is no record saying what the correct content was. What the checksum buys in the table is exactly this: a 32-block record that serves not to know what the correct content is, but to **realize that it is wrong**. #### Summary - A file system keeps a name ledger, a metadata ledger, a space ledger, and a consistency ledger in the same partition; the difference between families is how these four ledgers are kept, not a ranking of superiority. - Creating 30,000 files in a single directory is 449,985,000 comparisons in the sequential-list layout, 125,312 in the hashed layout, 59,800 in the tree layout; all three directories give the same list with `ls`. - An estimate measured on a small directory and scaled linearly deviates by more than double in six of fifteen cases; the bucket count being fixed at format time in the hashed layout overshoots the estimate by 12.71 times at 100,000 files. - In a design carrying no checksum, all 54 of the reads that touch a corrupted block return without error; this becomes 38 with a metadata checksum, 0 with a data checksum. The storage cost is 32 blocks, 0.1 percent. - Every number fixed at format time is a ceiling; the next lesson measures what hitting one of those ceilings looks like. #### Next Step This lesson counted the metadata ledger as a design axis but did not look inside it: the record itself, carrying everything about a file except its name, is a resource, and it can be exhausted. The next lesson measures that record and produces the course's most striking number — in the same file system, a write fails while block usage sits at 0.0125. The disk is one percent full and the system is full; the diagnosis drawn from the usage percentage is flatly wrong here. ### Inodes and Metadata Source: https://academia.sh/en/courses/system-administration/inodes-and-metadata In the same file system, while block usage is 0.0125, inode usage is 1.0000 and a write fails: the disk is one percent full and the system is full. A file whose name is deleted but whose handle stays open hides 120,000 blocks from du's output. The previous lesson counted the metadata ledger as a design axis: is it kept fixed, or does it grow as needed. This lesson takes the single record inside that ledger and shows that it is a **resource** — one that can be exhausted, and that produces a strange failure when it is. The fictional server's `/data` partition carries thirty thousand measurement files, and the partition's usage is a bit over one percent. This lesson is going to try writing a new file to that partition, and the attempt is going to fail. The contradiction here is not a bug; it is two separate limits producing the same error message. #### The Record an Inode Carries An **inode** is the record that carries everything about a file except its name. The file system abstraction lesson in the Operating System Concepts course defined this record and showed that the name sits in the directory, not in the file; that definition is not repeated here. The only thing added here is what the operator can see. The record is referred to by a number, and that number is unique within the file system. `ls -i` lists the number, `stat` opens the whole record, `find` can go back from a number to the file's names. Among the record's fields, the most often misread is the **link count**: how many directory entries point to this record. Once the link count drops to zero and no process is left holding the record open, the block list is released. Both conditions are required, and the second is the subject of this lesson's second measurement. The transcript below is a **sample transcript** showing the shape of `stat`'s output; it was not run, and no numeric claim in this lesson comes from it. ```text File: /data/metrics/2f9c.data Size: 1731 Blocks: 8 IO Block: 4096 regular file Device: 8,18 Inode: 1048604 Links: 1 Access: (0640/-rw-r-----) Uid: ( 997/ metrics) Gid: ( 997/ metrics) ``` In the transcript, `Size` and `Blocks` sit on separate lines and say two different things: the first is the byte count the file reports, the second is the space the file system actually allocated. This distinction will be the source of six wrong readings in the second measurement. #### Two Limits, One Error Message A file system consumes two separate resources. The first is **blocks**: where content is written. The second is **inodes**: one is needed per file, and in designs that keep a fixed table, the total count is fixed at format time. When either one runs out, a write is rejected with the same error message. - **ST14** — The shared definition's file system carries **2,621,440 blocks** and **32,768 inodes**; a block is 4096 bytes. The numbers come from the model, not from any real device. - **ST15** — In the small-file workload, sizes range from 1 to 2000 bytes; every file holds one block. In the large-file workload, the upper bound is 2,000,000 bytes. - **ST16** — The warning rule looks at a **0.90** usage threshold. The rule is tested in three forms: block percentage only, inode percentage only, both together. - **ST17** — The diagnosis tested is a single question: can a new 1 KB file be written in this state. ```python """M03/K03 shared definition (Section 6): two separate limits, one error message.""" SEED = 20260218 BLOCK = 4096 def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def file_system(seed=SEED, files=30000, upper=2000, inode_count=32768, block_count=2621440): r = generator(seed) sizes = [r(upper) + 1 for _ in range(files)] used_blocks = sum(-(-s // BLOCK) for s in sizes) return {"files": files, "used_blocks": used_blocks, "total_blocks": block_count, "block_usage": round(used_blocks / block_count, 4), "used_inodes": files, "total_inodes": inode_count, "inode_usage": round(files / inode_count, 4)} def attempt_write(fs, size): """Can a new file be written. Two separate limits, one error message.""" blocks_ok = fs["used_blocks"] + -(-size // BLOCK) <= fs["total_blocks"] inodes_ok = fs["used_inodes"] + 1 <= fs["total_inodes"] return blocks_ok and inodes_ok for n in (30000, 32768): fs = file_system(files=n) print(f"{n:6d} files | block usage {fs['block_usage']:.4f}" f" | inode usage {fs['inode_usage']:.4f}" f" | new 1 KB file: {attempt_write(fs, 1024)}") print() SMALL = [(n, 2000) for n in (4096, 12288, 20480, 28672, 32000, 32768)] LARGE = [(n, 2000000) for n in (2000, 4000, 6000, 8000, 10000, 12000)] RULE = {"block percentage only": lambda f: f["block_usage"] < 0.90, "inode percentage only": lambda f: f["inode_usage"] < 0.90, "both together": lambda f: f["block_usage"] < 0.90 and f["inode_usage"] < 0.90} print("workload files block usage inode usage writable") for name, load in (("small", SMALL), ("large", LARGE)): for n, upper in load: fs = file_system(files=n, upper=upper) print(f"{name:7s} {n:6d} {fs['block_usage']:13.4f} {fs['inode_usage']:14.4f}" f" {attempt_write(fs, 1024)}") print() print("rule missed false alarm wrong diagnosis") for name, k in RULE.items(): missed = false_alarm = 0 for _, load in (("small", SMALL), ("large", LARGE)): for n, upper in load: fs = file_system(files=n, upper=upper) safe, actual = k(fs), attempt_write(fs, 1024) missed += safe and not actual false_alarm += (not safe) and actual print(f" {name:20s} {missed:9d} {false_alarm:12d} {missed + false_alarm:11d}") print() for seed in (SEED, 20260219): row = [] for name, k in RULE.items(): y = 0 for _, load in (("small", SMALL), ("large", LARGE)): for n, upper in load: fs = file_system(seed, n, upper) y += k(fs) != attempt_write(fs, 1024) row.append(f"{name} {y}") print(f"seed {seed}: " + " | ".join(row) + " / 12") ``` ``` 30000 files | block usage 0.0114 | inode usage 0.9155 | new 1 KB file: True 32768 files | block usage 0.0125 | inode usage 1.0000 | new 1 KB file: False workload files block usage inode usage writable small 4096 0.0016 0.1250 True small 12288 0.0047 0.3750 True small 20480 0.0078 0.6250 True small 28672 0.0109 0.8750 True small 32000 0.0122 0.9766 True small 32768 0.0125 1.0000 False large 2000 0.1894 0.0610 True large 4000 0.3693 0.1221 True large 6000 0.5553 0.1831 True large 8000 0.7416 0.2441 True large 10000 0.9283 0.3052 True large 12000 1.1168 0.3662 False rule missed false alarm wrong diagnosis block percentage only 1 1 2 inode percentage only 1 1 2 both together 0 2 2 seed 20260218: block percentage only 2 | inode percentage only 2 | both together 2 / 12 seed 20260219: block percentage only 2 | inode percentage only 2 | both together 2 / 12 ``` Three numbers side by side. **The oracle:** in the 32,768-file case, the inode table is completely full and 98.75 percent of the blocks are empty; the write **fails**. **The tool's output:** the number that comes back when usage is asked is **0.0125** — one percent. **Wrong diagnosis:** the warning rule is mistaken in **two** of twelve cases, and these two mistakes are not eliminated by changing the rule. The shared definition's two lines are the course's cleanest example. At thirty thousand files, block usage is **0.0114**, inode usage is **0.9155**, and the write succeeds; at 32,768 files, block usage is **0.0125**, inode usage is **1.0000**, and the write fails. The difference between them is 2768 files, and a thousandth of the total space. **The disk is one percent full and the system is full.** The rule table pays off the second claim. The rule that looks only at the block percentage misses one failure and raises one false alarm. The rule that looks only at the inode percentage gives the same number — while blocks fill up in the large-file workload, the inode percentage stays at 0.3662. When both are used together, the missed count drops to zero, but the false alarm count rises to two, and the total is again **2**. Adding a column changed the **type** of the wrong diagnosis, not its count. In the second seed, all three rules again give 2; the result depends not on the fiction but on the two limits being independent of each other. #### The File With No Name The second measurement counts the space allocated in the same file system using three separate methods. `df` reads the file system's own ledger, `du` walks the directory tree and sums the blocks of the files it finds, and the total from a file listing sums the reported bytes. All three answer the same question, and the three do not give the same number. - **ST18** — Five moments pass in sequence: the start, giving 200 second names to the same files, writing 40 sparse files, deleting a 120,000-block log file, and closing the handle that kept that file open. - **ST19** — A second name given to the same inode is called a **hard link**. A **sparse file** reports a size of 25,600 blocks, and actually allocates 8. The space the metadata itself takes up is not counted in this measurement; the oracle is data blocks alone. ```python """How many blocks are really allocated, and which tool sees how many.""" SEED = 20260218 BLOCK = 4096 FILES = 30000 SPARSE_COUNT = 40 # number of sparse files SPARSE_LOGICAL = 25600 # reported size, in blocks SPARSE_ALLOCATED = 8 # actually allocated blocks LOG_BLOCKS = 120000 # large log file HARD_LINKS = 200 # second names given to the same file def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def base(seed=SEED): r = generator(seed) return [r(2000) + 1 for _ in range(FILES)] def state(moment, sizes): """moment 0..4 . Oracle: the data blocks the file system really allocated.""" sparse = moment >= 2 log_exists = moment < 4 # deleted at moment 3, handle open allocated = FILES + LOG_BLOCKS * log_exists + SPARSE_COUNT * SPARSE_ALLOCATED * sparse # du: walks only NAMED files, counts a hard link once du = FILES + LOG_BLOCKS * (moment < 3) + SPARSE_COUNT * SPARSE_ALLOCATED * sparse # file-listing total: reported bytes, every NAME counted separately ls = sum(sizes) + sum(sizes[:HARD_LINKS]) * (moment >= 1) ls += LOG_BLOCKS * BLOCK * (moment < 3) ls += SPARSE_COUNT * SPARSE_LOGICAL * BLOCK * sparse return {"oracle": allocated, "df": allocated, "du": du, "ls": ls // BLOCK} MOMENT = ("t0 start", "t1 hard link", "t2 sparse files", "t3 log deleted", "t4 handle closed") SIZES = base() print("moment oracle df du ls total") for i, name in enumerate(MOMENT): d = state(i, SIZES) print(f"{name:18s} {d['oracle']:8d} {d['df']:8d} {d['du']:8d} {d['ls']:15d}") print() print("tool correct reads wrong reads") for tool in ("df", "du", "ls"): wrong = sum(state(i, SIZES)[tool] != state(i, SIZES)["oracle"] for i in range(5)) print(f" {tool:9s} {5 - wrong:11d} {wrong:12d}") print("wrong diagnosis:", sum(state(i, SIZES)[a] != state(i, SIZES)["oracle"] for i in range(5) for a in ("df", "du", "ls")), "/ 15") print() for seed in (SEED, 20260219): s = base(seed) print(f"seed {seed}: t0 oracle {state(0, s)['oracle']} ls {state(0, s)['ls']}" f" | t3 oracle {state(3, s)['oracle']} du {state(3, s)['du']}" f" | t4 oracle {state(4, s)['oracle']} ls {state(4, s)['ls']}") ``` ``` moment oracle df du ls total t0 start 150000 150000 150000 127313 t1 hard link 150000 150000 150000 127361 t2 sparse files 150320 150320 150320 1151361 t3 log deleted 150320 150320 30320 1031361 t4 handle closed 30320 30320 30320 1031361 tool correct reads wrong reads df 5 0 du 4 1 ls 0 5 wrong diagnosis: 6 / 15 seed 20260218: t0 oracle 150000 ls 127313 | t3 oracle 150320 du 30320 | t4 oracle 30320 ls 1031361 seed 20260219: t0 oracle 150000 ls 127292 | t3 oracle 150320 du 30320 | t4 oracle 30320 ls 1031339 ``` Three numbers side by side. **The oracle:** what has really been allocated at the third moment is **150,320 blocks**. **The tool's output:** `du` sees **30,320** blocks at the same moment. **Wrong diagnosis:** **six** of fifteen readings are wrong — five from the file-listing total, one from `du`'s output. The third moment is the trap the operator falls into most often. The log file has been deleted, `du`'s output has dropped by 120,000 blocks, and that file is not in the directory tree; but because the process holding it open is still running, the blocks have not been released even though the link count has dropped to zero. `df` keeps telling the truth, and until the **120,000-block** gap between the two tools is explained, an attempt to free space goes nowhere. The fix is not deleting; it is restarting the process holding the file open, and which process is holding it is found with a tool that lists open handles. The file-listing total is wrong at all five of the five moments, and wrong in two separate directions. At the start it **undercounts**: the byte total reported by thirty thousand small files corresponds to 127,313 blocks, while in reality 150,000 blocks are allocated, because even a 1731-byte file takes up a full block. Once the sparse files are written, it **overcounts**: 40 files report a size of 1,024,000 blocks, while in reality they hold 320 blocks. In the second seed, these two numbers come out to 127,292 and 1,031,339; the direction does not change. #### Repairing an Exhausted Table When inodes run out, there are three things that can be done, and all three have a cost. The first is **deleting files**; this gives the record in the table back, and collecting single-line measurement files into an archive is usually the fastest fix. The second is **enlarging** the file system; in designs that keep a fixed table, enlarging can extend the table too, but the ratio depends on the decision made at format time. The third is **reformatting**, and this irreversibly erases everything on the partition. The third way is not given in a runnable, complete form in this lesson. When the formatting command, written together with the option that sets the inode count, is run with the wrong partition name, it erases all the data without producing a warning, and it is irreversible. The safe way to test it is three steps: verifying the target partition's identity with `blkid`, doing the trial on a file mounted as a loop device, and taking a snapshot of the partition before running it for real. The precaution itself is measuring. Unless inode usage is asked as often as block usage, the moment the table fills becomes not a warning but a failure; and when the failure arrives, the only number in the output will still read one percent. #### Summary - An inode is the record that carries everything about a file except its name; a link count dropping to zero does not by itself free space, no process holding the record open may remain either. - A file system consumes two separate resources, and both produce the same error message: at 30,000 files, with block usage 0.0114 and inode usage 0.9155, the write succeeds; at 32,768 files, with 0.0125 and 1.0000, it fails. - The warning rule gives 2 wrong diagnoses in twelve cases when it looks only at the block percentage; also 2 when it looks only at the inode percentage, and also 2 when both are used together. The missed count drops to zero, the false-alarm count rises to two. - When allocated space is counted three ways, six of fifteen readings are wrong: the file-listing total is wrong at all five of the five moments, and `du` cannot see the 120,000 blocks of a file whose name is deleted but whose handle stays open. - One of the three ways to repair an exhausted table is reformatting, and it is irreversible; this lesson does not give that command in complete form, the safe way to test it is identity verification, a loop device, and a snapshot. #### Next Step This lesson's measurements rested on a single assumption: that the directory being looked at really is that file system's directory. Yet a file system is **mounted** at a point in the tree, and that point is a writable directory both before and after mounting. The next lesson measures the mount point and counts where data written there before a file system is mounted goes, and which tools cannot see it once it is mounted. ### Mounting and Persistent Mounts Source: https://academia.sh/en/courses/system-administration/mounting-and-persistent-mounts A persistent mount entry written to a device name mounts the correct file system in only thirteen of sixty boots and silently mounts the wrong one in twenty-seven; a mount option that does not update access time makes a cleanup job wrongly delete 787 files. The previous lesson separated a file system's two limits and built all of its measurements on a single assumption: that the directory being looked at really is that file system's directory. This assumption is not free. A file system is **mounted** at a point in the tree, and that point is a writable directory both before and after mounting. This lesson measures mounting itself and automatic mounting at boot. Two numbers will come out: in how many boots a persistent mount entry can mount the wrong file system, and how many files a single mount option can make a maintenance job wrongly delete. Neither produces an error message. #### Mounting Is an Attach Operation In this system, every file system sits not under a separate letter but on a branch of **a single tree**. **Mounting** is attaching the file system on a block device to a point in this tree; that point is called the **mount point**, and a mount point is always a directory that already exists. Mounting adds an entry to a table in the kernel; nothing changes on the device. The same device can be mounted at multiple points, one point can be mounted twice in a row, and unmounting does not affect the data. What is mounted is read in tree form with `findmnt`, in flat-list form with `mount`. The transcript below is a **sample transcript** showing the shape of `findmnt`'s output; it was not run, and no numeric claim in this lesson comes from it. ```text TARGET SOURCE FSTYPE OPTIONS / /dev/sdb1 ext4 rw,relatime ├─/data /dev/sdb2 xfs rw,noatime,nodev └─/log /dev/sdb3 ext4 rw,relatime,noexec,nosuid ``` The mount point's content before mounting is the lesson's first trap. The `/data` directory does not have to be empty; if it has files in it, the moment the device is mounted there, those files **become invisible**. They are not deleted, not moved, and keep taking up space on the underlying file system; only their names become unreachable. The most common form of this is: a unit starts running before the device is mounted and writes its output to the mount point; the device is mounted later, and that data stays buried inside the root file system. `du` cannot see it when it walks the mount point, because it is no longer in the tree; `df` shows unaccounted-for usage on the root file system. This gap between the two outputs is the twin of the previous lesson's file with a deleted name: in both, the usage on display does not account for real consumption. But the way to find it differs — here, temporarily unmounting and looking at the same directory again is enough. #### Persistent Mounts and Device Names A manual mount is lost on restart. Automatic mounting at boot is defined by a **persistent mount** entry that binds the device to a mount point, a file system type, and an option list. The entry's first field names the device, and how this field is written is what this section measures. Device names are assigned in discovery order. If one controller responds before another, the names swap; when a device is added or fails, the whole order shifts. A device's **identity**, on the other hand, is written when the file system is built and does not change no matter where the device is attached. - **ST20** — The fictional server has **five block devices**; three carry the same type of data file system, one carries swap space, one carries a file system of a different type. - **ST21** — The discovery order can change on every boot; the order is drawn from the shared definition's generator, and names are assigned according to this order. **60 boots** are observed. - **ST22** — The `/data` entry is written to the name `device-b`, and its type field is the data file system. It is counted as **correct** when the entry mounts the right identity, **silently wrong** when it mounts a different device without incident because the type matches, **noisily wrong** when it fails because the type does not match. ```python """At boot, device name and device identity are not the same thing.""" SEED = 20260218 BOOTS = 60 # number of boots observed DEVICES = ({"id": "k1", "type": "ext4", "content": "root"}, {"id": "k2", "type": "ext4", "content": "data"}, {"id": "k3", "type": "ext4", "content": "log"}, {"id": "k4", "type": "swap", "content": "swap"}, {"id": "k5", "type": "xfs", "content": "backup"}) NAMES = ("device-a", "device-b", "device-c", "device-d", "device-e") RECORD = {"name": "device-b", "id": "k2", "type": "ext4"} # the /data entry def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def discovery_order(r): """Devices are named in discovery order; the order can change.""" remaining, order = list(DEVICES), [] while remaining: order.append(remaining.pop(r(len(remaining)))) return order def boot(seed=SEED, count=BOOTS): r = generator(seed) by_name = {"correct": 0, "silent_wrong": 0, "noisy_error": 0} by_id = {"correct": 0, "silent_wrong": 0, "noisy_error": 0} for _ in range(count): selected = dict(zip(NAMES, discovery_order(r)))[RECORD["name"]] if selected["id"] == RECORD["id"]: by_name["correct"] += 1 elif selected["type"] == RECORD["type"]: by_name["silent_wrong"] += 1 else: by_name["noisy_error"] += 1 by_id["correct"] += 1 return by_name, by_id A, K = boot() print("entry form correct silent wrong noisy error wrong diagnosis") for name, s in (("device name", A), ("device id", K)): print(f" {name:12s} {s['correct']:6d} {s['silent_wrong']:13d}" f" {s['noisy_error']:14d} {s['silent_wrong']:11d}") print("boots observed:", BOOTS) print() for seed in (SEED, 20260219): a, _ = boot(seed) print(f"seed {seed}: correct {a['correct']:3d} silent wrong" f" {a['silent_wrong']:3d} noisy error {a['noisy_error']:3d}") ``` ``` entry form correct silent wrong noisy error wrong diagnosis device name 13 27 20 27 device id 60 0 0 0 boots observed: 60 seed 20260218: correct 13 silent wrong 27 noisy error 20 seed 20260219: correct 10 silent wrong 27 noisy error 23 ``` Three numbers side by side. **The oracle:** the file system that should be mounted at `/data` is the same identity on every boot. **The tool's output:** the mount table shows a source for `/data` on every boot, and the name it shows is **`device-b` every time**; anyone looking at the table sees no difference. **Wrong diagnosis:** in **twenty-seven** of sixty boots, the wrong file system mounts without error. The twenty failing boots are actually good news: because the type does not match, the mount fails, the failure is seen, and it gets fixed. What is dangerous is the twenty-seven silent cases — the unit runs, data gets written, no log line is produced, and the written data does not sit where it is expected. In an entry written to the identity, this number is **zero**; in the second seed, the correct-boot count comes out to 10 instead of 13, silent wrong is again 27. What writing to the identity buys is not performance, but **determinism**. Writing the identity has a cost too, and it is not hidden: the identity is unreadable, writing it by hand is error-prone, and it changes when the device is reformatted. This is why entries are mostly produced not by hand but by reading them off the device. A middle path is the label given when the file system is built; a label is a readable name and, unlike a device name, does not depend on discovery order, but if two devices are given the same label, determinism is lost again. The entry's last fields affect the diagnosis too. When an entry fails, boot can halt, and the machine comes up only into a rescue shell; the option that prevents this makes the failure invisible, and the system keeps running with a file system missing. The choice between the two is a preference: halting noisily, or running silently incomplete. #### The Option That Silently Changes Meaning The option list is the least-read field of a mount. Mounting a partition read-only or closing it to running programs produces **noisy** results: writes are rejected, execution is rejected, an error message appears. The option that does not update access time, on the other hand, rejects nothing; it only freezes a recorded number. - **ST23** — The `/data` partition has **2000 files**; the creation, last-write, and last-read moments are drawn from the generator within a 600-second window. - **ST24** — The recorded access time depends on the option: with full updating, the real read moment is written; with relaxed updating, only the write moment; with the no-update option, the creation moment. - **ST25** — The cleanup job deletes files whose recorded access time is older than **300 seconds**. The oracle is the file's **real** last-read moment. ```python """A mount option changes recorded access time; a cleanup job deletes wrongly.""" SEED = 20260218 PERIOD = 600 THRESHOLD = 300 FILES = 2000 def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def files(seed=SEED, count=FILES): """Oracle: every file's REAL last-read moment.""" r = generator(seed) records = [] for _ in range(count): creation = r(400) last_write = creation + (r(PERIOD - creation) if r(10) < 3 else 0) base = max(creation, last_write) last_read = base + (r(PERIOD - base) if r(10) < 6 else 0) records.append({"creation": creation, "last_write": last_write, "last_read": last_read}) return records def recorded(d, option): if option == "atime": return d["last_read"] if option == "relatime": return d["last_write"] return d["creation"] # noatime: never updated def cleanup(records, option, threshold=THRESHOLD): """The job deletes files whose recorded access time is older than the threshold.""" wrong_delete = wrong_keep = deleted = 0 for d in records: delete = recorded(d, option) < threshold stale = d["last_read"] < threshold # oracle deleted += delete wrong_delete += delete and not stale wrong_keep += stale and not delete return {"deleted": deleted, "wrong_delete": wrong_delete, "wrong_keep": wrong_keep, "wrong": wrong_delete + wrong_keep} L = files() print("oracle: files really stale:", sum(d["last_read"] < THRESHOLD for d in L), "/", FILES) print() print("option deleted wrong delete wrong keep wrong diagnosis") for o in ("atime", "relatime", "noatime"): t = cleanup(L, o) print(f" {o:9s} {t['deleted']:7d} {t['wrong_delete']:12d}" f" {t['wrong_keep']:13d} {t['wrong']:11d}") print() for seed in (SEED, 20260219): l2 = files(seed) print(f"seed {seed}: stale {sum(d['last_read'] < THRESHOLD for d in l2):4d}" f" atime {cleanup(l2, 'atime')['wrong']:4d}" f" relatime {cleanup(l2, 'relatime')['wrong']:4d}" f" noatime {cleanup(l2, 'noatime')['wrong']:4d}") ``` ``` oracle: files really stale: 717 / 2000 option deleted wrong delete wrong keep wrong diagnosis atime 717 0 0 0 relatime 1204 487 0 487 noatime 1504 787 0 787 seed 20260218: stale 717 atime 0 relatime 487 noatime 787 seed 20260219: stale 728 atime 0 relatime 502 noatime 821 ``` Three numbers side by side. **The oracle:** **717** of the 2000 files really have not been read for three hundred seconds. **The tool's output:** with the no-update option, when the files' access time is read, **1504** of them look stale. **Wrong diagnosis:** the cleanup job wrongly deletes **787 files** and produces not a single error message. The difference between the three rows is a design trade-off. Updating access time on every read turns a read-only workload into a write workload; every file read produces a metadata write. Relaxed updating lowers this cost and brings 487 wrong deletions; the no-update option zeroes the cost and brings 787 wrong deletions. The wrong-keep column being zero on all three rows shows the error is **one-directional**: the recorded time lags behind reality, it never runs ahead of it. In the second seed, the numbers come out to 502 and 821; the ordering does not change. The rule that follows is this: a mount option changes the assumption of the jobs running on top of it. The option list appears in `findmnt`'s output, and if whoever looks at that list does not know which job relies on which field, they cannot draw any diagnosis from the row they see. #### Switching to Read-Only on Error Some mount options set not operations but **failure behavior**. A file system learns at mount time what to do when it detects inconsistency in its own ledgers or an error from the device: it can ignore it and continue, switch itself to read-only, or halt the machine. The common default behavior is the middle one. Switching to read-only has an odd side from the operator's point of view. The device is still mounted, still visible in the tree, reads work, and `df` keeps showing usage; only writes are rejected. Units that write take this rejection as an error, and the restart policy covers this error over — the condition measured in the Services topic holds here too and is not repeated. The result is a system that looks like it is running but writes nothing. The record of this transition sits in exactly one place: a single line dropped into the system log. The line is written at the moment of transition and never written again; it disappears once the log is rotated. This is the most expensive example of the third claim from the Logs topic — the only evidence of the failure is kept in an arrangement that erases evidence by itself. In the mount table's option list, the `ro` marker appears, and that marker does not say the file system transitioned into this state **afterward**; it would look the same if the entry had been written read-only from the start. #### A Wrong Entry and the Way Back The file holding persistent mount entries is one of the least-written and most damaging files on the system. A wrongly written line can halt boot; if the system is remote and there is no console access, the machine becomes unreachable. This is why the process of writing an entry is not done alone, but through a three-step procedure. The first is **syntax and target verification**: `findmnt --verify` reads the entries, checks whether the device identities have a match, and checks whether the mount points exist; it mounts nothing. The second is **the trial**: the entries are reread, whatever is not yet mounted gets mounted, and the result is seen in the same session. The third is **the way back**: a copy of the file is taken before the change is made, and the restart is attempted at a time when rescue-shell access is possible. The order entries are read in is a trap too. Mount points can be nested, and an inner point cannot be mounted before the outer one is; when the order breaks, the inner file system mounts onto a directory on the root file system and does not sit where it is expected. What resolves the dependency is not the entries' order in the file but the mount points' path depth; tools do this ordering themselves, but a manually issued mount command does not. Unmounting is not as harmless as it seems either. A file system cannot be detached while a process has an open handle on it, and forcing the detach with the forcing option leaves whatever those processes were writing half-done. This lesson does not give the forcing detach command in complete form; the correct order is stopping the units that use the file system first, then verifying no open handle remains, and detaching last. #### Summary - Mounting attaches the file system on a device to a point in the single tree; the mount point's previous content is not deleted, it becomes invisible and keeps taking up space on the underlying file system. - Device names are assigned in discovery order: a persistent mount entry written to the name `device-b` mounts the correct file system in only 13 of sixty boots, silently mounts the wrong one in 27, and produces a noisy error in 20. Written to the identity, the wrong diagnosis is zero. - The mount option that does not update access time rejects no operation; it only freezes the recorded time. While 717 files are really stale, the job deletes 1504 files, and 787 of these are wrong deletions. - The error is one-directional: the recorded access time lags behind reality, it never runs ahead of it; this is why the wrong-keep column is zero across all three options. - When a persistent mount entry is written wrongly, boot can halt; the verification command, trying it in the same session, and a copy of the file measurably lower this risk. #### Next Step All of this lesson's measurements assumed a partition's boundaries are fixed: a range drawn on the device and a file system built inside it. When it fills, the only thing to be done was deleting. The next lesson adds a layer that removes this fixedness — a layer that turns the partition into a pool made of volumes, can grow a volume while it runs, and can hold a copy of a moment. That layer has its own limit too, and what will be measured is the moment a snapshot fills. ### Logical Volume Management Source: https://academia.sh/en/courses/system-administration/logical-volume-management Growing is two steps, and seven of twenty-one readings point to the wrong link in the chain; a snapshot with 512 blocks allocated fills at the 413th second, and monitoring that looks once every sixty seconds sees a number sitting at 256 blocks until then. All of the previous lesson's measurements assumed a partition's boundaries are fixed: a range drawn on the device and a file system built inside it. Under this assumption, the only thing to be done for a partition that fills up was deleting; the free space on a neighboring partition stayed unreachable. This lesson measures the layer that removes this fixedness. The layer collects partitions into a pool and cuts pieces of a requested size from the pool; the cut piece can be grown while it runs, and a copy of one of its moments can be kept. The gain is real, and it brings two new sources of wrong diagnosis: which link in the chain has grown, and how long a copy of a moment will last. #### Three Layers and the Unit Between Them The layer consists of three objects. A **physical volume** is a block device or partition that has joined the pool. A **volume group** is one or more physical volumes presented as a single area. A **logical volume** is a piece cut from the pool that a file system can be built on. The cutting is done not byte by byte but in fixed-size **extents**. An extent's size is set when the pool is created; every logical volume holds a whole number of extents, and growing happens in multiples of these units too. This has two consequences: the requested size gets rounded up, and a logical volume can spread across **more than one** of the pool's physical volumes. Spreading is both the layer's strength and its silent risk. If a logical volume is spread across two physical volumes, either one failing makes the whole volume unusable; adding a device to the pool does not raise durability, it only raises capacity. The layer itself is not a redundancy scheme, and the diagnosis errs most expensively when it is assumed to be. The three layers are read with three tools: `pvs` lists physical volumes, `vgs` lists pools, `lvs` lists logical volumes. The transcript below is a **sample transcript** showing the shape of `lvs`'s output; it was not run, and no numeric claim in this lesson comes from it. ```text LV VG Attr LSize Pool Origin Data% Meta% data pool-1 -wi-ao---- 32.00g data-snapshot pool-1 swi-a-s--- 2.00g data 41.30 log pool-1 -wi-ao---- 8.00g ``` The `Origin` and `Data%` columns in the second row are the subject of this lesson's second measurement: a snapshot's source and its fill level. #### Growing Is Two Steps Growing a logical volume does not grow the file system. The layer adds new extents to the volume; the file system keeps holding its own ledgers according to the old boundary and does not see the new space. The two steps are separate, their order is binding, and in the gap between them, tools report different numbers. - **ST26** — The chain passes through seven stages: start, adding a physical volume to the pool, growing the logical volume, growing the file system, writing continuing, a snapshot getting taken, and the snapshot dropping. - **ST27** — All numbers are in blocks; the extent is a multiple of a block in this measurement, and rounding is not counted. - **ST28** — The diagnosis asked is a single one: **how many more blocks can be written right now.** The oracle is the file system's capacity minus what is used. ```python """Growing is two steps; every tool shows a different link in the chain.""" STAGE = ( ("0 start", 0, 4096, 4096, 3800), ("1 physical volume added", 8192, 4096, 4096, 3800), ("2 logical volume grown", 4096, 8192, 4096, 3800), ("3 file system grown", 4096, 8192, 8192, 3800), ("4 writing continued", 4096, 8192, 8192, 6000), ("5 snapshot taken", 3584, 8192, 8192, 6000), ("6 snapshot dropped", 4096, 8192, 8192, 6000), ) def readings(a): """Question: how many more blocks can be written right now.""" _, vg_free, lv, fs, used = a return {"oracle": fs - used, "vgs": vg_free + lv - used, # thinks the pool's free space is writable "lvs": lv - used, # thinks the volume's size is writable "df": fs - used} print("stage vg free lv fs used oracle" " vgs lvs df") for a in STAGE: o = readings(a) print(f"{a[0]:24s} {a[1]:7d} {a[2]:5d} {a[3]:5d} {a[4]:11d}" f" {o['oracle']:6d} {o['vgs']:6d} {o['lvs']:6d} {o['df']:6d}") print() print("tool correct reads wrong reads") total = 0 for tool in ("vgs", "lvs", "df"): y = sum(readings(a)[tool] != readings(a)["oracle"] for a in STAGE) total += y print(f" {tool:3s} {len(STAGE) - y:11d} {y:12d}") print("wrong diagnosis:", total, "/", len(STAGE) * 3) ``` ``` stage vg free lv fs used oracle vgs lvs df 0 start 0 4096 4096 3800 296 296 296 296 1 physical volume added 8192 4096 4096 3800 296 8488 296 296 2 logical volume grown 4096 8192 4096 3800 296 8488 4392 296 3 file system grown 4096 8192 8192 3800 4392 8488 4392 4392 4 writing continued 4096 8192 8192 6000 2192 6288 2192 2192 5 snapshot taken 3584 8192 8192 6000 2192 5776 2192 2192 6 snapshot dropped 4096 8192 8192 6000 2192 6288 2192 2192 tool correct reads wrong reads vgs 1 6 lvs 6 1 df 7 0 wrong diagnosis: 7 / 21 ``` Three numbers side by side. **The oracle:** at the second stage, the writable space is **296 blocks** — even though the logical volume has doubled. **The tool's output:** at the same stage, `lvs` shows 4392, `vgs` shows 8488 blocks. **Wrong diagnosis:** **seven** of twenty-one readings are wrong. The table shows the real difficulty the layer brings: the same question is answered by three separate layers, and only the topmost one is correct. The pool's free space is never directly writable space; a logical volume's size is not writable space unless the file system is grown too. A process stopped at the second stage — the growth command issued, the file system not grown — leaves the system in a state thought to have "opened up space" but that has not, and the full-disk failure surfaces **after the growth was done**. The order itself is binding too. In growing, the logical volume is grown first, then the file system; in shrinking, the order is reversed, and if it is not reversed, the file system keeps pointing at blocks left outside its own boundary. The difference between these two sentences is an unrecoverable data loss. #### How a Snapshot Fills A **snapshot** is a second volume that keeps a logical volume's state at a given moment readable. The copying is not done at the start; the first time a block in the source is written over, its **old state** is moved into the snapshot's space. This is why the space a snapshot consumes is proportional not to the source's size, but to the **number of blocks that change** in the source. When the allocated space runs out, the snapshot becomes invalid and drops. The source volume is unaffected and keeps running; what is lost is the validity of the backup being taken from that snapshot. - **ST29** — The source logical volume is **4096 blocks**, and the space allocated for the snapshot is **512 blocks**. The usual load renews two of the **256 blocks in the hot set** every second. - **ST30** — Between the 400th and 430th seconds, a backup job runs a broad scan and touches **20 blocks** every second; the touched blocks are drawn from the whole volume. - **ST31** — The monitoring tool reads fill level once every `interval` seconds; the first reading that crosses the threshold is a warning. If the warning arrives **after** the fill moment, or never arrives, the diagnosis counts as wrong. ```python """A snapshot copies on write; the space allocated for it invalidates once full.""" SEED = 20260218 PERIOD = 600 VOLUME_BLOCKS = 4096 # source logical volume's block count ALLOCATED = 512 # blocks allocated for the snapshot HOT_SET = 256 # block set the usual load touches BURST = (400, 430) # interval where the backup job runs a broad scan def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def copied(seed=SEED): """Oracle: blocks the snapshot holds at the end of every second.""" r = generator(seed) seen, curve = set(), [] for t in range(PERIOD): if BURST[0] <= t < BURST[1]: for _ in range(20): seen.add(r(VOLUME_BLOCKS)) else: for _ in range(2): seen.add(r(HOT_SET)) curve.append(min(len(seen), ALLOCATED)) return curve def fill_time(curve): for t, v in enumerate(curve): if v >= ALLOCATED: return t return None def diagnose(curve, interval, threshold): """The tool looks every `interval` seconds; a reading past the threshold is a warning.""" fills = fill_time(curve) looks = [t for t in range(0, PERIOD, interval) if t < fills] warning = next((t for t in range(0, PERIOD, interval) if curve[t] / ALLOCATED >= threshold), None) on_time = warning is not None and warning < fills return {"last_reading": curve[looks[-1]] if looks else 0, "warning": warning, "wrong": not on_time} E = copied() D = fill_time(E) print("oracle: snapshot fills at", D, "seconds |", ALLOCATED, "blocks allocated") print("curve:", [(t, E[t]) for t in (0, 60, 180, 300, 360, 400, 410, 420)]) print() print("interval threshold last reading before fill first warning wrong diagnosis") wrong = 0 for interval in (10, 30, 60, 120, 300): for threshold in (0.70, 0.90): s = diagnose(E, interval, threshold) wrong += s["wrong"] print(f"{interval:8d} {threshold:.2f} {s['last_reading']:24d}" f" {str(s['warning']):>9s} {str(s['wrong']):>11s}") print("wrong diagnosis:", wrong, "/ 10") print() for seed in (SEED, 20260219): e2 = copied(seed) y = sum(diagnose(e2, i, t)["wrong"] for i in (10, 30, 60, 120, 300) for t in (0.70, 0.90)) print(f"seed {seed}: fill time {fill_time(e2)} t=360 reading {e2[360]}" f" wrong diagnosis {y} / 10") ``` ``` oracle: snapshot fills at 413 seconds | 512 blocks allocated curve: [(0, 2), (60, 122), (180, 256), (300, 256), (360, 256), (400, 276), (410, 466), (420, 512)] interval threshold last reading before fill first warning wrong diagnosis 10 0.70 466 410 False 10 0.90 466 410 False 30 0.70 256 420 True 30 0.90 256 420 True 60 0.70 256 420 True 60 0.90 256 420 True 120 0.70 256 480 True 120 0.90 256 480 True 300 0.70 256 None True 300 0.90 256 None True wrong diagnosis: 8 / 10 seed 20260218: fill time 413 t=360 reading 256 wrong diagnosis 8 / 10 seed 20260219: fill time 413 t=360 reading 256 wrong diagnosis 8 / 10 ``` Three numbers side by side. **The oracle:** the snapshot fills at the **413th second**. **The tool's output:** monitoring that looks once every sixty seconds sees **256 blocks** at its last reading before filling — exactly half of the allocated space. **Wrong diagnosis:** **eight** of the ten sampling-and-threshold combinations fail to produce the warning on time. The curve shows why. In the first three hundred seconds, fill level sits at 256, because the usual load keeps touching the same hot set, and once that set has been copied once, it needs no new copy. The number does not budge for three hundred seconds and looks **stable** to whoever is watching. Then the backup job runs its broad scan, and fill level climbs from 256 to 512 in thirteen seconds. Lowering the threshold does not fix this: in the table, the 0.70 and 0.90 thresholds give the same result, because the problem is not how high the threshold is, but **the distance between two looks**. The same structure as the sampling measurement in the Process Management topic holds here too. In the second seed, neither the fill moment nor the wrong-diagnosis count changes. The reason is structural: the hot set saturates under every seed, and the burst's width is independent of the seed. The result depends not on the fiction but on copy-on-write's own behavior. The design rule that follows is this: the space allocated to a snapshot is chosen not by the source's size, but by **the number of blocks that will change over the time the snapshot will live**. A snapshot kept for a long time can demand as much space as the whole source. In practice, three precautions follow directly from this measurement. The first is **shortening the snapshot's lifetime**: a snapshot removed right after the backup is taken ends without any threshold being crossed. The second is reading fill level **together with the event** — two readings, taken as the backup job starts and finishes, say more than a regular sixty-second sample. The third is that the snapshot **can be grown while it runs**: new extents can be added from the pool before the allocated space runs out, but not after it has. What the three share is not leaving the decision to the sampling interval. There is also a point commonly misunderstood. A snapshot is not a backup; it sits on the same physical volumes as its source, and the two are lost together if the device underlying the source fails. What it provides is a **consistent read moment** — it makes it possible for the process taking the backup to read from a fixed image while the data keeps changing. Backup itself is outside this course and has been measured elsewhere. #### Shrinking and Deleting Among this layer's commands, the irreversible ones are not open-ended; their names are known. The command that deletes a logical volume returns that volume's extents to the pool, and the file system inside it can never be built again. The command that shrinks a logical volume returns the file system's last blocks to the pool if the file system has not been shrunk beforehand; if the file system is using those blocks, the data loss is silent and certain. This lesson does not give these two commands in a runnable, complete form. The safe way to test is three steps. The first is a **dry run**: volume-management tools carry an option that reports the result without applying the change, and the real command is not issued before that option's output is read. The second is a **snapshot**: a snapshot taken before the change makes recovery possible if the source is corrupted — as long as it is used before the allocated space fills. The third is **testing in a separate pool**: a small pool built from loop devices makes it possible to run the whole chain away from real data. #### Summary - The layer consists of three objects: physical volumes that have joined the pool, the volume group presenting them as a single area, and logical volumes cut from the pool in fixed-size extents. Adding a device to the pool raises capacity, not durability. - Growing is two steps, and the gap between tools is measurable: seven of twenty-one readings across seven stages are wrong; `vgs` errs six times, `lvs` once, `df` never. - When the logical volume is grown but the file system is not, the writable space stays at 296 blocks, but `lvs` shows 4392, `vgs` shows 8488. - A snapshot fills not by the source's size but by the number of blocks that change: a 512-block snapshot fills at the 413th second, and monitoring that looks once every sixty seconds sees a number sitting at 256 blocks until then. Eight of ten sampling-and-threshold combinations fail to produce the warning on time. - Lowering the threshold does not fix this miss; what distinguishes is the distance between two looks. The commands for deleting and shrinking a volume are irreversible, and this lesson does not give them in complete form. #### Next Step This lesson treated storage only as a layer that holds data. Yet a portion of the same block device holds no files at all: it is the space pages get written to under memory pressure. The next lesson measures that space and answers a question — in which indicator does swap usage show up, and how late. The measurement will not come out one-directional: some indicators report pressure late, one keeps reporting it even after the pressure has passed, and one shouts even when there is no pressure at all. ### Swap Space Source: https://academia.sh/en/courses/system-administration/swap-space Pressure lasts 84 seconds, but the used-swap indicator raises a false alarm for 272 seconds, free memory for 451; as the swap area grows, the closability diagnosis rises from 44 wrong to 354 wrong. The previous lessons treated storage only as a layer that holds data: partitions, file systems, mount points, and logical volumes. A portion of the same block device, though, holds no files at all and appears at no mount point. That portion is where pages are written when memory falls short. This lesson's question is not one of storage but of **seeing**: in which indicator does swap usage show up, and how late. The answer will not come out one-directional. Some indicators report pressure late, one keeps reporting it even after the pressure has passed, and one shouts even when there is no pressure at all. #### Swap Space and What It Is Not **Swap** is the disk space pages that do not fit in physical memory get written to. It can be a partition, it can be a file; both are formatted the same way and activated the same way. The file form is flexible because it can be added and removed afterward, the partition form is plain because it is independent of the file system's state. The difference between them is not a ranking of superiority but a management preference. The mechanics of paged memory were measured in the Operating System Concepts course: translating a virtual page to a physical page, the page fault, and the page-replacement procedure are defined there and are not repeated here. This lesson counts only what the operator can see. What can be seen is a handful of numbers: swap's total size, its used portion, pages written to swap and read from swap per second, free memory, and available memory. These are read through `swapon --show`, `free`, and the file holding memory statistics. The transcript below is a **sample transcript** showing the shape of the output; it was not run, and no numeric claim in this lesson comes from it. ```text NAME TYPE SIZE USED PRIO /dev/sdb4 partition 4G 1.2G -2 total used free shared buff/cache available Mem: 7.6G 4.1G 160M 88M 3.3G 3.1G Swap: 4.0G 1.2G 2.8G ``` The `free` and `available` columns being separate in the transcript is the subject of this lesson's first measurement: one does not count reclaimable cache as free, the other does. #### Four Indicators, Four Separate Errors The measurement is done in a 600-second window on the fictional server. The report-producing unit starts growing gradually from the two-hundredth second, finishes its work at the three-hundred-twentieth second; there is one more short, sharp spike at the four-hundred-eightieth second. - **ST32** — Physical memory is **1200 pages**, the usual resident demand is **900 pages**. Pressure is the second demand exceeds physical memory, and the oracle knows this. - **ST33** — A page written to swap stays there; used swap does **not decrease** until it is read back. This is the behavior that is the axis of the measurement. - **ST34** — The cache fills free memory and grows by **four pages** per second. Available memory counts reclaimable cache as free. - **ST35** — The warning threshold is **100 pages**. The tool looks once every `interval` seconds; between two looks, the operator's belief is the last reading. ```python """How late (or how early) swap usage shows up in each indicator.""" SEED = 20260218 PERIOD = 600 PHYSICAL = 1200 # physical memory, pages BASELINE = 900 # usual resident demand THRESHOLD = 100 # memory warning threshold, pages AMPLITUDE = 60 # fluctuation in demand def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def demand(seed=SEED): """Oracle: the resident pages really requested every second.""" r = generator(seed) values = [] for t in range(PERIOD): if 200 <= t < 320: base = BASELINE + (t - 200) * 6 # report job grows gradually elif 480 <= t < 495: base = 1700 # short, sharp spike else: base = BASELINE values.append(base + r(2 * AMPLITUDE + 1) - AMPLITUDE) return values def curves(seed=SEED): d_list = demand(seed) used, rate, free, available, pressure = [], [], [], [], [] peak = previous = 0 for t, d in enumerate(d_list): overflow = max(0, d - PHYSICAL) delta = abs(overflow - previous) # pages sent to and read from swap per second previous = overflow peak = max(peak, overflow) # written page stays until read back headroom = PHYSICAL - min(d, PHYSICAL) cache = min(4 * t, headroom) # cache fills the free space used.append(peak) rate.append(delta) free.append(headroom - cache) available.append(headroom) # reclaimable cache is not counted pressure.append(d > PHYSICAL) return {"used": used, "rate": rate, "free": free, "available": available, "pressure": pressure} RULE = {"used swap": lambda e, i: e["used"][i] > 0, "swap rate": lambda e, i: e["rate"][i] > 0, "free memory": lambda e, i: e["free"][i] < THRESHOLD, "available memory": lambda e, i: e["available"][i] < THRESHOLD} def count(e, rule, interval): missed = false_alarm = 0 last = False for t in range(PERIOD): if t % interval == 0: last = rule(e, t) missed += e["pressure"][t] and not last false_alarm += last and not e["pressure"][t] return missed, false_alarm E = curves() print("oracle: seconds under pressure:", sum(E["pressure"]), "/", PERIOD) print("indicator t=100 t=240 t=300 t=330 t=485 t=560") for name in ("used", "rate", "free", "available"): print(f" {name:14s}", " ".join(f"{E[name][t]:5d}" for t in (100, 240, 300, 330, 485, 560))) print() print("indicator interval missed false alarm wrong diagnosis") for name, rule in RULE.items(): for interval in (5, 30, 60, 120): missed, false_alarm = count(E, rule, interval) print(f" {name:21s} {interval:6d} {missed:9d} {false_alarm:12d}" f" {missed + false_alarm:11d}") print() for seed in (SEED, 20260219): e = curves(seed) s = " | ".join(f"{name.split()[0]} {sum(count(e, k, 5))}" for name, k in RULE.items()) print(f"seed {seed}: pressure {sum(e['pressure']):3d} seconds | interval 5 : {s}") ``` ``` oracle: seconds under pressure: 84 / 600 indicator t=100 t=240 t=300 t=330 t=485 t=560 used 0 0 331 423 533 535 rate 0 0 57 0 61 0 free 0 0 0 0 0 0 available 268 120 0 258 0 333 indicator interval missed false alarm wrong diagnosis used swap 5 1 272 273 used swap 30 19 265 284 used swap 60 49 265 314 used swap 120 69 225 294 swap rate 5 2 13 15 swap rate 30 19 25 44 swap rate 60 49 85 134 swap rate 120 69 105 174 free memory 5 0 451 451 free memory 30 0 426 426 free memory 60 0 396 396 free memory 120 0 396 396 available memory 5 1 12 13 available memory 30 19 25 44 available memory 60 49 85 134 available memory 120 69 105 174 seed 20260218: pressure 84 seconds | interval 5 : used 273 | swap 15 | free 451 | available 13 seed 20260219: pressure 86 seconds | interval 5 : used 271 | swap 16 | free 469 | available 9 ``` Three numbers side by side. **The oracle:** the window has **84 seconds** of real memory pressure. **The tool's output:** the indicator whose name directly names swap, used swap, sits at **423** at the three-hundred-thirtieth second, at **533** at the four-hundred-eighty-fifth and **535** at the five-hundred-sixtieth, and never comes down. **Wrong diagnosis:** even read once every five seconds, that indicator raises a false alarm for **272 seconds**; its total is **273**. The four indicators have four separate ways of erring. **Used swap** is cumulative: once a page is written to swap, it stays there until read back, so the indicator stays high even after the pressure ends and shows the past as if it were today; its false alarm reaches 272 seconds. **Swap rate** is instantaneous, and at a five-second interval it misses only 2 seconds and raises 13 false alarms; when the interval rises to sixty, the missed count jumps to 49, because the spike lasts fifteen seconds and falls between two looks. **Free memory** alarms almost the whole time — 451 seconds — because the cache fills the free space, and free memory sits near zero even on a healthy system. **Available memory** misses 1 second and raises 12 false alarms at the five-second interval. The two lowest totals are 13 and 15, and both depend on the same sampling interval: when the interval rises to sixty, all four climb to at least 134. In the second seed, pressure lasts 86 seconds, and the five-second-interval totals come out to 271, 16, 469, and 9; the ordering does not change. The course's rule finds its exact match here: **what distinguishes is not how many numbers are looked at, but which number is looked at.** The indicator that counts the swap amount directly turned out to be the worst one for finding swap pressure. #### The Cost of Turning It Off Turning off swap is a one-line command, and it has a one-line risk: every page in the area has to be **read back** into physical memory. If there is not enough room, the operation either fails or ends with a process being terminated for lack of memory. - **ST36** — Turning off is possible when used swap fits into available memory. The oracle makes this comparison every second. - **ST37** — The diagnosis tested is the common one: **if usage percentage is under half, it can be turned off.** ```python """To turn off swap, every page in it must be read back into memory.""" SEED = 20260218 PERIOD = 600 PHYSICAL = 1200 BASELINE = 900 AMPLITUDE = 60 def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def demand(seed=SEED): r = generator(seed) values = [] for t in range(PERIOD): if 200 <= t < 320: base = BASELINE + (t - 200) * 6 elif 480 <= t < 495: base = 1700 else: base = BASELINE values.append(base + r(2 * AMPLITUDE + 1) - AMPLITUDE) return values def state(seed=SEED): used, available, peak = [], [], 0 for d in demand(seed): peak = max(peak, max(0, d - PHYSICAL)) used.append(peak) available.append(PHYSICAL - min(d, PHYSICAL)) return used, available def tally(size, seed=SEED): used, available = state(seed) actual = diagnosed = wrong = 0 for t in range(PERIOD): g = used[t] <= available[t] # oracle: do the pages fit back in memory y = used[t] / size < 0.50 # diagnosis: usage under half actual += g diagnosed += y wrong += g != y return actual, diagnosed, wrong print("swap size closable seconds closable per DIAGNOSIS wrong diagnosis") for size in (512, 1024, 2048, 4096): g, t, y = tally(size) print(f"{size:9d} {g:16d} {t:21d} {y:11d}") print("observation window:", PERIOD, "seconds") print() for seed in (SEED, 20260219): print(f"seed {seed}: " + " | ".join( f"size {size}: wrong {tally(size, seed)[2]}" for size in (512, 4096))) ``` ``` swap size closable seconds closable per DIAGNOSIS wrong diagnosis 512 246 290 44 1024 246 483 237 2048 246 600 354 4096 246 600 354 observation window: 600 seconds seed 20260218: size 512: wrong 44 | size 4096: wrong 354 seed 20260219: size 512: wrong 45 | size 4096: wrong 355 ``` Three numbers side by side. **The oracle:** swap can be safely turned off in **246** of six hundred seconds. **The tool's output:** in a 2048-page swap area, usage never rises above fifty percent, meaning the indicator says "fine" in **600 of the window's 600 seconds**. **Wrong diagnosis:** the same rule is mistaken 44 times at 512 pages, and **354** times at 2048 and above. In the second seed, these two numbers come out to 45 and 355. The result runs against intuition, and precisely for that reason it is instructive: **growing the swap area makes the percentage-based diagnosis worse.** The same 533 pages appear as nearly all of a small area, an eighth of a large one; the percentage changes even though the page count does not. A percentage is a ratio, and the ratio's denominator here is a number the operator chose. The right question is not "how full is swap" but "do the pages inside it fit back into memory," and the number that answers that question is not in swap's output but in memory's. #### How Much It Should Be and What Not to Touch Swap's size is not a correctness decision but a budget decision, and it has a cost at both ends. On a system with no swap at all, a process gets terminated the moment demand exceeds physical memory; the failure is hard, fast, and easy to diagnose. On a system with generous swap, the same demand produces no termination, but pages go to disk and every read-back adds a wait; the failure is soft, slow, and hard to diagnose. The choice between the two is made according to which failure is manageable. There is also swap tendency: the kernel carries a setting that determines whether, when it applies pressure, it reclaims the page cache first or the rarely used memory pages first. Raising this setting sends pages to swap under even the mildest pressure; the used-swap indicator climbs early, and the false-alarm column in the table above grows even further. The destructive-command boundary applies in two places here. First, the command that formats a partition as swap **irreversibly** destroys the file system on that partition; if the wrong partition name is written, it runs without a warning. Second, turning off swap while under pressure can lead to process termination. This lesson does not give either command in a runnable, complete form. The safe way to test is verifying the target partition's identity beforehand, doing the trial with swap built on a file, and attempting the turn-off only at a moment when used swap is below available memory. #### Summary - Swap can be a partition or a file, and it appears at no mount point; the mechanics of paged memory are not repeated in this course, what is measured is what the operator can see. - Real memory pressure in the six-hundred-second window is 84 seconds; even read once every five seconds, the used-swap indicator raises a false alarm for 272 seconds, because a page written to swap stays there until it is read back. - Free memory raises a false alarm for 451 seconds, because the cache fills the free space; available memory gives the lowest total at the five-second interval, with 1 miss and 12 false alarms. - Swap rate misses 2 seconds and raises 13 false alarms at the same interval, but when the interval rises to sixty, the missed count jumps to 49; at coarse sampling, all four indicators climb to at least 134. - Swap can be safely turned off in 246 of the window's 600 seconds; the rule that looks at usage percentage is mistaken 44 times at 512 pages, 354 times at 2048 and above — growing the area makes the percentage-based diagnosis worse. #### Next Step Throughout this topic, storage was always something that already existed: a device, partitions on it, file systems inside them. The course's last lesson returns to the start of the chain and follows a new device joining the system from start to finish — recognition, partitioning, formatting, mounting, making it persistent, and extending it. At every step a tool says "done," and a layer above is not yet aware of it; what will be measured is how many steps of the chain give a wrong "the work is done" diagnosis. The lesson also closes the course. ### Adding and Expanding a Disk Source: https://academia.sh/en/courses/system-administration/adding-and-expanding-a-disk The chain for joining a new device to the system is nine steps, and thirteen of forty-five readings say the work is done when it is not; a persistent entry written to a device name mounts the correct file system in only ten of forty boots. Throughout this topic, storage was always something that already existed: a device, partitions on it, file systems inside them, mount points above them. The course's last lesson returns to the start of the chain and follows a device joining the system from start to finish. What is followed is not the steps themselves but the **gap** between them. At every step a tool says "done," and a layer above is not yet aware of it. The number to be measured is how many readings of the chain give a wrong "the work is done" diagnosis. #### The Chain's Links When a new device is physically attached, the system does not use it by itself. There are nine steps in between, and every step depends on the one before it. The device must first be **recognized**: the kernel scans the bus and adds the device to its tree. Then it is **partitioned**; the partition table gets written to the device, but the kernel can keep holding the old table in memory, and a **reread** is needed. After that, a **file system is built** on the partition. The built file system is not visible in the tree; it needs to be **mounted**. Because a manual mount is lost on restart, a **persistent mount entry** is written. Finally, the unit that will produce the data must **start writing** to the new path; the path changing does not by itself change the unit's configuration. - **ST38** — The chain is nine steps, and the order of steps is fixed in this measurement; the persistent mount entry is written **before** the manual mount, because this is common practice. - **ST39** — The oracle is the state "the new space is persistently in use": the file system is mounted, the entry is persistent, and the unit is writing to the new path. Until all three hold at once, the work is not done. - **ST40** — Five tools are read, and each sees a single link in the chain: `lsblk` the device, `blkid` the file system, `findmnt` and `df` the mount, entry checking the persistence. - **ST41** — In the verification boot, the discovery order is drawn from the shared definition's generator; **40 boots** are observed, and the entry is written in two forms. ```python """Chain for joining a new device to the system: what each tool says at every step.""" SEED = 20260218 BOOTS = 40 FIELDS = ("visible", "partition", "kernel_read", "fs", "mounted", "volume_writes", "persistent") STEP = ( ("0 device attached", 0, 0, 0, 0, 0, 0, 0), ("1 rescanned", 1, 0, 0, 0, 0, 0, 0), ("2 partition created", 1, 1, 0, 0, 0, 0, 0), ("3 kernel read table", 1, 1, 1, 0, 0, 0, 0), ("4 file system built", 1, 1, 1, 1, 0, 0, 0), ("5 persistent entry written", 1, 1, 1, 1, 0, 0, 1), ("6 manually mounted", 1, 1, 1, 1, 1, 0, 1), ("7 unit writes to new path", 1, 1, 1, 1, 1, 1, 1), ("8 restarted", 1, 1, 1, 1, 1, 1, 1), ) TOOL = {"lsblk": "visible", "blkid": "fs", "findmnt": "mounted", "df": "mounted", "entry check": "persistent"} def state(a): d = dict(zip(FIELDS, a[1:])) d["oracle"] = d["mounted"] and d["persistent"] and d["volume_writes"] return d print("step ", " ".join(f"{k:>14s}" for k in TOOL), " oracle") for a in STEP: d = state(a) print(f"{a[0]:28s}", " ".join(f"{str(bool(d[v])):>14s}" for v in TOOL.values()), f" {str(bool(d['oracle'])):>6s}") print() print("tool correct reads wrong reads") total = 0 for tool, field in TOOL.items(): y = sum(bool(state(a)[field]) != bool(state(a)["oracle"]) for a in STEP) total += y print(f" {tool:14s} {len(STEP) - y:11d} {y:12d}") print("wrong diagnosis:", total, "/", len(STEP) * len(TOOL)) print() def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def verify(seed, entry_format, device_count=5, count=BOOTS): """Persistent entry written to a DEVICE NAME shifts when discovery order changes.""" r = generator(seed) correct = 0 for _ in range(count): remaining, order = list(range(device_count)), [] while remaining: order.append(remaining.pop(r(len(remaining)))) correct += 1 if entry_format == "id" else order[1] == 1 return correct for seed in (SEED, 20260219): print(f"seed {seed}: {BOOTS} boots |" f" entry written to name correct {verify(seed, 'name'):3d}" f" | entry written to id correct {verify(seed, 'id'):3d}") ``` ``` step lsblk blkid findmnt df entry check oracle 0 device attached False False False False False False 1 rescanned True False False False False False 2 partition created True False False False False False 3 kernel read table True False False False False False 4 file system built True True False False False False 5 persistent entry written True True False False True False 6 manually mounted True True True True True False 7 unit writes to new path True True True True True True 8 restarted True True True True True True tool correct reads wrong reads lsblk 3 6 blkid 6 3 findmnt 8 1 df 8 1 entry check 7 2 wrong diagnosis: 13 / 45 seed 20260218: 40 boots | entry written to name correct 10 | entry written to id correct 40 seed 20260219: 40 boots | entry written to name correct 6 | entry written to id correct 40 ``` Three numbers side by side. **The oracle:** the work finishes at the **seventh step**; in none of the states before that is the new space persistently in use. **The tool's output:** `lsblk` shows the device from the first step onward, and what it shows is correct — just not the answer to the question being asked. **Wrong diagnosis:** **thirteen** of forty-five readings say the work is done. The distribution shows the chain's shape. The lowest-level tool is wrong the most: `lsblk` six times, `blkid` three times. The topmost tools are wrong the least, but not zero — `findmnt` and `df` once each, because at the sixth step the file system is mounted and the unit is still writing to the old path. Entry checking is wrong twice, because the persistent entry is written before mounting and the file is correct on paper. **No tool sees the whole chain**, and each tells the truth about its own link. This distribution's practical consequence is an ordering rule: verification starts not from the bottom of the chain but from the top. First it is asked whether the unit is really writing to the new path, then whether the mount is persistent, and last whether the device is visible; asked in reverse order, the first three answers come back positive and the questioning stops there. The last two lines measure the chain's eighth step. In the verification boot, if the entry is written to the device name, the correct file system mounts in only **ten** of forty boots; in the second seed, this number drops to **six**. With the entry written to the identity, forty of forty are correct. A chain not verified by a restart does not count as complete. #### Irreversible Steps Three of the chain's nine steps are irreversible, and all three gather in the same place: the steps that write to the device. Writing the partition table erases the old layout, building a file system erases everything on the partition, and formatting as swap does the same thing. The common risk of all three is choosing the wrong target, and none of them asks "are you sure." This lesson does not give these commands in a **runnable, complete form**. The safe path is four steps, and all four are measurable. The first is **verifying the target by its identity**: in `lsblk` and `blkid`'s output, it is read that the target device is empty, has no mounted file system on it, and is the expected size. The second is a **dry run**: the option that shows partitioning tools' result without applying it, and the table's text dump, print out beforehand what the change will be. The third is **testing on a separate device**: mounting a file as a loop device makes it possible to run the whole chain away from real data. The fourth is a **snapshot**: if the change is on a logical volume, a snapshot taken beforehand makes recovery possible. The expansion direction follows the same rule too, and it has an added ordering. When a device is grown, the kernel does not see the new size by itself; the logical volume cannot grow before the partition grows, the file system cannot grow before the logical volume grows. The order runs upward, and it reverses on shrinking. What comes out when it is not reversed is not an error message but a silent data loss. #### Summary - Joining a new device to the system is a nine-step chain: recognition, partitioning, rereading the table, building the file system, the persistent entry, mounting, and directing the unit to the new path. - Five tools see five separate links of the chain, and none sees the whole; thirteen of forty-five readings say the work is done when it is not. The lowest-level tool is wrong the most: `lsblk` six, `blkid` three times. - The chain does not count as complete until it is verified by a restart: an entry written to the device name mounts the correct file system in ten of forty boots, in six in the second seed; with the entry written to the identity, forty of forty are correct. - Three of the chain's steps are irreversible, and none of them asks for confirmation; the safe path is verifying the target by its identity, reading the dry-run output, testing on a separate device, and taking a snapshot. - Expansion is done bottom to top, shrinking top to bottom; reversing the order produces not an error message but silent data loss. #### Course Wrap-Up This course asked one question twenty-two times: **how big is the gap between what the output says and the system's reality.** In every lesson, three numbers stood side by side — the oracle, what the tool showed, and how many times the diagnosis drawn from that showing was wrong. The course's rule was this: a command's number is not the row it shows, but how many times the diagnosis drawn from those rows is wrong. The table below collects the measure of the twenty-two lessons in one place. **The rows left blank are not made up**: every row is written only from its own source lesson. | Lesson | Oracle | Tool's Output | Wrong Diagnosis | |---|---|---|---| | Listing and Searching Processes | **14** of 24 processes are heavy | interval 1/5/15/30/60 → 14, 14, 11, 9, 9 | **0, 0, 3, 5, 5**; lifetime-average column **7** | | Foreground and Background Jobs | of 13 jobs, `&` keeps **0** alive, detaching keeps **4** alive | `jobs` **13 lines** | `&` **13**, detaching **9**, the path that severs the hangup signal **0** | | Signals | **2** accept the request, **5** resist; with the forcing signal **7/7** | exit code **0** on every attempt | wait 0/10/30 seconds → **7, 6, 5** | | Process Priorities | total work **237,906 → 238,244**, load steady at 3.97 | `NI` column changed **6/6** | **6/6**; **94 percent** of the freed share goes to the other heavy processes | | Resource Limits | **8 failure causes** | **3 messages + 1 silence**, each message maps to two causes | message alone **21**, limit alone **29**, combined **13** | | The Service Manager Model | 6 units, **7** transitive dependencies | flat **0** / direct **4** / transitive **7** / ordered **9** lines | **3 / 2 / 0 / 2** — the reading that prints the most lines does not give the best diagnosis | | Service Status and Control | 6 of 6 units failed, **60** restarts | **4** units read "active," hidden time **600** | **4/6**; polling 1/5/15/30/60 → **480/99/33/18/10** | | Defining a New Service | **60** crashes, 9 of 9 combinations failed | 5 "active," 2 "exited," 2 "failed" | **7/9**; start limit none/20/10/5/3 → hidden time **600/600/106/62/41** | | The Boot Process | `ingest` → **4** affected, **5** total impacted | "**1 unit**"; **4** units never tried | **3/6** | | Bootloaders | **11** of 24 changes work | generator output **24**, command line reading **21**, behavior test **11** | **13 / 10 / 0**; remaining log lines are **0** in both | | Log Architecture | 86 errors in 4000 lines, real source **31** with `process` | six unit files: 28, 0, 31, 0, 27, 0 | **5/6**; three files **completely silent** | | Authentication Logs | **41** attack lines in 601 lines | four filters: 153, 67, 51, 43 lines | **114 / 28 / 12 / 2** | | Log Filtering and Rotation | **86 errors**, the first at the 6.1th second | kept after rotation: 86, 42, **19**, 12, **5** | of six questions **0/3/4/4/4**; ratio 0.2209 and **0.0581** | | System Health Metrics | instantaneous load crosses the threshold for **54 seconds** | the averaged band 6.18–6.25 **never** crosses the threshold | **54**; false alarm **0** | | Tracing Performance Problems | **18 events** (4/4/10) | five snapshots: 432, **1728**, 432, 54, 54 lines | **12 / 12 / 8 / 2 / 0** — the diagnosis **does not change** when lines quadruple | | Block Devices and Partitions | free space in four pieces, largest contiguous 2,881,536 sectors | partition list 3,702,784 free sectors | 6 / 12 decisions | | File Systems | 54 reads touching a corrupted block | 0 errors in a design with no checksum | 54 silent reads | | Inodes and Metadata | write fails at 32,768 files | block usage 0.0125 | 2 / 12 warnings | | Mounting and Persistent Mounts | 717 files really stale | 1504 files look stale | 787 wrong deletions | | Logical Volume Management | writable space 296 blocks | lvs 4392, vgs 8488 blocks | 7 / 21 readings | | Swap Space | memory pressure 84 seconds | used swap stays above 533 | 272 false alarms | | Adding and Expanding a Disk | work finishes at the seventh step | lsblk shows the device from the first step | 13 / 45 readings | The Storage topic's seven lessons showed one pattern over and over. Tools do not lie; each tells the truth about its own layer, and the question asked is not that layer's question. A partition list is not a gap map, a usage percentage is not writability, a logical volume's size is not a file system's size, used swap is not memory pressure. The wrong diagnosis is born in these gaps, and its count does not shrink by adding a column: in the inode measurement, all three separate warning rules gave two wrong readings each in twelve cases, only the type of the error changed. A second pattern concerned time. The snapshot filled at the four-hundred-thirteenth second, and monitoring that looked once every sixty seconds saw a number that had not budged until then; when the swap rate fell between two looks, the missed seconds rose from 2 to 49. Evidence disappears on its own, and in storage this happens through a filled area or a dropped snapshot leaving no trace behind. One who looks late sees little. Throughout the course, destructive commands were not written in complete form, and this was not an omission but part of the measurement. What the commands that write the partition table, build a file system, delete a logical volume, and format swap have in common is that they produce no warning at all when the target is chosen wrongly. The safe path was the same four steps in every lesson: verifying the target by its identity, reading the dry-run output, doing the trial on a separate device, and leaving a way back. The next course, **Linux Network Administration and Troubleshooting**, carries the same question outside a single machine. There too is an interface list, and an address appearing in the list does not mean a connection can be established; there too is a routing table, and a row in the table does not mean a packet reaches its destination. The layered-diagnosis method is this course's storage-chain layering carried over onto the network: a sequence of queries narrowing from the physical link toward the application, and at every layer a tool saying "correct here, not above." And there, evidence disappears even faster, because a packet passes through and leaves, sitting nowhere unless it is captured.