--- title: 'Listing and Searching Processes' source: 'https://academia.sh/en/courses/system-administration/listing-and-searching-processes' course: 'System Administration' language: en updated: '2026-08-17T18:10:04+00:00' license: 'CC BY-SA 4.0' --- # Listing and Searching Processes On a synthetic server of twenty-four processes, 14 are actually heavy: once the sampling interval rises to 15 seconds the tool misses 3 of them, and at 30 and 60 seconds it misses 5; a single glance gives 5–7 false diagnoses, the lifetime average gives 7, and selecting by name touches 9 processes wrongly in the best case. The Shell Programming course turned a script into a tool: it gained an argument interface, stopped on errors, cleaned up after itself, and was handed to a scheduler. Writing a script is *making something happen*. System administration begins with a different task — first, **seeing what is happening**. And seeing has its own errors. This course counts those errors. The Operating System Concepts course measured the process, the scheduler, and context switching **as mechanism**; what was measured there was the cost of the abstraction. Here the mechanism is not rebuilt: what is measured is **what the operator can see of that mechanism.** ## The Course's Measure In every lesson, three numbers stand side by side. The first is the **oracle**: the system's real state. Because we generate the fiction ourselves, we know which process is actually heavy. The second is the **tool's output**: what the tool shows. The third is the **false diagnosis** count: how many times the conclusion drawn from the output contradicts the oracle. The rule itself is one sentence: **a command's value is not the number of lines it shows, but how many times the diagnosis drawn from those lines is wrong.** An output whose diagnosis is not tested is counted as unmeasured. - **PM1.** The observation window is **600 seconds**. All durations are seconds in the model; no real time is measured. - **PM2.** The synthetic server has **24 processes**. Each process's real CPU usage is not a single number but a **time series**. - **PM3.** There are three patterns: **flat** (continuously low), **spike** (a five-second jump once every sixty seconds), **heavy** (continuously high). - **PM4.** A process is counted as **heavy** if its usage exceeds **50** even once during the window. The oracle applies this definition by looking at the full 600 seconds; the tool looks only at the instants it observes. - **PM5.** The monitoring tool looks once every `interval` seconds and keeps the **highest value it saw** per process. It knows nothing of what happened between glances. - **PM6.** The false diagnosis count is the sum of missed heavy processes and light processes mistaken for heavy. - **PM7.** The second seed is **20260219**. If the resulting diagnosis counts do not stay in the same order of magnitude, the result depends on the fiction, and is written as such. ## Reading the Process Table The command that gives the process list is `ps`; the command that gives a continuously refreshed view is `top` or one of its interactive derivatives. `ps` is a **snapshot**; the list it gives is the process table at the instant the command ran. The transcript below is **illustrative**, produced for this page, and has not been run: ```text $ ps -eo pid,user,ni,pcpu,pmem,etime,comm --sort=-pcpu PID USER NI %CPU %MEM ELAPSED COMMAND 1128 app 0 71.2 3.1 00:10:00 queue-worker 1043 monitor 5 68.4 1.2 00:10:00 metric-collector 1014 app -3 64.9 2.8 00:10:00 report-generator 1092 root 0 6.1 0.4 00:10:00 backup-job 1071 backup 10 3.0 0.2 00:10:00 cache-cleaner ``` Most columns explain themselves; two do not. `NI` is the **priority value** that will be measured in the fourth lesson. `%CPU` is the most misleading column: `ps` divides this value by the time elapsed **since the process's birth**, that is, it is a **lifetime average**. A process that has run for ten hours appears low in this column even if it consumed the core by itself for the last five minutes. `top` computes the same column differently: it represents the short interval between two refreshes. When the two commands are run on the same machine at the same time, they give different numbers for the same process, and **both are correct**; they are measuring different time spans. This distinction is exactly the source of what this lesson measures. ## The Synthetic Server The measurement rests not on a real command but on the course's common definition. The output of a command run on a real server cannot be reproduced: the machine differs, the load differs, the moment differs. Because we generate the synthetic server ourselves, we know the oracle. ```python """M03/K03 common definition: a synthetic server, the text the tools produce, and the diagnosis drawn from that text compared against the ORACLE. No real command is ever called. """ SEED = 20260218 SECOND_SEED = 20260219 PERIOD = 600 # observation window: 600 seconds def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def processes(seed=SEED, count=24): """Each process's real CPU usage is a TIME SERIES, not a single number.""" r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result def usage(process, second): """Real CPU usage, second by second. The oracle knows this; the tool does not.""" p = process["pattern"] if p == "flat": return 3 + (process["pid"] + second) % 4 if p == "heavy": return 60 + (process["pid"] + second) % 25 phase = (second + process["pid"]) % 60 # spike: a 5-second jump every 60 seconds return 88 + phase if phase < 5 else 2 + phase % 3 S = processes() print("synthetic server:", len(S), "processes |", {d: sum(1 for s in S if s["pattern"] == d) for d in ("flat", "spike", "heavy")}) print("processes actually crossing the threshold:", sum(1 for s in S if max(usage(s, t) for t in range(PERIOD)) >= 50)) ``` ``` synthetic server: 24 processes | {'flat': 10, 'spike': 7, 'heavy': 7} processes actually crossing the threshold: 14 ``` **The oracle is 14.** Seven processes are continuously heavy, and seven spike once every sixty seconds, comfortably exceeding the threshold when they do. Of the 24 processes in total, 14 cross the threshold somewhere in the window. Every number from here on is read against this 14. ## The Sampling Interval Determines the Diagnosis The monitoring tool has one setting: how often it looks. Enlarging the interval shrinks the output and reduces the load. What this does to the diagnosis can be measured. ```python # On top of the first block: S, usage, PERIOD, SECOND_SEED, processes come from there. def sample(processes_, interval, window=PERIOD): """A monitoring tool that looks once every `interval` seconds. Returns: highest seen per process.""" seen = {s["pid"]: 0 for s in processes_} glances = 0 for t in range(0, window, interval): glances += 1 for s in processes_: seen[s["pid"]] = max(seen[s["pid"]], usage(s, t)) return {"glances": glances, "seen": seen} def true_peak(processes_, window=PERIOD): return {s["pid"]: max(usage(s, t) for t in range(window)) for s in processes_} def diagnose_heavy(measurement, threshold=50): """The diagnosis drawn from the output: which processes are heavy.""" return {p for p, v in measurement.items() if v >= threshold} INTERVALS = (1, 5, 15, 30, 60) def sweep(processes_): truth = diagnose_heavy(true_peak(processes_)) rows = [] for a in INTERVALS: o = sample(processes_, a) t = diagnose_heavy(o["seen"]) rows.append((a, o["glances"], len(t), len(truth - t), len(truth - t) + len(t - truth))) return len(truth), rows oracle, rows = sweep(S) print("interval glances lines produced seen heavy missed false diagnosis") for a, glances, seen, missed, wrong in rows: print(f"{a:8d} {glances:7d} {glances * len(S):14d} {seen:10d}" f" {missed:6d} {wrong:15d}") print("oracle (actually heavy):", oracle, "| total processes:", len(S)) print() oracle2, rows2 = sweep(processes(SECOND_SEED)) print("second seed", SECOND_SEED, "- oracle:", oracle2) print(" false diagnosis:", {a: y for a, _, _, _, y in rows2}) print(" first seed :", {a: y for a, _, _, _, y in rows}) ``` ``` interval glances lines produced seen heavy missed false diagnosis 1 600 14400 14 0 0 5 120 2880 14 0 0 15 40 960 11 3 3 30 20 480 9 5 5 60 10 240 9 5 5 oracle (actually heavy): 14 | total processes: 24 second seed 20260219 - oracle: 12 false diagnosis: {1: 0, 5: 0, 15: 5, 30: 6, 60: 7} first seed : {1: 0, 5: 0, 15: 3, 30: 5, 60: 5} ``` A tool looking once every one or five seconds sees all 14 of the 14 heavy processes: **zero false diagnoses.** Once the interval rises to 15 seconds, **3** are missed; at 30 and 60 seconds, **5** are missed. The ones missed are always the same class — those that rise once every sixty seconds, for five seconds. A tool that looks once every thirty seconds is not obligated to catch a five-second spike; most of the time it does not. **The tool is not lying.** At the moment it looked, that process really was light. What is wrong is not the output but the diagnosis drawn from it: "it is not on the list, so it is not a problem." The second column carries the second claim. At interval 1, the tool produces **14,400 lines**; at interval 60, **240**. As the line count rises sixtyfold, the false diagnosis count drops from 5 to 0 — but the relationship between these two numbers is not linear: **2,880 lines give a diagnosis just as good as 14,400 lines.** Every line added past five seconds adds nothing to the diagnosis. With the second seed, the oracle drops to 12 and the false diagnoses come out as 5, 6, 7. The numbers are not the same; **the direction and order of magnitude are** — starting from zero, growing within a single-digit band as the interval grows. The result that stands is this: **enlarging the interval produces missed processes.** How many will be missed depends on the fiction. ## Single Glance vs. Lifetime Average The sampling interval is a problem of many glances. What the operator does most often, however, is a **single glance**: running `ps` once. The cost of this, and of `ps`'s own column average, is measured separately. ```python # On top of the previous blocks: S, usage, PERIOD, and diagnose_heavy come from there. TRUTH = diagnose_heavy({s["pid"]: max(usage(s, t) for t in range(PERIOD)) for s in S}) def glance(processes_, t): """The value read on a single glance: what it was using at that instant.""" return {s["pid"]: usage(s, t) for s in processes_} def lifetime_average(processes_, window=PERIOD): """The process's average since its birth. Dilutes the spike.""" return {s["pid"]: sum(usage(s, t) for t in range(window)) // window for s in processes_} readings = [(f"glance t={t:<3d} ", glance(S, t)) for t in (0, 3, 30, 300)] readings.append(("lifetime average ", lifetime_average(S))) print("reading form seen heavy missed false diagnosis") for name, measurement in readings: d = diagnose_heavy(measurement) print(f" {name}{len(d):10d} {len(TRUTH - d):9d}" f" {len(d - TRUTH) + len(TRUTH - d):11d}") print("oracle:", len(TRUTH), "heavy processes |", len(S), "processes total") ``` ``` reading form seen heavy missed false diagnosis glance t=0 9 5 5 glance t=3 8 6 6 glance t=30 7 7 7 glance t=300 9 5 5 lifetime average 7 7 7 oracle: 14 heavy processes | 24 processes total ``` A single glance gives **between 5 and 7** false diagnoses, depending on the moment it looks. Same machine, same 600 seconds, same command: the answer depends on which second it ran. This is the measure of the sentence "I ran `ps` to see the problem, and there was nothing there." The real result is in the last row. **The lifetime average is as wrong as the worst single glance:** 7 false diagnoses. Averaging does not erase the spike, it dilutes it — a five-second spike, divided over 600 seconds, drops far below the threshold. Looking at a longer window does not give a better diagnosis here; **which measure is taken comes before how much data was collected.** ## The Cost of Searching by Name There are two ways to search for a process. Filtering the list as text — narrowing `ps` output with `grep` — also catches the searching process itself, and brings back unrelated lines when the pattern is written wrong. The second way is `pgrep` and its counterpart `pkill`, which query the process table directly: ```text $ pgrep -a -u app queue-worker 1128 /opt/queue/queue-worker --shard 3 1135 /opt/queue/queue-worker --shard 4 ``` This transcript, too, is **illustrative and has not been run.** `pgrep` is more precise than text filtering: it does not list itself, and it can be narrowed by user and session. But what is precise is the **match**, not the diagnosis. Selecting by name silently accepts the assumption "the processes carrying this name are the problem." ```python # On top of the previous blocks: S and TRUTH come from there. name_table = {} for s in S: name_table.setdefault(s["command"], [0, 0]) name_table[s["command"]][0] += 1 name_table[s["command"]][1] += 1 if s["pid"] in TRUTH else 0 print("command name matched actually heavy wrong touches by name") for k in sorted(name_table): total, heavy = name_table[k] print(f" {k:22s} {total:6d} {heavy:14d} {min(heavy, total - heavy):21d}") print("total wrong touches (best case):", sum(min(a, t - a) for t, a in name_table.values())) print("how many heavy processes a single name catches at most:", max(a for _, a in name_table.values()), "/", len(TRUTH)) ``` ``` command name matched actually heavy wrong touches by name backup-job 5 3 2 cache-cleaner 2 1 1 data-receiver 1 1 0 metric-collector 5 2 2 queue-worker 6 3 3 report-generator 5 4 1 total wrong touches (best case): 9 how many heavy processes a single name catches at most: 4 / 14 ``` No command name coincides exactly with the heavy processes. Of the six processes carrying the name `queue-worker`, three are heavy and three are not; an operator selecting by name either sweeps in the three innocent processes too or leaves out the three heavy ones. Even in the best case, **9 processes are touched wrongly**, and a single name catches at most **4** of the heavy processes — the oracle is 14. The reason for this is not the fiction but the structure itself: **a command name is not an identity, it is a label.** The same program runs as multiple instances, and the instances' loads are independent of each other. Selecting by name is fast, and this speed turns into a cost when sending signals, in the third lesson of this topic. ## Summary - The course's measure is three numbers: the oracle, the tool's output, and the false diagnosis count. A command's value is not the number of lines it shows, but how many times the diagnosis drawn from those lines is wrong. - **14** of the 24 processes on the synthetic server are actually heavy; this is the oracle, and every number in the lesson is read against it. - When the sampling interval rises to 15 seconds, the tool misses **3** heavy processes; at 30 and 60 seconds, **5**. The ones missed are those that spike once every sixty seconds; the tool is correct at the instant it looks. - At a 1-second interval, **14,400** lines are produced; at 60 seconds, **240**. But 2,880 lines give a diagnosis just as good as 14,400. Past a certain point, more lines add nothing to the diagnosis. - A single glance gives between **5 and 7** false diagnoses depending on the moment it looks; the lifetime average taken since the process's birth is, with **7** false diagnoses, as bad as the worst single glance. - Searching by name touches **9 processes** wrongly in the best case, and a single name catches at most **4** of the heavy processes; a command name is not an identity but a label. ## Next Step This lesson looked at processes from outside: list, column, search. Processes also have a bond with the operator — which terminal they are attached to, which session they belong to, what happens when the session closes. The next lesson measures this bond: which processes survive when a script is backgrounded and the session is closed may not overlap at all with what `jobs` output shows.