Listing and Searching Processes

The Shell Programming course turned a script into a tool: it gained an argument interface, stopped on errors, cleaned up after itself, and was handed to a scheduler. Writing a script is making something happen. System administration begins with a different task — first, seeing what is happening. And seeing has its own errors.

This course counts those errors. The Operating System Concepts course measured the process, the scheduler, and context switching as mechanism; what was measured there was the cost of the abstraction. Here the mechanism is not rebuilt: what is measured is what the operator can see of that mechanism.

The Course’s Measure

In every lesson, three numbers stand side by side. The first is the oracle: the system’s real state. Because we generate the fiction ourselves, we know which process is actually heavy. The second is the tool’s output: what the tool shows. The third is the false diagnosis count: how many times the conclusion drawn from the output contradicts the oracle.

The rule itself is one sentence: a command’s value is not the number of lines it shows, but how many times the diagnosis drawn from those lines is wrong. An output whose diagnosis is not tested is counted as unmeasured.

• PM1. The observation window is 600 seconds. All durations are seconds in the model; no real time is measured.
• PM2. The synthetic server has 24 processes. Each process’s real CPU usage is not a single number but a time series.
• PM3. There are three patterns: flat (continuously low), spike (a five-second jump once every sixty seconds), heavy (continuously high).
• PM4. A process is counted as heavy if its usage exceeds 50 even once during the window. The oracle applies this definition by looking at the full 600 seconds; the tool looks only at the instants it observes.
• PM5. The monitoring tool looks once every interval seconds and keeps the highest value it saw per process. It knows nothing of what happened between glances.
• PM6. The false diagnosis count is the sum of missed heavy processes and light processes mistaken for heavy.
• PM7. The second seed is 20260219. If the resulting diagnosis counts do not stay in the same order of magnitude, the result depends on the fiction, and is written as such.

Reading the Process Table

The command that gives the process list is ps; the command that gives a continuously refreshed view is top or one of its interactive derivatives. ps is a snapshot; the list it gives is the process table at the instant the command ran. The transcript below is illustrative, produced for this page, and has not been run:

$ ps -eo pid,user,ni,pcpu,pmem,etime,comm --sort=-pcpu
    PID USER        NI %CPU %MEM   ELAPSED COMMAND
   1128 app          0 71.2  3.1  00:10:00 queue-worker
   1043 monitor      5 68.4  1.2  00:10:00 metric-collector
   1014 app         -3 64.9  2.8  00:10:00 report-generator
   1092 root         0  6.1  0.4  00:10:00 backup-job
   1071 backup      10  3.0  0.2  00:10:00 cache-cleaner

Most columns explain themselves; two do not. NI is the priority value that will be measured in the fourth lesson. %CPU is the most misleading column: ps divides this value by the time elapsed since the process’s birth, that is, it is a lifetime average. A process that has run for ten hours appears low in this column even if it consumed the core by itself for the last five minutes.

top computes the same column differently: it represents the short interval between two refreshes. When the two commands are run on the same machine at the same time, they give different numbers for the same process, and both are correct; they are measuring different time spans. This distinction is exactly the source of what this lesson measures.

The Synthetic Server

The measurement rests not on a real command but on the course’s common definition. The output of a command run on a real server cannot be reproduced: the machine differs, the load differs, the moment differs. Because we generate the synthetic server ourselves, we know the oracle.

"""M03/K03 common definition: a synthetic server, the text the tools produce, and the
diagnosis drawn from that text compared against the ORACLE. No real command is ever called.
"""
SEED = 20260218
SECOND_SEED = 20260219
PERIOD = 600                      # observation window: 600 seconds


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


USER = ("root", "app", "backup", "monitor")
COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector",
           "queue-worker", "cache-cleaner")


def processes(seed=SEED, count=24):
    """Each process's real CPU usage is a TIME SERIES, not a single number."""
    r = generator(seed)
    result = []
    for i in range(count):
        kind = r(10)
        pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy")
        result.append({"pid": 1000 + i * 7 + r(5),
                       "user": USER[r(len(USER))],
                       "command": COMMAND[r(len(COMMAND))], "pattern": pattern,
                       "priority": r(11) - 5, "memory": 20 + r(400),
                       "ignores_signal": r(10) < 2})
    return result


def usage(process, second):
    """Real CPU usage, second by second. The oracle knows this; the tool does not."""
    p = process["pattern"]
    if p == "flat":
        return 3 + (process["pid"] + second) % 4
    if p == "heavy":
        return 60 + (process["pid"] + second) % 25
    phase = (second + process["pid"]) % 60          # spike: a 5-second jump every 60 seconds
    return 88 + phase if phase < 5 else 2 + phase % 3


S = processes()
print("synthetic server:", len(S), "processes |",
      {d: sum(1 for s in S if s["pattern"] == d) for d in ("flat", "spike", "heavy")})
print("processes actually crossing the threshold:",
      sum(1 for s in S if max(usage(s, t) for t in range(PERIOD)) >= 50))

synthetic server: 24 processes | {'flat': 10, 'spike': 7, 'heavy': 7}
processes actually crossing the threshold: 14

The oracle is 14. Seven processes are continuously heavy, and seven spike once every sixty seconds, comfortably exceeding the threshold when they do. Of the 24 processes in total, 14 cross the threshold somewhere in the window. Every number from here on is read against this 14.

The Sampling Interval Determines the Diagnosis

The monitoring tool has one setting: how often it looks. Enlarging the interval shrinks the output and reduces the load. What this does to the diagnosis can be measured.

# On top of the first block: S, usage, PERIOD, SECOND_SEED, processes come from there.
def sample(processes_, interval, window=PERIOD):
    """A monitoring tool that looks once every `interval` seconds. Returns: highest seen per process."""
    seen = {s["pid"]: 0 for s in processes_}
    glances = 0
    for t in range(0, window, interval):
        glances += 1
        for s in processes_:
            seen[s["pid"]] = max(seen[s["pid"]], usage(s, t))
    return {"glances": glances, "seen": seen}


def true_peak(processes_, window=PERIOD):
    return {s["pid"]: max(usage(s, t) for t in range(window)) for s in processes_}


def diagnose_heavy(measurement, threshold=50):
    """The diagnosis drawn from the output: which processes are heavy."""
    return {p for p, v in measurement.items() if v >= threshold}


INTERVALS = (1, 5, 15, 30, 60)


def sweep(processes_):
    truth = diagnose_heavy(true_peak(processes_))
    rows = []
    for a in INTERVALS:
        o = sample(processes_, a)
        t = diagnose_heavy(o["seen"])
        rows.append((a, o["glances"], len(t), len(truth - t),
                     len(truth - t) + len(t - truth)))
    return len(truth), rows


oracle, rows = sweep(S)
print("interval  glances  lines produced  seen heavy  missed  false diagnosis")
for a, glances, seen, missed, wrong in rows:
    print(f"{a:8d}  {glances:7d}  {glances * len(S):14d}  {seen:10d}"
          f"  {missed:6d}  {wrong:15d}")
print("oracle (actually heavy):", oracle, "| total processes:", len(S))
print()
oracle2, rows2 = sweep(processes(SECOND_SEED))
print("second seed", SECOND_SEED, "- oracle:", oracle2)
print("  false diagnosis:", {a: y for a, _, _, _, y in rows2})
print("  first seed      :", {a: y for a, _, _, _, y in rows})

interval  glances  lines produced  seen heavy  missed  false diagnosis
       1      600           14400          14       0                0
       5      120            2880          14       0                0
      15       40             960          11       3                3
      30       20             480           9       5                5
      60       10             240           9       5                5
oracle (actually heavy): 14 | total processes: 24

second seed 20260219 - oracle: 12
  false diagnosis: {1: 0, 5: 0, 15: 5, 30: 6, 60: 7}
  first seed      : {1: 0, 5: 0, 15: 3, 30: 5, 60: 5}

A tool looking once every one or five seconds sees all 14 of the 14 heavy processes: zero false diagnoses. Once the interval rises to 15 seconds, 3 are missed; at 30 and 60 seconds, 5 are missed. The ones missed are always the same class — those that rise once every sixty seconds, for five seconds. A tool that looks once every thirty seconds is not obligated to catch a five-second spike; most of the time it does not.

The tool is not lying. At the moment it looked, that process really was light. What is wrong is not the output but the diagnosis drawn from it: “it is not on the list, so it is not a problem.”

The second column carries the second claim. At interval 1, the tool produces 14,400 lines; at interval 60, 240. As the line count rises sixtyfold, the false diagnosis count drops from 5 to 0 — but the relationship between these two numbers is not linear: 2,880 lines give a diagnosis just as good as 14,400 lines. Every line added past five seconds adds nothing to the diagnosis.

With the second seed, the oracle drops to 12 and the false diagnoses come out as 5, 6, 7. The numbers are not the same; the direction and order of magnitude are — starting from zero, growing within a single-digit band as the interval grows. The result that stands is this: enlarging the interval produces missed processes. How many will be missed depends on the fiction.

Single Glance vs. Lifetime Average

The sampling interval is a problem of many glances. What the operator does most often, however, is a single glance: running ps once. The cost of this, and of ps’s own column average, is measured separately.

# On top of the previous blocks: S, usage, PERIOD, and diagnose_heavy come from there.
TRUTH = diagnose_heavy({s["pid"]: max(usage(s, t) for t in range(PERIOD))
                        for s in S})


def glance(processes_, t):
    """The value read on a single glance: what it was using at that instant."""
    return {s["pid"]: usage(s, t) for s in processes_}


def lifetime_average(processes_, window=PERIOD):
    """The process's average since its birth. Dilutes the spike."""
    return {s["pid"]: sum(usage(s, t) for t in range(window)) // window
            for s in processes_}


readings = [(f"glance t={t:<3d}       ", glance(S, t)) for t in (0, 3, 30, 300)]
readings.append(("lifetime average  ", lifetime_average(S)))
print("reading form         seen heavy  missed  false diagnosis")
for name, measurement in readings:
    d = diagnose_heavy(measurement)
    print(f"  {name}{len(d):10d}  {len(TRUTH - d):9d}"
          f"  {len(d - TRUTH) + len(TRUTH - d):11d}")
print("oracle:", len(TRUTH), "heavy processes |", len(S), "processes total")

reading form         seen heavy  missed  false diagnosis
  glance t=0                  9          5            5
  glance t=3                  8          6            6
  glance t=30                 7          7            7
  glance t=300                9          5            5
  lifetime average           7          7            7
oracle: 14 heavy processes | 24 processes total

A single glance gives between 5 and 7 false diagnoses, depending on the moment it looks. Same machine, same 600 seconds, same command: the answer depends on which second it ran. This is the measure of the sentence “I ran ps to see the problem, and there was nothing there.”

The real result is in the last row. The lifetime average is as wrong as the worst single glance: 7 false diagnoses. Averaging does not erase the spike, it dilutes it — a five-second spike, divided over 600 seconds, drops far below the threshold. Looking at a longer window does not give a better diagnosis here; which measure is taken comes before how much data was collected.

The Cost of Searching by Name

There are two ways to search for a process. Filtering the list as text — narrowing ps output with grep — also catches the searching process itself, and brings back unrelated lines when the pattern is written wrong. The second way is pgrep and its counterpart pkill, which query the process table directly:

$ pgrep -a -u app queue-worker
1128 /opt/queue/queue-worker --shard 3
1135 /opt/queue/queue-worker --shard 4

This transcript, too, is illustrative and has not been run. pgrep is more precise than text filtering: it does not list itself, and it can be narrowed by user and session. But what is precise is the match, not the diagnosis. Selecting by name silently accepts the assumption “the processes carrying this name are the problem.”

# On top of the previous blocks: S and TRUTH come from there.
name_table = {}
for s in S:
    name_table.setdefault(s["command"], [0, 0])
    name_table[s["command"]][0] += 1
    name_table[s["command"]][1] += 1 if s["pid"] in TRUTH else 0

print("command name            matched  actually heavy  wrong touches by name")
for k in sorted(name_table):
    total, heavy = name_table[k]
    print(f"  {k:22s} {total:6d}  {heavy:14d}  {min(heavy, total - heavy):21d}")
print("total wrong touches (best case):",
      sum(min(a, t - a) for t, a in name_table.values()))
print("how many heavy processes a single name catches at most:",
      max(a for _, a in name_table.values()), "/", len(TRUTH))

command name            matched  actually heavy  wrong touches by name
  backup-job                  5               3                      2
  cache-cleaner               2               1                      1
  data-receiver               1               1                      0
  metric-collector            5               2                      2
  queue-worker                6               3                      3
  report-generator            5               4                      1
total wrong touches (best case): 9
how many heavy processes a single name catches at most: 4 / 14

No command name coincides exactly with the heavy processes. Of the six processes carrying the name queue-worker, three are heavy and three are not; an operator selecting by name either sweeps in the three innocent processes too or leaves out the three heavy ones. Even in the best case, 9 processes are touched wrongly, and a single name catches at most 4 of the heavy processes — the oracle is 14.

The reason for this is not the fiction but the structure itself: a command name is not an identity, it is a label. The same program runs as multiple instances, and the instances’ loads are independent of each other. Selecting by name is fast, and this speed turns into a cost when sending signals, in the third lesson of this topic.

Summary

• The course’s measure is three numbers: the oracle, the tool’s output, and the false diagnosis count. A command’s value is not the number of lines it shows, but how many times the diagnosis drawn from those lines is wrong.
• 14 of the 24 processes on the synthetic server are actually heavy; this is the oracle, and every number in the lesson is read against it.
• When the sampling interval rises to 15 seconds, the tool misses 3 heavy processes; at 30 and 60 seconds, 5. The ones missed are those that spike once every sixty seconds; the tool is correct at the instant it looks.
• At a 1-second interval, 14,400 lines are produced; at 60 seconds, 240. But 2,880 lines give a diagnosis just as good as 14,400. Past a certain point, more lines add nothing to the diagnosis.
• A single glance gives between 5 and 7 false diagnoses depending on the moment it looks; the lifetime average taken since the process’s birth is, with 7 false diagnoses, as bad as the worst single glance.
• Searching by name touches 9 processes wrongly in the best case, and a single name catches at most 4 of the heavy processes; a command name is not an identity but a label.

Next Step

This lesson looked at processes from outside: list, column, search. Processes also have a bond with the operator — which terminal they are attached to, which session they belong to, what happens when the session closes. The next lesson measures this bond: which processes survive when a script is backgrounded and the session is closed may not overlap at all with what jobs output shows.