System Health Metrics

The previous lesson measured a loss belonging to the past: in a rotated log, a deleted line did not come back. The second kind of data speaking about a machine’s state is not lines but numbers. Load, memory, disk fullness; each is presented as a single number, and that number is the first place looked at during a failure.

This lesson takes up how these numbers are produced. The problem that shows up belongs to the same family as rotation’s, but comes from a different mechanism: no line is deleted, the number is averaged. An averaging indicator mixes the past into the present, and the spike dissolves inside the average. What is measured is how many seconds that dissolving keeps the diagnosis wrong for.

A distinction has to be made from the start. A log line records an event: the trail of something that happened sits on disk and can be read later. A health indicator, by contrast, reads a state: it is produced at the moment it is read and leaves no trace for the moment it is not read. A log’s absence is explained by rotation; an indicator’s absence has no explanation, because a moment that was not recorded cannot be told apart from a moment that never happened.

What Load Average Counts

Load average is the average, over a time interval, of the number of processes that are ready to run or waiting uninterruptibly. Two points should be stated right away. First, this is not a percentage: it describes not processor occupancy but the amount of work queued up. Second, a load average always comes with a window; there is no such number as “the load” on its own. The common presentation places three windows side by side, and the three describe the same moment with three different histories.

Interpreting the number depends on core count. The same number can mean saturation on a single-core machine and comfort on a multi-core one. This is why there is no absolute threshold; the threshold is set per machine, and setting it uses the form divided by core count.

The scope of what enters the number can also mislead. Load average counts not only processes waiting on the processor but also processes sitting in uninterruptible I/O wait. The result is this: on a machine whose processor is nearly idle but whose disk is jammed, load average reads high and produces a “the processor is not enough” diagnosis. The three bottleneck types the next lesson tries to separate are all collected inside this one number.

The third point, and the one this lesson measures, is this: load average is a lagging indicator. Its value gives not the state at the moment being looked at but the average of the window up to that moment. When a spike begins, the indicator rises slowly; when the spike ends, it falls slowly. The gap in between is the truth the average misses.

The dump below is an example and has not been run; the three numbers are, in order, the one-, five-, and fifteen-minute windows.

# example dump , not run
$ uptime
 load average: 6.22, 6.19, 6.08

What Is Measured: The Spike the Average Suppresses

The measurement is built on the shared definition’s process setup. The setup’s assumptions: load is the sum of twenty-four processes’ second-by-second usage, presented divided by a hundred (GN14); measurement is done only from the sixtieth second onward, once the window has filled (GN15); the threshold is the value separating the busiest ten percent of the real load (GN16); the averaging windows tested are 15, 60, and 300 seconds (GN17); memory per process is constant across the window (GN18).

The oracle is the real load at every second; it is known second by second because we built the setup. The tool output is the same load averaged over a given window. The diagnosis is the decision “the system is at its busiest right now.” Wrong diagnosis is the number of seconds the indicator misses this decision.

SEED = 20260218
PERIOD = 600                     # observation window: 600 seconds


def generator(seed):
    d = seed

    def next_val(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_val


ACCOUNT = ("root", "app", "backup", "monitor")
COMMAND = ("data-receiver", "report-generator", "backup-task", "measurement-collector",
           "queue-worker", "cache-cleaner")


def processes(seed=SEED, count=24):
    """Every process: real CPU usage is a TIME SERIES, not a single number."""
    r = generator(seed)
    result = []
    for i in range(count):
        kind = r(10)
        if kind < 5:
            pattern = "flat"                    # steady low usage
        elif kind < 8:
            pattern = "burst"                   # short , high spikes
        else:
            pattern = "heavy"                   # steady high usage
        result.append({
            "pid": 1000 + i * 7 + r(5),
            "account": ACCOUNT[r(len(ACCOUNT))],
            "command": COMMAND[r(len(COMMAND))],
            "pattern": pattern,
            "priority": r(11) - 5,
            "memory": 20 + r(400),
            "ignores_signal": r(10) < 2,        # some processes ignore termination
        })
    return result


def usage(process, second):
    """Real CPU usage, second by second. The oracle knows this; the tool does not."""
    d = process["pattern"]
    if d == "flat":
        return 3 + (process["pid"] + second) % 4
    if d == "heavy":
        return 60 + (process["pid"] + second) % 25
    # burst: a 5-second spike every 60 seconds
    phase = (second + process["pid"]) % 60
    return 88 + phase if phase < 5 else 2 + phase % 3


def actual_load(processes_, second):
    return sum(usage(s, second) for s in processes_)


def load_average(processes_, second, window=60):
    """Load average is a LAGGING indicator: it averages the past window."""
    start = max(0, second - window)
    span = range(start, second + 1)
    return round(sum(actual_load(processes_, t) for t in span) / len(list(span)) / 100, 2)


def memory_share(processes_):
    """GN18: the tool lists the 'top five memory-using processes'. How much does this explain."""
    values = sorted((s["memory"] for s in processes_), reverse=True)
    cumulative, count = 0, 0
    for v in values:
        cumulative, count = cumulative + v, count + 1
        if cumulative / sum(values) >= 0.20:
            break
    return {"total": sum(values), "largest_share": round(values[0] / sum(values), 4),
            "top_five_share": round(sum(values[:5]) / sum(values), 4), "for_20_percent": count}


for seed in (SEED, 20260219):
    S = processes(seed)
    span = range(60, PERIOD)                    # GN15: after the window fills
    instant = {t: actual_load(S, t) / 100 for t in span}
    ordered = sorted(instant.values())
    threshold = ordered[int(len(ordered) * 0.9)]  # GN16 oracle: busiest ten percent
    print("seed", seed,
          f"| instant load {min(instant.values()):.2f}-{max(instant.values()):.2f}"
          f" | threshold {threshold:.2f} | seconds over threshold"
          f" {sum(1 for v in instant.values() if v >= threshold)}")
    print("  second 0/60/300 instant:", [round(instant.get(t, actual_load(S, t) / 100), 2)
                                          for t in (0, 60, 300)],
          "| average(60):", [load_average(S, t) for t in (0, 60, 300)])
    print("  window  lowest  highest  over threshold  missed  false alarm")
    for p in (15, 60, 300):                     # GN17
        series = {t: load_average(S, t, p) for t in span}
        missed = sum(1 for t in span if instant[t] >= threshold and series[t] < threshold)
        extra = sum(1 for t in span if series[t] >= threshold and instant[t] < threshold)
        print(f"  {p:6d}  {min(series.values()):6.2f}  {max(series.values()):7.2f}"
              f"  {sum(1 for v in series.values() if v >= threshold):14d}"
              f"  {missed:6d}  {extra:11d}")
    print("  memory:", memory_share(S))

seed 20260218 | instant load 5.36-7.80 | threshold 7.15 | seconds over threshold 54
  second 0/60/300 instant: [7.36, 7.31, 7.36] | average(60): [7.36, 6.25, 6.22]
  window  lowest  highest  over threshold  missed  false alarm
      15    5.62     6.80               0      54            0
      60    6.18     6.25               0      54            0
     300    6.13     6.29               0      54            0
  memory: {'total': 4876, 'largest_share': 0.0779, 'top_five_share': 0.3659, 'for_20_percent': 3}
seed 20260219 | instant load 4.04-6.29 | threshold 6.04 | seconds over threshold 54
  second 0/60/300 instant: [4.33, 4.33, 4.33] | average(60): [4.33, 4.85, 4.85]
  window  lowest  highest  over threshold  missed  false alarm
      15    4.52     5.23               0      54            0
      60    4.83     4.90               0      54            0
     300    4.80     4.88               0      54            0
  memory: {'total': 5444, 'largest_share': 0.0755, 'top_five_share': 0.3611, 'for_20_percent': 3}

What the Output Says Versus the System’s Truth

Three numbers sit side by side. The oracle: the real load fluctuates between 5.36 and 7.80 and stays above the 7.15 threshold for 54 seconds. The tool output: the sixty-second average sits between 6.18 and 6.25; it never crosses the threshold. Wrong diagnosis: 54. In every second the system is at its busiest, the indicator says “normal.”

The false-alarm count is zero, and this makes the indicator’s silence even more dangerous. If the indicator were noisy, the operator would learn not to trust it; this indicator is not wrong, it just never speaks. An averaged series produces no false alarms, because it produces no spikes in the first place.

Where the threshold comes from should also stand clearly. Here, the threshold is not a number chosen from outside; it is the value separating the busiest ten percent of the real load, so by definition 54 seconds exceed it. What is measured is not whether the threshold was chosen correctly but whether it is possible to see the same event from an averaged series at all. The answer is no, and lowering the threshold does not fix it: once the threshold is brought inside the average’s band, the indicator crosses it constantly instead and cannot distinguish any second from another.

The second line of the template shows the lag directly. At second zero, instantaneous load is 7.36 and the average is 7.36 too: because the window has not filled yet, the two are the same. At second sixty, instantaneous is 7.31 while the average has dropped to 6.25; at second three hundred, instantaneous is again 7.36 while the average is 6.22. The gap between the two numbers at the same moment is not an error; the two measure different things. The problem is that the two are presented under the same name.

The effect of window width is also read from the table. Instantaneous load has a range of 2.44 units (5.36 to 7.80). At the fifteen-second average, this range narrows to 1.18; at sixty seconds, to 0.07. At the three-hundred-second window, it rises to 0.16, but it still never crosses the threshold; the reason for the rise is not window length but the first seconds during which the window has not filled. The reason the sixty-second window gives the flattest series is that the spikes in the setup repeat on a sixty-second cycle: when a window covers an exact multiple of the cycle, it dilutes the spike by the same amount every time.

This is a situation encountered on real machines too. A recurring job — a scheduled task, a batch transfer, a cache cleanup — leaves no trace at all in the indicator if it runs at the same rhythm as the averaging window. The reason for the invisibility is not that the job is small, but that the rhythms overlap.

The rule that follows is this: load average is not a spike indicator, it is a trend indicator. It does not answer “what is happening right now”; it answers “what was the general direction over the last minute.” What was measured for the sampling interval in the Process Management topic is repeated here for the averaging window: the indicator is not lying, the question asked is a different one.

Three Windows Side by Side

There is a reason load average is presented with three windows: a single number gives level, three numbers give direction. If the short window is greater than the long one, load is rising; if smaller, it is falling; if the three are close to each other, the state is steady. This reading does not remove the average’s lag, but it makes the sign of the lag visible.

The direction itself determines the diagnosis too. In a rising trend, the real question is not “has the current value crossed the threshold” but “at this rate, when will it cross.” In a falling trend, a high value is the residue of a past event, and reacting to it means chasing a failure that has already ended. This second situation is common in operations: the indicator reads high, an investigation begins, and by the time the investigation ends the indicator has come down on its own; nobody learns what happened.

The limit of the three-window reading also sits in the measurement. The spikes in the setup last five seconds; the shortest window is fifteen seconds. A five-second event shrinks to a third in a fifteen-second average and leaves too little of a mark in any of the three windows to change direction. Reading direction requires the event to last longer than the shortest window.

The way to see short-lived events is not to read the average but to read an unaveraged counter: runnable process count, context-switch count, and interrupt count all give an instantaneous value on every read. These counters are noisy and a single reading of them is misleading, but they carry no lag. The choice between a lag-free, noisy counter and a lagging, smooth average is this lesson’s real decision; no indicator gives both at once.

Memory and Disk Indicators

The memory indicator’s trap is different. A tool lists “the top five memory-using processes,” and that list is the easiest output to read. The measurement counts how much of the picture this list explains: the largest process holds only 7.79 percent of total memory, the top five together 36.59 percent. The remaining nineteen processes hold 63 percent, and none of them is on the list. The decision “kill the largest process” recovers less than eight percent of the total; freeing twenty percent requires three processes. The list is not wrong; it just does not answer the question being asked. In this setup, the answer to “who is consuming the memory” is not a single name but a long queue.

The second trap is that the operating system does not leave free memory idle. Files that have been read are held in cache and released when needed; this memory looks “in use” but produces no pressure. A diagnosis that looks only at percentage used mistakes a healthy machine for a full one. The distinguishing indicator is not the percentage used but the portion that is not reclaimable.

The third trap is that memory reported per process is not summable. Shared libraries and shared pages are counted separately in every process’s own line; a calculation that sums process lines finds more memory than exists on the machine. A total drawn from the process list should not be used without comparing it against the system-level total.

There is a similar duality on the disk side too, and it goes beyond this topic’s scope. Percentage full is a single number, whereas in a file system two independent limits get exhausted and both produce the same error message. The Storage topic will measure this distinction; for now, what needs to be known is that a low percentage full does not guarantee a write will succeed.

Another property of the disk indicator is that it is insensitive to the moment of measurement. Load changes within seconds, fullness within hours. This does not make the disk indicator more reliable; it just gives a different kind of lag. What is dangerous in a filling disk is not the current percentage but the rate of filling; a rate cannot be derived from a single reading, and unless the difference between two readings is taken, the indicator does not say how many hours remain until failure.

The Second Seed

In the second seed, the setup’s load level drops: instantaneous load fluctuates between 4.04 and 6.29 and the threshold falls to 6.04. Even though the numbers change, the structure stays the same — the real load again crosses the threshold for 54 seconds, the averaged indicators again never cross it, and wrong diagnosis is again 54. The sixty-second average’s range stays at 0.07 units. The absolute value of the threshold and the load depends on the setup; what does not depend on the setup is that the average completely suppresses the spike and that the false-alarm count is zero.

Summary

• Load average is not a percentage; it is the average, over a window, of the amount of work queued up; interpreting it depends on core count.
• While the real load crosses the threshold for 54 seconds, the sixty-second average never crosses it; wrong diagnosis 54, false alarms 0.
• Instantaneous load’s 2.44-unit range narrows to 0.07 in the sixty-second average; widening the window calms the indicator, and a calmed indicator cannot show a spike.
• The largest memory-using process holds only 7.79 percent of the total; the top five hold 36.59 percent, and freeing twenty percent requires three processes.
• Load average is a trend indicator; it cannot answer “what is happening right now,” and when forced to, it stays silent.

Next Step

This lesson’s indicators reduced the entire system to a single number, and the reduction ate the spike. When a performance problem is investigated, the question asked is even more specific: is the slowdown caused by the processor, the disk, or memory. The three causes look very similar to each other in most tool output, and raising the level of detail does not reduce the similarity. The next lesson takes up separating the three bottleneck types and counts how a view producing four times as many lines never changes the wrong-diagnosis count at all, while a three-line view does.