Tracing Performance Problems

The previous lesson took up indicators that reduce a machine’s state to single numbers and measured that the reduction eats the spike. One gap remained there: load average collected processes waiting on the processor and processes waiting on disk inside the same number. When a machine slows down, the question asked requires exactly separating that sum: is the slowdown caused by the processor, the disk, or memory.

This lesson takes up separating the three bottleneck types. The expected solution is more detailed output: more columns, more lines, finer detail. What is measured is whether this expectation is correct, and the course’s second claim is paid here: more output does not mean a better diagnosis.

Where the expectation comes from is understandable too. Facing a failure, the only control knob at hand is mostly the level of detail: a longer list, a finer breakdown, more frequent sampling. All of these grow the output, and growing output gives a feeling that effort is increasing. The measurement tests this feeling, and the result is clear: if the direction of the effort is wrong, its amount does not change the outcome.

Three Bottlenecks and Their Shared Symptoms

A machine slowing down can arise from three mechanisms, and the three look the same to the user.

In a CPU bottleneck, the number of runnable processes exceeds the number of processors; processes queue up, each waiting for its turn. Processor busy percentage is high, and that high value corresponds to real work.

In an I/O bottleneck, processes wait not on the processor but on disk. While waiting, they are in uninterruptible wait; they do not consume the processor but they are counted. What shows up on the processor side is the wait share: the processor is idle, but the reason it is idle is not that no work is left but that data has not arrived. A view that reports busyness by folding the wait share into it cannot tell this situation apart from a CPU bottleneck.

In a memory bottleneck, physical memory runs short and pages are moved to swap. The trap here is two-fold. First, swap movement is itself a disk operation; heavy swapping produces I/O wait, and the memory bottleneck looks like a disk bottleneck. Second, memory indicators stay high not just during the pressure but after it has passed too: pages that went out to swap sit there until recalled.

There is a fourth possibility, and it falls outside this trio: the bottleneck can be not in the machine itself but in a counterpart it is waiting on. A process waiting on a network request is slow too, but it leaves no trace in any of the three metrics; the processor is idle, the disk is idle, memory is comfortable. This lesson does not measure that situation, and it is the one scenario where the machine is slow even though all three metrics say “normal.” Diagnosis on the network side is the subject of the next course.

The Operating System Concepts course built the mechanism of paged memory and scheduling; it is not repeated here. The question here is not the mechanism but which mechanism the operator can read from which output. The same distinction was set up at the opening of this course: measuring an abstraction’s cost and measuring whether the operator can see that abstraction are two separate jobs.

Five Views

The measurement compares five separate views, and each produces a different number of lines.

A — process list. Twenty-four lines, one meaningful column: CPU percentage per process. This is the most common first look.

B — thread detail. The same list, expanded to thread level: ninety-six lines. Four times as much output, the same columns. The level of detail has gone up; the metric set has not changed.

C — A plus a wait column. Twenty-four lines again, but the per-process I/O wait share has been added to the list. The line count is the same as A, the metric set is one unit larger.

D — three system metrics. Three lines: CPU busy, wait share, swap movement. No per-process detail at all.

E — the same three metrics, a different reading order. The exact same three lines as D; the only difference is which metric is checked first. D looks at wait share first, E at swap movement.

The dump below is an example and has not been run; it shows the three metrics view D carries.

# example dump , not run
$ vmstat 1 1
 r  b   swpd    free  ...  si  so  ...  us  sy  id  wa
 4  6  102400  81920  ...  38  44  ...  61  12   5  22

Where These Three Metrics Are Read From

Each of the three metrics comes from a separate counter, and which tool reads them is a secondary question; what matters is the counter itself.

CPU busy is computed from the time buckets the kernel keeps: time spent in user mode, time spent in kernel mode, idle time, and time spent waiting. These buckets are cumulative counters increasing since boot; the instantaneous percentage comes from the difference between two readings. A single reading gives the average since boot, and that average says almost nothing during a failure. This is another form of the lag problem from the previous lesson: the counter itself does not lag, but a single reading averages everything.

Wait share is the name of one of the same buckets, and it means exactly “the processor was idle, but at least one process was waiting on disk.” Two warnings are needed. First, this is a duration measure, not a queue measure; it does not say how many processes are waiting. Second, if the processor is genuinely busy, wait share reads low, because the processor has no idle time. That is, when the processor and the disk hit a bottleneck at the same time, wait share understates the size of the problem.

Swap movement is counted in two directions: pages read back into memory, and pages written to disk. The distinction matters. Writing to disk alone is not evidence of pressure; the operating system can move pages untouched for a long time out to disk even while things are comfortable. The evidence of pressure is pages being read back: a page in use was evicted and immediately requested again. A diagnosis that looks only at whether the swap area appears full searches the present for a pressure event that happened and ended in the past.

What Is Measured: Line Count or Metric Set

The setup’s assumptions: eighteen events are generated, and each event has a single real cause (GN19); the three metrics’ bands are derived from the cause (GN20); once swap movement rises above sixty, memory pressure also produces I/O wait (GN21); the views’ lines-per-event counts are 24, 96, 24, 3, and 3 (GN22); the decision thresholds are 60 for CPU, 25 for wait, 20 for swap (GN23).

The oracle is the real bottleneck cause of the eighteen events placed into the setup. The tool output is the line count each of the five views produces. The diagnosis is the cause each view derives from its own metrics; wrong diagnosis is the count of diagnoses that come out different from the real cause.

SEED = 20260218


def generator(seed):
    d = seed

    def next_val(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_val


CAUSE = ("cpu", "io", "memory")


def bottlenecks(seed=SEED, count=18):
    """GN19 ORACLE: every event's single real bottleneck. GN20: the three metrics
    are derived from the cause. GN21: swap over 60 also produces wait."""
    r = generator(seed)
    events = []
    for i in range(count):
        n = CAUSE[r(3)]
        if n == "cpu":
            cpu, swap, wait = 78 + r(18), r(3), 2 + r(4)
        elif n == "io":
            cpu, swap, wait = 74 + r(20), r(3), 46 + r(28)
        else:
            cpu, swap = 52 + r(30), 34 + r(40)
            wait = 30 + r(20) if swap >= 60 else 6 + r(9)
        events.append({"no": i + 1, "cause": n, "cpu": cpu,
                        "wait": wait, "swap": swap})
    return events


def view_a(o):
    """GN23 thresholds. Process list , single metric: CPU percentage."""
    return "cpu" if o["cpu"] >= 60 else "memory"


def view_c(o):
    """Same list , one more column: wait."""
    if o["wait"] >= 25:
        return "io"
    return "cpu" if o["cpu"] >= 60 else "memory"


def view_d(o):
    """Three system metrics , wait read first."""
    if o["wait"] >= 25:
        return "io"
    return "memory" if o["swap"] >= 20 else "cpu"


def view_e(o):
    """Same three metrics , swap read first."""
    if o["swap"] >= 20:
        return "memory"
    return "io" if o["wait"] >= 25 else "cpu"


VIEW = (("A process list", view_a, 24),        # GN22: lines/event
        ("B thread detail", view_a, 96),
        ("C A + wait column", view_c, 24),
        ("D three system metrics", view_d, 3),
        ("E same three metrics , swap first", view_e, 3))

for seed in (SEED, 20260219):
    O = bottlenecks(seed)
    print("seed", seed, "| events", len(O), "|",
          {n: sum(1 for o in O if o["cause"] == n) for n in CAUSE})
    print("  view                           lines/event  total lines  wrong diagnosis")
    for name, f, lines in VIEW:
        y = sum(1 for o in O if f(o) != o["cause"])
        print(f"  {name:30s} {lines:10d} {lines * len(O):13d} {y:12d}/{len(O)}")

seed 20260218 | events 18 | {'cpu': 4, 'io': 4, 'memory': 10}
  view                           lines/event  total lines  wrong diagnosis
  A process list                         24           432           12/18
  B thread detail                        96          1728           12/18
  C A + wait column                      24           432            8/18
  D three system metrics                  3            54            2/18
  E same three metrics , swap first          3            54            0/18
seed 20260219 | events 18 | {'cpu': 7, 'io': 6, 'memory': 5}
  view                           lines/event  total lines  wrong diagnosis
  A process list                         24           432           11/18
  B thread detail                        96          1728           11/18
  C A + wait column                      24           432            5/18
  D three system metrics                  3            54            2/18
  E same three metrics , swap first          3            54            0/18

What the Output Says Versus the System’s Truth

Three numbers sit side by side. The oracle: four of the eighteen events are CPU-caused, four I/O-caused, ten memory-caused. The tool output: the views produce, in order, 432, 1728, 432, 54, and 54 lines. Wrong diagnosis: 12, 12, 8, 2, and 0.

Comparing the first row against the second pays off the claim. View B produces exactly four times as many lines as A — 1728 instead of 432 — and the wrong-diagnosis count does not change at all: it stays at twelve. Raising the level of detail grew the text to be read; it corrected the diagnosis by not one bit. The reason is clear: the thread breakdown is the same metric sliced finer, and slicing a metric finer cannot see what that metric could not see in the first place.

The third row shows what makes the difference. View C produces a quarter as many lines as B — the same as A, 432 — but wrong diagnosis drops from 12 to 8. What produces the gain is not the line count but a single column added to the list: wait share. The correction of four wrong diagnoses comes from I/O bottlenecks becoming separable from CPU bottlenecks.

The fourth row takes this to the extreme. View D produces 54 lines; a thirty-second of B. Wrong diagnosis drops to 2. Three lines carrying no per-process detail at all give six times better diagnosis than a view carrying ninety-six lines per process. This is the measurement’s name: what separates them is not the line count but which metric is looked at.

The gap between C and D also shows where scale fits in. C looks per process and can answer “which process”; D looks at the system as a whole and cannot answer that question. Yet D is wrong less often, because the question being asked is not “which process” but “which resource.” The two questions are frequently mixed up: searching for the process consuming the most resource without first finding the source of the slowdown often ends up finding the victim. On a machine under memory pressure, the process waiting the most is not the process creating the pressure; it is the one most affected by it.

Choosing a view also has a consequence for order. The correct sequence is to look at the system as a whole first, then the process: D says “memory,” and then the process list is sorted by memory and the culprit is sought. The reverse sequence — process list first, system metric second — is represented in the measurement by row A, and it takes the wrong path on twelve of eighteen events.

The Remaining Two Misses and Reading Order

The fifth row pays off the remaining share. View E carries the exact same three metrics as D; the only difference is which one is asked first. D looks at wait share first and says “I/O” when it sees high wait. E looks at swap movement first and says “memory” when it sees high swap. Wrong diagnosis is 2 for D, 0 for E.

The events producing the difference are memory bottlenecks whose swap rises above sixty. There, because swap movement itself creates disk work, wait share rises too; the two metrics are high at the same time, and which one is read first determines the diagnosis. Wait share is a result, swap movement is a cause; a reading order that reads the result first shadows the cause.

This yields a rule for the operations side: if the causal direction between two metrics read high at the same time is not known, growing the metric set is not enough. Even an operator looking at just three numbers is wrong on two of eighteen events if they look in the wrong order. The metric set determines the diagnosis’s ceiling; reading order determines whether that ceiling is reached.

There is a general way to decide which metric a sequence should start with: the metric highest in the chain is read first. Swap movement is the direct result of memory pressure and arises from no other cause; wait share, on the other hand, can arise from disk slowness, from swap, or from an ordinary bulk read. A metric with multiple causes is read after a metric with a single cause. This rule is not specific to the setup; it works in every metric set where indicators feed into one another.

The decision tree has a cost too. A procedure that reads the three metrics in a fixed order reduces a situation where two bottlenecks genuinely exist at the same time to a single cause; it stops at the first matching branch and never reports the second cause. This does not show up in the measurement, because in the setup every event has a single real cause. On a real machine, reducing the diagnosis to a single name is itself a kind of wrong diagnosis.

The limit of the measurement has to be written clearly too. E’s zero comes from the setup being perfectly separable; a real machine is not expected to have three metrics separate every event. What is meaningful is not the zero but the sequence from 12 down to 0: quadrupling the line count gained nothing, adding the right metric removed four misses, narrowing and correcting the metric set removed ten misses, and fixing the order removed the remaining two.

The Second Seed

In the second seed, the distribution of events changes: seven CPU, six I/O, five memory. Views A and B again give the exact same number — eleven wrong diagnoses each — and the line-count difference again changes nothing. C drops to 5, D to 2, E to 0. The absolute numbers depend on the setup; what does not depend on the setup is that quadrupling the line count never changes wrong diagnosis, and that the three-line view outperforms the ninety-six-line view in both seeds.

Summary

• The three bottleneck types produce the same symptom; memory pressure also creates I/O wait through swap, and looks like a disk bottleneck.
• Quadrupling the line count (from 432 to 1728) never changes wrong diagnosis: it stays the same in both seeds.
• Adding a single correct column to the list brings wrong diagnosis from 12 to 8; dropping process detail and reducing to three system metrics brings it to 2.
• Changing the reading order of the same three metrics removes the remaining two misses too; wait share is a result, swap movement is a cause.
• What determines the diagnosis is not the volume of output but the metric set and the order that set is read in.

Next Step

What was measured throughout this topic was how much of the truth the text and numbers a machine produces about itself carry: a diagnosis looking at a single unit file was wrong on five of six, rotation deleted 81 of 86 errors, an averaged indicator never showed a fifty-four-second spike at all, and four times the output corrected not a single miss. The silent assumption behind all these measurements was that data could be written to disk. The next topic removes that assumption and descends into storage: block devices, partitions, file systems, and a file system’s two independent limits that run out. There, the situation where a disk is one percent full and yet a write fails is shown numerically.