Performance Profiling

The previous lesson measured an expensive channel: system call tracing brought back 2 of the 14 facts the strict bundle closes, and slowed the run by 34 percent. Cost per returning fact was 17.0 percentage points. This lesson sits at the other end of the scale. The sampling-based CPU profile is the cheapest of the model’s four channels: the slowdown it adds to the run is 3 percent.

The question is what that cheapness costs. If a channel gives information without touching the run, its power to pierce isolation might be low too; or, the other way around, a cheap channel might be exactly the one that opens the real gap, because it can be left on continuously. The measurement says which of the two is true: the profile is also the narrowest in the number of facts it brings back — the returning fact is 1.

The Profile as a Channel

A sampling profile stops the running process at regular intervals and records which code is executing at that moment. What it collects is not a sequence of events but a distribution: what share of the samples each function collected. How to read a profile — which metric to pick, which event the sampling rate will miss — was measured on the frontend side in Frontend Quality and on the operations side in the Observability and Operations course, and is not repeated here. In this lesson the profile is measured not as a performance tool but as an isolation channel.

The assumptions continue and widen. The three facts the profile channel shows are the list in the code (OD8). Costs are percentages in the model, not a real measurement; the baseline run is 1000 time units (OD9). The profile is on for the whole run, and its cost is taken as independent of sample count (OD10) — on a real system the cost rises as sample rate rises, and the model closes off this distinction. The sample count is a mock value and every sample produces one line (OD11). Sample count improves a fact’s estimate, it does not change the fact’s visibility (OD12). The profiling tool runs from outside the bundle, during the run (OD13).

# example dump , has not been run
$ perf record -F <sample-rate> -g -p <process-no> -- sleep <seconds>
[ perf record: <sample-count> samples written ]
$ perf report --stdio --sort symbol
# Overhead  Symbol
#  <percent>  [.] <function-a>
#  <percent>  [.] <function-b>
#  <percent>  [k] <kernel-function>

The values in the dump are mock and the block has not been run. What is measured is not the dump’s lines but the facts readable from it.

# M03/K05 common definition -- only the parts this lesson uses.
FACTS = [
    ("other-process-list", "process"), ("own-process-number", "process"),
    ("ancestor-process-chain", "process"),
    ("root-filesystem-tree", "mount"), ("other-mounts", "mount"),
    ("shared-temp-directory", "mount"),
    ("host-interfaces", "network"), ("host-routing-table", "network"),
    ("listening-ports", "network"),
    ("user-id-mapping", "user"), ("file-ownership", "user"),
    ("hostname", "host"),
    ("cpu-share", "resource"), ("memory-limit", "resource"),
    ("memory-used", "resource"), ("cpu-count", "resource"),
    ("kernel-version", "kernel"), ("kernel-settings", "kernel"),
    ("system-load", "kernel"), ("clock", "kernel"),
    ("capability-set", "privilege"), ("file-permissions", "privilege"),
    ("security-label", "label"), ("policy-rules", "label"),
]
MECHANISM = {
    "pid-namespace":     {"axis": ["process"], "left_open": ["own-process-number"]},
    "mount-namespace":   {"axis": ["mount"], "left_open": ["root-filesystem-tree"]},
    "network-namespace": {"axis": ["network"], "left_open": []},
    "user-namespace":    {"axis": ["user"], "left_open": ["file-ownership"]},
    "uts-namespace":     {"axis": ["host"], "left_open": []},
    "control-group":     {"axis": ["resource"], "left_open": ["cpu-count"]},
    "capability-dropping": {"axis": ["privilege"], "left_open": ["file-permissions"]},
    "mandatory-label":   {"axis": ["label"], "left_open": ["security-label"]},
}
COMMON_BUNDLE = ["pid-namespace", "mount-namespace", "network-namespace",
                 "uts-namespace", "control-group", "capability-dropping"]
STRICT_BUNDLE = COMMON_BUNDLE + ["user-namespace", "mandatory-label"]

CHANNEL = {  # OD8: facts the profile channel shows ; tracing channel for comparison
    "sampling-profile":     ["cpu-share", "cpu-count", "system-load"],
    "system-call-tracing": ["root-filesystem-tree", "other-mounts", "file-ownership",
                             "capability-set", "clock"],
}
# OD9: costs are percentages in the MODEL, not a real measurement
CHANNEL_COST = {"sampling-profile": 3, "system-call-tracing": 34}
BASELINE_TIME = 1000
# OD11-OD13: sample count is a construct; each sample produces one line
SAMPLE_COUNT = [100, 1000, 10000, 100000]


def visible(mechanisms, facts=FACTS):
    closed = set()
    for d in mechanisms:
        v = MECHANISM[d]
        for e in v["axis"]:
            closed |= {o for o, x in facts if x == e}
        closed -= set(v["left_open"])
    return [o for o, _ in facts if o not in closed]


def returned(mechanisms, channel, facts=FACTS):
    closed = {o for o, _ in facts} - set(visible(mechanisms, facts))
    return sorted(set(channel) & closed)


def surface(mechanisms, channel=(), facts=FACTS):
    g = set(visible(mechanisms, facts))
    back = set(returned(mechanisms, channel, facts))
    return {"visible": len(g), "returned": len(back), "open_surface": len(g | back),
            "open_axes": len({e for o, e in facts if o in (g | back)})}


PROFILE = CHANNEL["sampling-profile"]
g = set(visible(STRICT_BUNDLE))
ys = surface(STRICT_BUNDLE, PROFILE)
print("facts:", len(FACTS), "| axes:", len({e for _, e in FACTS}),
      "| strict bundle: visible", len(g), "closed", len(FACTS) - len(g))
print()
print("RESOURCE AXIS (the control group's axis)")
for o, e in FACTS:
    if e == "resource":
        status = ("left open" if o in MECHANISM["control-group"]["left_open"]
                   else "RETURNS" if o in PROFILE else "closed")
        print(f"  {o:20s}{status}")
print()
print("fact shown by channel   axis      status under strict bundle")
for o in PROFILE:
    print(f"  {o:24s}{dict(FACTS)[o]:10s}"
          f"{'already visible' if o in g else 'RETURNS'}")
print()
print("bundle          visible  returned  open surface  open axes")
for ad, d in (("plain", []), ("common", COMMON_BUNDLE), ("strict", STRICT_BUNDLE)):
    y = surface(d, PROFILE)
    print(f"  {ad:12s} {y['visible']:7d}  {y['returned']:10d}"
          f"  {y['open_surface']:10d}  {y['open_axes']:10d}")
print()
print("channel                 returned  open surface  cost  run  cost per fact")
for k in CHANNEL:
    y = surface(STRICT_BUNDLE, CHANNEL[k])
    b = CHANNEL_COST[k]
    print(f"  {k:22s} {y['returned']:4d}  {y['open_surface']:10d}  {b:5d}"
          f"  {BASELINE_TIME * (100 + b) // 100:5d}  {b / y['returned']:17.2f}")
print()
print("SAMPLE COUNT SWEEP (returned fact does not depend on sample count)")
print("  samples   lines  returned fact  lines per fact")
for n in SAMPLE_COUNT:
    print(f"  {n:6d}  {n:8d}  {ys['returned']:15d}  {n // ys['returned']:17d}")
print()
print("FACT SET SWEEP -- removal")
returned_falls = [o for o, _ in FACTS
                   if surface(STRICT_BUNDLE, PROFILE,
                              [x for x in FACTS if x[0] != o])["returned"] < ys["returned"]]
changed = [o for o, _ in FACTS
           if surface(STRICT_BUNDLE, PROFILE,
                      [x for x in FACTS if x[0] != o])["open_surface"] != ys["open_surface"]]
print("  baseline open surface", ys["open_surface"], "| changes it", len(changed),
      "| does not", len(FACTS) - len(changed))
print("  the only fact zeroing out the return:", returned_falls)
print()
print("FACT SET SWEEP -- addition (one new fact per axis)")
for e in sorted({e for _, e in FACTS}):
    y = surface(STRICT_BUNDLE, PROFILE, FACTS + [("new-fact", e)])
    print(f"  +{e:10s} facts 25  open surface {y['open_surface']:2d}", end="")
    print("   <-- grows" if y["open_surface"] > ys["open_surface"] else "")

facts: 24 | axes: 9 | strict bundle: visible 10 closed 14

RESOURCE AXIS (the control group's axis)
  cpu-share           RETURNS
  memory-limit        closed
  memory-used         closed
  cpu-count           left open

fact shown by channel   axis      status under strict bundle
  cpu-share               resource  RETURNS
  cpu-count               resource  already visible
  system-load             kernel    already visible

bundle          visible  returned  open surface  open axes
  plain             24           0          24           9
  common            12           1          13           7
  strict            10           1          11           7

channel                 returned  open surface  cost  run  cost per fact
  sampling-profile          1          11      3   1030               3.00
  system-call-tracing       2          12     34   1340              17.00

SAMPLE COUNT SWEEP (returned fact does not depend on sample count)
  samples   lines  returned fact  lines per fact
     100       100                1                100
    1000      1000                1               1000
   10000     10000                1              10000
  100000    100000                1             100000

FACT SET SWEEP -- removal
  baseline open surface 11 | changes it 11 | does not 13
  the only fact zeroing out the return: ['cpu-share']

FACT SET SWEEP -- addition (one new fact per axis)
  +host       facts 25  open surface 11
  +kernel     facts 25  open surface 12   <-- grows
  +label      facts 25  open surface 11
  +mount      facts 25  open surface 11
  +network    facts 25  open surface 11
  +privilege  facts 25  open surface 11
  +process    facts 25  open surface 11
  +resource   facts 25  open surface 11
  +user       facts 25  open surface 11

Oracle, Channel, Returning Fact

Three numbers sit side by side. Oracle: the process’s 24 facts, over nine axes. Channel: the sampling profile shows three facts. Returning fact and open surface: 1 of the three is a fact the strict bundle closed and it returns; the open surface climbs from 10 to 11, open axes stays at 7.

The single returning fact is cpu-share. The other two facts — cpu-count and system-load — were already visible under the strict bundle. This is the smallest open surface among the four channels: 11. The same number is 12 for system call tracing, 13 for the remaining two channels.

Why the two already-visible facts are visible reads separately too. cpu-count is a fact the control group deliberately leaves open: a process keeps seeing how many CPUs exist, only its share of them is limited. system-load sits on the kernel axis, and no mechanism closes that axis; the previous lesson’s addition sweep already showed this in one line. So two of the profile’s three facts sit exactly where isolation never touches; the channel opens no gap there, it only makes existing visibility readable.

The bundle table gives one more detail. In the common six-mechanism bundle too, the return is 1; tightening isolation does not narrow this channel’s gap. The reason shows up in the table: the mechanism that opens the gap is the control group, and it is present in the common bundle too. What the channel pierces is not the two mechanisms added later to the bundle, it is a mechanism at the bundle’s core.

The Control Group’s Axis Partly Reopens

The resource axis has four facts. The control group closes three of them and leaves cpu-count open; this was measured in the control groups lesson and is not repeated here. The profile brings back one of the three closed ones — cpu-share. Result: two of the resource axis’s four facts become visible. The control group alone left 25 percent visibility; with the profile on, this climbs to 50 percent.

The reason is structural and goes beyond the model. The control group limits how much resource a process can take; the profile counts how much resource the process uses. The two are two faces of the same quantity. Limiting requires measuring, and measuring makes visible. A control group’s own accounting counters hold this same quantity; the profile does not produce this information from scratch, it makes information already kept readable with per-process separation.

The remaining two facts — memory-limit and memory-used — stay closed on this channel. The profile is CPU-centered; the memory side will be opened by another channel, and that channel is measured in this topic’s last lesson.

Among the four channels, the profile is the only one whose returning fact is a quantity kept in a mechanism’s own accounting. The others read a fact from a place the mechanism never records at all. This difference has a counterpart on the narrowing side: closing visibility without stopping measurement is hard here, because measurement is part of isolation’s own operation. The narrowing question has to be asked not as “should this be measured” but as “who should be able to read it.”

More Samples, Same Fact

The sample-count sweep pays the course’s second claim on this channel. When the sample count grows from 100 to 100000, that is, a thousandfold, the line count grows a thousandfold too; the returning fact stays fixed at 1. Lines per fact climb from 100 to 100000. This relationship is defined in the model (OD12), but the definition is not arbitrary: a fact’s visibility is a binary property, while sample count determines the precision of that fact’s estimate.

The distinction holds on the operational side too. An operator raising the sample rate gets a finer distribution — a more precise answer to whether a function’s share is 40 percent or 42 percent. Raising it does not give a new fact: which processes are sharing the CPU is equally clear at the first sample and the hundred-thousandth. For someone looking at this from an isolation angle, this channel’s risk is independent of sample rate.

The cost side reads from the same place. The profile’s cost is 3 percent: the baseline run of 1000 time units becomes 1030 with the profile on. Cost per returning fact is 3.00 percentage points; against system call tracing’s 17.00 points, the profile is 5.67 times cheaper per fact. This cheapness has a consequence: an expensive channel is closed once diagnosis ends, a cheap channel can be left on continuously. Continuity makes a single fact’s visibility permanent.

Fact Set Sweep

In the removal sweep, the open surface changes with 11 of 24 facts, does not change with 13. The only fact that lowers the returned count is cpu-share; when it is removed, the channel brings back no fact and the open surface drops to 10. That is, this channel’s entire gap rests on a single fact. This shows the measurement’s dependence on the mock setup honestly: if the list changed, the profile’s gap could close entirely.

The addition sweep confirms the previous lesson’s result. When one fact is added to each of the nine axes, the open surface grows only on the kernel axis (from 11 to 12). None of the eight mechanisms closes that axis; a new fact added to the resource axis, in contrast, is closed by the control group and is not written to the surface. If an axis has a closer, a fact list growing on that axis does not grow the surface.

Narrowing the Surface

This channel’s gap is narrow, but it has narrowing paths, and they need to be written up next to the counted surface. The first is the sampling privilege: access to CPU counters is bound to a separate capability and can be added to the capability-dropping list; sampling can be turned off for unprivileged processes given kernel settings. The second is scope: the profile is turned on to target a single process or a single control group, not the whole machine — machine-wide profiling also makes bundle-external processes’ CPU share visible. The third is the profile file itself; the collected distribution carries function names and call stacks, and the file is bound to access control. The fourth is duration: it is turned on for the diagnosis and not left on continuously.

This lesson counts a gap, it does not describe its use. Escape, privilege escalation, and evasion procedures are written in no lesson of this course; what is measured is which fact stays visible.

Summary

• The sampling profile shows three facts; only 1 of them (cpu-share) returns under the strict bundle and the open surface becomes 11 — the smallest surface of the four channels.
• Open axes stay at 7; the profile opens no new axis, it thickens the resource axis: visible facts on that axis climb from 1 to 2 of its four.
• The cost is 3 percent (baseline 1000, run 1030) and 3.00 percentage points per returning fact — 5.67 times cheaper per fact than system call tracing’s 17.00 points. The percentages are model values, not a real measurement.
• When sample count grows a thousandfold, line count grows a thousandfold too, and the returning fact stays at 1: sample count improves the estimate, it does not grow the surface.
• Sweep: the channel’s whole gap rests on one fact (cpu-share); a newly added fact again grows the surface only on the kernel axis.
• Narrowing: dropping the sampling capability, narrowing the profile to a single process or control group, access control on the profile file, and a bounded duration.

Next Step

Two channels have been measured, and both worked in the same direction: they left the open-axis count at 7. The next channel breaks this. Event-based in-kernel tracing brings back 3 facts for a 6-percent cost, and one of them reopens the axis the network namespace closes fully; open axes climbs from 7 to 8. The next lesson measures this channel by splitting it into three tracepoints and writes up the result: the channel that brings back the most is not the most expensive channel.