--- title: 'Performance Profiling' source: 'https://academia.sh/en/courses/kernel-interfaces/performance-profiling' course: 'Kernel Interfaces and Isolation' language: en updated: '2026-08-17T18:09:58+00:00' license: 'CC BY-SA 4.0' --- # Performance Profiling The cheapest and narrowest of the four channels: the sampling-based CPU profile brings back only 1 of the 14 facts the strict bundle closes, the open surface becomes 11, and the run slows by 3 percent; the returned fact does not change when the sample count grows a thousandfold. The previous lesson measured an expensive channel: system call tracing brought back 2 of the 14 facts the strict bundle closes, and slowed the run by 34 percent. Cost per returning fact was 17.0 percentage points. This lesson sits at the other end of the scale. The sampling-based CPU profile is the **cheapest** of the model's four channels: the slowdown it adds to the run is 3 percent. The question is what that cheapness costs. If a channel gives information without touching the run, its power to pierce isolation might be low too; or, the other way around, a cheap channel might be exactly the one that opens the real gap, because it can be left on continuously. The measurement says which of the two is true: the profile is also the narrowest in the number of facts it brings back — the returning fact is **1**. ## The Profile as a Channel A sampling profile stops the running process at regular intervals and records which code is executing at that moment. What it collects is not a sequence of events but a **distribution**: what share of the samples each function collected. How to read a profile — which metric to pick, which event the sampling rate will miss — was measured on the frontend side in Frontend Quality and on the operations side in the Observability and Operations course, and is not repeated here. In this lesson the profile is measured not as a performance tool but **as an isolation channel**. The assumptions continue and widen. The three facts the profile channel shows are the list in the code (**OD8**). Costs are percentages in the model, **not a real measurement**; the baseline run is 1000 time units (**OD9**). The profile is on for the whole run, and its cost is taken as independent of sample count (**OD10**) — on a real system the cost rises as sample rate rises, and the model closes off this distinction. The sample count is a mock value and every sample produces one line (**OD11**). Sample count improves a fact's **estimate**, it does not change the fact's visibility (**OD12**). The profiling tool runs from outside the bundle, during the run (**OD13**). ```text # example dump , has not been run $ perf record -F -g -p -- sleep [ perf record: samples written ] $ perf report --stdio --sort symbol # Overhead Symbol # [.] # [.] # [k] ``` The values in the dump are mock and the block has not been run. What is measured is not the dump's lines but the **facts** readable from it. ```python # M03/K05 common definition -- only the parts this lesson uses. FACTS = [ ("other-process-list", "process"), ("own-process-number", "process"), ("ancestor-process-chain", "process"), ("root-filesystem-tree", "mount"), ("other-mounts", "mount"), ("shared-temp-directory", "mount"), ("host-interfaces", "network"), ("host-routing-table", "network"), ("listening-ports", "network"), ("user-id-mapping", "user"), ("file-ownership", "user"), ("hostname", "host"), ("cpu-share", "resource"), ("memory-limit", "resource"), ("memory-used", "resource"), ("cpu-count", "resource"), ("kernel-version", "kernel"), ("kernel-settings", "kernel"), ("system-load", "kernel"), ("clock", "kernel"), ("capability-set", "privilege"), ("file-permissions", "privilege"), ("security-label", "label"), ("policy-rules", "label"), ] MECHANISM = { "pid-namespace": {"axis": ["process"], "left_open": ["own-process-number"]}, "mount-namespace": {"axis": ["mount"], "left_open": ["root-filesystem-tree"]}, "network-namespace": {"axis": ["network"], "left_open": []}, "user-namespace": {"axis": ["user"], "left_open": ["file-ownership"]}, "uts-namespace": {"axis": ["host"], "left_open": []}, "control-group": {"axis": ["resource"], "left_open": ["cpu-count"]}, "capability-dropping": {"axis": ["privilege"], "left_open": ["file-permissions"]}, "mandatory-label": {"axis": ["label"], "left_open": ["security-label"]}, } COMMON_BUNDLE = ["pid-namespace", "mount-namespace", "network-namespace", "uts-namespace", "control-group", "capability-dropping"] STRICT_BUNDLE = COMMON_BUNDLE + ["user-namespace", "mandatory-label"] CHANNEL = { # OD8: facts the profile channel shows ; tracing channel for comparison "sampling-profile": ["cpu-share", "cpu-count", "system-load"], "system-call-tracing": ["root-filesystem-tree", "other-mounts", "file-ownership", "capability-set", "clock"], } # OD9: costs are percentages in the MODEL, not a real measurement CHANNEL_COST = {"sampling-profile": 3, "system-call-tracing": 34} BASELINE_TIME = 1000 # OD11-OD13: sample count is a construct; each sample produces one line SAMPLE_COUNT = [100, 1000, 10000, 100000] def visible(mechanisms, facts=FACTS): closed = set() for d in mechanisms: v = MECHANISM[d] for e in v["axis"]: closed |= {o for o, x in facts if x == e} closed -= set(v["left_open"]) return [o for o, _ in facts if o not in closed] def returned(mechanisms, channel, facts=FACTS): closed = {o for o, _ in facts} - set(visible(mechanisms, facts)) return sorted(set(channel) & closed) def surface(mechanisms, channel=(), facts=FACTS): g = set(visible(mechanisms, facts)) back = set(returned(mechanisms, channel, facts)) return {"visible": len(g), "returned": len(back), "open_surface": len(g | back), "open_axes": len({e for o, e in facts if o in (g | back)})} PROFILE = CHANNEL["sampling-profile"] g = set(visible(STRICT_BUNDLE)) ys = surface(STRICT_BUNDLE, PROFILE) print("facts:", len(FACTS), "| axes:", len({e for _, e in FACTS}), "| strict bundle: visible", len(g), "closed", len(FACTS) - len(g)) print() print("RESOURCE AXIS (the control group's axis)") for o, e in FACTS: if e == "resource": status = ("left open" if o in MECHANISM["control-group"]["left_open"] else "RETURNS" if o in PROFILE else "closed") print(f" {o:20s}{status}") print() print("fact shown by channel axis status under strict bundle") for o in PROFILE: print(f" {o:24s}{dict(FACTS)[o]:10s}" f"{'already visible' if o in g else 'RETURNS'}") print() print("bundle visible returned open surface open axes") for ad, d in (("plain", []), ("common", COMMON_BUNDLE), ("strict", STRICT_BUNDLE)): y = surface(d, PROFILE) print(f" {ad:12s} {y['visible']:7d} {y['returned']:10d}" f" {y['open_surface']:10d} {y['open_axes']:10d}") print() print("channel returned open surface cost run cost per fact") for k in CHANNEL: y = surface(STRICT_BUNDLE, CHANNEL[k]) b = CHANNEL_COST[k] print(f" {k:22s} {y['returned']:4d} {y['open_surface']:10d} {b:5d}" f" {BASELINE_TIME * (100 + b) // 100:5d} {b / y['returned']:17.2f}") print() print("SAMPLE COUNT SWEEP (returned fact does not depend on sample count)") print(" samples lines returned fact lines per fact") for n in SAMPLE_COUNT: print(f" {n:6d} {n:8d} {ys['returned']:15d} {n // ys['returned']:17d}") print() print("FACT SET SWEEP -- removal") returned_falls = [o for o, _ in FACTS if surface(STRICT_BUNDLE, PROFILE, [x for x in FACTS if x[0] != o])["returned"] < ys["returned"]] changed = [o for o, _ in FACTS if surface(STRICT_BUNDLE, PROFILE, [x for x in FACTS if x[0] != o])["open_surface"] != ys["open_surface"]] print(" baseline open surface", ys["open_surface"], "| changes it", len(changed), "| does not", len(FACTS) - len(changed)) print(" the only fact zeroing out the return:", returned_falls) print() print("FACT SET SWEEP -- addition (one new fact per axis)") for e in sorted({e for _, e in FACTS}): y = surface(STRICT_BUNDLE, PROFILE, FACTS + [("new-fact", e)]) print(f" +{e:10s} facts 25 open surface {y['open_surface']:2d}", end="") print(" <-- grows" if y["open_surface"] > ys["open_surface"] else "") ``` ``` facts: 24 | axes: 9 | strict bundle: visible 10 closed 14 RESOURCE AXIS (the control group's axis) cpu-share RETURNS memory-limit closed memory-used closed cpu-count left open fact shown by channel axis status under strict bundle cpu-share resource RETURNS cpu-count resource already visible system-load kernel already visible bundle visible returned open surface open axes plain 24 0 24 9 common 12 1 13 7 strict 10 1 11 7 channel returned open surface cost run cost per fact sampling-profile 1 11 3 1030 3.00 system-call-tracing 2 12 34 1340 17.00 SAMPLE COUNT SWEEP (returned fact does not depend on sample count) samples lines returned fact lines per fact 100 100 1 100 1000 1000 1 1000 10000 10000 1 10000 100000 100000 1 100000 FACT SET SWEEP -- removal baseline open surface 11 | changes it 11 | does not 13 the only fact zeroing out the return: ['cpu-share'] FACT SET SWEEP -- addition (one new fact per axis) +host facts 25 open surface 11 +kernel facts 25 open surface 12 <-- grows +label facts 25 open surface 11 +mount facts 25 open surface 11 +network facts 25 open surface 11 +privilege facts 25 open surface 11 +process facts 25 open surface 11 +resource facts 25 open surface 11 +user facts 25 open surface 11 ``` ## Oracle, Channel, Returning Fact Three numbers sit side by side. **Oracle:** the process's 24 facts, over nine axes. **Channel:** the sampling profile shows three facts. **Returning fact and open surface:** **1** of the three is a fact the strict bundle closed and it returns; the open surface climbs from **10 to 11**, open axes stays at **7**. The single returning fact is `cpu-share`. The other two facts — `cpu-count` and `system-load` — were already visible under the strict bundle. This is the smallest open surface among the four channels: **11**. The same number is 12 for system call tracing, 13 for the remaining two channels. Why the two already-visible facts are visible reads separately too. `cpu-count` is a fact the control group deliberately leaves open: a process keeps seeing how many CPUs exist, only its share of them is limited. `system-load` sits on the kernel axis, and no mechanism closes that axis; the previous lesson's addition sweep already showed this in one line. So two of the profile's three facts sit exactly where isolation never touches; the channel opens no gap there, it only makes existing visibility readable. The bundle table gives one more detail. In the common six-mechanism bundle too, the return is **1**; tightening isolation does not narrow this channel's gap. The reason shows up in the table: the mechanism that opens the gap is the **control group**, and it is present in the common bundle too. What the channel pierces is not the two mechanisms added later to the bundle, it is a mechanism at the bundle's **core**. ## The Control Group's Axis Partly Reopens The resource axis has four facts. The control group closes three of them and leaves `cpu-count` open; this was measured in the control groups lesson and is not repeated here. The profile brings back one of the three closed ones — `cpu-share`. Result: **two** of the resource axis's four facts become visible. The control group alone left 25 percent visibility; with the profile on, this climbs to 50 percent. The reason is structural and goes beyond the model. The control group limits **how much resource** a process can take; the profile counts **how much resource** the process uses. The two are two faces of the same quantity. Limiting requires measuring, and measuring makes visible. A control group's own accounting counters hold this same quantity; the profile does not produce this information from scratch, it makes information already kept readable with per-process separation. The remaining two facts — `memory-limit` and `memory-used` — stay closed on this channel. The profile is CPU-centered; the memory side will be opened by another channel, and that channel is measured in this topic's last lesson. Among the four channels, the profile is the only one whose returning fact is a quantity **kept in a mechanism's own accounting**. The others read a fact from a place the mechanism never records at all. This difference has a counterpart on the narrowing side: closing visibility without stopping measurement is hard here, because measurement is part of isolation's own operation. The narrowing question has to be asked not as "should this be measured" but as **"who should be able to read it."** ## More Samples, Same Fact The sample-count sweep pays the course's second claim on this channel. When the sample count grows from 100 to 100000, that is, **a thousandfold**, the line count grows a thousandfold too; the returning fact stays fixed at **1**. Lines per fact climb from 100 to 100000. This relationship is defined in the model (**OD12**), but the definition is not arbitrary: a fact's visibility is a binary property, while sample count determines the precision of that fact's **estimate**. The distinction holds on the operational side too. An operator raising the sample rate gets a finer distribution — a more precise answer to whether a function's share is 40 percent or 42 percent. Raising it does not give a **new fact**: which processes are sharing the CPU is equally clear at the first sample and the hundred-thousandth. For someone looking at this from an isolation angle, this channel's risk is independent of sample rate. The cost side reads from the same place. The profile's cost is **3 percent**: the baseline run of 1000 time units becomes 1030 with the profile on. Cost per returning fact is **3.00 percentage points**; against system call tracing's **17.00** points, the profile is **5.67 times** cheaper per fact. This cheapness has a consequence: an expensive channel is closed once diagnosis ends, a cheap channel **can be left on continuously**. Continuity makes a single fact's visibility permanent. ## Fact Set Sweep In the removal sweep, the open surface changes with **11** of 24 facts, does not change with **13**. The **only** fact that lowers the returned count is `cpu-share`; when it is removed, the channel brings back no fact and the open surface drops to 10. That is, this channel's entire gap rests on a single fact. This shows the measurement's dependence on the mock setup honestly: if the list changed, the profile's gap could close entirely. The addition sweep confirms the previous lesson's result. When one fact is added to each of the nine axes, the open surface grows only on the **`kernel`** axis (from 11 to 12). None of the eight mechanisms closes that axis; a new fact added to the resource axis, in contrast, is closed by the control group and is not written to the surface. If an axis has a closer, a fact list growing on that axis does not grow the surface. ## Narrowing the Surface This channel's gap is narrow, but it has narrowing paths, and they need to be written up next to the counted surface. The first is the sampling privilege: access to CPU counters is bound to a separate capability and can be added to the capability-dropping list; sampling can be turned off for unprivileged processes given kernel settings. The second is scope: the profile is turned on to target a single process or a single control group, not the whole machine — machine-wide profiling also makes bundle-external processes' CPU share visible. The third is the profile file itself; the collected distribution carries function names and call stacks, and the file is bound to access control. The fourth is duration: it is turned on for the diagnosis and not left on continuously. This lesson counts a gap, it does not describe its use. Escape, privilege escalation, and evasion procedures are written in no lesson of this course; what is measured is which fact stays visible. ## Summary - The sampling profile shows three facts; only **1** of them (`cpu-share`) returns under the strict bundle and the open surface becomes **11** — the smallest surface of the four channels. - Open axes stay at **7**; the profile opens no new axis, it thickens the resource axis: visible facts on that axis climb from 1 to 2 of its four. - The cost is **3 percent** (baseline 1000, run 1030) and **3.00 percentage points** per returning fact — **5.67 times** cheaper per fact than system call tracing's 17.00 points. The percentages are model values, not a real measurement. - When sample count grows **a thousandfold**, line count grows a thousandfold too, and the returning fact stays at **1**: sample count improves the estimate, it does not grow the surface. - Sweep: the channel's whole gap rests on one fact (`cpu-share`); a newly added fact again grows the surface only on the **kernel** axis. - Narrowing: dropping the sampling capability, narrowing the profile to a single process or control group, access control on the profile file, and a bounded duration. ## Next Step Two channels have been measured, and both worked in the same direction: they left the open-axis count at 7. The next channel breaks this. Event-based in-kernel tracing brings back **3** facts for a 6-percent cost, and one of them reopens the axis the network namespace closes **fully**; open axes climbs from 7 to **8**. The next lesson measures this channel by splitting it into three tracepoints and writes up the result: the channel that brings back the most is not the most expensive channel.