--- title: 'In-Kernel Tracing' source: 'https://academia.sh/en/courses/kernel-interfaces/in-kernel-tracing' course: 'Kernel Interfaces and Isolation' language: en updated: '2026-08-17T18:09:58+00:00' license: 'CC BY-SA 4.0' --- # In-Kernel Tracing Measuring event-based in-kernel tracing: 3 facts return for a 6-percent cost, the open surface becomes 13, and the axis the network namespace closes fully reopens, taking open axes from 7 to 8; at 2.00 points per fact, it is the lowest of the four channels. The previous two channels worked in the same direction. System call tracing brought back 2 facts for a 34-percent cost, the sampling profile 1 fact for 3 percent; both left the open-axis count at **7**. The open-axis count staying fixed was not a small detail — the channels thickened open axes, they did not open a closed one. This lesson breaks that pattern. Event-based in-kernel tracing brings back **3** of the 14 facts the strict bundle closes, and one of them belongs to an axis the network namespace closes **fully**. Open axes climb from 7 to **8**. The cost is 6 percent — less than a fifth of system call tracing. This topic's sharpest sentence follows from here: **the channel that brings back the most is not the most expensive channel.** ## Event-Based Observation System call tracing hands control to the observing side on every call; the sampling profile takes a snapshot at regular intervals. In-kernel tracing is a third way: **tracepoints** are placed inside the kernel, a small handler runs when an event occurs, and the result is written to a buffer. The process is not stopped, no sample is waited for; only the event of interest is recorded. Assumptions continue. The five facts the channel shows are the list in the code (**OD14**). Costs are **percentages in the model, not a real measurement**; the baseline run is 1000 time units (**OD15**). Tracing is set up on the kernel side and runs independently of the traced process's namespaces (**OD16**); this is the source of the channel's power to pierce isolation. When an event is recorded, the related fact becomes fully visible (**OD17**). The channel splits into three tracepoints, and the points' fact sets are the lists in the code (**OD18**). Point costs are 2, 3, and 1, and they sum to the channel cost (**OD19**). Points can be turned on and off independently of one another (**OD20**). Buffer overflow and event drops are not in the model (**OD21**) — on a real system, high event rates thin out recording, and this boundary falls outside the measurement. ```text # example dump , has not been run $ cat /sys/kernel/tracing/available_events | wc -l $ echo 1 > /sys/kernel/tracing/events/sched/sched_process_fork/enable $ cat /sys/kernel/tracing/trace_pipe - [] : sched_process_fork: parent= child= ``` The block has not been run and the values in it are mock. What is measured is not the lines' format, it is the facts readable from the lines. ```python # M03/K05 common definition -- only the parts this lesson uses. FACTS = [ ("other-process-list", "process"), ("own-process-number", "process"), ("ancestor-process-chain", "process"), ("root-filesystem-tree", "mount"), ("other-mounts", "mount"), ("shared-temp-directory", "mount"), ("host-interfaces", "network"), ("host-routing-table", "network"), ("listening-ports", "network"), ("user-id-mapping", "user"), ("file-ownership", "user"), ("hostname", "host"), ("cpu-share", "resource"), ("memory-limit", "resource"), ("memory-used", "resource"), ("cpu-count", "resource"), ("kernel-version", "kernel"), ("kernel-settings", "kernel"), ("system-load", "kernel"), ("clock", "kernel"), ("capability-set", "privilege"), ("file-permissions", "privilege"), ("security-label", "label"), ("policy-rules", "label"), ] MECHANISM = { "pid-namespace": {"axis": ["process"], "left_open": ["own-process-number"]}, "mount-namespace": {"axis": ["mount"], "left_open": ["root-filesystem-tree"]}, "network-namespace": {"axis": ["network"], "left_open": []}, "user-namespace": {"axis": ["user"], "left_open": ["file-ownership"]}, "uts-namespace": {"axis": ["host"], "left_open": []}, "control-group": {"axis": ["resource"], "left_open": ["cpu-count"]}, "capability-dropping": {"axis": ["privilege"], "left_open": ["file-permissions"]}, "mandatory-label": {"axis": ["label"], "left_open": ["security-label"]}, } STRICT_BUNDLE = ["pid-namespace", "mount-namespace", "network-namespace", "uts-namespace", "control-group", "capability-dropping", "user-namespace", "mandatory-label"] CHANNEL = { "system-call-tracing": ["root-filesystem-tree", "other-mounts", "file-ownership", "capability-set", "clock"], "sampling-profile": ["cpu-share", "cpu-count", "system-load"], "in-kernel-tracing": ["other-process-list", "ancestor-process-chain", "listening-ports", "kernel-settings", "system-load"], "core-dump": ["memory-used", "capability-set", "security-label", "user-id-mapping"], } # OD15: costs are MODEL percentages , not a real measurement CHANNEL_COST = {"system-call-tracing": 34, "sampling-profile": 3, "in-kernel-tracing": 6, "core-dump": 0} BASELINE_TIME = 1000 # OD18-OD20: the channel splits into three tracepoints ; point costs 2+3+1 = 6 POINT = { "process-events": (["other-process-list", "ancestor-process-chain"], 2), "network-events": (["listening-ports"], 3), "kernel-setting-events": (["kernel-settings", "system-load"], 1), } def visible(mechanisms, facts=FACTS): closed = set() for d in mechanisms: v = MECHANISM[d] for e in v["axis"]: closed |= {o for o, x in facts if x == e} closed -= set(v["left_open"]) return [o for o, _ in facts if o not in closed] def returned(mechanisms, channel, facts=FACTS): closed = {o for o, _ in facts} - set(visible(mechanisms, facts)) return sorted(set(channel) & closed) def surface(mechanisms, channel=(), facts=FACTS): g = set(visible(mechanisms, facts)) back = set(returned(mechanisms, channel, facts)) return {"visible": len(g), "returned": len(back), "open_surface": len(g | back), "open_axes": len({e for o, e in facts if o in (g | back)})} IN_KERNEL = CHANNEL["in-kernel-tracing"] g = set(visible(STRICT_BUNDLE)) ys = surface(STRICT_BUNDLE, IN_KERNEL) print("facts:", len(FACTS), "| axes:", len({e for _, e in FACTS}), "| strict bundle: visible", len(g), "closed", len(FACTS) - len(g), "open axes", surface(STRICT_BUNDLE)["open_axes"]) print() print("fact shown by channel axis status under strict bundle") for o in IN_KERNEL: print(f" {o:28s}{dict(FACTS)[o]:10s}" f"{'already visible' if o in g else 'RETURNS'}") print() print("AXIS TABLE (fact count | visible under strict | visible with channel on)") back = set(returned(STRICT_BUNDLE, IN_KERNEL)) for e in sorted({e for _, e in FACTS}): all_ = [o for o, x in FACTS if x == e] a = len([o for o in all_ if o in g]) b = len([o for o in all_ if o in g | back]) note = " <-- axis reopened" if a == 0 < b else "" print(f" {e:10s} {len(all_):5d} {a:24d} {b:26d}{note}") print() print("channel returned open surface open axes cost cost per fact") for k in CHANNEL: y = surface(STRICT_BUNDLE, CHANNEL[k]) b = CHANNEL_COST[k] ratio = f"{b / y['returned']:.2f}" if y["returned"] else "-" print(f" {k:22s} {y['returned']:4d} {y['open_surface']:10d} {y['open_axes']:10d}" f" {b:5d} {ratio:>17s}") print() print("TRACEPOINT SWEEP (points open one after another)") print(" point new fact total returned total cost marginal cost") open_, total = [], 0 for ad, (facts_, cost) in POINT.items(): before = len(returned(STRICT_BUNDLE, open_)) open_ = open_ + facts_ now = len(returned(STRICT_BUNDLE, open_)) total += cost print(f" {ad:22s} {now - before:10d} {now:12d} {total:13d} {cost:15d}") print(" total tracepoint cost:", total, "= channel cost", CHANNEL_COST["in-kernel-tracing"], "| run", BASELINE_TIME * (100 + total) // 100) print() print("FACT SET SWEEP -- removal") lowers_return = [o for o, _ in FACTS if surface(STRICT_BUNDLE, IN_KERNEL, [x for x in FACTS if x[0] != o])["returned"] < ys["returned"]] changes = [o for o, _ in FACTS if surface(STRICT_BUNDLE, IN_KERNEL, [x for x in FACTS if x[0] != o])["open_surface"] != ys["open_surface"]] lowers_axes = [o for o, _ in FACTS if surface(STRICT_BUNDLE, IN_KERNEL, [x for x in FACTS if x[0] != o])["open_axes"] < ys["open_axes"]] print(" baseline open surface", ys["open_surface"], "open axes", ys["open_axes"], "| changes it", len(changes), "| does not", len(FACTS) - len(changes)) print(" lowers returned:", ", ".join(lowers_return)) print(" fact that lowers open axes from 8 to 7:", len(lowers_axes), "| of these , the one the channel brings back:", [o for o in lowers_axes if o in returned(STRICT_BUNDLE, IN_KERNEL)]) print() grows = [e for e in sorted({e for _, e in FACTS}) if surface(STRICT_BUNDLE, IN_KERNEL, FACTS + [("new-fact", e)])["open_surface"] > ys["open_surface"]] print("FACT SET SWEEP -- addition (one new fact per axis)") print(" axis growing the surface:", grows, "-> open surface", ys["open_surface"] + 1) print(" the other", 9 - len(grows), "axes stay at open surface", ys["open_surface"]) ``` ``` facts: 24 | axes: 9 | strict bundle: visible 10 closed 14 open axes 7 fact shown by channel axis status under strict bundle other-process-list process RETURNS ancestor-process-chain process RETURNS listening-ports network RETURNS kernel-settings kernel already visible system-load kernel already visible AXIS TABLE (fact count | visible under strict | visible with channel on) host 1 0 0 kernel 4 4 4 label 2 1 1 mount 3 1 1 network 3 0 1 <-- axis reopened privilege 2 1 1 process 3 1 3 resource 4 1 1 user 2 1 1 channel returned open surface open axes cost cost per fact system-call-tracing 2 12 7 34 17.00 sampling-profile 1 11 7 3 3.00 in-kernel-tracing 3 13 8 6 2.00 core-dump 3 13 7 0 0.00 TRACEPOINT SWEEP (points open one after another) point new fact total returned total cost marginal cost process-events 2 2 2 2 network-events 1 3 5 3 kernel-setting-events 0 3 6 1 total tracepoint cost: 6 = channel cost 6 | run 1060 FACT SET SWEEP -- removal baseline open surface 13 open axes 8 | changes it 13 | does not 11 lowers returned: other-process-list, ancestor-process-chain, listening-ports fact that lowers open axes from 8 to 7: 6 | of these , the one the channel brings back: ['listening-ports'] FACT SET SWEEP -- addition (one new fact per axis) axis growing the surface: ['kernel'] -> open surface 14 the other 8 axes stay at open surface 13 ``` ## Oracle, Channel, Returning Fact Three numbers sit side by side. **Oracle:** the process's 24 facts, over nine axes. **Channel:** in-kernel tracing shows five facts. **Returning fact and open surface:** **3** of the five are facts the strict bundle closed and they return; the open surface climbs from **10 to 13**, open axes from **7 to 8**. The returning trio is `other-process-list`, `ancestor-process-chain`, and `listening-ports`. The first two are facts the pid namespace closes: even if a process sees only itself inside its own namespace, a process-event tracker placed on the kernel side sees fork events regardless of namespace boundaries. The third is a fact the network namespace closes, and that is exactly where the real break happens. The two already-visible facts — `kernel-settings` and `system-load` — sit on the kernel axis. The same result came up in the previous two lessons: no mechanism closes that axis. So two of the channel's five facts open nothing new for this reason. ## A Closed Axis Reopens The axis table changes in this channel alone, among the four. The network axis has three facts and the network namespace closes all three; in the common definition's first reading, network and uts namespaces were the only pair that closes their own axes **fully**. In-kernel tracing pierces that full closure: `listening-ports` returns, and the number of visible facts on the network axis climbs from 0 to **1**. The distinction matters operationally. An axis thickening is one more thing becoming visible while something on that axis was already visible; it grows the measure but does not change the isolation decision. An axis being **reopened**, though, invalidates the isolation decision itself. The decision made when setting up the network namespace was "this process will not see the machine's network"; with the channel on, that decision no longer holds. That the strict bundle leaves something open on seven of nine axes was a known result; the eighth axis opening **because of observation** is a separate result. There is also a thickening on the process axis: the visible count out of three facts climbs from 1 to **3**, that is, the axis becomes fully visible. Because the axis was already counted open, no increase shows in the table; on the fact side, though, this is the channel's biggest contribution. ## The Channel That Brings Back the Most Is Not the Most Expensive The four-channel comparison table gives this topic's main claim in a single glance. System call tracing brings **2** facts for **34 percent**, in-kernel tracing brings **3** facts for **6 percent**. Cost per fact is **17.00** for the first, **2.00** percentage points for the second: in-kernel tracing is, per fact, **one eighth and a half** as expensive as system call tracing, and brings back more facts. There is no link between expense and piercing power in this model. The reason is structural. System call tracing's cost is paid **on every call**; it grows with the measured process's call density. In-kernel tracing's cost is paid **only on the chosen events**; however much the process runs, nothing is paid for events not chosen. The first is wide and blind, the second is narrow and selective. Selectivity is the source of both the cheapness and the piercing power: when the right event is chosen, a single tracepoint gives directly the fact a namespace closes. The table's fourth row is already visible too: core dump brings **3** facts for **0 percent** cost. Its cost is not in the run, it sits elsewhere, and it is measured in this topic's last lesson. ## Tracepoint Sweep Once the channel is split into three points, marginal return shows plainly. The process-events point brings **2** new facts and costs **2** points. The network-events point brings **1** new fact and costs **3** points — the most expensive point is the point that brings back the fewest facts; but the fact it brings is exactly the one that reopens the axis. The kernel-setting-events point brings **0** new facts and still charges **1** point. The three points sum to 6, equal to the channel cost; the run climbs from 1000 to 1060. The third point is the most instructive one operationally. It is on, it produces records, it charges a cost, and it adds **nothing** on the isolation side; the two facts it shows were already visible. The question to ask when auditing an observation setup should not be "what does this point show," it should be "what does this point show that is **new**." A point with zero marginal return only charges a cost. ## Fact Set Sweep In the removal sweep, the open surface changes with **13** of 24 facts, does not change with **11**. The three facts lowering the returned count are exactly the channel's own three returning facts. There are **6** facts that lower open axes from 8 to 7; only one of them — `listening-ports` — is a fact the channel brings back. The remaining five are removed and close their own axis entirely, because they were the only fact visible on that axis under the strict bundle. This second result shows the fragility of the axis measure. An axis being counted "open" can rest on a single fact; if the list changes, the axis count changes. The fact count is more robust. The addition sweep gives the same answer as the previous two lessons: among the nine axes, only the `kernel` axis grows the open surface with a new fact (13 to 14); the other eight stay at surface **13**. ## Narrowing the Surface A narrowing path is written up next to the counted surface. The first is the privilege to place a tracepoint: in-kernel tracing requires a separate privilege and it can be added to the capability-dropping list; tracing can be turned off for unprivileged processes given kernel settings. The second is point selection — as the sweep showed, points with zero marginal return are turned off; as the number of open points falls, both cost and surface shrink. The third is event filters: a tracepoint can be narrowed to a single control group or a single namespace; machine-wide tracing also makes bundle-external processes' events visible. The fourth is on the recording side; the collected event stream carries process names, ancestor chains, and ports, and its access has to be restricted. This lesson counts a gap, it does not describe its use. Escape, privilege escalation, and evasion procedures are written in no lesson of this course. ## Summary - In-kernel tracing shows five facts; **3** return and the open surface becomes **13** — one of the two biggest returners among the four channels. - Open axes climb from **7 to 8**; the axis the network namespace closes **fully** reopens, and this is the only channel that does so. - The cost is **6 percent** (baseline 1000, run 1060) and **2.00 percentage points** per fact — **one eighth and a half** of system call tracing's 17.00 points. The percentages are model values, not a real measurement. - The three tracepoints' marginal returns are 2, 1, and **0** facts; their costs are 2, 3, and 1 points. The most expensive point brings the fewest facts, the third point charges a cost while bringing no new fact. - Sweep: only one of the 6 facts that lower the open axes is a fact the channel brings; a newly added fact again grows the surface only on the **kernel** axis. - Narrowing: dropping the tracing capability, turning off points with zero marginal return, narrowing events to a single control group, and access control on the event stream. ## Next Step All three channels ran during the run, and all three added a cost to the run. The next channel touches the run **not at all**: a core dump looks **after** a process has crashed, and its run cost is **0 percent**. Yet the number of facts it brings back is **3**, and among what it brings are user ID mapping and the capability set. Why a channel with zero cost has one of the largest surfaces is explained by **what the dump carries** — and this is the last lesson of both this course and the M03 curriculum.