In-Kernel Tracing

The previous two channels worked in the same direction. System call tracing brought back 2 facts for a 34-percent cost, the sampling profile 1 fact for 3 percent; both left the open-axis count at 7. The open-axis count staying fixed was not a small detail — the channels thickened open axes, they did not open a closed one.

This lesson breaks that pattern. Event-based in-kernel tracing brings back 3 of the 14 facts the strict bundle closes, and one of them belongs to an axis the network namespace closes fully. Open axes climb from 7 to 8. The cost is 6 percent — less than a fifth of system call tracing. This topic’s sharpest sentence follows from here: the channel that brings back the most is not the most expensive channel.

Event-Based Observation

System call tracing hands control to the observing side on every call; the sampling profile takes a snapshot at regular intervals. In-kernel tracing is a third way: tracepoints are placed inside the kernel, a small handler runs when an event occurs, and the result is written to a buffer. The process is not stopped, no sample is waited for; only the event of interest is recorded.

Assumptions continue. The five facts the channel shows are the list in the code (OD14). Costs are percentages in the model, not a real measurement; the baseline run is 1000 time units (OD15). Tracing is set up on the kernel side and runs independently of the traced process’s namespaces (OD16); this is the source of the channel’s power to pierce isolation. When an event is recorded, the related fact becomes fully visible (OD17). The channel splits into three tracepoints, and the points’ fact sets are the lists in the code (OD18). Point costs are 2, 3, and 1, and they sum to the channel cost (OD19). Points can be turned on and off independently of one another (OD20). Buffer overflow and event drops are not in the model (OD21) — on a real system, high event rates thin out recording, and this boundary falls outside the measurement.

# example dump , has not been run
$ cat /sys/kernel/tracing/available_events | wc -l
<event-count>
$ echo 1 > /sys/kernel/tracing/events/sched/sched_process_fork/enable
$ cat /sys/kernel/tracing/trace_pipe
  <process-name>-<no> [<cpu>] <time>: sched_process_fork: parent=<no> child=<no>

The block has not been run and the values in it are mock. What is measured is not the lines’ format, it is the facts readable from the lines.

# M03/K05 common definition -- only the parts this lesson uses.
FACTS = [
    ("other-process-list", "process"), ("own-process-number", "process"),
    ("ancestor-process-chain", "process"),
    ("root-filesystem-tree", "mount"), ("other-mounts", "mount"),
    ("shared-temp-directory", "mount"),
    ("host-interfaces", "network"), ("host-routing-table", "network"),
    ("listening-ports", "network"),
    ("user-id-mapping", "user"), ("file-ownership", "user"),
    ("hostname", "host"),
    ("cpu-share", "resource"), ("memory-limit", "resource"),
    ("memory-used", "resource"), ("cpu-count", "resource"),
    ("kernel-version", "kernel"), ("kernel-settings", "kernel"),
    ("system-load", "kernel"), ("clock", "kernel"),
    ("capability-set", "privilege"), ("file-permissions", "privilege"),
    ("security-label", "label"), ("policy-rules", "label"),
]
MECHANISM = {
    "pid-namespace":     {"axis": ["process"], "left_open": ["own-process-number"]},
    "mount-namespace":   {"axis": ["mount"], "left_open": ["root-filesystem-tree"]},
    "network-namespace": {"axis": ["network"], "left_open": []},
    "user-namespace":    {"axis": ["user"], "left_open": ["file-ownership"]},
    "uts-namespace":     {"axis": ["host"], "left_open": []},
    "control-group":     {"axis": ["resource"], "left_open": ["cpu-count"]},
    "capability-dropping": {"axis": ["privilege"], "left_open": ["file-permissions"]},
    "mandatory-label":   {"axis": ["label"], "left_open": ["security-label"]},
}
STRICT_BUNDLE = ["pid-namespace", "mount-namespace", "network-namespace", "uts-namespace",
                 "control-group", "capability-dropping", "user-namespace", "mandatory-label"]

CHANNEL = {
    "system-call-tracing": ["root-filesystem-tree", "other-mounts", "file-ownership",
                             "capability-set", "clock"],
    "sampling-profile":     ["cpu-share", "cpu-count", "system-load"],
    "in-kernel-tracing":   ["other-process-list", "ancestor-process-chain",
                             "listening-ports", "kernel-settings", "system-load"],
    "core-dump":           ["memory-used", "capability-set", "security-label",
                             "user-id-mapping"],
}
# OD15: costs are MODEL percentages , not a real measurement
CHANNEL_COST = {"system-call-tracing": 34, "sampling-profile": 3,
                "in-kernel-tracing": 6, "core-dump": 0}
BASELINE_TIME = 1000
# OD18-OD20: the channel splits into three tracepoints ; point costs 2+3+1 = 6
POINT = {
    "process-events":       (["other-process-list", "ancestor-process-chain"], 2),
    "network-events":       (["listening-ports"], 3),
    "kernel-setting-events": (["kernel-settings", "system-load"], 1),
}


def visible(mechanisms, facts=FACTS):
    closed = set()
    for d in mechanisms:
        v = MECHANISM[d]
        for e in v["axis"]:
            closed |= {o for o, x in facts if x == e}
        closed -= set(v["left_open"])
    return [o for o, _ in facts if o not in closed]


def returned(mechanisms, channel, facts=FACTS):
    closed = {o for o, _ in facts} - set(visible(mechanisms, facts))
    return sorted(set(channel) & closed)


def surface(mechanisms, channel=(), facts=FACTS):
    g = set(visible(mechanisms, facts))
    back = set(returned(mechanisms, channel, facts))
    return {"visible": len(g), "returned": len(back), "open_surface": len(g | back),
            "open_axes": len({e for o, e in facts if o in (g | back)})}


IN_KERNEL = CHANNEL["in-kernel-tracing"]
g = set(visible(STRICT_BUNDLE))
ys = surface(STRICT_BUNDLE, IN_KERNEL)
print("facts:", len(FACTS), "| axes:", len({e for _, e in FACTS}),
      "| strict bundle: visible", len(g), "closed", len(FACTS) - len(g),
      "open axes", surface(STRICT_BUNDLE)["open_axes"])
print()
print("fact shown by channel         axis      status under strict bundle")
for o in IN_KERNEL:
    print(f"  {o:28s}{dict(FACTS)[o]:10s}"
          f"{'already visible' if o in g else 'RETURNS'}")
print()
print("AXIS TABLE (fact count | visible under strict | visible with channel on)")
back = set(returned(STRICT_BUNDLE, IN_KERNEL))
for e in sorted({e for _, e in FACTS}):
    all_ = [o for o, x in FACTS if x == e]
    a = len([o for o in all_ if o in g])
    b = len([o for o in all_ if o in g | back])
    note = "  <-- axis reopened" if a == 0 < b else ""
    print(f"  {e:10s} {len(all_):5d} {a:24d} {b:26d}{note}")
print()
print("channel                 returned  open surface  open axes  cost  cost per fact")
for k in CHANNEL:
    y = surface(STRICT_BUNDLE, CHANNEL[k])
    b = CHANNEL_COST[k]
    ratio = f"{b / y['returned']:.2f}" if y["returned"] else "-"
    print(f"  {k:22s} {y['returned']:4d}  {y['open_surface']:10d}  {y['open_axes']:10d}"
          f"  {b:5d}  {ratio:>17s}")
print()
print("TRACEPOINT SWEEP (points open one after another)")
print("  point                   new fact  total returned  total cost  marginal cost")
open_, total = [], 0
for ad, (facts_, cost) in POINT.items():
    before = len(returned(STRICT_BUNDLE, open_))
    open_ = open_ + facts_
    now = len(returned(STRICT_BUNDLE, open_))
    total += cost
    print(f"  {ad:22s} {now - before:10d} {now:12d} {total:13d} {cost:15d}")
print("  total tracepoint cost:", total, "= channel cost", CHANNEL_COST["in-kernel-tracing"],
      "| run", BASELINE_TIME * (100 + total) // 100)
print()
print("FACT SET SWEEP -- removal")
lowers_return = [o for o, _ in FACTS
                 if surface(STRICT_BUNDLE, IN_KERNEL,
                            [x for x in FACTS if x[0] != o])["returned"] < ys["returned"]]
changes = [o for o, _ in FACTS
           if surface(STRICT_BUNDLE, IN_KERNEL,
                      [x for x in FACTS if x[0] != o])["open_surface"] != ys["open_surface"]]
lowers_axes = [o for o, _ in FACTS
               if surface(STRICT_BUNDLE, IN_KERNEL,
                          [x for x in FACTS if x[0] != o])["open_axes"] < ys["open_axes"]]
print("  baseline open surface", ys["open_surface"], "open axes", ys["open_axes"],
      "| changes it", len(changes), "| does not", len(FACTS) - len(changes))
print("  lowers returned:", ", ".join(lowers_return))
print("  fact that lowers open axes from 8 to 7:", len(lowers_axes),
      "| of these , the one the channel brings back:",
      [o for o in lowers_axes if o in returned(STRICT_BUNDLE, IN_KERNEL)])
print()
grows = [e for e in sorted({e for _, e in FACTS})
         if surface(STRICT_BUNDLE, IN_KERNEL,
                    FACTS + [("new-fact", e)])["open_surface"] > ys["open_surface"]]
print("FACT SET SWEEP -- addition (one new fact per axis)")
print("  axis growing the surface:", grows, "-> open surface", ys["open_surface"] + 1)
print("  the other", 9 - len(grows), "axes stay at open surface", ys["open_surface"])

facts: 24 | axes: 9 | strict bundle: visible 10 closed 14 open axes 7

fact shown by channel         axis      status under strict bundle
  other-process-list          process   RETURNS
  ancestor-process-chain      process   RETURNS
  listening-ports             network   RETURNS
  kernel-settings             kernel    already visible
  system-load                 kernel    already visible

AXIS TABLE (fact count | visible under strict | visible with channel on)
  host           1                        0                          0
  kernel         4                        4                          4
  label          2                        1                          1
  mount          3                        1                          1
  network        3                        0                          1  <-- axis reopened
  privilege      2                        1                          1
  process        3                        1                          3
  resource       4                        1                          1
  user           2                        1                          1

channel                 returned  open surface  open axes  cost  cost per fact
  system-call-tracing       2          12           7     34              17.00
  sampling-profile          1          11           7      3               3.00
  in-kernel-tracing         3          13           8      6               2.00
  core-dump                 3          13           7      0               0.00

TRACEPOINT SWEEP (points open one after another)
  point                   new fact  total returned  total cost  marginal cost
  process-events                  2            2             2               2
  network-events                  1            3             5               3
  kernel-setting-events           0            3             6               1
  total tracepoint cost: 6 = channel cost 6 | run 1060

FACT SET SWEEP -- removal
  baseline open surface 13 open axes 8 | changes it 13 | does not 11
  lowers returned: other-process-list, ancestor-process-chain, listening-ports
  fact that lowers open axes from 8 to 7: 6 | of these , the one the channel brings back: ['listening-ports']

FACT SET SWEEP -- addition (one new fact per axis)
  axis growing the surface: ['kernel'] -> open surface 14
  the other 8 axes stay at open surface 13

Oracle, Channel, Returning Fact

Three numbers sit side by side. Oracle: the process’s 24 facts, over nine axes. Channel: in-kernel tracing shows five facts. Returning fact and open surface: 3 of the five are facts the strict bundle closed and they return; the open surface climbs from 10 to 13, open axes from 7 to 8.

The returning trio is other-process-list, ancestor-process-chain, and listening-ports. The first two are facts the pid namespace closes: even if a process sees only itself inside its own namespace, a process-event tracker placed on the kernel side sees fork events regardless of namespace boundaries. The third is a fact the network namespace closes, and that is exactly where the real break happens.

The two already-visible facts — kernel-settings and system-load — sit on the kernel axis. The same result came up in the previous two lessons: no mechanism closes that axis. So two of the channel’s five facts open nothing new for this reason.

A Closed Axis Reopens

The axis table changes in this channel alone, among the four. The network axis has three facts and the network namespace closes all three; in the common definition’s first reading, network and uts namespaces were the only pair that closes their own axes fully. In-kernel tracing pierces that full closure: listening-ports returns, and the number of visible facts on the network axis climbs from 0 to 1.

The distinction matters operationally. An axis thickening is one more thing becoming visible while something on that axis was already visible; it grows the measure but does not change the isolation decision. An axis being reopened, though, invalidates the isolation decision itself. The decision made when setting up the network namespace was “this process will not see the machine’s network”; with the channel on, that decision no longer holds. That the strict bundle leaves something open on seven of nine axes was a known result; the eighth axis opening because of observation is a separate result.

There is also a thickening on the process axis: the visible count out of three facts climbs from 1 to 3, that is, the axis becomes fully visible. Because the axis was already counted open, no increase shows in the table; on the fact side, though, this is the channel’s biggest contribution.

The Channel That Brings Back the Most Is Not the Most Expensive

The four-channel comparison table gives this topic’s main claim in a single glance. System call tracing brings 2 facts for 34 percent, in-kernel tracing brings 3 facts for 6 percent. Cost per fact is 17.00 for the first, 2.00 percentage points for the second: in-kernel tracing is, per fact, one eighth and a half as expensive as system call tracing, and brings back more facts. There is no link between expense and piercing power in this model.

The reason is structural. System call tracing’s cost is paid on every call; it grows with the measured process’s call density. In-kernel tracing’s cost is paid only on the chosen events; however much the process runs, nothing is paid for events not chosen. The first is wide and blind, the second is narrow and selective. Selectivity is the source of both the cheapness and the piercing power: when the right event is chosen, a single tracepoint gives directly the fact a namespace closes.

The table’s fourth row is already visible too: core dump brings 3 facts for 0 percent cost. Its cost is not in the run, it sits elsewhere, and it is measured in this topic’s last lesson.

Tracepoint Sweep

Once the channel is split into three points, marginal return shows plainly. The process-events point brings 2 new facts and costs 2 points. The network-events point brings 1 new fact and costs 3 points — the most expensive point is the point that brings back the fewest facts; but the fact it brings is exactly the one that reopens the axis. The kernel-setting-events point brings 0 new facts and still charges 1 point. The three points sum to 6, equal to the channel cost; the run climbs from 1000 to 1060.

The third point is the most instructive one operationally. It is on, it produces records, it charges a cost, and it adds nothing on the isolation side; the two facts it shows were already visible. The question to ask when auditing an observation setup should not be “what does this point show,” it should be “what does this point show that is new.” A point with zero marginal return only charges a cost.

Fact Set Sweep

In the removal sweep, the open surface changes with 13 of 24 facts, does not change with 11. The three facts lowering the returned count are exactly the channel’s own three returning facts. There are 6 facts that lower open axes from 8 to 7; only one of them — listening-ports — is a fact the channel brings back. The remaining five are removed and close their own axis entirely, because they were the only fact visible on that axis under the strict bundle.

This second result shows the fragility of the axis measure. An axis being counted “open” can rest on a single fact; if the list changes, the axis count changes. The fact count is more robust. The addition sweep gives the same answer as the previous two lessons: among the nine axes, only the kernel axis grows the open surface with a new fact (13 to 14); the other eight stay at surface 13.

Narrowing the Surface

A narrowing path is written up next to the counted surface. The first is the privilege to place a tracepoint: in-kernel tracing requires a separate privilege and it can be added to the capability-dropping list; tracing can be turned off for unprivileged processes given kernel settings. The second is point selection — as the sweep showed, points with zero marginal return are turned off; as the number of open points falls, both cost and surface shrink. The third is event filters: a tracepoint can be narrowed to a single control group or a single namespace; machine-wide tracing also makes bundle-external processes’ events visible. The fourth is on the recording side; the collected event stream carries process names, ancestor chains, and ports, and its access has to be restricted.

This lesson counts a gap, it does not describe its use. Escape, privilege escalation, and evasion procedures are written in no lesson of this course.

Summary

• In-kernel tracing shows five facts; 3 return and the open surface becomes 13 — one of the two biggest returners among the four channels.
• Open axes climb from 7 to 8; the axis the network namespace closes fully reopens, and this is the only channel that does so.
• The cost is 6 percent (baseline 1000, run 1060) and 2.00 percentage points per fact — one eighth and a half of system call tracing’s 17.00 points. The percentages are model values, not a real measurement.
• The three tracepoints’ marginal returns are 2, 1, and 0 facts; their costs are 2, 3, and 1 points. The most expensive point brings the fewest facts, the third point charges a cost while bringing no new fact.
• Sweep: only one of the 6 facts that lower the open axes is a fact the channel brings; a newly added fact again grows the surface only on the kernel axis.
• Narrowing: dropping the tracing capability, turning off points with zero marginal return, narrowing events to a single control group, and access control on the event stream.

Next Step

All three channels ran during the run, and all three added a cost to the run. The next channel touches the run not at all: a core dump looks after a process has crashed, and its run cost is 0 percent. Yet the number of facts it brings back is 3, and among what it brings are user ID mapping and the capability set. Why a channel with zero cost has one of the largest surfaces is explained by what the dump carries — and this is the last lesson of both this course and the M03 curriculum.