Container Runtime Detection

The previous lesson stopped at the service boundary: the allow list per caller closed thirty-five pairs, and the path to the secret store grew without closing. What is inside the boundary was never asked. A shell opened inside a service’s own pod violates no call pair: neither the network policy nor mutual verification sees it, because there is no call.

This lesson measures that side, and the unit of measurement is the process launch: how many processes started in a week of records, how many of them deviate from the baseline, and how many of the deviating ones are a real anomaly and how many a legitimate change. Container run options — dropped capabilities, a read-only root, a user ID — were measured in the Containers course and are taken here as input. The question is a different one: when a mechanism is built that looks at the behavior of the running container, what happens to the three numbers.

• IP7. The environment is the same as before: nine services, thirty-one replicas, one cluster. No real container is run; what is measured is a model built with node.
• IP8. The record consists only of process launch events; file access and network connections are not modeled. The process class and its weekly count are fixed, every event in the same class carries the same binary and the same life; in reality life is a distribution.
• IP9. The true class of an event is known in the model. In production that class is known only after review; here it is assumed known so that false positives can be counted.
• IP10. The anomalous classes are written as models: which binary is unexpected in which service and how many seconds the process ran. No exploitation step or evasion method is written.
• IP11. The learning window is seven days, and it is assumed that no anomalous behavior leaks into that window. Detection latency is the sum of the pipeline stage times; wall-clock time is not measured.

The Record and the Four Baselines

// process.mjs — a week of process launch records and four baselines. IT IS A MODEL, not a real cluster.
export const REPLICAS = { 'portal-frontend': 6, 'subscriber-service': 4, 'billing': 3, 'work-order': 3,
  'notification': 2, 'report': 2, 'measurement-collector': 6, 'validator': 3, 'field-sync': 2 };
// Number of binaries in the service image (base image + application layer).
export const IMAGE_BINARIES = { 'portal-frontend': 34, 'subscriber-service': 41, 'billing': 41,
  'work-order': 38, 'notification': 29, 'report': 38, 'measurement-collector': 44, 'validator': 29,
  'field-sync': 48 };
export const SERVICES = Object.keys(REPLICAS);
export const scope = (s) => (s[1] === '*' ? SERVICES : [s[1]]);

// [name, service ('*' = all nine), binary, weekly events, true class, in the image?,
//  seen in the learning window?, process life (s), earlier closing stage]
export const CLASSES = [
  ['application entry point', '*', 'app', 620, 'legitimate', 1, 1, 0, '-'],
  ['health probe', '*', 'health', 62496, 'legitimate', 1, 1, 0.4, '-'],
  ['log rotator', '*', 'log', 5208, 'legitimate', 1, 1, 1.2, '-'],
  ['certificate renewal client', '*', 'cert', 217, 'legitimate', 1, 1, 2, '-'],
  ['nightly backup job', 'measurement-collector', 'backup', 7, 'legitimate', 1, 1, 340, '-'],
  ['scheduled report generation', 'report', 'report-gen', 168, 'legitimate', 1, 1, 26, '-'],
  ['monthly reconciliation job', 'billing', 'reconcile', 1, 'legitimate', 1, 0, 880, '-'],
  ['quarterly archive job', 'report', 'archive', 1, 'legitimate', 1, 0, 410, '-'],
  ['operator diagnostic shell', 'work-order', 'shell', 4, 'legitimate', 1, 0, 210, '-'],
  ['transition step of the new release', 'subscriber-service', 'migrate', 3, 'legitimate', 1, 0, 64, '-'],
  ['shell opened inside the pod', 'validator', 'shell', 6, 'anomalous', 1, 0, 95, 'build'],
  ['network tool not in the image', 'field-sync', 'net-tool', 3, 'anomalous', 0, 0, 12, 'build'],
  ['binary brought in from outside', 'portal-frontend', 'temp-binary', 2, 'anomalous', 0, 0, 40, 'none'],
  ['short-lived timer record', 'work-order', 'timer', 1, 'anomalous', 1, 0, 0.5, 'none'],
  ['execution inside an interpreter', 'subscriber-service', 'app', 9, 'anomalous', 1, 1, 6, 'code'],
];

// The hand-written allow list in two forms: a (service, binary) pair or the binary name alone.
export const PAIR_LIST = new Set(['billing|reconcile', 'work-order|shell']);
export const BINARY_LIST = new Set(['reconcile', 'shell']);

export const BASELINES = {
  'image inventory': (s) => !s[5],
  '7-day learning': (s) => !s[6],
  'learning + pair list': (s) => !s[6] && !PAIR_LIST.has(`${s[1]}|${s[2]}`),
  'learning + binary list': (s) => !s[6] && !BINARY_LIST.has(s[2]),
};

export function measure(deviates) {
  let caught = 0, missed = 0, falsePositive = 0, deviating = 0;
  const alerts = new Set(), real = new Set();
  for (const s of CLASSES) {
    const d = deviates(s);
    if (d) { deviating += s[3]; alerts.add(`${s[1]}|${s[2]}`); if (s[4] === 'anomalous') real.add(`${s[1]}|${s[2]}`); }
    if (s[4] === 'anomalous') d ? (caught += s[3]) : (missed += s[3]);
    else if (d) falsePositive += s[3];
  }
  return { deviating, caught, missed, falsePositive, alerts: alerts.size, real: real.size };
}

export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join('');
export const total = (f) => CLASSES.filter(f).reduce((a, s) => a + s[3], 0);

In this model the baseline is a single thing: which binary is considered legitimate in which service. The four definitions differ only in where that list comes from — what is found in the image, what is observed, and an allow list written at two different widths on top of what is observed.

The Same Week, Four Baselines

// run1.mjs — the same weekly record; the only thing that changes is where the baseline comes from
import { CLASSES, SERVICES, REPLICAS, IMAGE_BINARIES, BASELINES, measure, gen, total, scope } from './process.mjs';

const sum = (o) => Object.values(o).reduce((a, b) => a + b, 0);
const learned = new Set(CLASSES.filter((s) => s[6]).flatMap((s) => scope(s).map((v) => `${v}|${s[2]}`)));
console.log(`${SERVICES.length} services, ${sum(REPLICAS)} replicas; ${total(() => true)} process launches in a week ` +
  `(${total((s) => s[4] === 'legitimate')} legitimate / ${total((s) => s[4] === 'anomalous')} anomalous)`);
console.log(`baseline width: image inventory ${sum(IMAGE_BINARIES)} pairs, learning window ${learned.size} pairs\n`);

const hdr = gen([26, 11, 7, 16, 7, 14]);
console.log(hdr(['baseline', 'caught', 'missed', 'false positive', 'alert', 'real alert']));
for (const [name, deviates] of Object.entries(BASELINES)) {
  const r = measure(deviates);
  console.log(hdr([name, r.caught, r.missed, r.falsePositive, r.alerts, r.real]));
}

9 services, 31 replicas; 68746 process launches in a week (68725 legitimate / 21 anomalous)
baseline width: image inventory 342 pairs, learning window 38 pairs

baseline                       caught missed  false positive  alert    real alert
image inventory                     5     16               0      2             2
7-day learning                     12      9               9      8             4
learning + pair list               12      9               4      6             4
learning + binary list              6     15               4      5             3

The first row is the place to read exactly. The image inventory baseline produces no false positives at all — zero — and on a day-to-day dashboard that looks like a flawless layer. The same row has sixteen missed events, three quarters of the twenty-one anomalous ones. The cause is not a narrow definition but a wide one: the baseline is three hundred forty-two pairs, and because every binary sitting in a service image is counted as legitimate, opening a shell that is already in the image never violates the baseline. Zero false positives here is not a measure of success; it is a sign of how wide the baseline is.

The second row cuts the baseline to thirty-eight pairs, and the caught events rise from five to twelve. The price is written in the same row: nine legitimate events turn into false positives, and those nine do not come at random but from four different kinds. The monthly reconciliation and the quarterly archive job never entered the learning window; the operator diagnostic shell is something an engineer really does; the transition step is a normal part of a deployment. The alert column gives the number the operator sees: the deviating events fall into eight separate groups and four of them are real, so half of the alerts are records that will be closed after review.

The Width of the Allow List

The third and fourth rows close the same flaw at two different widths. The pair list writes service|binary, the binary list writes only the binary name; both close the same two false positives, and the false positives fall from nine to four.

The difference is in the caught column. With the pair list the caught events stay at twelve; with the binary list they fall to six. The six lost events are the shell opened in the validator service — the moment shell is written into the list, that binary becomes legitimate in all nine of nine services. The difference between the two rows is one column, and its price is half of the caught events. Writing an exclusion without counting the set it covers is the same mistake as the endpoint-based exclusion in the first lesson of this course: the list is not a setting but an inventory whose scope has to be measured.

Latency and the Events the Pipeline Carries

Detection latency is the sum of four stages: event buffer, send window, rule evaluation and notification. The send window is zero seconds in stream mode and thirty seconds in batch mode.

// run2.mjs — detection latency, the events the pipeline carries and narrowing the baseline from the image
import { CLASSES, SERVICES, REPLICAS, IMAGE_BINARIES, BASELINES, measure, gen, total, scope } from './process.mjs';

// Pipeline stages: event buffer 2 s + send window 0/30 s + evaluation 3 s + notification 12 s
const STREAM = 2 + 0 + 3 + 12, BATCH = 2 + 30 + 3 + 12;
const deviates = BASELINES['learning + pair list'];
const caughtClasses = CLASSES.filter((s) => s[4] === 'anomalous' && deviates(s));
const hdr = gen([32, 7, 8, 14, 15]);
console.log(hdr(['caught anomalous class', 'events', 'life s', `stream ${STREAM} s`, `batch ${BATCH} s`]));
for (const s of caughtClasses) console.log(hdr([s[0], s[3], s[7], s[7] > STREAM ? 'yes' : 'no', s[7] > BATCH ? 'yes' : 'no']));
const count = (b) => caughtClasses.filter((s) => s[7] > b).reduce((a, s) => a + s[3], 0);
const caughtTotal = caughtClasses.reduce((a, s) => a + s[3], 0);
console.log(`detections arriving while the process is still running: stream ${count(STREAM)}/${caughtTotal} events, batch ${count(BATCH)}/${caughtTotal} events`);

const events = total(() => true), replicas = Object.values(REPLICAS).reduce((a, b) => a + b, 0);
const health = total((s) => s[2] === 'health'), r = measure(deviates);
const rate = events / replicas / (7 * 24 * 3600 / 30), releases = 2 * SERVICES.length;
console.log(`\nevents carried by the pipeline ${events}/week; share of the health probe ${health} events ` +
  `(${(100 * health / events).toFixed(1)}%) and it deviates in no baseline`);
console.log(`deviating events in this baseline ${r.deviating} (${(100 * r.deviating / events).toFixed(3)}%); expected events per replica ` +
  `in a 30 s window ${rate.toFixed(2)}, compression factor ${Math.max(1, rate).toFixed(2)} -> latency ${BATCH} s instead of ${STREAM} s`);
console.log(`${releases} releases a week, deployment interval per service ${(7 * SERVICES.length / releases).toFixed(1)} days: if the baseline ` +
  `is tied to the image digest the 7-day learning window is cut in half, if it is tied to the service every ` +
  `new binary produces ${total((s) => s[2] === 'migrate')} false positives a week`);

const MINIMAL = new Set(CLASSES.filter((s) => s[4] === 'legitimate').flatMap((s) => scope(s).map((v) => `${v}|${s[2]}`)));
MINIMAL.delete('work-order|shell');
const m = measure((s) => scope(s).some((v) => !MINIMAL.has(`${v}|${s[2]}`)));
const imageTotal = Object.values(IMAGE_BINARIES).reduce((a, b) => a + b, 0);
console.log(`\nif the image is narrowed from ${imageTotal} binaries to ${MINIMAL.size} pairs (the diagnostic shell is removed):`);
const hdr2 = gen([26, 11, 7, 16, 7]);
console.log(hdr2(['baseline', 'caught', 'missed', 'false positive', 'alert']));
for (const [name, o] of [['learning + pair list', r], ['minimal image inventory', m]])
  console.log(hdr2([name, o.caught, o.missed, o.falsePositive, o.alerts]));
const stage = {};
for (const s of caughtClasses) stage[s[8]] = (stage[s[8]] || 0) + s[3];
console.log(`early closing stage of the ${caughtTotal} caught events: ${Object.entries(stage).map(([a, n]) => `${a} ${n}`).join(', ')}; ` +
  `the ${r.missed} missed events are a single class: ${CLASSES.find((s) => s[4] === 'anomalous' && !deviates(s))[0]}, early stage code`);

caught anomalous class           events  life s   stream 17 s     batch 47 s
shell opened inside the pod           6      95           yes            yes
network tool not in the image         3      12            no             no
binary brought in from outside        2      40           yes             no
short-lived timer record              1     0.5            no             no
detections arriving while the process is still running: stream 8/12 events, batch 6/12 events

events carried by the pipeline 68746/week; share of the health probe 62496 events (90.9%) and it deviates in no baseline
deviating events in this baseline 16 (0.023%); expected events per replica in a 30 s window 0.11, compression factor 1.00 -> latency 47 s instead of 17 s
18 releases a week, deployment interval per service 3.5 days: if the baseline is tied to the image digest the 7-day learning window is cut in half, if it is tied to the service every new binary produces 3 false positives a week

if the image is narrowed from 342 binaries to 41 pairs (the diagnostic shell is removed):
baseline                       caught missed  false positive  alert
learning + pair list               12      9               4      6
minimal image inventory            12      9               4      5
early closing stage of the 12 caught events: build 9, none 3; the 9 missed events are a single class: execution inside an interpreter, early stage code

The upper table separates what detection is good for. Of the twelve events caught in stream mode, the alert arrives while the process is still running for eight; in batch mode that number falls to six. For the rest there is no point of intervention, not because the alert did not come but because it came late: the network tool ran for twelve seconds and exited, the timer record for half a second. For these two classes detection is not a prevention but a record.

The middle block writes the price of the pipeline. Sixty-eight thousand seven hundred forty-six events are collected in a week, and ninety point nine percent of them are the health probe: a process that runs every five minutes on every replica and deviates in no baseline. The deviating events are sixteen, that is zero point zero two three percent of the volume collected. The thirty-second batch send does not shrink this volume: because the expected number of events falling into a window per replica is less than one, the compression factor stays at one and the thirty-second latency is bought for nothing. The third line says how long the baseline holds: with eighteen releases a week, if the baseline is tied to the image digest no seven-day window ever closes, and if it is tied to the service the window closes but every new binary produces a false positive. A baseline is not something set up once and left alone; it is a maintenance item tied to the speed of deployment.

Where It Would Have Closed Earlier

The two rows of the last table are the same in all three numbers: twelve caught, nine missed, four false positives. The upper row requires a learning window, an allow list and a maintenance schedule; the lower row requires only that the image be narrowed — to the forty-one pairs the application really runs instead of three hundred forty-two binaries. The same three numbers are obtained without a runtime mechanism, at the build stage and permanently, and with one alert group fewer besides.

The early-stage distribution says the same result a second time. Nine of the twelve caught events can be closed at the build stage: had the shell not been in the validator, six events, and had the network tool not been in field sync, three events would never have been born. For the remaining three events there is no early stage — the binary brought in from outside and the timer record are items only a running mechanism can see. The measure of the layer is not twelve, it is three.

The nine missed events say the opposite. Execution inside an interpreter does not launch a new process; the application’s own binary is already running. A mechanism that looks at process behavior by definition misses a behavior that spawns no process; this item belongs to the code stage.

Summary

• A week of records has 68,746 process launches; 21 are anomalous. The image inventory baseline, 342 pairs wide, catches 5 events, misses 16 and produces zero false positives — this zero is the sign of a wide definition, not a narrow one.
• The seven-day learning baseline drops to 38 pairs, raises the caught events to 12 and turns 9 legitimate events into false positives: rare jobs, the operator diagnostic and the transition step of the new release.
• When the allow list is written as a service|binary pair the false positives fall to 4 and the 12 caught events stay; when written with the binary name alone the false positives are again 4 but the caught events fall to 6.
• Detection latency is 17 seconds in stream mode and 47 seconds in batch mode; of the 12 caught events 8 and 6 respectively arrive while the process is still running. Batch send provides no compression at this event density.
• 90.9% of the events the pipeline carries are the health probe; the deviating share is 0.023%. With 18 releases a week, a baseline tied to the image digest is never completed.
• Narrowing the image to 41 pairs gives the same three numbers at the build stage. 9 of the 12 caught events can be closed at the build stage; the share that can be closed only at runtime is 3 events.

Next Step

All the measurement of this lesson was made after the container started running: the process started, the event was collected, the rule was evaluated, and seventeen or forty-seven seconds later an alert was produced. The presence of the shell in the validator and the privileges with which the thirty-one replicas run were known before the run began; all of them are fields that can be read at the moment an object is sent to the cluster. The next lesson measures that moment: admission controller rules are run over twenty-two cluster objects, and when enforce mode is switched on it counts how many deployments would be rejected and how many of those are legitimate. Whom and when a rejection touches is written in the same table: how many teams are affected and at which stage of the pipeline the same rule could have run without making anyone wait.