Network Policies

The previous lesson’s matrix bounded what a subject could do through the cluster interface. Traffic between two pods does not pass through that matrix: the verifier connects directly to the billing service’s endpoint and never touches the cluster interface at all. Two subjects that cannot reach each other in the privilege matrix can reach each other freely over the network. A reachability matrix was already built and measured earlier, in the context of single-machine network isolation — which pod can reach which pod. That measurement is not repeated here; this lesson’s question is building the same matrix with policy.

The delegated decision is this: the network proxy decides whether to admit a connection request by looking at who the source and destination are. It needs the following items for this: which group a pod belongs to, which flow between which groups is legitimate, and through which channel. The first item is already in the cluster — pod labels. So is the third. What is missing is the second, and nobody measures it; someone has to write it. This lesson measures the cost of writing it, and of not writing it.

• OO19. There are ten service groups and three channel types, all belonging to the fictional regional measurement network. No real cluster is run; what is measured is a model built with node.
• OO20. The set of legitimate flows is twenty-two, and four of them do not appear in the thirty-day trace. This distinction was hand-written; in reality, which flow is sparse can only be known by extending the observation period.
• OO21. Channel type is reduced to three values. In reality, port number and protocol are separate axes; they would enlarge the matrix without changing the result’s direction.
• OO22. Policy is written only by destination, meaning it bounds incoming traffic. There are no rules in this model that separately bound outgoing traffic; if there were, the rule count would grow without the closed-path count growing.
• OO23. Rule count is counted with our own small schema, defined within the lesson. A label assignment does not count as a rule; it is shown in a separate column.
• OO24. A path is either open or closed. Rate limiting, mutual authentication, and per-connection authorization are not modeled.

Policy Model

Policy is written in our own small schema: a rule specifies the target label, the label the source has to carry, and the channel. Rules only grant; under a default-closed setup, every path that is not written is closed.

// network.mjs — reachability matrix between pod groups and policy; a model, not a real cluster
export const CHANNEL = ['api', 'data', 'metric'];
export const SERVICE = {                                // service -> labels
  ingress: ['edge'], 'reading-collector': ['app'], verifier: ['app'],
  billing: ['app'], 'work-order': ['app'], 'nightly-batch-job': ['batch'],
  'data-store': ['store'], queue: ['store'], 'metric-collector': ['monitoring'], dashboard: ['monitoring'],
};
// Genuinely legitimate flows; "sparse" ones did not appear in the thirty-day trace.
export const LEGITIMATE = [
  ['ingress', 'reading-collector', 'api'], ['ingress', 'billing', 'api'],
  ['ingress', 'work-order', 'api'], ['reading-collector', 'queue', 'data'],
  ['verifier', 'queue', 'data'], ['verifier', 'data-store', 'data'],
  ['reading-collector', 'data-store', 'data'], ['billing', 'data-store', 'data'],
  ['work-order', 'data-store', 'data'], ['nightly-batch-job', 'data-store', 'data'],
  ['nightly-batch-job', 'queue', 'data'], ['metric-collector', 'reading-collector', 'metric'],
  ['metric-collector', 'verifier', 'metric'], ['metric-collector', 'billing', 'metric'],
  ['metric-collector', 'work-order', 'metric'], ['dashboard', 'metric-collector', 'api'],
  ['billing', 'work-order', 'api'], ['work-order', 'queue', 'data'],
  ['nightly-batch-job', 'billing', 'api', 'sparse'], ['metric-collector', 'nightly-batch-job', 'metric', 'sparse'],
  ['verifier', 'billing', 'api', 'sparse'], ['ingress', 'dashboard', 'api', 'sparse'],
];
// Label policy, in our own schema: "target <label> <- <source label> : <channel>"
export const LABEL_RULE = [
  ['store', 'app', 'data'], ['store', 'batch', 'data'], ['app', 'edge', 'api'],
  ['app', 'monitoring', 'metric'], ['monitoring', 'monitoring', 'api'], ['app', 'app', 'api'],
];
export const path = (s, d, c) => `${s}->${d}:${c}`;
export const allPaths = (s = SERVICE) => {
  const a = [];
  for (const so of Object.keys(s)) for (const d of Object.keys(s)) if (so !== d) for (const c of CHANNEL) a.push(path(so, d, c));
  return a;
};
export const defaultOpen = (s = SERVICE) => new Set(allPaths(s));
export const ruleFromTrace = () => LEGITIMATE.filter((m) => !m[3]).map((m) => [m[0], m[1], m[2]]);
export const openFromTrace = (rules) => new Set(rules.map(([s, d, c]) => path(s, d, c)));
export function openFromLabels(rules = LABEL_RULE, s = SERVICE) {
  const open = new Set();
  for (const [dLabel, sLabel, c] of rules)
    for (const [d, dl] of Object.entries(s)) if (dl.includes(dLabel))
      for (const [so, sl] of Object.entries(s)) if (so !== d && sl.includes(sLabel)) open.add(path(so, d, c));
  return open;
}
export const tally = (open, legit = LEGITIMATE) => {
  const m = new Set(legit.map((x) => path(x[0], x[1], x[2])));
  return { open: open.size, unnecessaryOpen: [...open].filter((y) => !m.has(y)).length,
    legitClosed: [...m].filter((y) => !open.has(y)).length, closed: allPaths().length - open.size };
};

There are two policy-writing styles, and they do not derive from the same matrix. ruleFromTrace turns the observed flow itself into a rule: each line is one source, one destination, one channel. openFromLabels, on the other hand, writes through labels; one rule opens every source–destination pair that carries the label at once.

Default-Open and Default-Closed

// run7.mjs — default-open is compared with default-closed
import { SERVICE, CHANNEL, LEGITIMATE, LABEL_RULE, allPaths, defaultOpen, ruleFromTrace, openFromTrace, openFromLabels, tally } from './network.mjs';

const trace = ruleFromTrace();
console.log(`${Object.keys(SERVICE).length} services x ${Object.keys(SERVICE).length - 1} targets x ${CHANNEL.length} channels` +
  ` = ${allPaths().length} possible paths`);
console.log(`legitimate flows ${LEGITIMATE.length}, ${LEGITIMATE.filter((m) => m[3]).length} of which do not appear in the thirty-day trace\n`);
console.log('policy                 rules  open paths  closed paths  unnecessary open  legit closed');
for (const [name, rules, open] of [
  ['default-open', 0, defaultOpen()],
  ['trace-derived', trace.length, openFromTrace(trace)],
  ['label-based', LABEL_RULE.length, openFromLabels()],
]) {
  const s = tally(open);
  console.log(`${name.padEnd(22)} ${String(rules).padEnd(6)} ${String(s.open).padEnd(11)} ` +
    `${String(s.closed).padEnd(13)} ${String(s.unnecessaryOpen).padEnd(17)} ${s.legitClosed}`);
}
console.log('\nlabel policy\'s first three rules and how many paths each opens:');
for (const k of LABEL_RULE.slice(0, 3))
  console.log(`  target ${k[0]} <- ${k[1]} : ${k[2]}   ->  ${openFromLabels([k]).size} paths`);

10 services x 9 targets x 3 channels = 270 possible paths
legitimate flows 22, 4 of which do not appear in the thirty-day trace

policy                 rules  open paths  closed paths  unnecessary open  legit closed
default-open           0      270         0             248               0
trace-derived          18     18          252           0                 4
label-based            6      36          234           17                3

label policy's first three rules and how many paths each opens:
  target store <- app : data   ->  8 paths
  target store <- batch : data   ->  2 paths
  target app <- edge : api   ->  4 paths

The default-open row produces two perfect numbers with zero rules: no legitimate path closes, and no path closes at all. The third number is the cost — two hundred forty-eight paths sit open despite corresponding to no legitimate flow. As a percentage, that is ninety-two percent of the matrix. In a cluster with no network segmentation, this is the count of places a compromised pod could reach: the secret-bearing data store, the queue, the billing flow, all of it.

The trace-derived policy finishes the job with eighteen rules, closing two hundred fifty-two paths and leaving no unnecessary open path — all eighteen of the eighteen open paths are genuinely used. But the last column shows four: four legitimate flows closed by mistake. These are the sparse flows absent from the thirty-day trace — the end-of-month reconciliation, the correction flow, the weekly report. Nothing breaks on the day the policy is written; it breaks when the end of the month arrives, and what shows up then is a network error, not a policy line. What makes this false negative expensive is that it is delayed.

The label-based policy achieves a similar closure with a third of the rules — six — and two hundred thirty-four paths. Its cost is in the middle column: seventeen unnecessary open paths. The three rows below show where that comes from. “Data channel from app-labeled sources to store-labeled targets” is a single rule that opens eight paths; seven of those are in the trace, one is not. A label rule treats everything carrying the same label the same way — wherever it cannot distinguish, it opens more than it should.

What is interesting is the last column: the label-based policy closes not four but three legitimate flows. The label’s generality accidentally caught one of the sparse flows. The same looseness both opens seventeen unnecessary paths and prevents one false negative; the two are opposite faces of a single setting.

When a New Service Is Added

Policy’s real cost shows up not on the day it is written, but on the day the cluster changes.

// run8.mjs — what each policy updates once a new service is added to the cluster
import { SERVICE, LEGITIMATE, LABEL_RULE, ruleFromTrace, openFromTrace, openFromLabels, path, tally } from './network.mjs';

const NEW_FLOWS = [['nightly-batch-job', 'archiver', 'api'], ['archiver', 'data-store', 'data'],
  ['metric-collector', 'archiver', 'metric']];
const covered = (rules, service) => {                  // how many new flows do existing rules already cover
  const open = openFromLabels(rules, service);
  return NEW_FLOWS.filter(([s, d, c]) => open.has(path(s, d, c))).length;
};
console.log('new service: archiver, three new legitimate flows\n');
console.log('policy                     label     covered  rules to  unnecessary open');
console.log('                           assigned  flows    add       before -> after');
const traceBefore = tally(openFromTrace(ruleFromTrace())).unnecessaryOpen;
const traceAfter = tally(openFromTrace([...ruleFromTrace(), ...NEW_FLOWS]), [...LEGITIMATE, ...NEW_FLOWS]).unnecessaryOpen;
console.log(`${'trace-derived'.padEnd(27)} ${'-'.padEnd(9)} ${String(0).padEnd(8)} ` +
  `${String(NEW_FLOWS.length).padEnd(9)} ${traceBefore} -> ${traceAfter}`);
for (const label of [['batch'], ['batch', 'app']]) {
  const service = { ...SERVICE, archiver: label };
  const k = covered(LABEL_RULE, service);
  const missing = NEW_FLOWS.filter(([a, b, c]) => !openFromLabels(LABEL_RULE, service).has(path(a, b, c)));
  const rules = [...LABEL_RULE, ...missing.map(([a, b, c]) => [service[b][0], service[a][0], c])];
  const before = tally(openFromLabels(LABEL_RULE)).unnecessaryOpen;
  const after = tally(openFromLabels(rules, service), [...LEGITIMATE, ...NEW_FLOWS]).unnecessaryOpen;
  console.log(`${`label: ${label.join('+')}`.padEnd(27)} ${String(label.length).padEnd(9)} ` +
    `${String(k).padEnd(8)} ${String(rules.length - LABEL_RULE.length).padEnd(9)} ${before} -> ${after}`);
}

new service: archiver, three new legitimate flows

policy                     label     covered  rules to  unnecessary open
                           assigned  flows    add       before -> after
trace-derived               -         0        3         0 -> 0
label: batch                1         1        2         17 -> 21
label: batch+app            2         2        1         17 -> 29

In the trace-derived policy, none of the new service’s three flows are covered by existing rules — covered flows are zero — and three rules get written for the three of them. In return, unnecessary open paths stay at zero. This style responds to change linearly: one line per new flow.

In the label-based policy there are two options, and the difference between them is this lesson’s result. When archiver arrives with only the batch label, existing rules cover one of the three flows, two rules need to be written, and unnecessary open paths go from seventeen to twenty-one. When archiver is given a second label, app, covered flows rise to two, the rules to write drop to one — but unnecessary open paths jump to twenty-nine.

The eight-path gap between them is the second label’s price. The app label puts archiver within the scope of four rules at once; most of the paths those rules open are unnecessary for archiver. So the way to avoid writing a rule runs through widening the label, and widening the label opens up the matrix. The choice between the two-rule option and the one-rule option is a choice between the labor of writing one line and eight open paths.

The silent side is here too. The eight extra open paths produce no error, land in no log, and may never be used at all; their existence shows up only when the matrix is recounted. A rule left out, by contrast — say one of the trace-derived style’s three lines were forgotten — produces an immediate connection error and gets fixed. The two policy styles therefore fail in different directions: one is noisy and correctable, the other silent and cumulative.

As the Cluster Grows

Adding a single service produced a small difference between the two styles. The difference reveals its direction once the addition is repeated.

// run9.mjs — how each policy's rule count grows as the cluster grows
import { SERVICE, LEGITIMATE, LABEL_RULE, ruleFromTrace, openFromTrace, openFromLabels, path, tally } from './network.mjs';

const flows = (name) => [['nightly-batch-job', name, 'api'], [name, 'data-store', 'data'],
  ['metric-collector', name, 'metric']];
let service = { ...SERVICE }, rules = [...LABEL_RULE], legit = [...LEGITIMATE], traceRules = ruleFromTrace();
console.log('new    trace-derived            label-based (new service label: batch+app)');
console.log('svc    rules  unnecessary open  rules  unnecessary open  rules added this step');
for (let n = 1; n <= 6; n += 1) {
  const name = `archive-${n}`;
  service = { ...service, [name]: ['batch', 'app'] };
  const newFlows = flows(name);
  legit = [...legit, ...newFlows]; traceRules = [...traceRules, ...newFlows];
  const missing = newFlows.filter(([a, b, c]) => !openFromLabels(rules, service).has(path(a, b, c)));
  const added = missing.map(([a, b, c]) => [service[b][0], service[a][0], c]);
  rules = [...rules, ...added];
  const t = tally(openFromTrace(traceRules), legit), l = tally(openFromLabels(rules, service), legit);
  console.log(`${String(n).padEnd(6)} ${String(traceRules.length).padEnd(6)} ${String(t.unnecessaryOpen).padEnd(18)} ` +
    `${String(rules.length).padEnd(6)} ${String(l.unnecessaryOpen).padEnd(18)} ${added.length}`);
}

new    trace-derived            label-based (new service label: batch+app)
svc    rules  unnecessary open  rules  unnecessary open  rules added this step
1      21     0                  7      29                 1
2      24     0                  7      43                 0
3      27     0                  7      59                 0
4      30     0                  7      77                 0
5      33     0                  7      97                 0
6      36     0                  7      119                0

Two columns trace two separate curves. In the trace-derived policy, the rule count grows by three with every service — from twenty-one to thirty-six — and unnecessary open paths stay at zero across all six services. Maintenance cost is linear, openness constant.

The label-based policy does the opposite. The rule count grows by one at the first service and stops there: seven rules, and seven for the following five services too. The last column reads zero five times — because the new service carries the existing labels, existing rules cover it automatically. Maintenance cost drops to zero. The counterpart to that accumulates in the third number column: unnecessary open paths climb from twenty-nine to a hundred nineteen, and the climb accelerates — fourteen, sixteen, eighteen, twenty, twenty-two, in turn. Every new app-labeled service opens a mutual path with every app-labeled service that came before it; the growth is therefore not linear in the number of services but proportional to the number of pairs.

The delegated decision’s cost is what this table reads out. To delegate the “is this connection legitimate” decision to the network proxy, someone has to report the set of legitimate flows to it. Reporting one by one costs labor at every change, but what is reported is correct. Reporting through labels drops the labor to zero and makes what is reported roughly correct; the size of that roughness grows along with the cluster, and stays invisible as long as it is measured nowhere.

Summary

• Ten services, three channels, and nine targets build a 270-path matrix; legitimate flow is 22, meaning 248 paths sit open with no corresponding flow under default-open.
• The trace-derived policy closes 252 paths with 18 rules and leaves no unnecessary openness, but closes 4 sparse flows absent from the trace by mistake; that false negative shows up only the first time the flow is attempted.
• The label-based policy closes 234 paths with 6 rules; its cost is 17 unnecessary open paths — a single rule can open eight paths at once — and the same looseness catches one of the sparse flows by accident, cutting the false-negative count from four to three.
• When a new service is added, the trace-derived policy needs three rules and keeps unnecessary open paths at zero; in the label-based policy, the narrow label produces two rules and 21 unnecessary paths, the wide label one rule and 29 unnecessary paths.
• Once six services are added, the trace-derived policy grows to 36 rules and keeps unnecessary open paths at zero; the label-based policy stays at 7 rules, but unnecessary open paths climb from 29 to 119, because the growth is proportional to the number of pairs, not the number of services.
• A missing rule fails noisily and gets fixed; an extra open path fails silently and accumulates. Choosing a policy style is choosing between these two failure modes.

Next Step

This lesson’s policy bounds traffic between pods; it does not bound what a pod can do inside itself. A process that runs privileged, reaches into the host machine’s filesystem, or is opened with a wide capability set opens up another domain on the node it sits on, even with a fully written network policy. What these settings are and what they cost on a single machine was measured earlier. The next lesson’s question is not the settings themselves but how the cluster enforces them: an admission-control rule is written and applied to the existing pod definitions; how many definitions violate policy, how many deployments the enforcement stops, the difference between warn mode and block mode, and how many definitions stay exposed if an exemption is granted are all counted.