Network Segmentation and Service Zero Trust

The previous topic shortened the life of secrets and wrote the revocation flow after a leak. All of that measurement leaned on a single assumption: the party that reads the secret is the party that is supposed to read it. Nothing in the production environment tests that assumption. If services trust one another without question, the compromise of a single service is a door open to the others, and shortening the life of a secret does not close that door — it only limits how long the key in the hand of whoever walks through it will keep working.

This topic measures the layer beneath the application, and the first lesson counts lateral movement. From here on the unit of measurement is the service pair: how many directed pairs exist, how many correspond to a legitimate call, which verification scheme closes how many. The rule count of network segmentation, the label width and the growth rate of the matrix were measured in the Container Orchestration and Network Security courses; they are not repeated here, they are taken as input.

• IP1. The environment is the production environment of the fictional regional measurement network: nine services, one secret store, one cluster. No real cluster is run; what is measured is a model built with node.
• IP2. A call is directed: a->b and b->a are separate pairs. Endpoints, port numbers and protocols are not modeled; they would enlarge the matrix and would not change the direction of the result.
• IP3. The set of legitimate calls is seventeen and was read from the call contract. In reality this list can be extracted only with a long observation window or from the design document.
• IP4. The network policy is written at the label level and is taken as given; it has five rules and four labels.
• IP5. Compromise is a model: process control of one service falls into the attacker’s hands and every pair that service is open on becomes usable. No exploitation step, call sequence or escape method is written.
• IP6. The secret store is a node that makes no calls; four services read it legitimately, and the network policy lets all nine reach it.

The Pair Set and Four Schemes

// services.mjs — nine services, the pair set and the verification schemes. IT IS A MODEL, not a real network.
const splitRows = (s) => s.trim().replace(/\s{2,}/g, "\n").split("\n").map((l) => l.trim().split("|"));

// Service: name|zone labels (segmentation was measured in earlier courses, here it is INPUT).
export const SERVICES = Object.fromEntries(splitRows(`
  portal-frontend|edge         subscriber-service|app
  billing|app                  work-order|app,collector-client
  notification|app             report|app,collector-client
  measurement-collector|collection  validator|collection
  field-sync|collection`).map(([a, e]) => [a, e.split(",")]));

// Legitimate call pair: caller|callee. Seventeen pairs, from the call contract.
export const LEGIT = splitRows(`
  portal-frontend|subscriber-service     portal-frontend|billing
  portal-frontend|work-order             portal-frontend|report
  subscriber-service|notification        measurement-collector|validator
  validator|billing                      validator|measurement-collector
  billing|subscriber-service             billing|notification
  work-order|field-sync                  work-order|notification
  field-sync|measurement-collector       field-sync|work-order
  report|billing                         report|measurement-collector
  notification|subscriber-service`).map(([a, b]) => `${a}->${b}`);

// The network policy is written at the label level: target label <- source label.
export const NET_RULES = [["app", "edge"], ["app", "app"],
  ["app", "collection"], ["collection", "collection"], ["collection", "collector-client"]];

export const NAMES = Object.keys(SERVICES);
export const allPairs = () => NAMES.flatMap((a) => NAMES.filter((b) => b !== a).map((b) => `${a}->${b}`));

export function netOpen(s = SERVICES) {
  const open = new Set(), names = Object.keys(s);
  for (const [t, f] of NET_RULES)
    for (const target of names) if (s[target].includes(t))
      for (const source of names) if (source !== target && s[source].includes(f))
        open.add(`${source}->${target}`);
  return open;
}

// Scheme: how much the called service verifies who the caller is.
// none      : the caller identity is never verified
// one-way   : only the called party proves itself, the caller identity is still not verified
// mutual    : the caller proves itself too, but any identity from the internal authority is accepted
// authorized: mutual verification + a per-caller allow list
export function pairs(scheme, net = netOpen()) {
  if (scheme === "authorized") return new Set(LEGIT.filter((c) => net.has(c)));
  return new Set(net);
}

// The set of services reached when one service is compromised, and the hop counts (breadth-first search).
export function spread(start, edges) {
  const distance = { [start]: 0 }, queue = [start];
  while (queue.length) {
    const n = queue.shift();
    for (const c of edges) {
      const [f, t] = c.split("->");
      if (f === n && !(t in distance)) { distance[t] = distance[n] + 1; queue.push(t); }
    }
  }
  return distance;
}

export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join("");

// The secret store is a separate node and makes no calls. Four services read it legitimately;
// the network policy lets all nine reach it (the store label is open to app and collection).
export const SECRET_READERS = new Set(["subscriber-service", "billing", "measurement-collector", "field-sync"]);

export function secretHops(start, edges, authorized) {
  const u = spread(start, edges);
  const reachable = authorized ? [...SECRET_READERS] : NAMES;
  const d = reachable.filter((s) => s in u).map((s) => u[s] + 1);
  return d.length ? Math.min(...d) : Infinity;
}

The body of the pairs function is short because what is measured is short: three of the four schemes return the same set. This is not a shortcoming, it is the result of the lesson.

The Closed Pairs

// run1.mjs — four verification schemes run on the same pair set
import { NAMES, LEGIT, allPairs, netOpen, pairs, spread, secretHops, gen } from './services.mjs';

const net = netOpen();
console.log(`${NAMES.length} services -> ${allPairs().length} directed pairs; network policy allows ${net.size} pairs`);
console.log(`legitimate pairs in the call contract ${LEGIT.length}, all pass the network policy: ` +
  `${LEGIT.every((c) => net.has(c)) ? 'yes' : 'no'}`);
console.log(`pairs that pass the network policy but match no legitimate call: ${net.size - LEGIT.length}\n`);

const fmtScheme = gen([28, 11, 10, 15, 17, 15]);
console.log(fmtScheme(['scheme', 'open pairs', 'closed', 'avg reached', 'lowest reach', 'secret hops']));
for (const [name, scheme] of [['no verification', 'none'], ['one-way transport', 'one-way'],
  ['mutual, no allow list', 'mutual'], ['mutual + allow list', 'authorized']]) {
  const k = pairs(scheme, net), authorized = scheme === 'authorized';
  const reached = NAMES.map((s) => Object.keys(spread(s, k)).length - 1);
  const hops = NAMES.map((s) => secretHops(s, k, authorized));
  console.log(fmtScheme([name, k.size, net.size - k.size, (reached.reduce((a, b) => a + b, 0) / NAMES.length).toFixed(1),
    Math.min(...reached), `${Math.min(...hops)}-${Math.max(...hops)}`]));
}

const authorizedEdges = pairs('authorized', net);
console.log('\nspread per service with the allow list on:');
const fmtSpread = gen([23, 11, 9, 16]);
console.log(fmtSpread(['compromised service', 'reached', 'before', 'secret hops']));
for (const s of NAMES) {
  const after = Object.keys(spread(s, authorizedEdges)).length - 1;
  const before = Object.keys(spread(s, net)).length - 1;
  console.log(fmtSpread([s, after, before, `${secretHops(s, net, false)} -> ${secretHops(s, authorizedEdges, true)}`]));
}

9 services -> 72 directed pairs; network policy allows 52 pairs
legitimate pairs in the call contract 17, all pass the network policy: yes
pairs that pass the network policy but match no legitimate call: 35

scheme                       open pairs    closed    avg reached     lowest reach    secret hops
no verification                      52         0            7.1                7            1-1
one-way transport                    52         0            7.1                7            1-1
mutual, no allow list                52         0            7.1                7            1-1
mutual + allow list                  17        35            4.1                1            1-2

spread per service with the allow list on:
compromised service        reached   before     secret hops
portal-frontend                  8        8          1 -> 2
subscriber-service               1        7          1 -> 1
billing                          2        7          1 -> 1
work-order                       6        7          1 -> 2
notification                     1        7          1 -> 2
report                           5        7          1 -> 2
measurement-collector            4        7          1 -> 1
validator                        4        7          1 -> 2
field-sync                       6        7          1 -> 1

The first three rows carry the same numbers and this is where the table has to be read. One-way transport verification closes zero pairs. The called party proves itself, the caller proves nothing; since who is calling is not known, no pair can be rejected. The difference between the second and third rows is sharper: mutual verification on its own also closes zero pairs. Every service carries a valid workload identity, so a compromised service also presents a valid identity. If you ask the verification question as “is it a valid internal identity”, the answer is always yes.

The one row that closes is the fourth: a per-caller allow list. Thirty-five of fifty-two pairs close and the average number of reached services drops from 7.1 to 4.1. This is not a closing that can be thought free because thirty-five pairs correspond to no legitimate call; the list itself is seventeen rows, and someone has to write those seventeen rows.

The Path That Does Not Close

The lower table shows where the gain is distributed, and the distribution is not even. When the notification service is compromised, the number of reached services drops from seven to one; so does the subscriber service. These are the leaves of the call graph and the allow list really does isolate them. When the portal frontend is compromised, on the other hand, the number drops from eight to eight: it does not change at all. The edge service makes four legitimate calls, those four call the others, and the chain covers the whole cluster. The allow list closes only the edges of the call graph that do not exist; it does not stop a spread that walks over the edges that do.

The secret hops column ties the same limit to a number. With no verification the secret store is one hop from every service. With the allow list on it is still one hop for four services — they read the store legitimately — and two hops for the remaining five. So the highest value goes from one to two. Lateral movement does not close, it lengthens: the attacker first has to use an intermediate service, and the use of that intermediate service produces a log line. That is the gain, and it has to be named correctly — thirty-five pairs closed, the path to the secret store did not.

What Verification Brings

// run2.mjs — the items the allow list and certificate management bring
import { NAMES, LEGIT, SERVICES, netOpen, gen } from './services.mjs';

const REPLICAS = { 'portal-frontend': 6, 'subscriber-service': 4, 'billing': 3, 'work-order': 3, 'notification': 2,
  'report': 2, 'measurement-collector': 6, 'validator': 3, 'field-sync': 2 };
const replicaTotal = Object.values(REPLICAS).reduce((a, b) => a + b, 0);
console.log(`${NAMES.length} services, ${replicaTotal} replicas; each replica carries a workload identity`);
const fmtCert = gen([18, 13, 17, 13, 16]);
console.log('\n' + fmtCert(['certificate life', 'certificates', 'annual renewals', 'manual items', 'exposure window']));
for (const [name, days, manual] of [['90 days', 90, replicaTotal], ['24 hours', 1, 0], ['1 hour', 1 / 24, 0]])
  console.log(fmtCert([name, replicaTotal, Math.round((365 / days) * replicaTotal), manual, `${(days * 24).toFixed(0)} h`]));
console.log(`allow list ${LEGIT.length} rows; network policy 5 rules and ` +
  `${new Set(Object.values(SERVICES).flat()).size} labels; root trust anchor 1 item, rotated by hand once a year`);

console.log('\nlegitimate pairs lost if the certificate of a service cannot be renewed:');
const fmtLost = gen([23, 10, 11, 9]);
console.log(fmtLost(['service', 'caller', 'callee', 'total']));
const lost = NAMES.map((s) => ({ s, c: LEGIT.filter((x) => x.startsWith(`${s}->`)).length,
  h: LEGIT.filter((x) => x.endsWith(`->${s}`)).length })).sort((a, b) => b.c + b.h - a.c - a.h);
for (const d of lost.slice(0, 3)) console.log(fmtLost([d.s, d.c, d.h, d.c + d.h]));
const last = lost[lost.length - 1];
console.log(`average ${(lost.reduce((a, b) => a + b.c + b.h, 0) / NAMES.length).toFixed(1)} pairs, ` +
  `lowest ${last.c + last.h} pairs (${last.s})`);

const NEW_PAIRS = ['work-order->archive', 'archive->billing'];
const before = netOpen().size, after = netOpen({ ...SERVICES, archive: ['app'] }).size;
console.log('\nwhen a tenth service (archive, 2 replicas, app label) is added:');
console.log(`  network rules 5 -> 5, pairs opened by the network policy ${before} -> ${after}`);
console.log(`  allow list ${LEGIT.length} -> ${LEGIT.length + NEW_PAIRS.length} rows, certificates ${replicaTotal} -> ${replicaTotal + 2}`);
console.log(`  unneeded open pairs of the new service: ${after - before - NEW_PAIRS.length} under the network scheme, 0 under the allow list`);

9 services, 31 replicas; each replica carries a workload identity

certificate life   certificates  annual renewals manual items exposure window
90 days                      31              126           31          2160 h
24 hours                     31            11315            0            24 h
1 hour                       31           271560            0             1 h
allow list 17 rows; network policy 5 rules and 4 labels; root trust anchor 1 item, rotated by hand once a year

legitimate pairs lost if the certificate of a service cannot be renewed:
service                    caller     callee    total
billing                         2          3        5
portal-frontend                 4          0        4
subscriber-service              1          3        4
average 3.8 pairs, lowest 3 pairs (field-sync)

when a tenth service (archive, 2 replicas, app label) is added:
  network rules 5 -> 5, pairs opened by the network policy 52 -> 66
  allow list 17 -> 19 rows, certificates 31 -> 33
  unneeded open pairs of the new service: 12 under the network scheme, 0 under the allow list

A ninety-day certificate life produces one hundred twenty-six renewals a year, and if they are done by hand thirty-one items enter the calendar. Cutting the life to twenty-four hours zeroes the manual items because renewals at this count cannot be done by hand; in return annual renewals climb to eleven thousand and an automated distribution mechanism becomes mandatory. A one-hour life asks two hundred seventy thousand renewals on the same mechanism and cuts the exposure window from twenty-four hours to one hour. The three rows are three values of the same setting; which one is chosen is determined by looking at how long an identity can be abused.

The middle table counts the failure face of this mechanism. If the certificate of the billing service cannot be renewed, five legitimate pairs fall — the two calls it makes and the three it receives. The average is three point eight pairs and even the lowest value is three. With verification on, a renewal failure does not look like a security incident; it looks like an outage, and its cause is first sought in the application layer. This is exactly the price of closing for legitimate work: in return for the thirty-five closed pairs, the closing mechanism itself is a new source of failure that can drop three to five legitimate pairs.

The last block adds the tenth service. The network policy does not change at all — five rules, five rules — but the pairs it opens rise from fifty-two to sixty-six: while the need of the new service is two pairs, twelve unneeded pairs open on their own. Under the allow list scheme two rows are written and unneeded open pairs stay zero. A policy written with labels adapts to growth without effort and accumulates openness; a per-caller list asks two rows for each service and does not accumulate openness.

Where It Would Have Closed Earlier

The seventeen rows in the allow list look like a runtime configuration but their content does not belong to runtime. That list is the list of which service calls which service, and this information is born at design time. The moment the call contract is written the seventeen pairs are known; the thirty-five unneeded pairs come from the contract having been written nowhere.

The difference between the prices of closing the same item at two stages can be read in the table. Closing at design time is a document and a review. Closing at runtime is seventeen policy rows, thirty-one certificates, eleven thousand renewals a year, a root trust anchor, a rotation calendar and a source of failure that can drop three to five legitimate pairs per service. The runtime layer must do this, because the contract having been written does not mean it is enforced — but if the reason the layer was set up is the absence of the contract, the whole of the counted cost could have been closed more cheaply.

Summary

• Nine services produce 72 directed pairs; the network policy allows 52 of them, the legitimate call pairs are 17 and 35 pairs stay open without corresponding to any call.
• One-way transport verification and mutual verification without an allow list close zero pairs; a compromised service also presents a valid internal identity.
• A per-caller allow list closes 35 pairs and brings the average number of reached services from 7.1 to 4.1; on leaf services reach drops from seven to one, on the edge service from eight to eight — it does not change at all.
• The number of hops to the secret store rises from 1 to at most 2: the path does not close, it lengthens, and the use of the intermediate service produces a log line.
• The price is 31 certificates, 126 renewals a year and 31 manual items at a ninety-day life, 11,315 renewals and an automated distribution mechanism at a twenty-four-hour life; the loss of one renewal cuts 3.8 legitimate pairs on average.
• When the tenth service is added the label policy opens 12 unneeded pairs and the allow list opens 0 with 2 rows; the content of the list is the call contract, born at design time.

Next Step

All the measurement of this lesson stopped at the service boundary: who can call whom, what happens when a call is rejected. The inside of the boundary was never opened. Even if the allow list were written completely, a shell opened inside the service’s own pod, an unexpected binary or an unusual file access violates no pair — because it makes no call. The next lesson measures this side: a baseline is extracted for the process behavior of the nine services, the processes that deviate from the baseline are counted over a week of records, and how many of the deviations are real anomalies and how many legitimate changes are separated. How many seconds late detection arrives is written in the same table.