Admission Controller

The previous lesson measured the behavior of the running container and wrote that detection arrives seventeen or forty-seven seconds late. For nine of the twelve caught events there was an earlier stage: a binary that should not have been in the image could have been removed before the run began. The place where that decision is made is a gate, and that gate has not been measured yet.

This lesson measures the admission controller: the gate that, when an object is sent to the cluster, looks at the declared fields of the object and gives accept or reject. The mechanics of cluster objects — object types, namespaces, authorization — were measured in the Container Orchestration course and are taken here as input. The measure here has three columns: how many defects close, how many legitimate deployments are rejected, how many teams are affected. When the third is not written, the first two measure nothing, because moving to enforce mode is not a configuration change but a decision that touches the deployment flow of five teams.

• IP12. The environment is the same fictional regional measurement network: one cluster, the objects of nine services, scheduled job objects and platform workloads; five teams. The object set is a model built with node.
• IP13. The only thing modeled is which rules an object violates. The field structure of the objects, the relation between a release object and a pod, and the mechanics of namespaces are not modeled.
• IP14. Violations are divided into three classes: defect (an item the owner must fix), justified (a documented legitimate need), mismatch (the rule does not fit that object type). In production this separation is the result of a review and it is the most expensive step; in the model it is taken as given.
• IP15. An object is rejected if it violates one of the enforced rules; there is no partial acceptance.
• IP16. Weekly deployment counts are fixed per object. A decimal value represents a rarer frequency: 0.2 means a deployment every five weeks.
• IP17. The gate’s own failure — whether deployment should open or stop when the gate is down — is not modeled; that is a separate decision.

The Object Set and the Rule Set

// objects.mjs — twenty-two cluster objects, ten admission rules and the class of the violations. IT IS A MODEL.
export const RULES = ['no resource limit', 'root filesystem not read-only', 'privileged mode',
  'host network namespace', 'root user', 'mutable image tag', 'unsigned image',
  'no health probe', 'no owner label', 'secret in environment variable'];
export const R = Object.fromEntries(RULES.map((a, i) => [`R${i + 1}`, a]));

// [name, team, goes through the pipeline?, weekly deployments, {rule: violation class}]
// violation class: defect = an item the owner must fix, justified = a documented legitimate
// need, mismatch = the rule does not fit this object type (one-off job object).
export const OBJECTS = [
  ['portal-frontend', 'subscriber', 1, 2, { R6: 'defect' }],
  ['subscriber-service', 'subscriber', 1, 2, { R1: 'defect', R10: 'defect' }],
  ['notification', 'subscriber', 1, 1, { R2: 'defect' }],
  ['portal-batch-job', 'subscriber', 1, 0.5, { R1: 'defect', R8: 'mismatch' }],
  ['billing', 'billing', 1, 2, { R5: 'defect', R10: 'defect' }],
  ['report', 'billing', 1, 1, { R6: 'defect' }],
  ['monthly-reconciliation', 'billing', 1, 0.2, { R8: 'mismatch' }],
  ['quarterly-archive', 'billing', 1, 0.1, { R8: 'mismatch', R9: 'defect' }],
  ['work-order', 'field', 1, 2, { R2: 'defect' }],
  ['field-sync', 'field', 1, 2, { R1: 'defect', R2: 'defect', R7: 'defect' }],
  ['field-upload-job', 'field', 1, 0.5, { R8: 'mismatch' }],
  ['measurement-collector', 'measurement', 1, 2, {}],
  ['validator', 'measurement', 1, 2, { R5: 'defect' }],
  ['measurement-backup-job', 'measurement', 1, 0.3, { R2: 'defect', R8: 'mismatch' }],
  ['ingress-controller', 'platform', 1, 0.5, { R4: 'justified' }],
  ['log-collector', 'platform', 1, 0.5, { R2: 'defect', R3: 'justified' }],
  ['monitoring-collector', 'platform', 1, 0.5, { R3: 'justified' }],
  ['certificate-distributor', 'platform', 1, 0.3, { R10: 'defect' }],
  ['registry-mirror', 'platform', 1, 0.3, { R6: 'defect', R7: 'defect' }],
  ['backup-driver', 'platform', 0, 0.2, { R1: 'defect', R5: 'defect' }],
  ['network-plugin', 'platform', 0, 0.2, { R3: 'justified', R4: 'justified', R5: 'justified' }],
  ['node-agent', 'platform', 0, 0.1, { R3: 'justified', R5: 'justified', R9: 'defect' }],
];

export const TEAMS = [...new Set(OBJECTS.map((n) => n[1]))];
export const violations = (n, E) => Object.entries(n[4]).filter(([k]) => E.includes(k));

// In enforce mode an object is rejected if it violates one of the enforced rules.
// If a rejected object has no defect among the enforced rules, this is a LEGITIMATE rejection.
export function enforce(E) {
  const rejected = OBJECTS.filter((n) => violations(n, E).length);
  const legit = rejected.filter((n) => violations(n, E).every(([, s]) => s !== 'defect'));
  return { rejected, legit, defects: OBJECTS.reduce((a, n) => a + violations(n, E).filter(([, s]) => s === 'defect').length, 0),
    teams: new Set(rejected.map((n) => n[1])).size, legitTeams: new Set(legit.map((n) => n[1])).size,
    deployments: rejected.reduce((a, n) => a + n[3], 0), legitDeployments: legit.reduce((a, n) => a + n[3], 0) };
}

export const ALL = Object.keys(R);
export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join('');

The rule set looks only at the declared fields of an object; not at something that is running. This is also the basis of the comparison at the end of the lesson.

Three Modes and the Balance Per Rule

// run1.mjs — three modes, the balance per rule and enforcing the rules in order
import { OBJECTS, RULES, R, ALL, TEAMS, violations, enforce, gen } from './objects.mjs';

const count = (s) => OBJECTS.reduce((a, n) => a + Object.values(n[4]).filter((x) => x === s).length, 0);
const z = enforce(ALL);
console.log(`${OBJECTS.length} objects, ${TEAMS.length} teams, ${RULES.length} rules; violations ${count('defect') + count('justified') + count('mismatch')} ` +
  `(defect ${count('defect')} / justified ${count('justified')} / mismatch ${count('mismatch')})\n`);
const hm = gen([10, 15, 12, 18, 16, 17]);
console.log(hm(['mode', 'audit records', 'rejected', 'legit. rejection', 'affected teams', 'rejections/week']));
console.log(hm(['off', 0, 0, 0, 0, 0]));
console.log(hm(['audit', z.rejected.reduce((a, n) => a + violations(n, ALL).length, 0), 0, 0, 0, 0]));
console.log(hm(['enforce', 0, z.rejected.length, z.legit.length, `${z.teams}/${TEAMS.length}`, z.deployments.toFixed(1)]));

const balance = ALL.map((k) => ({ k, defects: enforce([k]).defects, legit: enforce([k]).legit.length }));
console.log('\n' + gen([6, 32, 9, 18])(['rule', 'name', 'defects', 'legit. rejection']));
for (const b of balance) console.log(gen([6, 32, 9, 18])([b.k, R[b.k], b.defects, b.legit]));

const order = [...balance].sort((a, b) => b.defects / (1 + b.legit) - a.defects / (1 + a.legit)).map((b) => b.k);
console.log(`\nrules are ordered by the defect/(1+legitimate rejection) ratio and enforced one at a time:`);
const hs = gen([8, 32, 16, 12, 18, 14]);
console.log(hs(['step', 'rule added', 'defects closed', 'rejected', 'legit. rejection', 'legit. teams']));
for (let i = 1; i <= order.length; i++) {
  const r = enforce(order.slice(0, i));
  console.log(hs([i, R[order[i - 1]], r.defects, r.rejected.length, r.legit.length, r.legitTeams]));
}

22 objects, 5 teams, 10 rules; violations 35 (defect 22 / justified 8 / mismatch 5)

mode        audit records    rejected  legit. rejection  affected teams  rejections/week
off                     0           0                 0               0                0
audit                  35           0                 0               0                0
enforce                 0          21                 5             5/5             18.2

rule                              name  defects  legit. rejection
R1                   no resource limit        4                 0
R2       root filesystem not read-only        5                 0
R3                     privileged mode        0                 4
R4              host network namespace        0                 2
R5                           root user        3                 2
R6                   mutable image tag        3                 0
R7                      unsigned image        2                 0
R8                     no health probe        0                 5
R9                      no owner label        2                 0
R10     secret in environment variable        3                 0

rules are ordered by the defect/(1+legitimate rejection) ratio and enforced one at a time:
step                          rule added  defects closed    rejected  legit. rejection  legit. teams
1          root filesystem not read-only               5           5                 0             0
2                      no resource limit               9           8                 0             0
3                      mutable image tag              12          11                 0             0
4         secret in environment variable              15          13                 0             0
5                         unsigned image              17          13                 0             0
6                         no owner label              19          15                 0             0
7                              root user              22          17                 1             1
8                        privileged mode              22          18                 2             1
9                 host network namespace              22          19                 3             1
10                       no health probe              22          21                 5             3

The upper table gives the price of moving to enforce mode all at once. Audit mode produces thirty-five records and stops no deployment; this is the stage where the gate is installed but changes nothing. When enforcement is turned on, twenty-one objects are rejected, eighteen point two deployments a week drop out, and five of the five teams are affected. Five of the rejected objects have not a single defect the gate complains about: every rule they violate is either a documented justification or a rule that does not fit that object type.

The middle table shows where these five legitimate rejections come from, one by one. Three rules — privileged mode, host network namespace and health probe — close no defect at all and produce the entire legitimate rejection. The health probe rule alone gives five legitimate rejections, because all five of its objects are one-off job objects and the rule should never have been applied to them. The root user rule stands between the two: it closes three defects and gives two legitimate rejections. The decision to enforce a rule is the ratio of these two columns, not the name of the rule.

How Far Staged Enforcement Is Free

The lower table orders the rules by that ratio and enforces them one at a time. In the first six steps legitimate rejection is zero and the defects closed rise to nineteen: eighty-six percent of the twenty-two defects close without stopping a single legitimate deployment. The seventh step adds the root user rule, closes three more defects to complete the twenty-two, and its price is a single legitimate rejection.

The last three steps close zero defects. In the eighth, ninth and tenth rows the defects closed column stays fixed at twenty-two while legitimate rejection rises from one to five and the teams with a legitimate rejection from one to three. These three rules are the source of the gate stopping the deployments of three teams within a week without justification, and the gain of all three is zero. The wrongness of moving to enforce mode all at once is tied to a number here: the same twenty-two defects could be closed by an ordered rollout with a single legitimate rejection.

The Width of the Exclusion and the Pipeline Gate

// run2.mjs — the width of the exclusion and the same rule running at the pipeline gate
import { OBJECTS, RULES, TEAMS, gen } from './objects.mjs';

// The exclusion has two granularities: per object (name|rule) or per namespace (team|rule).
const keys = (i) => OBJECTS.flatMap((n) => Object.entries(n[4])
  .filter(([, s]) => s !== 'defect').map(([k]) => `${n[i]}|${k}`));
const OBJECT_EXCL = new Set(keys(0)), TEAM_EXCL = new Set(keys(1));
const cells = (M, i) => OBJECTS.reduce((a, n) => a + RULES.filter((_, j) => M.has(`${n[i]}|R${j + 1}`)).length, 0);

const measure = (excluded) => {
  const open = (n) => Object.entries(n[4]).filter(([k]) => !excluded(n, k));
  const rejected = OBJECTS.filter((n) => open(n).length);
  return { rejected: rejected.length, legit: rejected.filter((n) => open(n).every(([, s]) => s !== 'defect')).length,
    defects: OBJECTS.reduce((a, n) => a + open(n).filter(([, s]) => s === 'defect').length, 0),
    deployments: rejected.reduce((a, n) => a + n[3], 0) };
};

const SETUPS = [['no exclusion', () => false, 0, 0],
  ['per object', (n, k) => OBJECT_EXCL.has(`${n[0]}|${k}`), OBJECT_EXCL.size, cells(OBJECT_EXCL, 0)],
  ['per namespace', (n, k) => TEAM_EXCL.has(`${n[1]}|${k}`), TEAM_EXCL.size, cells(TEAM_EXCL, 1)]];
const hx = gen([18, 8, 12, 13, 13, 18, 17]);
console.log(hx(['exclusion form', 'rows', 'covered', 'open defects', 'rejected', 'legit. rejection', 'rejections/week']));
for (const [name, f, rows, covered] of SETUPS) {
  const r = measure(f);
  console.log(hx([name, rows, covered, r.defects, r.rejected, r.legit, r.deployments.toFixed(1)]));
}

const pipeline = OBJECTS.filter((n) => n[2]), outside = OBJECTS.filter((n) => !n[2]);
const defects = (l) => l.reduce((a, n) => a + Object.values(n[4]).filter((s) => s === 'defect').length, 0);
console.log(`\nten of ten rules look at the declared fields of the object: the same rule can run at the pipeline gate`);
console.log(`  objects through the pipeline ${pipeline.length}/${OBJECTS.length}, defects they carry ${defects(pipeline)}/${defects(OBJECTS)}`);
console.log(`  objects created outside the pipeline ${outside.length} (${outside.map((n) => n[0]).join(', ')}), defects ${defects(outside)}`);

const legitObjects = OBJECTS.filter((n) => Object.values(n[4]).length && Object.values(n[4]).every((s) => s !== 'defect'));
const total = OBJECTS.reduce((a, n) => a + n[3], 0), permanent = legitObjects.reduce((a, n) => a + n[3], 0);
console.log(`\nweekly deployments ${total.toFixed(1)}; rejected in the week enforcement is turned on ${measure(() => false).deployments.toFixed(1)}`);
console.log(`  after ${defects(OBJECTS)} defects are fixed, permanent legitimate rejection ${permanent.toFixed(1)}/week, ` +
  `in ${new Set(legitObjects.map((n) => n[1])).size} of ${TEAMS.length} teams; the cause of its permanence is not a defect but rule scope`);

exclusion form        rows     covered open defects     rejected  legit. rejection  rejections/week
no exclusion             0           0           22           21                 5             18.2
per object              13          13           22           16                 0             16.3
per namespace            7          38           21           16                 0             16.3

ten of ten rules look at the declared fields of the object: the same rule can run at the pipeline gate
  objects through the pipeline 19/22, defects they carry 19/22
  objects created outside the pipeline 3 (backup-driver, network-plugin, node-agent), defects 3

weekly deployments 20.2; rejected in the week enforcement is turned on 18.2
  after 22 defects are fixed, permanent legitimate rejection 1.9/week, in 3 of 5 teams; the cause of its permanence is not a defect but rule scope

The second and third rows of the upper table both zero out legitimate rejection, and both bring the rejected objects down to sixteen. The difference is in two columns. The per-object exclusion is thirteen rows and covers exactly thirteen cells; the per-namespace exclusion is seven rows and covers thirty-eight cells — roughly half the rows, close to three times the coverage. The price is in the third column: the open defects go from twenty-two to twenty-one. The backup driver running as the root user quietly becomes legitimate under the platform exclusion written for the network plugin and the node agent.

The number of covered cells includes future objects too: every object created later in that namespace finds the exclusion ready. For this reason the exclusion list is of the same kind as the lists measured in the previous two lessons — not a setting but an inventory that, when its scope is not counted, is more decisive than the rule set.

Where It Would Have Closed Earlier

Ten of the ten rules look at the declared fields of the object; none reads something that is running. This means that all of the rules can run at the pipeline gate, before the object is sent to the cluster. The difference is not in the rule itself but in whom the rejection touches and when: at the pipeline gate the rejection reaches the person who wrote the change at merge time; at the admission gate it comes during deployment, often to someone else who happens to be on call at that moment, and it blocks a release window.

Nineteen of the twenty-two objects go through the pipeline, and nineteen of the twenty-two defects are in those objects. That is, eighty-six percent of the defects could have been closed at the owner’s desk without stopping anyone’s deployment. The remaining three defects are in the three platform objects created by hand outside the pipeline, and the pipeline gate by definition never sees them. The share that belongs to the admission gate alone is these three; for the other nineteen the gate is the second line of defense, not the first.

The last line writes a permanent item. Even after the twenty-two defects are fixed, one point nine deployments a week continue to be rejected, and this concerns three teams. The cause is not a defect but a rule scope that does not match the object type. This item can be closed with an exclusion, and when it is closed an inventory of thirteen or seven rows is born — or the rule is rewritten to read the object type and the inventory is never born.

Summary

• There are 35 violations in 22 objects: 22 defects, 8 documented justifications and 5 objects the rule does not fit. Audit mode produces 35 records and stops no deployment.
• Moving to enforce mode all at once rejects 21 objects, drops 18.2 deployments a week and affects 5 of 5 teams; 5 of the rejected objects are legitimate deployments that carry no defect.
• Three rules (privileged mode, host network namespace, health probe) produce the entire legitimate rejection without closing any defect; the health probe alone gives 5 legitimate rejections.
• When the rules are ordered by defect ratio, the first 6 steps close 19 of the 22 defects with zero legitimate rejections, and the 7th step closes the remaining 3 defects with a single legitimate rejection; the last 3 steps close zero defects and raise legitimate rejection to 5 and the affected teams to 3.
• An exclusion covers 13 cells with 13 rows per object and 38 cells with 7 rows per namespace; the second quietly exempts one real defect and also covers future objects in the namespace.
• All ten rules can run at the pipeline gate; 19 of the 22 defects are in objects that go through the pipeline. The defects that can be closed only at the admission gate number 3, and all three are in platform objects created by hand.

Next Step

The gate of this lesson saw only the object that was sent. Objects already running in the cluster never pass through the gate again; even if the rule set is tightened today, the twenty-two objects accepted yesterday stay where they are. Infrastructure resources outside the cluster — storage, network, registry, identity definitions — do not pass through any admission gate to begin with. Only a continuously running audit scans them, and that audit produces a list of findings. The next lesson measures that list, but not the number of findings: whose job each finding falls to, how many have no owner at all, and at what rate the unowned findings accumulate.