Continuous Configuration Audit

The gate of the previous lesson saw only the object that was sent, and all ten rules read the declared fields of the object. Even if the gate is tightened today, objects accepted yesterday stay where they are; storage, network and identity definitions outside the cluster pass through no admission gate at all. Only a continuously running audit scans that area.

This lesson measures that audit, but not the number of findings. Which misconfiguration closes at which gate was measured in the Cloud and Virtualization Security course and is not repeated here. The measure here is a single question: whose job does a finding fall to. How many findings return to an application team, how many stay with the platform team and how many have no owner at all. An unowned finding is different from an unmeasured finding: it is visible on the dashboard, it enters the count, it is reported, and no one closes it.

• IP18. The audit scans the whole of the fictional regional measurement network: cluster objects and the storage, network, identity and registry resources outside the cluster. The finding inventory is a model built with node.
• IP19. The content of the rule set is not measured; the numbers are read from the open finding inventory. As long as the same resource violates the same rule it is counted as a single finding.
• IP20. The ownership resolution chain has three steps and they are tried in order: the owner label of the resource, the namespace ownership table, the repository path mapping. The first matching step wins; if none matches the finding is unowned.
• IP21. At which step a finding is resolved and to which team it falls is given in the model. In production this is read from the actual state of the inventory that day and changes over time.
• IP22. Capacity is fixed per week and counts only the item set aside for these findings. The queue runs by max(0, queue + new - capacity); the difference in how hard the findings are to close is not modeled.

The Finding Inventory and the Ownership Chain

// finding.mjs — the open finding inventory of continuous audit and the ownership chain. IT IS A MODEL.
// [name, resource type, open findings, severity, step where ownership resolves, team, earlier closing stage]
export const FINDINGS = [
  ['no pod resource limit', 'cluster', 18, 'medium', 'label', 'subscriber', 'deploy'],
  ['root filesystem not read-only', 'cluster', 14, 'medium', 'label', 'field', 'deploy'],
  ['secret in environment variable', 'cluster', 6, 'high', 'label', 'billing', 'code'],
  ['mutable image tag', 'cluster', 9, 'low', 'namespace', 'subscriber', 'build'],
  ['unsigned image', 'registry', 7, 'high', 'namespace', 'field', 'build'],
  ['broadly accessible storage bucket', 'storage', 11, 'high', 'label', 'measurement', 'design'],
  ['unencrypted backup volume', 'storage', 5, 'high', 'repo', 'platform', 'design'],
  ['open management port', 'network', 9, 'high', 'none', '-', 'design'],
  ['unused identity definition', 'identity', 16, 'low', 'none', '-', 'none'],
  ['overly permissive role binding', 'identity', 8, 'high', 'namespace', 'platform', 'design'],
  ['resource with logging disabled', 'network', 13, 'low', 'repo', 'platform', 'deploy'],
  ['outdated runtime version', 'cluster', 16, 'medium', 'label', 'measurement', 'build'],
  ['old namespace left without an owner', 'cluster', 9, 'medium', 'none', '-', 'none'],
  ['hand-created network rule', 'network', 6, 'medium', 'none', '-', 'deploy'],
];

export const CHAIN = ['label', 'namespace', 'repo'];
export const STEP_NAME = { label: 'owner label', namespace: 'namespace ownership', repo: 'repo path mapping' };
// The item a team sets aside for security findings per week; the unowned finding has no capacity.
export const CAPACITY = { subscriber: 4, billing: 2, field: 3, measurement: 5, platform: 3, unowned: 0 };
export const NEW_PER_WEEK = 22; // new findings per week; tied to the deployment rate, given in the model

export const side = (b) => (b[4] === 'none' ? 'unowned' : b[5] === 'platform' ? 'platform' : 'application');
export const bucket = (b) => (b[4] === 'none' ? 'unowned' : b[5]);
export const sum = (f) => FINDINGS.filter(f).reduce((a, b) => a + b[2], 0);
export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join('');

The three steps of the chain read from three different places and all three are an inventory decision: has a label been put on the resource, is the namespace tied to a team, does the file that defines the resource sit under a repository path. None of them is a decision of the audit itself.

Whose Job the Finding Falls To

// run1.mjs — the ownership chain is run: whose job does the finding fall to
import { FINDINGS, CHAIN, STEP_NAME, side, sum, gen } from './finding.mjs';

const SEV = ['high', 'medium', 'low'];
const TOTAL = sum(() => true);
console.log(`${FINDINGS.length} finding classes; open findings ${TOTAL} ` +
  `(${SEV.map((s) => `${s} ${sum((b) => b[3] === s)}`).join(' / ')})\n`);

const hs = gen([22, 8, 12, 9]);
console.log(hs(['resolution step', 'findings', 'cumulative', 'share']));
let k = 0;
for (const a of CHAIN) {
  k += sum((b) => b[4] === a);
  console.log(hs([STEP_NAME[a], sum((b) => b[4] === a), k, `${(100 * k / TOTAL).toFixed(1)}%`]));
}
console.log(hs(['unresolved', TOTAL - k, TOTAL, `${(100 * (TOTAL - k) / TOTAL).toFixed(1)}%`]));

const ht = gen([22, 10, 9, 8, 9]);
console.log('\n' + ht(['who gets the finding', 'findings', ...SEV]));
for (const [name, t] of [['application teams', 'application'], ['platform team', 'platform'], ['unowned', 'unowned']])
  console.log(ht([name, sum((b) => side(b) === t), ...SEV.map((s) => sum((b) => side(b) === t && b[3] === s))]));
const un = FINDINGS.filter((b) => side(b) === 'unowned');
console.log(`unowned findings in ${un.length} classes: ` + un.map((b) => `${b[0]} (${b[2]})`).join(', '));

14 finding classes; open findings 147 (high 46 / medium 63 / low 38)

resolution step       findings  cumulative    share
owner label                 65          65    44.2%
namespace ownership         24          89    60.5%
repo path mapping           18         107    72.8%
unresolved                  40         147    27.2%

who gets the finding    findings     high  medium      low
application teams             81       24      48        9
platform team                 26       13       0       13
unowned                       40        9      15       16
unowned findings in 4 classes: open management port (9), unused identity definition (16), old namespace left without an owner (9), hand-created network rule (6)

The first step of the chain is the strongest: the owner label alone finds the owner of sixty-five findings, forty-four percent of the inventory. The second and third steps resolve twenty-four and eighteen findings more. The three steps together reach seventy-three percent and stop there. The remaining forty findings are items the audit sees but cannot send to anyone.

The lower table gives the real distribution. Eighty-one findings return to application teams, twenty-six stay with the platform team and forty are unowned. The severity columns separate these three buckets from each other: half of the platform team’s findings are high severity, and the majority of the application teams’ findings are medium. The unowned bucket holds nine high severity findings, and all nine are open management ports. This shows that being unowned is not limited to low severity items — the reason a finding is unowned is not that it is unimportant but that the resource has not been tied to anyone in the inventory.

If the Queue Runs for Twelve Weeks

// run2.mjs — the twelve-week course of the queue, the source of unownedness and the early stage
import { FINDINGS, CAPACITY, NEW_PER_WEEK, bucket, sum, gen } from './finding.mjs';

const TOTAL = sum(() => true), WEEKS = 12;
const hq = gen([12, 7, 15, 10, 14, 11]);
console.log(hq(['bucket', 'open', 'new per week', 'capacity', 'net per week', `week ${WEEKS}`]));
const last = {};
for (const bk of Object.keys(CAPACITY)) {
  const open = sum((b) => bucket(b) === bk), added = NEW_PER_WEEK * open / TOTAL, net = added - CAPACITY[bk];
  let q = open;
  for (let w = 0; w < WEEKS; w++) q = Math.max(0, q + net);
  last[bk] = Math.round(q);
  console.log(hq([bk, open, added.toFixed(1), CAPACITY[bk], net.toFixed(1), last[bk]]));
}
const owned = Object.keys(CAPACITY).filter((k) => k !== 'unowned');
const cap = owned.reduce((a, k) => a + CAPACITY[k], 0);
const unOpen = sum((b) => bucket(b) === 'unowned'), need = NEW_PER_WEEK * unOpen / TOTAL + unOpen / WEEKS;
console.log(`total backlog ${TOTAL} -> ${Object.values(last).reduce((a, b) => a + b, 0)}; owned buckets ` +
  `${TOTAL - unOpen} -> ${owned.reduce((a, k) => a + last[k], 0)}, unowned bucket ${unOpen} -> ${last.unowned}`);
console.log(`clearing the unowned bucket in ${WEEKS} weeks takes ${need.toFixed(1)} items a week: ` +
  `${(need / Math.max(...owned.map((k) => CAPACITY[k]))).toFixed(1)} times the capacity of the largest single team, ` +
  `${(100 * need / cap).toFixed(0)}% of the total capacity of the five teams (${cap})`);

const un = (f) => sum((b) => bucket(b) === 'unowned' && f(b));
const inside = un((b) => b[1] === 'cluster'), allUn = un(() => true), high = un((b) => b[3] === 'high');
console.log(`\nof the ${allUn} unowned findings ${inside} are in cluster objects, ${allUn - inside} in resources outside the cluster ` +
  `(${[...new Set(FINDINGS.filter((b) => bucket(b) === 'unowned' && b[1] !== 'cluster').map((b) => b[1]))].join(', ')})`);

const LEVERS = [['namespace ownership row', inside, high - un((b) => b[1] === 'cluster' && b[3] === 'high'), 'only cluster objects'],
  ['owner field in resource inventory', allUn, 0, 'all resources'],
  ['muting the low severity class', un((b) => b[3] === 'low'), high, 'finding disappears, resource stays'],
  ['owner label at the admission gate', 0, high, 'only newly created resources']];
const hl = gen([34, 10, 11, 16, 35]);
console.log('\n' + hl(['lever', 'resolved', 'remaining', 'remaining high', 'scope']));
for (const [name, c, h, scope] of LEVERS) console.log(hl([name, c, allUn - c, h, scope]));
console.log(`the admission gate does not touch the existing ${allUn} findings; it lowers the cluster share of weekly new unowned ` +
  `from ${(NEW_PER_WEEK * inside / TOTAL).toFixed(1)} to 0`);

const STAGES = ['design', 'build', 'code', 'deploy', 'none'];
const hg = gen([14, 10, 9]);
console.log('\n' + hg(['earlier stage', 'findings', 'unowned']));
for (const s of STAGES) console.log(hg([s, sum((b) => b[6] === s), un((b) => b[6] === s)]));
const early = sum((b) => b[6] !== 'none');
console.log(`${early} findings (${(100 * early / TOTAL).toFixed(0)}%) could have been closed at an earlier stage; ` +
  `for ${un((b) => b[6] === 'none')} of the ${allUn} unowned findings there is no earlier stage`);

bucket         open   new per week  capacity  net per week    week 12
subscriber       27            4.0         4           0.0         27
billing           6            0.9         2          -1.1          0
field            21            3.1         3           0.1         23
measurement      27            4.0         5          -1.0         15
platform         26            3.9         3           0.9         37
unowned          40            6.0         0           6.0        112
total backlog 147 -> 214; owned buckets 107 -> 102, unowned bucket 40 -> 112
clearing the unowned bucket in 12 weeks takes 9.3 items a week: 1.9 times the capacity of the largest single team, 55% of the total capacity of the five teams (17)

of the 40 unowned findings 9 are in cluster objects, 31 in resources outside the cluster (network, identity)

lever                               resolved  remaining  remaining high                              scope
namespace ownership row                    9         31               9               only cluster objects
owner field in resource inventory         40          0               0                      all resources
muting the low severity class             16         24               9 finding disappears, resource stays
owner label at the admission gate          0         40               9       only newly created resources
the admission gate does not touch the existing 40 findings; it lowers the cluster share of weekly new unowned from 1.3 to 0

earlier stage   findings  unowned
design                33        9
build                 32        0
code                   6        0
deploy                51        6
none                  25       25
122 findings (83%) could have been closed at an earlier stage; for 25 of the 40 unowned findings there is no earlier stage

The upper table sends six buckets in different directions. The capacity of the billing and measurement teams is above the incoming load, so their backlog shrinks: billing goes to zero in twelve weeks, measurement goes from twenty-seven to fifteen. The subscriber team stands in exact balance. The field and platform teams fall a little behind every week, and the platform’s backlog rises from twenty-six to thirty-seven. All of these are manageable numbers, and what they have in common is that each has an owner.

The sixth row is something else. In the unowned bucket capacity is zero, because capacity is a concept that belongs to a team and this bucket has no team. The forty findings receive six new findings every week, none ever closes, and in twelve weeks they rise to one hundred twelve. While the total backlog of the five buckets that have an owner falls from one hundred seven to one hundred two in the same period, the total inventory rises from one hundred forty-seven to two hundred fourteen: the whole of the increase comes from a single bucket, and that bucket responds to no fixing effort.

The last line writes the price of this. Clearing the unowned bucket in twelve weeks takes a capacity of nine point three items a week. This number is one point nine times the capacity of the largest single team and fifty-five percent of the total capacity of the five teams. That is, if the unowned findings were handed over to one team in bulk, that team’s whole capacity would not be enough; not even the capacity of the two largest teams would be enough, the share of a third team would be needed, and those teams would have to give up their own findings. The cost of being unowned is not that no one looks, but that the number to be paid if someone were to look is this large.

Who Will Give the Ownership

The middle block separates where the unownedness comes from: of the forty findings only nine are in a cluster object, and thirty-one are in network and identity resources. This separation determines the whole of the lever table. Writing a row into the namespace ownership table resolves nine findings and leaves thirty-one in place — because those resources are not in a namespace. The owner label rule added to the admission gate of the previous lesson touches none of the existing forty findings; it affects only newly created cluster objects and lowers the cluster share of weekly new unowned from one point three to zero. Muting the low severity class lowers the inventory from forty to twenty-four but never touches the remaining nine high severity findings; the resource stays where it is, it only becomes invisible.

The only row that resolves all forty of the forty is the second: an owner field in the resource inventory. That field is not a mechanism of the cluster and not an output of the audit either — it is metadata filled in when the resource is created. Continuous audit cannot produce ownership; it can only report its absence.

Where It Would Have Closed Earlier

Looked at with two columns, the last table gives the result of the lesson. One hundred twenty-two of the hundred forty-seven findings — eighty-three percent — could have been closed at an earlier stage: fifty-one with a manifest check at the deploy stage, thirty-three with an access decision at design, thirty-two with a rebuild at the build stage, six in code. Continuous audit finds these one hundred twenty-two again in every scan and reports them again in every scan.

The second column reads in the opposite direction. For twenty-five of the forty unowned findings an earlier stage does not exist, and this is not a coincidence: unused identity definitions and the old namespace left without an owner are resources that were born at some stage and forgotten. The decision that produced them belongs not to a stage but to a record that was never kept. The measure of this course — how many times cheaper it is to close an item earlier — takes another form here: ownership is not an item that can be closed at a stage but a field the resource has to carry from the first moment it exists. A finding that appears when the field is left empty becomes cheaper at no stage; it merely accumulates in a list that no one looks at.

Summary

• The ownership of 147 open findings is resolved by a three-step chain: owner label 65, namespace ownership 24, repository path mapping 18. The chain stops at 72.8% and 40 findings (27.2%) remain unowned.
• 81 of the findings fall to application teams and 26 stay with the platform team. The unowned bucket has 9 high severity findings; being unowned is not limited to low severity.
• The capacity of the unowned bucket is zero: while the 40 findings rise to 112 in twelve weeks, the total backlog of the five owned buckets falls from 107 to 102. Clearing this bucket in the same period takes 9.3 items a week, that is a share of 55% of the total capacity of the five teams.
• 31 of the 40 unowned findings are in network and identity resources outside the cluster; namespace ownership resolves only 9 of them, the admission gate touches none of the existing ones, and muting makes 16 of them invisible and leaves the resource in place. The only thing that resolves all forty is an owner field in the resource inventory.
• 122 of the 147 findings (83%) could have been closed at an earlier stage; for 25 of the 40 unowned findings there is no earlier stage, because ownership is a decision of resource creation and not of a stage.

Next Step

In the earlier stage column thirty-two findings read build: the mutable image tag, the unsigned image and the outdated runtime version. What these three classes have in common is that none of them can be fixed in place — to close them a new image has to be built and thirty-one replicas redeployed. The audit finds them again in every scan, because the distance between scan and fix is not a configuration change but a release cycle. The next lesson measures that cycle: after a fix is published, how many days does it take to reach the replicas, what is the difference in days between in-place patching and rebuilding, and how many replicas does the disruption that rebuilding brings in return touch.