Configuration Compliance Auditing

The previous lesson recorded who did what: the audit log is a record in the past tense, and its question is “did this operation happen.” This lesson’s question is in the present tense — does the resource’s current configuration comply with the rule? The log stays silent on whether something is wrong; an operation that opens a store to public access lands in the log cleanly, and the log does not say that operation was wrong.

Configuration compliance auditing is running a rule set against a set of resources and flagging what is noncompliant. Configuration drift was measured in M22/K01, and the reconciler loop that closes drift was measured in M22/K05; neither is repeated here. The distinction is this: there, the side that wrote the desired state and the side that closed the drift were the same side. Here the resource is rented, someone else writes the rule set, and remediation asks for write access into the rented layer — remediation itself becomes an access line item.

The Rule Set and Noncompliant Resources

The example course is the fictional regional measurement network: five teams, three environments. CS25 — resource table: twenty-two resources; each row carries a resource’s owning team, environment, and six configuration fields. CS26 — rule set: six rules; each rule is made up of a predicate, the field the fix changes, and the fixed value. CS27 — exception records: four records; three are written at the resource level, one at the team level. CS28 — actual-noncompliance label: thirty-eight resource-rule pairs are manually labeled noncompliant; two are of a type with no counterpart in the rule set. CS29 — change stream: twenty changes, six legit-tagged. CS30 — unit inputs: a manual fix costs 25 minutes per item, the auditor’s per-scan evaluation quota is 240 (fictional unit).

The file below builds the tables, the rules, and the three modes. It is a process-internal model; no real cloud resource, auditor, or remediation exists.

// compliance/audit.mjs — configuration compliance auditing in the rented layer. Process-internal
// model: the regional measurement network (fictional); no real cloud resource, auditor, or fix exists.

const parseRows = (s) => s.trim().split("\n").map((l) => l.trim().split("|"));
const row = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));

// Resource: name|team|env|public access|encryption at rest|audit logging|management port wide|owner tag|key age (days)
const RESOURCE = parseRows(`
  measurement-store|measurement|production|0|1|1|0|1|120
  measurement-archive|measurement|production|0|0|1|0|1|900
  measurement-staging|measurement|staging|1|0|0|1|0|500
  stream-queue|stream|production|0|1|1|0|1|200
  stream-store|stream|production|1|1|0|0|1|410
  stream-development|stream|development|1|0|0|1|0|730
  catalog-store|catalog|production|0|1|1|0|1|90
  catalog-gateway|catalog|production|1|1|1|0|1|150
  catalog-staging|catalog|staging|0|0|1|1|1|380
  gateway-public|gateway|production|1|1|1|0|1|60
  gateway-management|gateway|production|0|1|1|1|1|340
  gateway-development|gateway|development|1|1|0|1|0|620
  ops-log|ops|production|0|1|1|0|1|280
  ops-backup|ops|production|0|0|1|0|1|1100
  ops-tool|ops|staging|0|1|0|1|1|450
  ops-development|ops|development|1|1|0|0|0|800
  measurement-edge|measurement|production|1|1|1|0|1|170
  stream-archive|stream|production|0|0|1|0|0|960
  catalog-backup|catalog|production|0|1|0|0|1|520
  gateway-staging|gateway|staging|0|1|1|1|1|300
  ops-quota|ops|production|0|1|1|0|1|110
  measurement-report|measurement|staging|0|1|0|0|0|240`)
  .map(([name, team, env, pub, enc, log, port, tag, age]) =>
    ({ name, team, env, public: +pub, encrypted: +enc, log: +log,
       port: +port, tag: +tag, age: +age }));

// Rule set: rule id -> [predicate, field the fix changes, fixed value]
const RULE = {
  Y1: [(k) => k.public === 1, "public", 0],
  Y2: [(k) => k.encrypted === 0, "encrypted", 1],
  Y3: [(k) => k.log === 0, "log", 1],
  Y4: [(k) => k.port === 1, "port", 0],
  Y5: [(k) => k.tag === 0, "tag", 1],
  Y6: [(k) => k.age > 365, "age", 90],
};
const LABEL = { Y1: "public access closed", Y2: "encryption at rest", Y3: "audit logging",
  Y4: "management port narrow", Y5: "owner tag", Y6: "key age <= 365",
  Y7: "encryption in transit" };

// Declared exception records (model input). Three are per resource, one is at team level.
const EXCEPTION = parseRows(`
  gateway-public|Y1
  measurement-edge|Y1
  ops-backup|Y6
  team:stream|Y6`).map((x) => x.join("|"));

const exempted = (k, r) =>
  EXCEPTION.includes(`${k.name}|${r}`) || EXCEPTION.includes(`team:${k.team}|${r}`);

// Manually labeled ground-truth noncompliance (model input): resource-rule pairs where the drift is not legitimate.
// Y7 is not in the rule set; two actual noncompliance pairs are of this type.
const ACTUAL = parseRows(`
  measurement-staging|Y1 Y2 Y3 Y4 Y5 Y6
  stream-store|Y1 Y3 Y6
  stream-development|Y1 Y2 Y6
  gateway-development|Y1 Y3 Y6
  ops-development|Y1 Y3 Y5 Y6
  measurement-archive|Y2
  catalog-staging|Y2 Y4 Y6
  ops-backup|Y2 Y6
  stream-archive|Y2 Y5 Y6
  ops-tool|Y3 Y6
  catalog-backup|Y3 Y6
  measurement-report|Y3 Y5
  gateway-management|Y4
  gateway-staging|Y4
  stream-queue|Y7
  catalog-store|Y7`).flatMap(([name, r]) => r.split(" ").map((x) => `${name}|${x}`));

const RULES = Object.keys(RULE);
const ALL = [...RULES, "Y7"];

// Fired: predicate is true. Flagged: fired and no exception record.
const fire = (set) => {
  const a = [], f = [];
  for (const k of set) for (const r of RULES) if (RULE[r][0](k)) {
    a.push(`${k.name}|${r}`);
    if (!exempted(k, r)) f.push(`${k.name}|${r}`);
  }
  return { fired: a, flagged: f };
};

const baseline = fire(RESOURCE);
const caught = baseline.flagged.filter((x) => ACTUAL.includes(x));
const falseFlag = baseline.flagged.filter((x) => !ACTUAL.includes(x));
const missed = ACTUAL.filter((x) => !baseline.flagged.includes(x));
const nonCompliant = new Set(baseline.flagged.map((x) => x.split("|")[0]));

console.log(`${RESOURCE.length} resources, ${RULES.length} rules, ${EXCEPTION.length} exception records, ` +
  `${ACTUAL.length} manually labeled actual noncompliance`);
console.log(`flagged resources: ${nonCompliant.size}/${RESOURCE.length}, ` +
  `flagged pairs: ${baseline.flagged.length} (fired ${baseline.fired.length})`);

const K = [-8, -25, 11, 12, 11, 8, 15];
console.log();
row(K, "rule", "checks", "fired", "flagged", "caught", "missed", "false flag");
for (const r of ALL) {
  const at = baseline.fired.filter((x) => x.endsWith(`|${r}`)).length;
  const fl = baseline.flagged.filter((x) => x.endsWith(`|${r}`));
  const y = fl.filter((x) => ACTUAL.includes(x)).length;
  const mi = ACTUAL.filter((x) => x.endsWith(`|${r}`) && !fl.includes(x)).length;
  row(K, r, LABEL[r], at, fl.length, y, mi, fl.length - y);
}
row(K, "total", "", baseline.fired.length, baseline.flagged.length, caught.length, missed.length,
  falseFlag.length);

// Change stream: resource|field|new value|legitimate (model input)
const CHANGE = parseRows(`
  catalog-store|public|1|1
  catalog-store|log|0|0
  measurement-store|port|1|0
  measurement-store|age|900|0
  stream-queue|public|1|1
  stream-queue|encrypted|0|0
  ops-quota|port|1|1
  ops-quota|tag|0|0
  gateway-public|log|0|0
  gateway-public|age|700|0
  measurement-edge|port|1|1
  catalog-gateway|encrypted|0|0
  catalog-gateway|age|800|0
  ops-log|public|1|1
  ops-log|port|1|0
  measurement-store|tag|0|0
  stream-queue|age|900|1
  gateway-management|log|0|0
  ops-quota|age|600|0
  catalog-store|tag|0|0`)
  .map(([name, field, value, legit]) => ({ name, field, value: +value, legit: legit === "1" }));

// Three modes: report only, block, fix. Every mode sees the same stream in the same order.
const runMode = (mode) => {
  const state = RESOURCE.map((k) => ({ ...k }));
  const find = (name) => state.find((k) => k.name === name);
  let blocked = 0, reverted = 0, broken = 0, fixed = 0;

  if (mode === "fix") for (const k of state) for (const r of RULES)
    if (RULE[r][0](k) && !exempted(k, r)) {
      if (!ACTUAL.includes(`${k.name}|${r}`)) broken++;
      k[RULE[r][1]] = RULE[r][2];
      fixed++;
    }

  for (const c of CHANGE) {
    const k = find(c.name);
    const before = { ...k };
    k[c.field] = c.value;
    const now = RULES.filter((r) => RULE[r][0](k) && !RULE[r][0](before) && !exempted(k, r));
    if (now.length === 0) continue;
    if (mode === "block") { Object.assign(k, before); if (c.legit) blocked++; }
    else if (mode === "fix") {
      for (const r of now) { k[RULE[r][1]] = RULE[r][2]; fixed++; }
      if (c.legit) reverted++;
    }
  }

  // Left open at the end of the window: resources and pairs still carrying actual noncompliance
  const open = new Set(), pair = [];
  for (const k of state) {
    for (const r of RULES) if (RULE[r][0](k) && ACTUAL.includes(`${k.name}|${r}`))
      { open.add(k.name); pair.push(`${k.name}|${r}`); }
    if (ACTUAL.includes(`${k.name}|Y7`)) { open.add(k.name); pair.push(`${k.name}|Y7`); }
  }
  // The auditor's write scope: only fix mode writes fields
  const scope = mode === "fix" ? RESOURCE.length * RULES.length : 0;
  return { open: open.size, pair: pair.length, blocked, reverted, broken, fixed, scope };
};

const MODE = ["report", "block", "fix"], RESULT = {};
for (const mode of MODE) RESULT[mode] = runMode(mode);

const D = [-10, 17, 13, 11, 23, 22, 15];
console.log();
row(D, "mode", "open resources", "open pairs", "fixes", "blocked/reverted",
  "broken legit config", "write scope");
for (const mode of MODE) {
  const s = RESULT[mode];
  row(D, mode, s.open, s.pair, s.fixed, s.blocked + s.reverted, s.broken,
    `${s.scope} fields`);
}

// Manual-fix load (fictional unit): in report mode, every flagged pair is one work item
const MINUTES = 25, personHours = (n) => ((n * MINUTES) / 60).toFixed(1);
const legitCount = CHANGE.filter((c) => c.legit).length;
console.log(`\nreport-mode queue: ${baseline.flagged.length} pairs x ${MINUTES} min = ` +
  `${personHours(baseline.flagged.length)} person-hours; ${legitCount} of the ${CHANGE.length} ` +
  `changes in the stream are legit-tagged`);
console.log(`block: ${RESULT.block.blocked}/${legitCount} legit changes were blocked; ` +
  `fix: ${RESULT.fix.reverted}/${legitCount} were reverted`);
console.log(`missed ${missed.length} pairs: ${missed.filter((x) => x.endsWith("|Y7")).length} are a ` +
  `type with no counterpart in the rule set, ` +
  `${missed.filter((x) => !x.endsWith("|Y7")).length} are pairs an exception record covers`);
const teamException = ACTUAL.filter((x) => {
  const [name, r] = x.split("|");
  const k = RESOURCE.find((z) => z.name === name);
  return EXCEPTION.includes(`team:${k.team}|${r}`);
}).length;
console.log(`the single team-level exception record covers ${teamException} actual noncompliance pairs`);

// Limit and remaining (fictional): the auditor's per-scan evaluation quota
const QUOTA = 240, used = RESOURCE.length * RULES.length;
console.log(`\nevaluation quota ${QUOTA}/scan; used ${RESOURCE.length} x ` +
  `${RULES.length} = ${used}, remaining ${QUOTA - used} -> ` +
  `${Math.floor((QUOTA - used) / RULES.length)} more resources with the same rule set, or ` +
  `${Math.floor(QUOTA / RESOURCE.length) - RULES.length} more rules with the same resource set`);

22 resources, 6 rules, 4 exception records, 38 manually labeled actual noncompliance
flagged resources: 15/22, flagged pairs: 40 (fired 46)

rule    checks                         fired     flagged     caught  missed     false flag
Y1      public access closed               8           6          5       0              1
Y2      encryption at rest                 6           6          6       0              0
Y3      audit logging                      8           8          7       0              1
Y4      management port narrow             7           7          4       0              3
Y5      owner tag                          6           6          4       0              2
Y6      key age <= 365                    11           7          6       4              1
Y7      encryption in transit              0           0          0       2              0
total                                     46          40         32       6              8

mode         open resources   open pairs      fixes       blocked/reverted   broken legit config    write scope
report                   16           38          0                      0                     0       0 fields
block                    16           38          0                      5                     0       0 fields
fix                       6            6         59                      5                     8     132 fields

report-mode queue: 40 pairs x 25 min = 16.7 person-hours; 6 of the 20 changes in the stream are legit-tagged
block: 5/6 legit changes were blocked; fix: 5/6 were reverted
missed 6 pairs: 2 are a type with no counterpart in the rule set, 4 are pairs an exception record covers
the single team-level exception record covers 3 actual noncompliance pairs

evaluation quota 240/scan; used 22 x 6 = 132, remaining 108 -> 18 more resources with the same rule set, or 4 more rules with the same resource set

The numbers are of the measurement class; their inputs are the assumptions above. Six rules fire in forty-six places across twenty-two resources, the exception records filter out six of them, and forty flagged pairs remain. Looked at by resource, fifteen of the twenty-two look noncompliant — that is the number a first scan reads, and it is misleading on its own, because not every flagged pair is actually noncompliant.

Caught and Missed, Rule by Rule

The table below separates the rules and shows they do not behave alike. Y2 runs clean: it fires in six places, all six are actually noncompliant, no false flags — because its predicate is independent of a resource’s intent; there is no legitimate reading of encryption at rest being off. Y4 is written the same way but behaves differently: of seven firings, four are real and three flag a legitimate configuration; a management tool’s port being open in a development environment is that environment’s own arrangement. Y5 is the same: two of six firings belong to a temporary development resource whose owner tag is deliberately left empty.

Of the forty flagged pairs, thirty-two are actual noncompliance and eight are false flags. The six missed pairs come from two separate places, and neither is about a rule’s threshold. Four sit under exception records: Y6 fires in eleven places but only seven get flagged, because one exception record is written at the team level and covers three actual noncompliance pairs by itself. The exception’s scope is more decisive than the noncompliance itself — write access to an exception record is an audit line item, and who can write it has to be counted. The remaining two missed pairs are of a type with no counterpart in the rule set: no rule is written for encryption in transit, so no number is produced for those two resources either. Tightening the rules does not close this gap; the gap is not in the threshold, it is in the rule set’s scope.

Three Remediation Modes

The twenty-change stream runs in the same order under all three modes. Report-only mode touches nothing: thirty-eight open pairs and sixteen open resources remain, the auditor’s write access in the rented layer is zero fields, and in exchange a forty-item manual-fix queue is born — 16.7 person-hours at 25 minutes an item. Block mode rejects new changes but does not touch what already exists; the open count does not move, sixteen resources and thirty-eight pairs stand exactly as they were. The cost of blocking sits in a different column: five of the six legitimate changes in the stream were stopped.

How the one legitimate change that got through made it is worth saying. The change that extends the queue’s key age was not rejected, because that team has a team-level exception record for Y6 — meaning the record that lets legitimate work through and the record that covers three actual noncompliance pairs are the same record. Narrowing the exception would drop missed from four to one and raise blocked legitimate changes from five to six; both columns are tied to the same switch.

Fix mode drops the open count from sixteen resources to six, and from thirty-eight pairs to six. The remaining six pairs are already the missed ones: four are covered by the exception record, two are of the type with no counterpart in the rule set. The auditor writes fifty-nine fixes for this and asks for write access to six fields on each of twenty-two resources — a hundred thirty-two fields. In the rented layer this is a scope of access no human holds: the single subject that opens a resource to the network, flips encryption, and rotates the key becomes the audit tool itself.

The Cost of Wrong Remediation

Fix mode’s second column is the real measure: eight legitimate configurations broke. These are the entirety of the eight falsely flagged pairs — the catalog gateway that was supposed to stay publicly open got closed, three management ports got narrowed, two temporary resources got an owner tag written onto them, one archive key’s lifetime got shortened, one staging resource’s logging got turned on. The auditor does not know these are legitimate, because the only carrier of legitimacy information is the exception record, and it was not written for these eight pairs.

The rate deserves a name: eight of the forty fixes in the baseline scan, meaning one in every five fixes, break a legitimate configuration. On top of that, five legitimate changes coming from the stream get reverted after being applied. Mode choice is therefore not a choice about rule strictness but a choice between two columns: report-only mode stops no legitimate work and leaves sixteen resources open; fix mode closes ten resources and breaks or reverts thirteen legitimate operations. Block mode in the middle does neither — it does not let new noncompliance in, but it does not close the old noncompliance either; the sixteen existing resources still need a manual-fix queue.

Limit and Remaining

The auditor’s own limit is also counted. The per-scan evaluation quota is 240 units; twenty-two resources times six rules uses 132 units and leaves 108. Eighteen more resources could be added with the same rule set, or four more rules could be written with the same resource set; not both at once. An encryption-in-transit rule written to close the scope gap would take up twenty-two evaluations — a fifth of the remaining budget. The quota takes rule-writing out of being free: growing the rule set and growing the resource count run into the same limit.

Summary

• Six rules fire in forty-six places across twenty-two resources, and four exception records filter out six of them; of the forty flagged pairs, thirty-two are actual noncompliance and eight are false flags.
• Four of the six missed pairs sit under exception records, and a single team-level record covers three actual noncompliance pairs by itself; two are of a type with no counterpart in the rule set and do not close by tightening the threshold.
• Report-only mode leaves sixteen resources open, stops no legitimate work, and produces a forty-item queue (16.7 person-hours, fictional unit).
• Block mode does not change the open count but stops five of six legitimate changes; the one legitimate change that gets through does so because of the exception record covering three actual noncompliance pairs.
• Fix mode drops the open count from sixteen resources to six; in exchange it breaks eight legitimate configurations, reverts five legitimate changes, and asks for write access to a hundred thirty-two fields.
• 132 of the auditor’s 240-evaluation per-scan quota are used; the remaining 108 units are enough for eighteen more resources or four more rules, not both.

Next Step

Compliance auditing read the resource’s configuration, and the first rule was about public access: Y1 fired in eight places, six got flagged. But this rule reads a single flag for whether a resource is open. Once that flag is closed, how many paths really remain to that resource, which endpoint the service is reached through, and how the name resolves were never asked. The next lesson counts how many paths provide access to a managed service, and measures how many legitimate access paths break once the public endpoint is closed.