Audit Control Sets

The previous two lessons measured the control itself: how institutionalized it is, and how far the audit reaches. The question asked by the party reading the certificate is different — it brings its own requirement and asks which control satisfies it. The structure that answers this question is the audit control set: a set of controls grouped by trust principle, proven to have operated over a specific period.

The phrase “over a specific period” is decisive for this lesson. Showing that a control exists as a design is done with a document; showing that it operated can only be done by sampling records produced over the period. M23/K05 counted where evidence comes from in the leased layer; that distinction is an input here. The provider’s document shows the design, not that it operated over the period — and that may not be enough to count a requirement as “satisfied.”

• FW14. The control set has 28 items and is split across five trust principles: security, availability, processing integrity, confidentiality, privacy. The set and the numbers are fictional.
• FW15. Every control has an evidence source: its own record, a provider’s document, or none.
• FW16. Every control produces a certain number of evidence items in a period; a control producing zero cannot be sampled in an audit.
• FW17. The requirement set has 25 items; each requirement maps to zero or more controls. The mapping is a claim, not evidence.
• FW18. A requirement’s status falls into four categories: operating evidence if it has a control carrying at least one piece of operating evidence; attestation only if it rests solely on a provider’s document; no evidence if the controls it maps to produce no evidence at all; unmapped if it has no control at all.
• FW19. A shared control’s evidence load is divided equally among the requirements it satisfies.
• FW20. An auditor examines at most 25 items per control and spends 0.25 person-hours compiling per item examined (fictional).

// matrix/mapping.mjs — the control-requirement matrix in an audit control set.
// MODEL: regional measurement network (fictional); controls, requirements, and evidence counts are fictional.
export const row = (s) => s.trim().split("\n").map((l) => l.trim().split("|"));
export const gen = (B) => (...h) => h.map((v, j) =>
  (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join("");

// code | trust principle | evidence source (own|provider|none) | evidence items produced in the period
export const CONTROL = Object.fromEntries(row(`
  C01|security|own|240
  C02|security|own|180
  C03|security|own|60
  C04|security|own|48
  C05|security|own|365
  C06|security|own|52
  C07|security|own|12
  C08|security|provider|1
  C09|security|provider|1
  C10|security|own|4
  C11|security|own|3
  C12|security|own|2
  C13|availability|own|365
  C14|availability|own|4
  C15|availability|own|365
  C16|availability|provider|1
  C17|availability|own|1
  C18|processing-integrity|own|365
  C19|processing-integrity|own|12
  C20|processing-integrity|own|52
  C21|processing-integrity|own|310
  C22|confidentiality|own|1
  C23|confidentiality|own|365
  C24|confidentiality|own|365
  C25|confidentiality|provider|1
  C26|privacy|none|0
  C27|privacy|own|30
  C28|privacy|none|0`)
  .map(([code, principle, source, item]) => [code, { code, principle, source, item: +item }]));

// requirement | controls claimed to satisfy it
export const REQUIREMENT = row(`
  R01 blocking of unauthorized access|C01 C03 C04
  R02 monitoring of privileged operations|C01 C02
  R03 reflection of personnel turnover|C04
  R04 continuity of endpoint protection|C05 C06
  R05 closing of a vulnerability within a reasonable time|C06
  R06 review of network boundary rules|C07
  R07 audit of physical access|C08
  R08 secure disposal of hardware|C09
  R09 awareness training of personnel|C10
  R10 evaluation of third parties|C11
  R11 incident readiness|C12
  R12 prevention of data loss|C13 C14
  R13 continuity of service without interruption|C15 C16
  R14 recovery after an outage|C14 C17
  R15 validation of input|C18
  R16 completeness of processing|C19 C20
  R17 authorization of change|C21
  R18 classification of data|C22
  R19 compliance with the retention period|C23
  R20 encryption of data|C24 C25
  R21 visibility of where data flows|C26
  R22 fulfillment of deletion requests|C26 C27
  R23 basis for processing|C28
  R24 visibility of the subprocessor chain|
  R25 immutability of records|`)
  .map(([name, k]) => ({ name, code: name.split(" ")[0], controls: k ? k.split(" ") : [] }));

export const PRINCIPLE = ["security", "availability", "processing-integrity", "confidentiality", "privacy"];
// Operating evidence: an item produced from its own record over a period. A provider's
// document shows the design, not that it operated over the period (model assumption).
export const operating = (k) => CONTROL[k].source === "own" && CONTROL[k].item > 0;
export const status = (g) => !g.controls.length ? "unmapped"
  : g.controls.some(operating) ? "operating evidence"
  : g.controls.some((k) => CONTROL[k].source === "provider") ? "attestation only" : "no evidence";

if (import.meta.url === `file://${process.argv[1]}`) {
  const links = REQUIREMENT.reduce((a, g) => a + g.controls.length, 0);
  console.log(`model: ${Object.keys(CONTROL).length} controls, ${REQUIREMENT.length} requirements, ` +
    `${links} links, ${PRINCIPLE.length} trust principles`);

  const y = gen([22, 10, 11, 15, 14, 12]);
  console.log("\n" + y("trust principle", "controls", "operating", "reqs. covered",
    "period items", "items/req."));
  for (const i of PRINCIPLE) {
    const set = Object.values(CONTROL).filter((k) => k.principle === i);
    const codes = set.map((k) => k.code);
    const reqs = REQUIREMENT.filter((g) => g.controls.some((k) => codes.includes(k)));
    const item = set.reduce((a, k) => a + k.item, 0);
    console.log(y(i, set.length, set.filter((k) => operating(k.code)).length, reqs.length,
      item, reqs.length ? (item / reqs.length).toFixed(1) : "-"));
  }

  const z = gen([20, 8, 10, 11]);
  const STATUS = ["operating evidence", "attestation only", "no evidence", "unmapped"];
  console.log("\n" + z("requirement status", "count", "percent", "controls"));
  for (const d of STATUS) {
    const k = REQUIREMENT.filter((g) => status(g) === d);
    console.log(z(d, k.length, (100 * k.length / REQUIREMENT.length).toFixed(0),
      k.reduce((a, g) => a + g.controls.length, 0)));
  }
  for (const d of STATUS.slice(1)) {
    console.log(`${d}: ${REQUIREMENT.filter((g) => status(g) === d)
      .map((g) => g.code).join(", ")}`);
  }

  // Leverage: how many requirements a control satisfies.
  const leverage = Object.values(CONTROL).map((k) =>
    ({ ...k, g: REQUIREMENT.filter((x) => x.controls.includes(k.code)).length }));
  const many = leverage.filter((k) => k.g > 1), single = leverage.filter((k) => k.g === 1);
  console.log(`\ncontrols satisfying more than one requirement ${many.length}/${leverage.length}: ` +
    `${many.map((k) => `${k.code} (${k.g})`).join(", ")}`);
  console.log(`controls carrying a single requirement ${single.length}; controls tied to no ` +
    `requirement ${leverage.filter((k) => k.g === 0).length}`);
  const total = Object.values(CONTROL).reduce((a, k) => a + k.item, 0);
  console.log(`evidence items produced in the period ${total}; per control ` +
    `${(total / leverage.length).toFixed(1)}, per satisfied requirement ` +
    `${(total / REQUIREMENT.filter((g) => status(g) === "operating evidence").length).toFixed(1)}`);
}

model: 28 controls, 25 requirements, 33 links, 5 trust principles

trust principle         controls  operating  reqs. covered  period items  items/req.
security                      12         10             11           968        88.0
availability                   5          4              3           736       245.3
processing-integrity           4          4              3           739       246.3
confidentiality                4          3              3           732       244.0
privacy                        3          1              3            30        10.0

requirement status     count   percent   controls
operating evidence        19        76         29
attestation only           2         8          2
no evidence                2         8          2
unmapped                   2         8          0
attestation only: R07, R08
no evidence: R21, R23
unmapped: R24, R25

controls satisfying more than one requirement 5/28: C01 (2), C04 (2), C06 (2), C14 (2), C26 (2)
controls carrying a single requirement 23; controls tied to no requirement 0
evidence items produced in the period 3205; per control 114.5, per satisfied requirement 168.7

The second table is the lesson’s main result. Of the twenty-five requirements, only nineteen are in a state where operating over a period can be proven; the remaining six are unevidenced for three separate reasons, and the three reasons are nothing alike. R24 and R25 are unmapped: they have rows in the matrix, but the cells are empty — nobody has proposed a control. R21 and R23 are mapped, but the controls they map to produce no evidence items at all; they look filled in the matrix and stand bare in an audit. R07 and R08, meanwhile, rest on a provider’s document: the design can be shown, that it operated over the period cannot.

This three-way split says why reading the matrix as “how many cells are filled” is misleading. Three of the thirty-three links — R21’s and R22’s link to C26, and R23’s link to C28 — produce no evidence at all. Going by fill rate, twenty-three of the twenty-five requirements are mapped (92 percent); going by operating evidence, nineteen are satisfied (76 percent). A mapping is a claim; the item that turns it into evidence must be counted separately.

The leverage row gives the second observation: of the twenty-eight controls, only five satisfy more than one requirement, and twenty-three carry a single requirement. Leveraged controls buy scale in evidence production — C01 alone is enough for two requirements. But the fifth leveraged control is C26, and it produces no evidence at all: of the two requirements it satisfies, it leaves one (R21) without evidence, while its backup control C27 saves the other (R22). Leverage is valuable only on the condition that the control operates; a leveraged control that does not operate is a multiplier for the spread of failure.

Evidence Load and Critical Control

// matrix/evidence.mjs — evidence load, critical control, and two rankings are derived from the matrix.
import { CONTROL, REQUIREMENT, PRINCIPLE, operating, status, gen } from "./mapping.mjs";

const SAMPLE_CAP = 25;   // an auditor examines at most 25 items per control (fictional)
const COMPILE = 0.25;    // 0.25 person-hours to compile per item examined (fictional)

// If a control drops, which requirements lose their operating evidence
const dropImpact = (code) => REQUIREMENT.filter((g) => status(g) === "operating evidence" &&
  g.controls.filter(operating).every((k) => k === code));
const critical = Object.keys(CONTROL).map((k) => ({ k, g: dropImpact(k) }))
  .filter((x) => x.g.length > 0);

// A shared control's evidence load is divided among the requirements it satisfies.
const reqCount = (code) => REQUIREMENT.filter((g) => g.controls.includes(code)).length || 1;
const load = (g) => g.controls.filter(operating).reduce((a, k) => a + CONTROL[k].item / reqCount(k), 0);
const TOTAL = REQUIREMENT.reduce((a, g) => a + load(g), 0);

const y = gen([48, 10, 7, 9, 10]);
console.log(`${REQUIREMENT.length} requirements, total evidence load ${TOTAL.toFixed(0)} items/period`);
console.log("\n" + y("highest evidence-load requirement", "controls", "items", "share %",
  "cumul. %"));
let cumulative = 0;
for (const g of [...REQUIREMENT].sort((a, b) => load(b) - load(a)).slice(0, 5)) {
  cumulative += 100 * load(g) / TOTAL;
  console.log(y(g.name, g.controls.length, load(g).toFixed(0),
    (100 * load(g) / TOTAL).toFixed(1), cumulative.toFixed(1)));
}
const zero = REQUIREMENT.filter((g) => load(g) === 0);
console.log(`\nrequirements producing no evidence load ${zero.length}: ` +
  `${zero.map((g) => g.code).join(", ")}`);

// Two rankings: by control count and by evidence load
const byControlCount = [...REQUIREMENT].sort((a, b) => b.controls.length - a.controls.length ||
  a.code.localeCompare(b.code)).map((g) => g.code);
const byLoad = [...REQUIREMENT].sort((a, b) => load(b) - load(a) || a.code.localeCompare(b.code))
  .map((g) => g.code);
const common = byLoad.slice(0, 5).filter((k) => byControlCount.slice(0, 5).includes(k));
console.log(`top five by control count: ${byControlCount.slice(0, 5).join(", ")}`);
console.log(`top five by evidence load  : ${byLoad.slice(0, 5).join(", ")}`);
console.log(`common: ${common.length}/5`);

// Critical control and auditor workload
console.log(`\ncontrols whose drop leaves at least one requirement without evidence ${critical.length}/` +
  `${Object.keys(CONTROL).length}; ${critical.filter((x) => reqCount(x.k) === 1).length} of these ` +
  `are already single-requirement controls`);
const singleBacked = REQUIREMENT.filter((g) => g.controls.filter(operating).length === 1);
console.log(`requirements standing on a single operating control ${singleBacked.length}/` +
  `${REQUIREMENT.length}: ${singleBacked.map((g) => g.code).join(", ")}`);

const z = gen([22, 10, 10, 10, 12]);
console.log("\n" + z("trust principle", "produced", "reviewed", "review %", "person-hrs"));
for (const i of PRINCIPLE) {
  const set = Object.values(CONTROL).filter((k) => k.principle === i);
  const produced = set.reduce((a, k) => a + k.item, 0);
  const reviewed = set.reduce((a, k) => a + Math.min(SAMPLE_CAP, k.item), 0);
  console.log(z(i, produced, reviewed, produced ? (100 * reviewed / produced).toFixed(1) : "-",
    (reviewed * COMPILE).toFixed(1)));
}
const produced = Object.values(CONTROL).reduce((a, k) => a + k.item, 0);
const reviewed = Object.values(CONTROL).reduce((a, k) => a + Math.min(SAMPLE_CAP, k.item), 0);
console.log(`total: ${produced} items produced, ${reviewed} items reviewed (` +
  `${(100 * reviewed / produced).toFixed(1)}%), ${(reviewed * COMPILE).toFixed(1)} person-hours ` +
  `to compile`);

25 requirements, total evidence load 3201 items/period

highest evidence-load requirement                 controls  items  share %  cumul. %
R04 continuity of endpoint protection                    2    391     12.2      12.2
R12 prevention of data loss                              2    367     11.5      23.7
R13 continuity of service without interruption           2    365     11.4      35.1
R15 validation of input                                  1    365     11.4      46.5
R19 compliance with the retention period                 1    365     11.4      57.9

requirements producing no evidence load 6: R07, R08, R21, R23, R24, R25
top five by control count: R01, R02, R04, R12, R13
top five by evidence load  : R04, R12, R13, R15, R19
common: 3/5

controls whose drop leaves at least one requirement without evidence 13/28; 11 of these are already single-requirement controls
requirements standing on a single operating control 13/25: R03, R05, R06, R09, R10, R11, R13, R15, R17, R18, R19, R20, R22

trust principle         produced  reviewed  review %  person-hrs
security                     968       173      17.9        43.3
availability                 736        56       7.6        14.0
processing-integrity         739        87      11.8        21.8
confidentiality              732        52       7.1        13.0
privacy                       30        25      83.3         6.3
total: 3205 items produced, 393 items reviewed (12.3%), 98.3 person-hours to compile

Evidence load is concentrated in five requirements: five of the twenty-five requirements carry 57.9 percent of the total load. What they have in common is that they depend on a control that runs daily — endpoint status, backup, capacity monitoring, input validation, and the retention-period run each produce 365 items a year. By contrast, the six requirements that produce no load at all are exactly the six unevidenced requirements from above. A requirement with zero evidence load is not cheap; it is unsatisfied. If a compliance budget is allocated by looking only at the number of items produced, these six rows never show up at all.

The top five of the two rankings overlap in only three members. Sorting by control count puts R01 and R02 first — with three and two controls, they look the most “covered.” Sorting by evidence load puts R15 and R19 first instead; both have a single control, but that single control produces 365 items in the period. Control count is a measure of coverage, evidence load is a measure of cost, and the two order the same list differently.

The critical-control count gives the third result: if any of thirteen of the twenty-eight controls drops, at least one requirement is left without evidence, and eleven of these are already single-requirement controls. So the fragility is concentrated not in leveraged controls but in unbacked ones: thirteen requirements stand on a single operating control. Losing a leveraged control usually does not drop a requirement, because another control stands beside it.

The last table gives the audit’s workload. 3,205 evidence items are produced in the period, the auditor reviews 12.3 percent of them — 393 items — and this requires 98.3 person-hours to compile. Because of the sample cap, high-volume controls are reviewed at a proportionally lower rate: 7.6 percent for availability, 83.3 percent for privacy. This shows that the real cost of producing evidence is not being audited — 2,812 of the 3,205 items produced are stored without ever being read, but they still have to be produced and stored so that they can be shown when needed.

Summary

• An audit control set proves operation over a period, not design; a provider’s document shows the design, not that it operated over the period.
• Nineteen of the twenty-five requirements carry operating evidence; the remaining six are unevidenced for three separate reasons: two are not mapped at all, two are mapped but produce no evidence, and two rest on attestation only.
• The fill rate is 92 percent, the operating-evidence rate is 76 percent: a mapping is a claim, and the item that turns it into evidence must be counted separately.
• Five of the twenty-eight controls satisfy more than one requirement; but because one of them produces no evidence at all, it leaves one of its two requirements without evidence — leverage is valuable only if the control operates.
• Fragility lies in unbacked controls: if any of thirteen controls drops, a requirement is left without evidence, and thirteen requirements stand on a single operating control.
• 57.9 percent of the evidence load is concentrated in five requirements; the six requirements producing no load at all are exactly the unevidenced ones. 3,205 items are produced in the period, and 12.3 percent are reviewed.

Next Step

This lesson’s matrix took the requirements independent of their sources: each row was a sentence, and where it came from was never asked. Part of the requirements, though, arise not from what the organization does but from what data it touches. A system that processes payment card data, a system that holds health data, and a system that stores personal data carry different control lists even if they have no other difference at all.

The next lesson ties the requirement to the data type: how many extra controls each type brings, how the lists stack up when a single system holds two types at once, and how many controls the harshest form of narrowing the scope — not holding the data at all — eliminates. The remaining controls that cannot be eliminated will be counted too.