Engineering Equivalent of Controls

The previous topic measured the incident itself: the log was designed, application-specific detection scenarios were run, the records each response step needs were counted, and the external report flow was measured by time. All of that practice left one shared gap — the evidence that would show someone looking at the organization from outside that the work was done is collected by hand at the end of the period. The four lessons of this topic measure that gap.

The first step is the mapping itself. In the Governance, Risk and Compliance course, the matrix between controls and requirements was run, and the requirements left without evidence and the controls satisfying more than one requirement were counted; that measurement is the input here and is not repeated. The ends of the matrix here are different: at one end is an audit item, at the other the work an application team actually runs. The question is different too — which of the team’s work does an audit item fall on, how many of its work items does it fall on, and how many items fall on none of its work.

The Control Set and the Practice Set

AC1. The control set has 26 items and is gathered from four compliance sources: a function-based framework, a certifiable management system standard, an audit control set, and a sector compliance requirement. The set is fictional; no source, regulation, or organization is named, and the items are written only in terms of what they ask. AC2. The practice set has 18 items and consists of the work run by the subscriber portal team of the fictional regional measurement network. AC3. Every practice has a site — pipeline, repo, runtime, or manual — and a trace: a machine-readable record or a document. AC4. The mapping is a claim: a control falling on a practice says that the practice carries out the control, not that it produces evidence; evidence is separated out in the third section. AC5. The matrix sees only the work the team itself carries out; work done by the platform or by another team does not appear here, and this makes the number of controls with no equivalent look higher than it is.

// compliance/set.mjs — audit controls and team practices are a MODEL: fictional regional measurement network.
// The controls come from four compliance sources, the practices are the work of an application team.
export const split = (s) => s.trim().split("\n").map((l) => l.trim().split("|"));
export const gen = (W) => (...h) => h.map((v, j) =>
  (j ? String(v).padStart(W[j]) : String(v).padEnd(W[0]))).join("");

// code | compliance source | what the control asks
export const CONTROLS = split(`
  FF-01|function-framework|component inventory is kept current
  FF-02|function-framework|access is granted with least privilege
  FF-03|function-framework|data in transit and at rest is encrypted
  FF-04|function-framework|security events are collected centrally
  FF-05|function-framework|anomalous behavior raises an alert
  FF-06|function-framework|incident response roles are defined
  FF-07|function-framework|recovery objective is tested
  FF-08|function-framework|supplier component is assessed
  MS-01|management-system|scope statement is updated
  MS-02|management-system|applicability statement is justified
  MS-03|management-system|internal audit runs periodically
  MS-04|management-system|corrective action is closed
  MS-05|management-system|management review is recorded
  MS-06|management-system|staff competence is recorded
  CS-01|audit-set|change goes through approval
  CS-02|audit-set|production access is granted with a second pair of eyes
  CS-03|audit-set|backup restore is tested
  CS-04|audit-set|privileged access is reviewed
  CS-05|audit-set|code change gets a second reviewer
  CS-06|audit-set|closed within a defined period
  CS-07|audit-set|system clocks are synchronized
  SR-01|sector|payment data is processed in a separate segment
  SR-02|sector|personal data item is classified
  SR-03|sector|data retention period is enforced
  SR-04|sector|data subject request is traceable
  SR-05|sector|transfer to a third party is recorded
`).map(([code, source, text]) => ({ code, source, text }));

// code | site where the practice runs | trace it leaves | name of the practice
export const PRACTICES = Object.fromEntries(split(`
  P01|pipeline|machine|dependency component analysis
  P02|pipeline|machine|static analysis
  P03|pipeline|machine|build output signing
  P04|pipeline|machine|provenance generation
  P05|pipeline|machine|image scanning
  P06|repo|machine|second reviewer on merge request
  P07|repo|machine|branch protection rule
  P08|runtime|machine|short-lived credential from secret store
  P09|runtime|machine|key rotation job
  P10|runtime|machine|network segmentation rule
  P11|runtime|machine|admission controller
  P12|runtime|machine|application audit log
  P13|runtime|machine|alert rule set
  P14|manual|document|backup restore drill
  P15|manual|document|access review meeting
  P16|manual|document|incident response drill
  P17|manual|document|data classification exercise
  P18|manual|document|management review meeting
`).map(([code, site, trace, name]) => [code, { code, site, trace, name }]));

// control | practices that actually carry it out; empty cell: no engineering equivalent
export const MAPPING = Object.fromEntries(split(`
  FF-01|P01,P04
  FF-02|P08,P15
  FF-03|
  FF-04|P12
  FF-05|P13
  FF-06|P16
  FF-07|P14
  FF-08|P01,P05
  MS-01|
  MS-02|
  MS-03|
  MS-04|
  MS-05|P18
  MS-06|
  CS-01|P06,P07
  CS-02|P08
  CS-03|P14
  CS-04|P15
  CS-05|P06,P07
  CS-06|P01,P02,P05
  CS-07|
  SR-01|P10,P11
  SR-02|P17
  SR-03|
  SR-04|P12
  SR-05|
`).map(([c, p]) => [c, p ? p.split(",") : []]));

export const SOURCES = ["function-framework", "management-system", "audit-set", "sector"];

if (import.meta.url === `file://${process.argv[1]}`) {
  const practices = Object.values(PRACTICES);
  const links = Object.values(MAPPING).reduce((t, p) => t + p.length, 0);
  console.log(`model: ${CONTROLS.length} controls, ${practices.length} practices, ${links} links`);
  const y = gen([20, 10]);
  console.log("\n" + y("compliance source", "controls"));
  for (const u of SOURCES) console.log(y(u, CONTROLS.filter((c) => c.source === u).length));
  const z = gen([20, 10, 16]);
  console.log("\n" + z("practice site", "practices", "trace left"));
  for (const site of ["pipeline", "repo", "runtime", "manual"]) {
    const p = practices.filter((x) => x.site === site);
    console.log(z(site, p.length, p[0].trace));
  }
}

model: 26 controls, 18 practices, 25 links

compliance source     controls
function-framework           8
management-system            6
audit-set                    7
sector                       5

practice site        practices      trace left
pipeline                     5         machine
repo                         2         machine
runtime                      6         machine
manual                       5        document

The model consists of two lists and the 25 links between them. The control side is evenly distributed: the largest source contributes eight items, the smallest five. The distribution on the practice side is not even, and the real information is there: of the eighteen practices, thirteen run in the pipeline, in the repo, or at runtime and leave a machine-readable trace; five run as a meeting or a drill and leave only a document behind. These two kinds of trace look equal in the matrix; in an audit they are not.

Running the Matrix

An empty cell in the mapping table is the unit of measure for this lesson: if a control has no practice opposite it, that item falls on none of the team’s work. The second number is its reverse — how many work items an item falls on.

The matrix can be read in two directions, and the two readings answer two separate questions. Read from control to practice, the question is coverage: is the item being carried out, and if so by how many work items. Read from practice to control, the question is leverage: how many items does this one piece of work satisfy at once. Both readings run over the same 25 links and bring different entries to the front; that is why both are written down. There is also a third reading, and it is mostly skipped: the practice that falls on no control. That column counts the work the control set does not see.

// compliance/matrix.mjs — the control-practice matrix is run and controls with no equivalent are counted.
import { CONTROLS, PRACTICES, MAPPING, SOURCES, gen } from "./set.mjs";

const y = gen([18, 9, 15, 15, 18]);
console.log(y("compliance source", "controls", "has equivalent", "no equivalent", "practices/control"));
const rate = (k) => {
  const v = k.filter((c) => MAPPING[c.code].length > 0);
  const b = v.reduce((t, c) => t + MAPPING[c.code].length, 0);
  return [v.length, k.length - v.length, v.length ? (b / v.length).toFixed(2) : "-"];
};
for (const u of SOURCES) {
  const k = CONTROLS.filter((c) => c.source === u);
  console.log(y(u, k.length, ...rate(k)));
}
console.log(y("TOTAL", CONTROLS.length, ...rate(CONTROLS)));

const none = CONTROLS.filter((c) => MAPPING[c.code].length === 0);
console.log(`\n${none.length} controls with no engineering equivalent:`);
for (const c of none) console.log(`  ${c.code}  ${c.text}`);

const controlCount = (p) => Object.values(MAPPING).filter((l) => l.includes(p)).length;
const ranked = Object.values(PRACTICES).map((p) => ({ ...p, n: controlCount(p.code) }))
  .sort((a, b) => b.n - a.n);
const single = CONTROLS.filter((c) => MAPPING[c.code].length === 1).length;
const heavy = CONTROLS.filter((c) => MAPPING[c.code].length > 2).map((c) => c.code);
console.log(`\n${ranked.filter((p) => p.n > 1).length} practices fall on more than one control: ` +
  ranked.filter((p) => p.n > 1).map((p) => `${p.code} (${p.n})`).join(", "));
console.log(`controls resting on a single practice: ${single}; falling on more than two practices: ${heavy.join(", ")}`);
console.log(`${ranked.filter((p) => p.n === 0).length} practices fall on no control: ` +
  ranked.filter((p) => p.n === 0).map((p) => `${p.code} ${p.name}`).join(", "));

compliance source  controls has equivalent  no equivalent practices/control
function-framework        8              7              1              1.43
management-system         6              1              5              1.00
audit-set                 7              6              1              1.67
sector                    5              3              2              1.33
TOTAL                    26             17              9              1.47

9 controls with no engineering equivalent:
  FF-03  data in transit and at rest is encrypted
  MS-01  scope statement is updated
  MS-02  applicability statement is justified
  MS-03  internal audit runs periodically
  MS-04  corrective action is closed
  MS-06  staff competence is recorded
  CS-07  system clocks are synchronized
  SR-03  data retention period is enforced
  SR-05  transfer to a third party is recorded

8 practices fall on more than one control: P01 (3), P05 (2), P06 (2), P07 (2), P08 (2), P12 (2), P14 (2), P15 (2)
controls resting on a single practice: 10; falling on more than two practices: CS-06
2 practices fall on no control: P03 build output signing, P09 key rotation job

Of the twenty-six controls, seventeen have an equivalent and nine do not. The number nine is not a defect on its own; whether it is a defect is understood by looking at where they gather. Five of them are gathered in a single compliance source, the certifiable management system standard — the scope statement, the applicability statement, the internal audit, the corrective action, and staff competence. None of these items is the work of an application team, and their absence is not counted as a deficiency; their equivalent lies not in the team but in the organization’s management layer. The matrix does not say this; it only shows an empty cell. Reading an empty cell requires writing down separately whose job it is.

The remaining four are of a completely different class. FF-03, CS-07, SR-03, and SR-05 belong directly to the place where this application runs: data encryption, clock synchronization, the retention period, and transfer to a third party. All four are the team’s responsibility, and for none of the four is there a practice running. CS-07 is the sneakiest of the four: if system clocks are not synchronized, all the records the previous topic produced cannot be ordered against one another — that is, the absence of an equivalent for one control also spoils the evidence of other controls that do have one.

The number of practices per control is 1.47, and ten controls rest on a single practice. A control that rests on a single practice is left without an equivalent the moment that practice stops; the matrix shows this only when it is run again. Read in the opposite direction, eight practices satisfy more than one control — dependency component analysis alone is enough for three items. The last line is the least discussed box of the matrix: two practices fall on no control. Build output signing and key rotation run in the team, spend resources, and have no equivalent in this control set. This does not show that the practice is unnecessary; it shows that the control set does not ask for it — a control set is not a list of engineering priorities.

What Produces the Answer

AC6. The answer to a control can come from three sources: by query from a machine-readable record, by manual compilation from a document collected at the end of the period, or by an individual declaration that rests on no record. The source has a second property: a query covers the whole period, while a document covers only the day it was produced.

// compliance/trace.mjs — the trace a practice leaves is examined: what answers the control's question.
import { CONTROLS, PRACTICES, MAPPING, SOURCES, gen } from "./set.mjs";

// Three groups: if the practice leaves a machine-readable trace it is "machine-readable"; if it leaves
// only a meeting or drill document it is "document-only"; if it has no practice at all it is "no trace".
const group = (c) => {
  const p = MAPPING[c.code].map((x) => PRACTICES[x]);
  return !p.length ? "no trace" : p.some((x) => x.trace === "machine") ? "machine-readable" : "document-only";
};
const GROUPS = ["machine-readable", "document-only", "no trace"];
const y = gen([18, 18, 14, 9]);
console.log(y("compliance source", ...GROUPS));
for (const u of SOURCES) {
  const k = CONTROLS.filter((c) => c.source === u);
  console.log(y(u, ...GROUPS.map((o) => k.filter((c) => group(c) === o).length)));
}
console.log(y("TOTAL", ...GROUPS.map((o) => CONTROLS.filter((c) => group(c) === o).length)));

// Coverage of the answer: a machine trace is written throughout the period, a document belongs only to the day it was produced.
const COVERAGE = { "machine-readable": "whole period", "document-only": "single day", "no trace": "none" };
const SOURCE = { "machine-readable": "query", "document-only": "manual compilation", "no trace": "self-declaration" };
const z = gen([16, 10, 18, 20]);
console.log("\n" + z("group", "controls", "answer coverage", "answer source"));
for (const o of GROUPS) {
  const k = CONTROLS.filter((c) => group(c) === o);
  console.log(z(o, k.length, COVERAGE[o], SOURCE[o]));
}
const doc = CONTROLS.filter((c) => group(c) === "document-only");
console.log(`\ndocument-only group: ${doc.map((c) => c.code).join(", ")}`);
const machine = CONTROLS.filter((c) => group(c) === "machine-readable").length;
console.log(`Of the 26 audit questions, ${machine} are answered by querying the record, ` +
  `${doc.length} by manual compilation at period end; ` +
  `${CONTROLS.length - machine - doc.length} are not answered at all`);

compliance source   machine-readable document-only no trace
function-framework                 5             2        1
management-system                  0             1        5
audit-set                          4             2        1
sector                             2             1        2
TOTAL                             11             6        9

group             controls   answer coverage       answer source
machine-readable        11      whole period               query
document-only            6        single day  manual compilation
no trace                 9              none    self-declaration

document-only group: FF-06, FF-07, MS-05, CS-03, CS-04, SR-02
Of the 26 audit questions, 11 are answered by querying the record, 6 by manual compilation at period end; 9 are not answered at all

The fill rate of the matrix was 65%; looking at the source of the answer, the number falls to 11/26, that is 42%. The six controls in between look filled in the matrix, are met with a document in the audit, and that document describes a single day. The backup restore drill ran on one day, the access review meeting was held on one day, the data classification exercise was finished once. The control’s question, however, is about the whole period: was privileged access reviewed throughout the period. A one-day document does not fully answer this question; it turns it into a sampling.

The three groups correspond to three separate kinds of work. For the nine controls with no trace, the work to be done is setting up a practice — four in the team’s own area, five in another layer. For the six document-only controls, the work to be done is not setting up a practice; the practice is already running, and the work is extracting a machine-readable trace from it. For the eleven machine-readable controls, the work is only to compile the trace in the form the audit asks for. These three kinds of work carry very different costs, and a table that summarizes a compliance effort as “65% ready” writes all three into the same cell.

Summary

• Between twenty-six audit items and the eighteen practices of an application team there are 25 links; 17 controls have an engineering equivalent, 9 have none.
• Five of the nine controls with no equivalent are gathered in a single compliance source and their equivalent lies not in the team but in the organization’s management layer; the remaining four are directly in the team’s area. Reading an empty cell requires writing down separately whose job it is.
• A control with an equivalent falls on 1.47 practices on average and 10 controls rest on a single practice; eight practices satisfy more than one control, and the one with the most leverage is enough for three items.
• Two practices fall on no control. This shows not that the practice is unnecessary, but that the control set is not a list of engineering priorities.
• Looking at the source of the answer, the rate falls from 65% to 42%: 11 controls are answered by querying the record, 6 by a document describing a single day, and 9 are not answered at all.

Next Step

The matrix ran and three separate kinds of work were separated. Two of them stay outside this lesson: setting up practices for the controls with no trace is the job of an engineering plan, and those whose equivalent is in another layer are not even on the team’s list. What remains is what can be measured — the six document-backed controls and the eleven controls that already leave a machine trace.

The next lesson takes these seventeen items up again along the axis of evidence production and follows a single number: how far the number of items collected by hand at period end falls. Does generating evidence in the pipeline end manual compilation, which item never becomes automatic under any condition, and how many times the volume, how many personal data items, and which retention constraint does automatically produced evidence bring with it?