Evidence Automation

The previous lesson matched twenty-six audit items with the team’s eighteen practices and separated three kinds of work. Two of them could not be measured: nine controls have no practice opposite them, and the equivalent of five of those lies in another layer. What remained measurable was seventeen controls — eleven leave a machine-readable trace, six leave only a document.

This lesson follows the evidence of those seventeen controls. The question is not whether the practice exists, but where the evidence is generated. A practice running and the fact that it is running being provable are two separate jobs, and the second is mostly done at the end of the period, by hand. The unit of measure comes from here too: the number of items touched by hand and the annual compilation load of those items.

The Evidence Item Set

AC7. The evidence item set has 24 items and is tied to the 17 controls with an equivalent from the previous lesson; the set belongs to the fictional regional measurement network and is a model. AC8. Every item has a kind: a trace is a record that arises on its own in a system, a decision is a person’s justification, and a meeting is the minutes of a drill or a meeting. AC9. Manually compiled evidence is produced on the day of compilation, and the interval it covers is not written on the record; evidence generated in the pipeline carries a period stamp — this distinction is an assumption of the model. AC10. The minutes are given per period as data, not measured; compilation is done three times a year, and this too is fiction.

// evidence/item.mjs — audit evidence items are a MODEL: fictional regional measurement network, one period.
// The items tie to the 17 controls that had an equivalent in the previous lesson; the numbers are fiction.
export const gen = (W) => (...h) => h.map((v, j) =>
  (j ? String(v).padStart(W[j]) : String(v).padEnd(W[0]))).join("");

// code|control|kind|site|manual minutes|records produced in the period|personal data
export const ITEMS = `
  K01|FF-01|trace|pipeline|120|312|no|dependency inventory
  K02|FF-01|trace|pipeline|90|312|no|provenance information
  K03|FF-02|trace|runtime|150|8640|yes|short-lived credential issuance
  K04|FF-02|decision|runtime|240|24|yes|access review decision
  K05|FF-04|trace|runtime|180|54000|yes|application audit log
  K06|FF-05|trace|runtime|100|960|no|alert rule firing
  K07|FF-06|meeting|manual|300|2|no|incident response drill report
  K08|FF-06|decision|manual|120|6|no|role assignment decision
  K09|FF-07|meeting|manual|360|2|no|recovery objective measurement report
  K10|FF-08|trace|pipeline|110|312|no|image scan report
  K11|MS-05|meeting|manual|180|4|no|management review minutes
  K12|CS-01|trace|repo|130|486|no|change approval record
  K13|CS-01|decision|repo|90|18|no|emergency change justification
  K14|CS-02|trace|runtime|140|420|yes|production access issuance
  K15|CS-03|meeting|manual|300|2|no|restore drill report
  K16|CS-04|decision|runtime|280|24|yes|privileged access review
  K17|CS-05|trace|repo|120|486|no|code review approval
  K18|CS-05|trace|repo|80|486|no|branch protection rule status
  K19|CS-06|trace|pipeline|160|312|no|open scan result
  K20|CS-06|trace|pipeline|140|312|no|static analysis result
  K21|SR-01|trace|runtime|150|1440|no|segment rule status
  K22|SR-01|trace|runtime|130|1440|no|admission controller rejection record
  K23|SR-02|decision|manual|420|1|yes|data classification table
  K24|SR-04|trace|runtime|170|2400|yes|data subject request trace
`.trim().split("\n").map((l) => {
  const [code, control, kind, site, min, records, personal, name] = l.trim().split("|");
  return { code, control, kind, site, min: +min, records: +records, personal, name };
});

export const KINDS = ["trace", "decision", "meeting"];
export const COMPILATIONS = 3;            // three evidence compilations a year (fiction)
export const SETUP = { pipeline: 6, repo: 8, runtime: 12, manual: 0 };   // person-hours per item

if (import.meta.url === `file://${process.argv[1]}`) {
  const y = gen([12, 8, 10, 12, 16, 16]);
  console.log(y("item kind", "items", "in system", "manual", "period minutes", "period records"));
  for (const t of KINDS) {
    const k = ITEMS.filter((x) => x.kind === t);
    console.log(y(t, k.length, k.filter((x) => x.site !== "manual").length,
      k.filter((x) => x.site === "manual").length,
      k.reduce((a, x) => a + x.min, 0), k.reduce((a, x) => a + x.records, 0)));
  }
  const min = ITEMS.reduce((a, x) => a + x.min, 0);
  console.log(y("TOTAL", ITEMS.length, ITEMS.filter((x) => x.site !== "manual").length,
    ITEMS.filter((x) => x.site === "manual").length, min,
    ITEMS.reduce((a, x) => a + x.records, 0)));
  console.log(`\ninitial state: ${ITEMS.length} of ${ITEMS.length} items are compiled by hand`);
  console.log(`per compilation ${(min / 60).toFixed(1)} hours, ${COMPILATIONS} compilations a year -> ` +
    `${(COMPILATIONS * min / 60).toFixed(1)} hours`);
  console.log(`items carrying a personal data type ` +
    `${ITEMS.filter((x) => x.personal === "yes").length}/${ITEMS.length}`);
}

item kind      items in system      manual  period minutes  period records
trace             15        15           0            1970           72318
decision           5         3           2            1150              73
meeting            4         0           4            1140              10
TOTAL             24        18           6            4260           72401

initial state: 24 of 24 items are compiled by hand
per compilation 71.0 hours, 3 compilations a year -> 213.0 hours
items carrying a personal data type 7/24

The second and third columns of the table set up the real tension of this lesson. Of the twenty-four items, eighteen already arise in a system — in the pipeline, in the repo, or at runtime. Even so, in the initial state the evidence of all twenty-four is compiled by hand. The record exists in the system, and at the end of the period a person exports it, puts it in a directory, and enters it into a table. The difference is not an engineering gap but a connection gap: a record existing does not mean it stands in the form the audit asks for.

That there are more items than controls is also information: seventeen controls ask for twenty-four items. Some controls are not met with a single item — showing that a code change passed a second pair of eyes requires both the review approval and the status of the branch protection rule, because the first says the event took place and the second says its not taking place was prevented. The number of evidence items is determined not by the number of controls but by how many parts the control’s question consists of.

The third row also gives the limit of this. None of the four meeting items arises in a system, and this comes not from a lack of tooling but from the nature of the item: the result of a drill is written by a person. The minutes column, meanwhile, says the distribution is unbalanced — the four meeting items take 1,140 minutes, more than a quarter of the total load, yet exactly one sixth of the item count.

Moving Generation into the Pipeline

AC11. For a decision-kind item, the record becomes automatic — that the decision was made, the person who made it, and the timestamp are read from a system; the justification text stays manual. The remaining manual work was counted as 0.4 times the item’s minutes, and this ratio is an assumption of the model. AC12. The setup cost is given in person-hours per item (pipeline 6, repo 8, runtime 12) and is data; the maintenance cost was not modeled, which makes the payback period look shorter than it is.

// evidence/automation.mjs — evidence generation is moved into the pipeline: how far the items collected by hand fall.
import { ITEMS, COMPILATIONS, SETUP, gen } from "./item.mjs";

// A trace item arises in a system on its own: it is generated in the pipeline. The record of a decision
// becomes automatic, its justification stays manual (0.4 times the item's minutes). A meeting does not become automatic.
const status = (k) => k.site === "manual" ? "stays manual"
  : k.kind === "trace" ? "generated in pipeline"
  : k.kind === "decision" ? "semi-automatic" : "stays manual";
const remaining = (k) => status(k) === "generated in pipeline" ? 0
  : status(k) === "semi-automatic" ? Math.round(0.4 * k.min) : k.min;

const STATUSES = ["generated in pipeline", "semi-automatic", "stays manual"];
const y = gen([24, 8, 22, 16]);
console.log(y("automation result", "items", "manual minutes left", "setup hours"));
for (const d of STATUSES) {
  const k = ITEMS.filter((x) => status(x) === d);
  console.log(y(d, k.length, k.reduce((a, x) => a + remaining(x), 0),
    k.reduce((a, x) => a + SETUP[x.site], 0)));
}
const remainingMin = ITEMS.reduce((a, x) => a + remaining(x), 0);
const setup = ITEMS.reduce((a, x) => a + (status(x) === "stays manual" ? 0 : SETUP[x.site]), 0);
console.log(y("TOTAL", ITEMS.length, remainingMin, setup));

const before = COMPILATIONS * ITEMS.reduce((a, x) => a + x.min, 0) / 60;
const after = COMPILATIONS * remainingMin / 60;
const manualItems = ITEMS.filter((x) => status(x) !== "generated in pipeline");
console.log(`\nitems touched by hand ${ITEMS.length} -> ${manualItems.length} ` +
  `(${ITEMS.filter((x) => status(x) === "stays manual").length} full, ` +
  `${ITEMS.filter((x) => status(x) === "semi-automatic").length} partial)`);
console.log(`annual compilation load ${before.toFixed(1)} -> ${after.toFixed(1)} hours; ` +
  `gain ${(before - after).toFixed(1)} hours/year`);
console.log(`setup ${setup} hours -> payback ${(setup / (before - after)).toFixed(2)} years`);
console.log(`items that never become automatic: ` +
  ITEMS.filter((x) => status(x) === "stays manual").map((x) => `${x.code} ${x.kind}`).join(", "));

automation result          items   manual minutes left     setup hours
generated in pipeline         15                     0             138
semi-automatic                 3                   244              32
stays manual                   6                  1680               0
TOTAL                         24                  1924             170

items touched by hand 24 -> 9 (6 full, 3 partial)
annual compilation load 213.0 -> 96.2 hours; gain 116.8 hours/year
setup 170 hours -> payback 1.46 years
items that never become automatic: K07 meeting, K08 decision, K09 meeting, K11 meeting, K15 meeting, K23 decision

The items touched by hand fall from 24 to 9: six fully manual, three partial. The annual compilation load falls from 213 hours to 96.2 hours and the gain is 116.8 hours a year. The setup takes 170 hours, so the investment pays back in 1.46 years. These three numbers must be read together: the sentence “automation cut the load by more than half” on its own is true, but the net balance of the first year is negative.

The last line gives the ceiling of automation. Six items never become automatic under any condition, two of them of the decision kind and four of the meeting kind. These six items carry 1,680 minutes of the remaining manual load, that is 87% of the 1,924 minutes. Automation took not most of the load but most of the item count; the remaining nine items are few in number and heavy in minutes. Where the next improvement should look is written in this column, and it is not a pipeline but the arrangement of a meeting.

The three partially automated items are a separate class and the one most often misunderstood. An access review decision is marked in a system, but why that access is maintained is a sentence, and a pipeline cannot write that sentence. A record becoming automatic does not mean the decision becomes automatic; what becomes automatic is the proof that the decision exists, not its content.

The Cost of Automatic Evidence

AC13. The auditor inspects at most 25 records per item; the audit period is 365 days, and the retention window of an evidence item carrying a personal data type is 90 days. All three are fictional data, and the window decision comes from the data type the item carries.

// evidence/cost.mjs — the cost of automatic evidence: volume, sampling rate, and retention window.
import { ITEMS, gen } from "./item.mjs";

const SAMPLE = 25, PERIOD_DAYS = 365, RETENTION_DAYS = 90;   // all three are fictional data
const automatic = (k) => k.site !== "manual" && k.kind !== "meeting";
const records = ITEMS.reduce((a, k) => a + k.records, 0);
const inspected = ITEMS.reduce((a, k) => a + Math.min(k.records, SAMPLE), 0);

const y = gen([26, 20, 22]);
console.log(y("evidence volume", "before compilation", "pipeline-generated"));
console.log(y("evidence records/period", ITEMS.length, records));
console.log(y("inspected by auditor", ITEMS.length, inspected));
console.log(y("sampling rate", "100%", `${(100 * inspected / records).toFixed(2)}%`));

// An evidence item carrying a personal data type is subject to the retention window; when the window is
// shorter than the period, the item answers only a part of the period.
const personal = ITEMS.filter((k) => automatic(k) && k.personal === "yes");
const full = ITEMS.filter((k) => automatic(k) && k.personal === "no");
const manual = ITEMS.filter((k) => !automatic(k));
const share = (100 * RETENTION_DAYS / PERIOD_DAYS).toFixed(1);
const z = gen([32, 10, 16]);
console.log("\n" + z("interval covered by evidence", "items", "period share"));
console.log(z(`whole period (${PERIOD_DAYS} days)`, full.length, "100%"));
console.log(z(`last ${RETENTION_DAYS} days (retention limit)`, personal.length, `${share}%`));
console.log(z("single day (manual compilation)", manual.length, "-"));

console.log(`\nautomatic items carrying a personal data type: ` +
  personal.map((k) => k.code).join(", "));
console.log(`Of ${ITEMS.length} evidence items, ${full.length} answer the whole period, ` +
  `${personal.length} the last ${RETENTION_DAYS} days, and ${manual.length} a single day`);
console.log(`evidence records ${ITEMS.length} -> ${records} (${Math.round(records / ITEMS.length)} times); ` +
  `inspected records ${inspected}`);

evidence volume             before compilation    pipeline-generated
evidence records/period                     24                 72401
inspected by auditor                        24                   458
sampling rate                             100%                 0.63%

interval covered by evidence         items    period share
whole period (365 days)                 12            100%
last 90 days (retention limit)           6           24.7%
single day (manual compilation)          6               -

automatic items carrying a personal data type: K03, K04, K05, K14, K16, K24
Of 24 evidence items, 12 answer the whole period, 6 the last 90 days, and 6 a single day
evidence records 24 -> 72401 (3017 times); inspected records 458

At first glance the first table reads against automation: evidence records rise from 24 to 72,401, a factor of 3,017, and the sampling rate falls from 100% to 0.63%. This reading is wrong, and the reason it is wrong lies in the measure of the course. The 100% before compilation says all twenty-four documents were read; it does not say which interval those twenty-four documents describe. In the later state the 458 records inspected are a sample, but all 72,401 records behind them can be queried — the auditor’s next question requires changing the query, not enlarging the sample. More records mean less inspection; they do not have to mean fewer answers.

The condition of this conditional sentence shows in the second table. The six items carrying a personal data type — credential issuance, audit log, production access, data subject request, and two review decisions — are subject to a 90-day retention window. The audit period is 365 days. These six items answer 24.7% of the period; every question about the first nine months of the period is unanswered, and that it is unanswered is understood only when the question is asked. Automation did not leave these items without evidence; it made the lifetime of the evidence a constraint separate from evidence generation.

The queryability condition is not free either. For an evidence record generated in the pipeline to be useful in an audit, it must carry three fields: which control it belongs to, which moment it belongs to, and that it can be shown not to have been changed after it was generated. These three fields are inside the setup hours, and all three can be missing. A pile of automatic evidence with missing fields answers less than twenty-four manually compiled documents — its count is large, it cannot be queried, and it does not say which period it describes. The gain from automation comes not from volume but from these three fields.

Read together, the three rows give this table: of the twenty-four items, twelve answer the whole period, six the last ninety days, and six a single day. Before automation these three classes could not be told apart, because all of them stood in the same form — as a document compiled at the end of the period. The first gain of measuring is not the number itself but that the three classes have been separated.

Summary

• Eighteen of the twenty-four evidence items already arose in a system, yet the evidence of all twenty-four was compiled by hand at the end of the period; the gap is not engineering but connection.
• When generation is moved into the pipeline, the items touched by hand fall from 24 to 9 and the annual compilation load from 213 hours to 96.2 hours; the 170-hour setup pays back in 1.46 years and the balance of the first year is negative.
• Six items never become automatic and carry 87% of the remaining manual load: automation took not most of the load but most of the item count.
• The record of a decision becomes automatic and its justification stays manual; what becomes automatic is the proof that the decision exists, not its content.
• The cost appears in the opposite column: evidence records rise by a factor of 3,017, the sampling rate falls to 0.63%, and the six items carrying a personal data type answer 24.7% of the period because of the 90-day retention window.

Next Step

Evidence generation moved into the pipeline and three classes were separated: items answering the whole period, the last ninety days, and a single day. All of these measurements were about whether the evidence exists. A part of the audit items asks not for existence but for a number: was the finding closed within its target time, and how long have those not closed been open.

The next lesson computes these two numbers at the scale of an application team — mean time to remediate and open finding age. Both come from the same set of findings, go in opposite directions in the same period, and the ability to compute either depends on certain timestamps being present in the record. If a timestamp is missing, the item never enters the calculation; are the items that do not enter chosen at random, or do they always come from the same class?