Reporting

The previous lesson put the 61 evidenced findings in order along two separate axes and measured what a 40-person-day remediation budget closes. The next step is for the finding to go to a remediation team, and that team will read the finding for the first time. The test team having evidence on hand does not mean that evidence can be reproduced on the other side.

This lesson tests the report with a single measure: reproducibility. A finding is reproducible if someone who did not produce it can observe the same result; if it cannot be observed, whether the remediation worked can never be verified either. Reproduction steps are therefore a report item, not a narrative. No example step is written in this lesson; what is measured is what information a step must carry and how many records actually contain that information.

• TP25. The report set is the previous lesson’s 61 findings. The records are fictional and contain no attack steps; what is counted is field names.
• TP26. Eight fields are defined: precondition, input definition, observed result, timestamp, evidence id, environment info, expected behavior, and remediation suggestion.
• TP27. The first five fields are mandatory for reproduction. The remaining three complete the record but do not enter the step set itself.
• TP28. A record can carry an environment variable: even when environment info is written down, its value changes from run to run. These records cannot be reproduced step by step.
• TP29. A second reader can reproduce a finding only if the step set is complete, environment info is written, and there is no environment variable.
• TP30. A finding can be called “closed” in a retest round only if it is reproducible and its expected behavior is written. Field-writing durations are fictional.

Field Audit

// report.mjs — reproducibility audit of report records.
// THIS IS A MODEL; the records are fictional and contain no attack steps — only the
// FIELD NAMES a step must carry are counted.
export const SEED = 20260519, N = 61;
const makeRng = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = makeRng(SEED);

// A reproduction step is a report item. Fields are the information the step must carry;
// the fill rate is fictional and reflects the field's difficulty.
export const FIELD = [
  { name: "precondition", fillRate: 0.82, mandatory: true, minutes: 4 },
  { name: "input definition", fillRate: 0.74, mandatory: true, minutes: 6 },
  { name: "observed result", fillRate: 0.91, mandatory: true, minutes: 5 },
  { name: "timestamp", fillRate: 0.88, mandatory: true, minutes: 1 },
  { name: "evidence id", fillRate: 0.63, mandatory: true, minutes: 3 },
  { name: "environment info", fillRate: 0.57, mandatory: false, minutes: 4 },
  { name: "expected behavior", fillRate: 0.49, mandatory: false, minutes: 7 },
  { name: "remediation suggestion", fillRate: 0.86, mandatory: false, minutes: 12 },
];
// Remediation suggestion class: directly actionable, decision-required, structural.
const SUGGESTION = [["direct", 0.41], ["decision-required", 0.34], ["structural", 0.25]];
const pickSuggestion = (u) => { let t = 0;
  for (const [name, p] of SUGGESTION) { t += p; if (u < t) return name; } return SUGGESTION.at(-1)[0]; };

export const RECORD = Array.from({ length: N }, (_, i) => {
  const filled = {};
  for (const f of FIELD) filled[f.name] = r() < f.fillRate;
  return { no: i + 1, filled, suggestion: filled["remediation suggestion"] ? pickSuggestion(r()) : "none",
    // Environment-dependent: even when the record carries environment info, its value
    // changes from run to run.
    variable: r() < 0.23 };
});

const MANDATORY = FIELD.filter((f) => f.mandatory).map((f) => f.name);
export const stepsComplete = (k) => MANDATORY.every((name) => k.filled[name]);
// A second reader can reproduce a finding only if the step set is complete, environment
// info is present, and there is no environment variable.
export const isReproducible = (k) => stepsComplete(k) && k.filled["environment info"] && !k.variable;

if (import.meta.url === `file://${process.argv[1]}`) {
  const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const yz = gen([23, 10, 9, 10, 12, 13]);
  console.log(`model (seed ${SEED}): ${N} report records, ${FIELD.length} fields, ` +
    `${MANDATORY.length} mandatory for reproduction`);
  console.log("\n" + yz(["field", "mandatory", "filled", "missing", "sole gap", "min/finding"]));
  for (const f of FIELD) {
    const filled = RECORD.filter((k) => k.filled[f.name]).length;
    // Sole gap in: records that would complete their step set if only this field were filled.
    const sole = RECORD.filter((k) => !k.filled[f.name] && f.mandatory &&
      MANDATORY.filter((z) => !k.filled[z]).length === 1).length;
    console.log(yz([f.name, f.mandatory ? "yes" : "no", filled, N - filled, sole, f.minutes]));
  }

  const complete = RECORD.filter(stepsComplete), reproducible = RECORD.filter(isReproducible);
  const noEnv = complete.filter((k) => !k.filled["environment info"]).length;
  const variable = complete.filter((k) => k.filled["environment info"] && k.variable).length;
  const y2 = gen([48, 9, 9]);
  console.log("\n" + y2(["record class", "findings", "percent"]));
  for (const [name, s] of [["step set complete", complete.length],
    ["step set incomplete", N - complete.length],
    ["complete but no environment info", noEnv],
    ["complete, environment written, value variable", variable],
    ["second reader can reproduce", reproducible.length]]) {
    console.log(y2([name, s, `${(100 * s / N).toFixed(1)}`]));
  }
  const missingMost = MANDATORY.map((z) => [z, RECORD.filter((k) => !k.filled[z]).length])
    .sort((a, b) => b[1] - a[1])[0];
  console.log(`\nthe mandatory field missing most often is "${missingMost[0]}" ` +
    `(missing from ${missingMost[1]} records); writing all fields for one record ` +
    `takes ${FIELD.reduce((a, x) => a + x.minutes, 0)} minutes`);
}

model (seed 20260519): 61 report records, 8 fields, 5 mandatory for reproduction

field                   mandatory   filled   missing    sole gap  min/finding
precondition                  yes       55         6           1            4
input definition              yes       41        20           9            6
observed result               yes       56         5           2            5
timestamp                     yes       54         7           1            1
evidence id                   yes       36        25          11            3
environment info               no       37        24           0            4
expected behavior              no       30        31           0            7
remediation suggestion         no       51        10           0           12

record class                                     findings  percent
step set complete                                      19     31.1
step set incomplete                                    42     68.9
complete but no environment info                        7     11.5
complete, environment written, value variable           4      6.6
second reader can reproduce                             8     13.1

the mandatory field missing most often is "evidence id" (missing from 25 records); writing all fields for one record takes 42 minutes

Looked at field by field, the first table makes the report look good: observed result is written in 56 of 61 records, precondition in 55, timestamp in 54. No single field is a disaster on its own. The second table, though, leaves the count of records where all five mandatory fields are filled together at 19 — 31.1 percent. Independent gaps do not add, they multiply: a team that fills each field at a high individual rate can still end up at roughly one-third on the combined measure. Report quality must be measured per record, not per field.

The “sole gap” column shows where the cheapest gain lies. Evidence id is the sole missing field in 11 records: filling only that field raises the count of records with a complete step set from 19 to 30. The field costs 3 minutes per record, and comes to 75 minutes across the 25 records where it is missing — a 57 percent increase for under an hour and a half of work. Input definition is the sole gap in 9 records, and the 20 records where it is missing come to 120 minutes. An empty column carries information too: environment info, expected behavior, and remediation suggestion are never the sole gap in any record, because they are not part of the mandatory field set and cannot complete the step set on their own.

The third number is the real warning. Of the 19 records with a complete step set, only 8 can be reproduced by a second reader. Of the rest, 7 have no environment info written at all, and 4 have environment info written but its value changes from run to run. Reproducibility does not end with the steps being written down; it also requires that the steps be executable under the same conditions.

The Cost of Completion and the Retest

// closure.mjs — the cost of completing missing fields and how many findings can be
// called "closed" in a retest round.
import { RECORD, FIELD, stepsComplete, isReproducible } from "./report.mjs";

const N = RECORD.length;
const minutesOf = (name) => FIELD.find((f) => f.name === name).minutes;
const MANDATORY = FIELD.filter((f) => f.mandatory).map((f) => f.name);
const missingMinutes = (fields) => RECORD.reduce((a, k) =>
  a + fields.filter((name) => !k.filled[name]).reduce((s, name) => s + minutesOf(name), 0), 0);

// A finding can be called "closed" in a retest round only if it is reproducible AND its
// expected behavior is written.
const closes = (k) => isReproducible(k) && k.filled["expected behavior"];
const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = gen([32, 15, 13, 18, 16]);

const STEP = [
  ["today's report", [], (k) => isReproducible(k), closes],
  ["mandatory fields completed", MANDATORY,
    (k) => k.filled["environment info"] && !k.variable,
    (k) => k.filled["environment info"] && !k.variable && k.filled["expected behavior"]],
  ["environment info also written", [...MANDATORY, "environment info"],
    (k) => !k.variable, (k) => !k.variable && k.filled["expected behavior"]],
  ["expected behavior also written", [...MANDATORY, "environment info", "expected behavior"],
    (k) => !k.variable, (k) => !k.variable],
];
console.log(`${N} findings; writing a missing field adds overhead to a record equal to ` +
  `that field's duration`);
console.log(yz(["completion step", "added minutes", "added hours", "reproducible", "closeable"]));
for (const [name, fields, u, kap] of STEP) {
  const d = missingMinutes(fields);
  console.log(yz([name, d, (d / 60).toFixed(1), RECORD.filter(u).length, RECORD.filter(kap).length]));
}
const variable = RECORD.filter((k) => k.variable).length;
console.log(`even if every field were written, ${variable} findings carry an environment ` +
  `variable and cannot be reproduced step by step; for these, the evidence record itself ` +
  `is the only basis`);

// Remediation suggestion class: how many items are directly actionable.
const CLASS = ["direct", "decision-required", "structural", "none"];
const y2 = gen([20, 9, 18, 16, 18]);
console.log("\n" + y2(["suggestion class", "findings", "reproducible", "closeable",
  "needs decision"]));
for (const s of CLASS) {
  const l = RECORD.filter((k) => k.suggestion === s);
  console.log(y2([s, l.length, l.filter(isReproducible).length, l.filter(closes).length,
    s === "direct" || s === "none" ? 0 : l.length]));
}
console.log(`\nof ${N} findings, ${RECORD.filter((k) => k.suggestion === "direct").length} have a ` +
  `directly actionable suggestion; ${RECORD.filter((k) => k.suggestion === "decision-required" ||
  k.suggestion === "structural").length} require a separate decision before remediation, ` +
  `${RECORD.filter((k) => k.suggestion === "none").length} have no suggestion at all`);
console.log(`findings that can be called closed in a retest round: ${RECORD.filter(closes).length}; ` +
  `even among the ${RECORD.filter(stepsComplete).length} findings with a complete step set, ` +
  `a portion falls short of this bar`);

61 findings; writing a missing field adds overhead to a record equal to that field's duration
completion step                   added minutes  added hours      reproducible       closeable
today's report                                0          0.0                 8               3
mandatory fields completed                  251          4.2                27              14
environment info also written               347          5.8                49              25
expected behavior also written              564          9.4                49              49
even if every field were written, 12 findings carry an environment variable and cannot be reproduced step by step; for these, the evidence record itself is the only basis

suggestion class     findings      reproducible       closeable    needs decision
direct                     29                 5               1                 0
decision-required          12                 1               1                12
structural                 10                 1               1                10
none                       10                 1               0                 0

of 61 findings, 29 have a directly actionable suggestion; 22 require a separate decision before remediation, 10 have no suggestion at all
findings that can be called closed in a retest round: 3; even among the 19 findings with a complete step set, a portion falls short of this bar

The first row gives this course’s sharpest number: in today’s report, 3 of the 61 findings are in a state that can be called closed after a retest round. For the remaining 58, there is no basis to answer the question “did it work” once remediation is applied; the fix gets made, resources get spent, and the result cannot be measured.

The rest of the ladder writes the price of this gap. Completing the mandatory fields takes 251 minutes, or 4.2 hours, and raises the closeable count to 14. When environment info is also written, 25 findings are reached at a total of 5.8 hours; when expected behavior is also written, 49 findings are reached at 9.4 hours. The previous lesson’s remediation budget was 40 person-days; 9.4 hours is roughly 3 percent of that budget. When report writing is seen as overhead, it is the first item to get cut — but what actually gets cut is the measurability of the remediation budget.

12 findings fall outside this ladder. These carry an environment variable, and even if every field were written, step-by-step reproduction would not give the same result. For these records, the only basis is the evidence record itself, taken at the moment of observation: under what identifier it is stored, when it was captured, and how its integrity is demonstrated. A finding that cannot be reproduced is not deleted from the report; it stays with an “environment-dependent” tag and its evidence id. This also explains why the evidence id field, despite being one of the cheapest fields, is the mandatory field missing most often: it takes three minutes to write, but when it is missing it both breaks the step set and removes the only basis for environment-dependent findings. Only an identifier string sits in the record; where the evidence that identifier points to is stored and how it is verified is a separate item.

The second table separates the two axes from each other. The remediation suggestion is directly actionable for 29 findings, but only 5 of those 29 are reproducible and only 1 can be called closed. Suggestion quality and evidence quality are two independent axes: a well-written suggestion attached to an unverifiable finding gets implemented, but its result cannot be measured. The reverse also happens — 10 findings have no suggestion at all, and these stop at the first step once they enter the remediation queue.

The remaining 22 findings require a separate decision before remediation: for 12, the suggestion offers a choice; for 10, a structural change is needed. This distinction being written into the report is necessary for the remediation team to order the queue correctly; a “decision-required” item cannot be put into the same sprint as a directly actionable one.

Summary

• Even though each of the five mandatory fields is individually filled at a high rate, all five together are filled in only 19 records — 31.1 percent: independent gaps do not add, they multiply.
• Evidence id is the sole missing field in 11 records, and completing it raises the count of records with a complete step set from 19 to 30; its cost is 75 minutes for the 25 records where it is missing.
• Of the 19 records with a complete step set, only 8 can be reproduced by a second reader; 7 lack environment info, and 4 have environment info written but with a variable value.
• In today’s report, 3 of the 61 findings can be called closed in a retest round; writing the missing fields takes 9.4 hours and raises this number to 49 — roughly 3 percent of the remediation budget.
• 12 findings cannot be reproduced step by step under any circumstance because they carry an environment variable; for these, the only basis is the evidence record’s identifier, timestamp, and integrity.
• Suggestion quality and evidence quality are independent: of the 29 findings with a directly actionable suggestion, only 1 can be called closed, while 22 findings require a separate decision before remediation.

Next Step

The report is written, and which findings are verifiable has been counted. All of this measurement rested on a single assumption: where the report is going is known, and a remediation team is there. That assumption falls away when the finding comes from outside the organization’s own contracted assessment rather than from it. Then who it gets reported to, through which channel, and within how much time; how long remediation should be given; and when the finding can be published become the subject of a separate process. The next lesson measures responsible disclosure in terms of time and steps, and closes out the course.