The Risk Register

The previous three lessons measured a given decision’s record, the process that produced it, and the table it rested on. In all three, the decision settles at some point and the record closes. Something stays open: the chance that the assumption the decision rests on does not hold. Giving a criterion a weight of 0.50 in a trade-off analysis rests on the assumption that quality is critical; if the assumption is wrong, the decision is wrong too, and only time reveals it.

This is not a decision to record — it is an uncertainty to track, and it needs its own record. How risk splits the test budget was measured in test planning; it is not repeated here. The question here concerns the record itself: what a record catches when it is kept, what escapes it, and how the frequency at which the record is reviewed changes that.

The Risk Register and a Period

A risk register is modeled with two fields: likelihood and impact. Likelihood is an estimate written into the record; impact is the scale of how much work it will cause once it happens. Of the fourteen risks in the model, eight are in the record and six are not — the unrecorded ones exist in the system, no one wrote them down.

Every risk has two likelihoods kept separate: the estimate written into the record, and the actual likelihood that determines what is observed across the period. The second is the model’s input (DR12); the person keeping the record does not see it — they only measure it as periods pass. Five hundred periods are run, the generator is written in-house, and the seed is visible.

// risk/record.mjs — a risk record is modeled with likelihood and impact, one period is run
// [name, recorded estimate (null if not in the record), actual likelihood, impact]; all model input (DR12)
export const RISKS = [
  ["catalog-version-breakage", 0.60, 0.55, 4],
  ["branch-network-outage", 0.70, 0.75, 3],
  ["cache-freshness-overrun", 0.50, 0.15, 3],
  ["identity-service-outage", 0.40, 0.35, 5],
  ["backup-window-overrun", 0.30, 0.45, 2],
  ["member-data-mismatch", 0.20, 0.20, 5],
  ["card-printing-delay", 0.15, 0.10, 2],
  ["rule-engine-growth", 0.10, 0.50, 3],
  ["search-index-inconsistency", null, 0.40, 3],
  ["report-server-shortfall", null, 0.35, 4],
  ["notification-queue-backlog", null, 0.30, 2],
  ["license-renewal-delay", null, 0.25, 4],
  ["branch-staff-training-gap", null, 0.20, 2],
  ["loan-limit-exception", null, 0.15, 1],
].map(([name, estimate, actual, impact]) => ({ name, estimate, actual, impact, recorded: estimate !== null }));

export const MONTHS = 12;                    // one period
export const random = (seed) => () => {      // the generator is written here, the seed is visible
  seed = (seed + 0x6d2b79f5) | 0;
  let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};

// One period: each risk occurs at its actual likelihood; if it occurs, the month is spread evenly.
export function period(rnd) {
  return RISKS.map((r) => {
    const occurred = rnd() < r.actual;
    return { risk: r, occurred, month: occurred ? 1 + Math.floor(rnd() * MONTHS) : null };
  });
}

export const BAND = (p) => (p >= 0.5 ? "high" : p >= 0.2 ? "mid" : "low");

if (import.meta.url.endsWith(process.argv[1].split("/").pop())) {
  const TRIALS = 500, SEED = 20240517, rnd = random(SEED);
  const hits = RISKS.map(() => 0);
  for (let i = 0; i < TRIALS; i++)
    period(rnd).forEach((s, j) => { if (s.occurred) hits[j] += 1; });
  const measured = hits.map((h) => h / TRIALS);
  const recorded = RISKS.filter((r) => r.recorded);
  const total = hits.reduce((t, h) => t + h, 0);
  const unrecordedHits = RISKS.reduce((t, r, j) => t + (r.recorded ? 0 : hits[j]), 0);
  console.log(`${RISKS.length} risks: ${recorded.length} recorded, ${RISKS.length - recorded.length} ` +
    `unrecorded; ${TRIALS} periods, seed ${SEED}`);
  console.log(`realized risk ${total}, of which ${unrecordedHits} were unrecorded ` +
    `(${(100 * unrecordedHits / total).toFixed(1)}%)\n`);

  console.log("recorded estimate band vs. measured realization rate:");
  for (const band of ["high", "mid", "low"]) {
    const group = RISKS.map((r, j) => ({ r, m: measured[j] })).filter((x) => x.r.recorded && BAND(x.r.estimate) === band);
    console.log(`  ${band.padEnd(7)} ${group.length} risks, average estimate ` +
      `${(group.reduce((t, x) => t + x.r.estimate, 0) / group.length).toFixed(2)}, measured rate ` +
      `${(group.reduce((t, x) => t + x.m, 0) / group.length).toFixed(2)}`);
  }
  console.log("\nrecorded risks with the largest gap between estimate and measured rate:");
  for (const x of RISKS.map((r, j) => ({ r, m: measured[j] })).filter((x) => x.r.recorded)
    .sort((a, b) => Math.abs(b.m - b.r.estimate) - Math.abs(a.m - a.r.estimate)).slice(0, 3))
    console.log(`  ${x.r.name.padEnd(28)} estimate ${x.r.estimate.toFixed(2)} measured ${x.m.toFixed(2)} ` +
      `gap ${(x.m - x.r.estimate > 0 ? "+" : "") + (x.m - x.r.estimate).toFixed(2)} impact ${x.r.impact}`);
  console.log("\nunrecorded risks that were realized (measured rate x impact):");
  for (const x of RISKS.map((r, j) => ({ r, m: measured[j] })).filter((x) => !x.r.recorded)
    .sort((a, b) => b.m * b.r.impact - a.m * a.r.impact))
    console.log(`  ${x.r.name.padEnd(28)} measured ${x.m.toFixed(2)} impact ${x.r.impact} ` +
      `product ${(x.m * x.r.impact).toFixed(2)}`);
}

14 risks: 8 recorded, 6 unrecorded; 500 periods, seed 20240517
realized risk 2348, of which 808 were unrecorded (34.4%)

recorded estimate band vs. measured realization rate:
  high    3 risks, average estimate 0.60, measured rate 0.49
  mid     3 risks, average estimate 0.30, measured rate 0.34
  low     2 risks, average estimate 0.13, measured rate 0.30

recorded risks with the largest gap between estimate and measured rate:
  rule-engine-growth           estimate 0.10 measured 0.50 gap +0.40 impact 3
  cache-freshness-overrun      estimate 0.50 measured 0.15 gap -0.35 impact 3
  backup-window-overrun        estimate 0.30 measured 0.45 gap +0.15 impact 2

unrecorded risks that were realized (measured rate x impact):
  report-server-shortfall      measured 0.35 impact 4 product 1.41
  search-index-inconsistency   measured 0.39 impact 3 product 1.18
  license-renewal-delay        measured 0.24 impact 4 product 0.98
  notification-queue-backlog   measured 0.29 impact 2 product 0.59
  branch-staff-training-gap    measured 0.18 impact 2 product 0.36
  loan-limit-exception         measured 0.15 impact 1 product 0.15

The first number gives the record’s coverage: 808 of the 2348 realized risks, that is 34.4 percent, were never in the record at all. A risk register’s first measurable quality is not how accurately its contents were estimated — it is how much was left out. Someone looking at an eight-item register has never seen a third of the events that actually happened.

The band table gives the second number and shows the record’s accuracy within itself. The high band’s average estimate is 0.60, its measured rate 0.49; the low band’s estimate is 0.13, its measured rate 0.30. The bands do not even preserve the ordering: risks marked low occur at a rate close to the mid band. A risk register’s likelihood column, unless compared against the measured rate, is not a ranking — it is a list of impressions.

The gap table shows where this breakdown comes from. rule-engine-growth was estimated at 0.10, measured at 0.50; cache-freshness-overrun was estimated at 0.50, measured at 0.15. The two gaps run in opposite directions and both are large. Someone looking at the record prepares for the second risk and runs into the first. This is the number that shows why reviewing the record is not a formality: an estimate written once and left alone does not reveal its own wrongness.

The last table ranks the unrecorded risks by impact. The top two items, report-server-shortfall and search-index-inconsistency, outscore many of the recorded items in product. They are not unrecorded because they are unimportant — they are unrecorded because no one thought of them.

Review Frequency

An unrecorded risk does not enter the record on its own; someone has to notice it. Noticing happens at the moments the record is reviewed. The run below compares four frequencies over the same periods: never reviewing, once a year, once a quarter, and once a month. The probability of noticing an unrecorded risk in one review and the person-minutes of a review are the model’s inputs (DR13).

What is measured is not the number of risks noticed — it is the number of risks noticed before they occur. Recording a risk after it has occurred fixes the record; it does not prevent the event.

// risk/review.mjs — how many risks does review frequency catch before they are realized
import { RISKS, MONTHS, random, period } from "./record.mjs";

const NOTICE = 0.20;      // probability of noticing an unrecorded risk in one review (DR13)
const PERSON_MIN = 45;    // person-minutes of one review (DR13)
const FREQUENCY = {
  never: [], yearly: [12], quarterly: [3, 6, 9, 12], monthly: [...Array(MONTHS)].map((_, i) => i + 1),
};

const TRIALS = 500, SEED = 20240518;
console.log(`${TRIALS} periods, seed ${SEED}; notice probability ${NOTICE}, ` +
  `${PERSON_MIN} person-min per review`);
console.log("frequency   reviews  noticed      caught before      unrecorded realized  person-min");

const result = {};
for (const [name, months] of Object.entries(FREQUENCY)) {
  const rnd = random(SEED);          // every frequency runs on the same seed, over the same periods
  let noticed = 0, before = 0, missed = 0;
  for (let i = 0; i < TRIALS; i++)
    for (const s of period(rnd)) {
      if (s.risk.recorded) continue;
      let foundMonth = null;
      for (const m of months) if (rnd() < NOTICE) { foundMonth = m; break; }
      if (foundMonth !== null) noticed += 1;
      if (s.occurred && (foundMonth === null || foundMonth >= s.month)) missed += 1;
      else if (s.occurred) before += 1;
    }
  const personMin = months.length * PERSON_MIN;
  result[name] = { before: before / TRIALS, missed: missed / TRIALS, personMin, noticed: noticed / TRIALS };
  console.log(`  ${name.padEnd(10)} ${String(months.length).padStart(4)} ` +
    `${(noticed / TRIALS).toFixed(2).padStart(11)} ${(before / TRIALS).toFixed(2).padStart(17)} ` +
    `${(missed / TRIALS).toFixed(2).padStart(22)} ${String(personMin).padStart(9)}`);
}

console.log("\nwhat increasing frequency buys (per period):");
const names = Object.keys(FREQUENCY);
for (let i = 1; i < names.length; i++) {
  const a = result[names[i - 1]], b = result[names[i]];
  const extra = b.before - a.before;
  console.log(`  ${names[i - 1]} -> ${names[i]}: +${(b.personMin - a.personMin)} person-min, ` +
    `+${extra.toFixed(2)} risks caught before -> per risk ` +
    `${extra > 0 ? `${((b.personMin - a.personMin) / extra).toFixed(0)} person-min` : "no return"}`);
}

// Question set: 14 risks x 3 questions; expected value based on how many risks are in the record by period's end
const RECORDED_COUNT = RISKS.filter((r) => r.recorded).length, N = RISKS.length;
console.log(`\nquestion set (${N} risks x 3 questions = ${3 * N} answers; R1 was this risk considered, ` +
  `R2 what likelihood was assumed, R3 did the estimate hold):`);
for (const [name, months] of Object.entries(FREQUENCY)) {
  const inRecord = RECORDED_COUNT + result[name].noticed, outside = N - inRecord;
  const correct = inRecord * (months.length ? 3 : 2);
  const missing = outside * 2 + (months.length ? 0 : inRecord);
  console.log(`  ${name.padEnd(10)} in record ${inRecord.toFixed(2)}/${N} risks -> correct ` +
    `${correct.toFixed(1)}, missing ${missing.toFixed(1)}, wrong ${outside.toFixed(1)}`);
}

500 periods, seed 20240518; notice probability 0.2, 45 person-min per review
frequency   reviews  noticed      caught before      unrecorded realized  person-min
  never         0        0.00              0.00                   1.67         0
  yearly        1        1.26              0.00                   1.63        45
  quarterly     4        3.54              0.44                   1.22       180
  monthly      12        5.62              1.03                   0.59       540

what increasing frequency buys (per period):
  never -> yearly: +45 person-min, +0.00 risks caught before -> per risk no return
  yearly -> quarterly: +135 person-min, +0.44 risks caught before -> per risk 310 person-min
  quarterly -> monthly: +360 person-min, +0.60 risks caught before -> per risk 602 person-min

question set (14 risks x 3 questions = 42 answers; R1 was this risk considered, R2 what likelihood was assumed, R3 did the estimate hold):
  never      in record 8.00/14 risks -> correct 16.0, missing 20.0, wrong 6.0
  yearly     in record 9.26/14 risks -> correct 27.8, missing 9.5, wrong 4.7
  quarterly  in record 11.54/14 risks -> correct 34.6, missing 4.9, wrong 2.5
  monthly    in record 13.62/14 risks -> correct 40.9, missing 0.8, wrong 0.4

Yearly review notices 1.26 risks per period and catches none of them before they occur. The number being zero is not a coincidence: review happens in the twelfth month while realization is spread across all twelve, so noticing almost always comes after the event. A risk register reviewed once a year produces a correct record for the following period and does nothing for the period it is in.

Quarterly frequency catches 0.44 risks before they occur, monthly 1.03; risks that stay unrecorded and occur drop from 1.67 to 0.59. What it buys, though, is a rising curve: moving to quarterly costs 310 person-minutes per risk caught early, moving to monthly costs 602. The second transition’s unit cost is nearly double the first, because the risks most readily noticed are already caught at the sparser frequency. The return on increasing frequency diminishes; where to stop becomes a question of comparing the risk’s impact against the person-minutes.

The question set table shows the same curve from the record’s readability side. Of forty-two answers, the never-reviewed record gives 16 correct, 20 missing, 6 wrong. The wrong ones are the answers to “was this risk considered”: for an unrecorded risk, the answer given is “no,” and that answer looks correct, because the record really is silent on it. Under monthly review, correct answers rise to 40.9. A record’s value is set not by what was written into it once, but by how often it gets read again.

Summary

• 808 of the 2348 realized risks, that is 34.4 percent, were never in the record; a risk register’s first measure is not the accuracy of what it holds — it is the size of what it leaves out.
• The recorded bands did not preserve ordering: the high band’s estimate was 0.60 against a measured rate of 0.49, the low band’s estimate was 0.13 against a measured rate of 0.30.
• The two large gaps run in opposite directions — rule-engine-growth was estimated at 0.10 and measured at 0.50, cache-freshness-overrun at 0.50 and measured at 0.15; an estimate written once and left alone does not reveal its own wrongness.
• Yearly review noticed 1.26 risks per period and caught none of them before they occurred; quarterly caught 0.44, monthly 1.03, and risks missed dropped from 1.67 to 0.59.
• The unit cost of increasing frequency rises: 310 and 602 person-minutes per risk caught early; against that, in the question set, correct answers rose from 16 to 40.9 while wrong answers dropped from 6 to 0.4.

Next Step

A risk register tracks something not yet realized: an event with a likelihood that has not happened yet. Next to it stands one more record, and that one tracks something already realized. When a decision is made, sometimes an incomplete solution is chosen deliberately — to ship faster, or because the right solution is not yet known. That gap stays in the system and produces a little extra work with every change. Sometimes the gap is not chosen deliberately, and is noticed only later. The next lesson separates these two cases, tests whether the difference between them is measurable, and counts the extra work the gap produces per change.