Security Metrics

The previous lesson’s reported-delay column is this lesson’s starting point. That column was a real number, correctly measured, and it was written into the drill report; even so, it spent a six-person-day budget for zero gain. Its flaw was not that it was wrong — it was that it was never tied to a decision.

This lesson sorts an entire program metric set with the same eye. The test is mechanical and has four questions: does the metric have a defined threshold, was the threshold exceeded during the period, is the action to take when it is exceeded defined, and is there a named owner to do that work. If all four hold, the metric has changed behavior; if one is missing, the metric is a report line. The second number to write beside it is the collection cost: how many hours per period it takes to produce that metric.

• SP25. Eighteen metrics, twelve periods of observations, thresholds, and collection hours belong to the fictional regional measurement network and are a model. No real measurement runs.
• SP26. A metric that exceeds its threshold was counted as producing a decision if it had a defined action and a named owner. In reality the defined action is not always carried out; this overstates the decision count.
• SP27. Approaching the threshold was counted as at least one observation being less than fifteen percent away from the threshold; if that cutoff changes, the boundary between groups changes too.
• SP28. The collection cost is a fixed number of hours per period; setup and automation cost were not modeled.
• SP29. The metrics were treated as independent; in reality, one falls while another rises.
• SP30. The randomness generator is written inline in the lesson and the seed is visible.

// metrics/model.mjs — the MODEL of program metrics: eighteen metrics, twelve periods of
// observations, threshold, defined action, and collection cost in a fictional regional
// measurement network.
export const SEED = 20260829, PERIOD = 12;
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
// name, base value, periodic amplitude, trend, threshold, threshold direction (+: above is
// bad), defined action, named owner, collection hours per period
export const METRICS = [
  ['open queue', 175, 30, 4, 200, 1, true, true, 3],
  ['lagging item', 41, 12, 2, 60, 1, true, true, 2],
  ['average open age', 9, 3, 0.4, 12, 1, true, true, 2],
  ['patch window', 9, 2.5, 0, 12, 1, true, true, 1],
  ['unplanned outage', 29, 9, -0.6, 35, 1, true, true, 1],
  ['test coverage', 43, 8, 1.2, 40, -1, true, true, 4],
  ['training attendance rate', 68, 8, 0.3, 70, -1, false, true, 6],
  ['notification rate', 21, 5, 0.8, 19, -1, true, true, 5],
  ['failed drill item', 9, 2, -0.2, 6, 1, true, true, 8],
  ['recovery time', 421, 40, -6, 300, 1, true, true, 12],
  ['control coverage', 76, 5, 1.1, 70, -1, false, true, 9],
  ['requirement w/o evidence', 14, 4, -0.5, 8, 1, true, true, 7],
  ['unverified supplier control', 23, 3, 0.1, 40, 1, false, false, 6],
  ['asset not in inventory', 31, 9, 1.4, 25, 1, true, true, 14],
  ['risk-list top-ten change', 4, 2, 0, 8, 1, false, false, 5],
  ['audit finding', 11, 5, -0.4, 10, 1, true, true, 3],
  ['policy read rate', 94, 4, 0.2, 60, -1, false, false, 16],
  ['security incident count', 11, 5, 0.1, 12, 1, false, true, 2],
].map(([name, base, amplitude, trend, threshold, direction, action, owner, hours], i) =>
  ({ no: i + 1, name, base, amplitude, trend, threshold, direction, action, owner, hours }));

export const observe = (o) => { const r = generator(SEED + o.no * 977);
  return Array.from({ length: PERIOD }, (_, d) =>
    +(o.base + o.trend * d + (r() - 0.5) * 2 * o.amplitude).toFixed(1)); };
export const exceeds = (o) => observe(o).filter((v) => (v - o.threshold) * o.direction > 0).length;
// A crossing produces a decision: the threshold must be exceeded, and a defined action and
// named owner must both exist.
export const decisions = (o) => (o.action && o.owner ? exceeds(o) : 0);
// If the threshold was never approached, the metric is counted as never having been tested
// by the threshold at all.
export const near = (o) => { const g = observe(o);
  const dist = g.map((v) => Math.abs(v - o.threshold) / Math.max(1, Math.abs(o.threshold)));
  return Math.min(...dist) <= 0.15; };
// Four groups: decision-bearing / signaling but not tied to a decision / near threshold /
// inactive threshold
export const group = (o) => (exceeds(o) > 0
  ? (o.action && o.owner ? 'decision-bearing' : 'signal, no decision')
  : (near(o) ? 'near threshold' : 'inactive threshold'));
export const GROUPS = ['decision-bearing', 'signal, no decision', 'near threshold', 'inactive threshold'];
export const annualHours = (o) => o.hours * PERIOD;
export const TOTAL_HOURS = METRICS.reduce((a, o) => a + annualHours(o), 0);

Decision-Bearing Metric versus Report Line

// metrics/decisions.mjs — sorts metrics by whether they changed behavior or not and
// puts the decision count next to the collection cost
import { METRICS, GROUPS, PERIOD, TOTAL_HOURS, exceeds, decisions, near, group, annualHours } from './model.mjs';

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());
const rank = (l, f) => { const s = [...l].sort((a, b) => f(b) - f(a));
  return new Map(s.map((o, i) => [o.name, i + 1])); };

console.log(`${METRICS.length} metrics, ${PERIOD} periods, collection cost ${TOTAL_HOURS} hours/yr`);
const A = [-20, 8, 10, 9, 11];
console.log('\n1. the four groups of metrics');
y(A, 'group', 'metrics', 'hours/yr', '% hours', 'decis./yr');
for (const g of GROUPS) {
  const l = METRICS.filter((o) => group(o) === g), s = l.reduce((a, o) => a + annualHours(o), 0);
  y(A, g, l.length, s, (100 * s / TOTAL_HOURS).toFixed(1), l.reduce((a, o) => a + decisions(o), 0));
}

const db = METRICS.filter((o) => decisions(o) > 0);
const rD = rank(db, (o) => decisions(o)), rR = rank(db, (o) => decisions(o) / annualHours(o));
const B = [-26, 8, 10, 12, 8, 12];
console.log('\n2. decision-bearing metrics: two rankings');
y(B, 'metric', 'decis.', 'hours/yr', 'decis./hr', 'rank', 'rank');
y(B, '', 'yearly', '', '', 'decis.', 'decis./hr');
for (const o of [...db].sort((a, b) => decisions(b) - decisions(a)))
  y(B, o.name, decisions(o), annualHours(o), (decisions(o) / annualHours(o)).toFixed(3), rD.get(o.name), rR.get(o.name));
console.log(`metrics that swap rank between the two orderings: ` +
  `${db.filter((o) => rD.get(o.name) !== rR.get(o.name)).length}/${db.length}`);

const priciest = [...METRICS].sort((a, b) => annualHours(b) - annualHours(a))[0];
console.log(`\nmost expensive metric: ${priciest.name}, ${annualHours(priciest)} hours/yr, ` +
  `threshold exceeded in ${exceeds(priciest)} periods, defined action ${priciest.action ? 'yes' : 'no'}, ` +
  `owner ${priciest.owner ? 'yes' : 'no'}, near threshold: ${near(priciest) ? 'yes' : 'no'}`);
const inact = METRICS.filter((o) => group(o) === 'inactive threshold');
console.log(`removing the ${inact.length} metrics with an inactive threshold ` +
  `drops ${inact.reduce((a, o) => a + annualHours(o), 0)} hours/yr (` +
  `${(100 * inact.reduce((a, o) => a + annualHours(o), 0) / TOTAL_HOURS).toFixed(0)} percent of the total), decisions lost: ` +
  `${inact.reduce((a, o) => a + decisions(o), 0)}`);
const sig = METRICS.filter((o) => group(o) === 'signal, no decision');
console.log(`adding a defined action and owner to the ${sig.length} signaling metrics ` +
  `adds ${sig.reduce((a, o) => a + exceeds(o), 0)} decisions per year; the collection cost stays at ` +
  `${sig.reduce((a, o) => a + annualHours(o), 0)} hours/yr`);

18 metrics, 12 periods, collection cost 1272 hours/yr

1. the four groups of metrics
group                metrics  hours/yr  % hours  decis./yr
decision-bearing           8       612     48.1         56
signal, no decision        2        96      7.5          0
near threshold             5       240     18.9          0
inactive threshold         3       324     25.5          0

2. decision-bearing metrics: two rankings
metric                      decis.  hours/yr   decis./hr    rank        rank
                            yearly                        decis.   decis./hr
recovery time                   12       144       0.083       1           7
asset not in inventory          12       168       0.071       2           8
failed drill item               10        96       0.104       3           3
requirement w/o evidence         8        84       0.095       4           4
audit finding                    6        36       0.167       5           1
open queue                       3        36       0.083       6           5
average open age                 3        24       0.125       7           2
lagging item                     2        24       0.083       8           6
metrics that swap rank between the two orderings: 6/8

most expensive metric: policy read rate, 192 hours/yr, threshold exceeded in 0 periods, defined action no, owner no, near threshold: no
removing the 3 metrics with an inactive threshold drops 324 hours/yr (25 percent of the total), decisions lost: 0
adding a defined action and owner to the 2 signaling metrics adds 12 decisions per year; the collection cost stays at 96 hours/yr

The first table splits the eighteen metrics into four groups, and the first number worth reading is how the collection cost is distributed. The eight decision-bearing metrics cost 612 hours a year; the remaining ten metrics cost 660 hours and produce zero decisions. More than half of the annual measurement effort goes to a set of numbers that never changed behavior at all.

The conclusion here is not “drop the metric that produces no decision,” because zero has three separate causes. The second group — the two metrics that exceed their threshold but have no defined action — is the worst case: the signal arrived and was dropped. The fix is not in measurement but in definition; the last line counts it: adding a defined action and owner brings twelve more decisions a year, and the collection cost does not change. The cost of a number that rises from zero to twelve is writing two lines into a document.

The third group runs the opposite way. The five metrics that approach the threshold never produced an exceedance, but at least one observation of each passed within fifteen percent of the threshold. These are the only evidence that the threshold is binding and that the control holds; remove them and what remains is not evidence but optimism. They cost 240 hours a year, and that time was spent not per decision but per assurance.

The fourth group is the one that can be dropped: three metrics whose threshold was never exceeded, never approached, and that have no defined action or owner. 324 hours a year, twenty-five percent of the total, zero decisions lost. The most expensive metric is also in this group: 192 hours a year on its own and zero exceedances. Being expensive does not make a metric important, because a number nobody has tied to a decision has never been questioned enough to be trimmed.

The second table repeats this course’s main evidence one last time. Ranked by decision count, recovery time and asset not in inventory come out on top; ranked by hours per decision, the two of them drop to seventh and eighth place, while audit finding climbs from fifth to first. Six of the eight metrics swap places: the two metrics that trigger the most decisions are the two most expensive per decision.

Summary

• A metric changes behavior only when it satisfies all four conditions at once: a defined threshold, an exceedance during the period, a defined action, and a named owner. If one is missing, the metric is a report line.
• Eight of the eighteen metrics carry decisions (56 decisions a year, 612 hours); the remaining ten metrics cost 660 hours and produce zero decisions — 52 percent of the annual measurement effort.
• Adding a defined action and owner to the two metrics that signal but are never tied to a decision adds twelve decisions a year without changing the collection cost.
• The five metrics that approach the threshold and never exceed it (240 hours a year) are not dropped: they are the only evidence that the threshold is binding and that the control holds.
• The group that can be dropped is the three metrics whose threshold is never even approached: 324 hours a year, 25 percent of the total, zero decisions lost. The most expensive metric (192 hours a year) is in this group.
• Six of the eight decision-bearing metrics swap places between the decision-count ranking and the hours-per-decision ranking; the two metrics that trigger the most decisions are the two most expensive per decision.

Course Wrap-Up

These seventeen lessons repeated a single pattern: a set of items was defined, ranked by a measure, the risk each option reduced was written beside the unit it cost, and then the measure was changed and the point where the ranking broke was counted. The last column said the same thing in every lesson — two different priorities can come out of the same data; if the measure that produced the ranking is not written down, the list is not a decision, it is a preference with no visible justification.

This course is also the end of the curriculum. Ten courses built ten separate measurement axes, and each defined its own unit:

Across the ten courses there was one question asked: what a defense closes and what it stops in return. The unit changed from course to course — path, item, claim, link, minutes, radius, rank — but the question did not. No course used the word “secure” as an outcome, because what was always measured was the path that remained: the foothold that could not be closed, the claim that could not be verified, the link that would not break, the requirement without evidence, the item that could not be kept up with.

The close is therefore written as a single rule: a defense whose cost is not written down is counted as unmeasured. How many legitimate requests were denied, how many outages it caused, how many hours it took, how many items were managed by hand — if that column is empty, the number left over is not a measurement, it is a claim. What this curriculum leaves behind is not a list of defenses, it is the habit of filling in that column.