Application Security Metrics

The previous lesson measured where evidence is produced and separated the items into those that answer the whole period, the last ninety days, or a single day; what was measured was the existence of evidence. Some of the audit items, however, ask not for existence but for a number: was the finding closed within its target time, and how long has the item that was not closed been open.

This lesson calculates those two numbers — mean time to remediate and open finding age — at the scale of one application team. The Governance, Risk and Compliance course covered the same metrics at the program level; that queue is the input here. The question asked is narrow: what the metrics say, what they do not say, and how many questions what they do not say leaves unanswered.

Finding Set and Question Set

AC14. The finding set is 142 items, comes from five sources, belongs to a 180-day period, and is a model of the subscriber portal team of the fictional regional measurement network; the generator is in this block, and the seed is visible. AC15. An item can have three timestamps: opened exists on all of them, verification is never written in some sources, and closure exists only on an item that closed. AC16. The target durations of the severity classes are data (7, 21, 45, and 90 days) and are fictional. AC17. Midway through the period the priority rule changed: items from the pipeline were moved up, items from outside the pipeline were held back — this is an assumption of the model.

// metrics/finding.mjs — the finding set is a MODEL: fictional regional measurement network, a single 180-day period.
// The generator is written here and the seed is visible; the set comes out the same on every run.
export const gen = (B) => (...h) => h.map((v, j) =>
  (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join("");

export const PERIOD = 180, SEED = 20250, PIPELINE_SOURCES = ["dependency", "static", "image"];
let s = SEED;
const rnd = () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296;

// source | findings | base days | triage days | probability the verification stamp is missing
const SOURCES = [["dependency", 44, 16, 1, 0], ["static", 34, 26, 2, 0], ["image", 24, 22, 2, 0],
  ["audit", 16, 52, 7, .5], ["report", 24, 70, 11, .7]];
export const TARGET = { critical: 7, high: 21, medium: 45, low: 90 };
const FACTOR = { critical: .4, high: .7, medium: 1, low: 1.4 };
const ORDER = { pipeline: "medium low medium high low medium critical high".split(" "),
  external: "high medium critical high low medium high critical".split(" ") };

export const FINDINGS = [];
for (const [source, total, base, triage, missing] of SOURCES) {
  for (let i = 0; i < total; i++) {
    const inPipeline = PIPELINE_SOURCES.includes(source);
    const severity = ORDER[inPipeline ? "pipeline" : "external"][Math.floor(rnd() * 8)];
    const openedAt = 1 + Math.floor(rnd() * PERIOD);
    // The duration distribution is skewed: most items take a short time, a few take very long.
    let duration = base * FACTOR[severity] * (0.15 + 4 * Math.pow(rnd(), 3.5));
    // The priority rule changed midway through the period: pipeline items moved up, the others held back.
    duration *= inPipeline ? (openedAt > 90 ? .45 : 1) : (openedAt > 90 ? 3.2 : 1.8);
    const kind = inPipeline && rnd() < .12 ? "invalid" : "fixed";
    duration = Math.max(1, Math.round(kind === "invalid" ? duration * .15 : duration));
    const closedAt = openedAt + duration <= PERIOD ? openedAt + duration : null;
    const d = rnd() < missing ? null : openedAt + Math.round(triage * (.4 + 1.2 * rnd()));
    FINDINGS.push({ source, severity, target: TARGET[severity], openedAt, closedAt, kind,
      verifiedAt: d === null ? null : Math.min(d, closedAt ?? PERIOD) });
  }
}
export const closed = FINDINGS.filter((f) => f.closedAt !== null);
export const unstamped = FINDINGS.filter((f) => f.verifiedAt === null);

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`model: ${FINDINGS.length} findings, ${SOURCES.length} sources, ${PERIOD} days, seed ${SEED}`);
  console.log(`${closed.length} items closed, ${FINDINGS.length - closed.length} items still open`);
  console.log(`verification stamp missing on ${unstamped.length} items: ` + SOURCES.map(([k]) =>
    `${k} ${FINDINGS.filter((f) => f.source === k && !f.verifiedAt).length}/` +
    `${FINDINGS.filter((f) => f.source === k).length}`).join(", "));

  // Questions to be asked at the end of the period, the record field each one needs, and the number of items carrying that field.
  const K = closed.length, D = FINDINGS.length - unstamped.length;
  const QUESTIONS = [["mean time to remediate", "opened+closed", K],
    ["within-target rate", "opened+closed", K], ["age of an open item", "opened", FINDINGS.length],
    ["how much of the duration was triage", "verification", D],
    ["how many of the closed were fixed", "closure kind", K],
    ["how many times an item was reopened", "state transition", 0],
    ["which change made the fix", "change id", 0],
    ["why the metric changed", "opened+source", K]];
  const answer = (t) => t === 0 ? "none" : t >= K ? "full" : "partial";
  const z = gen([36, 20, 10, 8]);
  console.log("\n" + z("question", "record field needed", "carrying", "answer"));
  for (const [m, d, t] of QUESTIONS) console.log(z(m, d, t, answer(t)));
  const n = (c) => QUESTIONS.filter(([, , t]) => answer(t) === c).length;
  console.log(`\nof ${QUESTIONS.length} questions, ${n("full")} are answered fully, ${n("partial")} partially, ` +
    `${n("none")} are unanswered`);
}

model: 142 findings, 5 sources, 180 days, seed 20250
121 items closed, 21 items still open
verification stamp missing on 25 items: dependency 0/44, static 0/34, image 0/24, audit 10/16, report 15/24

question                             record field needed  carrying  answer
mean time to remediate                     opened+closed       121    full
within-target rate                         opened+closed       121    full
age of an open item                               opened       142    full
how much of the duration was triage         verification       117 partial
how many of the closed were fixed           closure kind       121    full
how many times an item was reopened     state transition         0    none
which change made the fix                      change id         0    none
why the metric changed                     opened+source       121    full

of 8 questions, 5 are answered fully, 1 partially, 2 are unanswered

The third line answers the question the previous lesson left behind. The 25 items missing the verification stamp are not randomly chosen: it is missing on none of the 102 items from the pipeline, on 10 of those from the audit log, and on 15 of those from the external report intake. The place where the stamp is missing is the same as the place where the duration is longest.

The two unanswered questions are of the same class: they are the questions that audit the metric. A reopened item appears in the metric not as one long duration but as two short ones; if the change that closed it is unknown, it cannot be verified that the closure was a fix. The cost of making them answerable — assuming four state transitions per item — is 568 additional records and a link to be built between the finding and the change.

Two Metrics, One Set

AC18. Mean time to remediate is calculated only from closed items; an open item never enters the metric. Open age, on the other hand, is an instantaneous measurement. The base of the two metrics is therefore not the same set, and the two can move in opposite directions in the same period.

// metrics/metrics.mjs — two metrics are calculated from the same set: mean time to remediate and open finding age.
import { FINDINGS, PERIOD, PIPELINE_SOURCES, closed, gen } from "./finding.mjs";

const mean = (a) => a.reduce((t, x) => t + x, 0) / a.length;
const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(p * (a.length - 1))];
const duration = (f) => f.closedAt - f.openedAt;
const openAt = (t) => FINDINGS.filter((f) => f.openedAt <= t && (!f.closedAt || f.closedAt > t));
const age = (t) => openAt(t).map((f) => t - f.openedAt);

const y = gen([26, 8, 10, 9, 8, 10]);
console.log(y("metric (days)", "items", "mean", "median", "p90", "max"));
for (const [name, a] of [["time to remediate", closed.map(duration)], ["open finding age (day 180)", age(PERIOD)]])
  console.log(y(name, a.length, mean(a).toFixed(1), percentile(a, .5), percentile(a, .9), Math.max(...a)));

const d = closed.map(duration), m = mean(d), above = d.filter((x) => x > m);
console.log(`\n${above.length}/${d.length} items above the mean (${m.toFixed(1)}) carry ` +
  `${(100 * above.reduce((t, x) => t + x, 0) / d.reduce((t, x) => t + x, 0)).toFixed(0)}% of the total days`);

// The same set under two grouping rules, then a snapshot of the open queue at two measurement points.
const z = gen([30, 12, 12, 12]);
console.log("\n" + z("grouping rule", "days 1-90", "days 91-180", "direction"));
for (const [name, f] of [["counted in the half it opened", (x) => x.openedAt],
                       ["counted in the half it closed", (x) => x.closedAt]]) {
  const a = mean(closed.filter((x) => f(x) <= 90).map(duration));
  const c = mean(closed.filter((x) => f(x) > 90).map(duration));
  console.log(z(name, a.toFixed(1), c.toFixed(1), c < a ? "falls" : "rises"));
}
for (const t of [90, PERIOD])
  console.log(z(`open queue (day ${t})`, `${age(t).length} items`,
    `mean ${mean(age(t)).toFixed(1)}`, `median ${percentile(age(t), .5)}`));

const w = gen([16, 9, 18, 8, 22]);
console.log("\n" + w("source group", "closed", "mean remediation", "open", "mean open finding age"));
for (const [name, f] of [["pipeline", (x) => PIPELINE_SOURCES.includes(x.source)],
                       ["non-pipeline", (x) => !PIPELINE_SOURCES.includes(x.source)]]) {
  const kk = closed.filter(f), aa = openAt(PERIOD).filter(f);
  console.log(w(name, kk.length, mean(kk.map(duration)).toFixed(1), aa.length,
    mean(aa.map((x) => PERIOD - x.openedAt)).toFixed(1)));
}
const h2 = FINDINGS.filter((x) => x.openedAt > 90 && !x.closedAt);
console.log(`\n${h2.length} items opened between days 91 and 180 are still open and never enter the metric; ` +
  `${h2.filter((x) => !PIPELINE_SOURCES.includes(x.source)).length} of them come from non-pipeline sources. ` +
  `Removing the ${FINDINGS.filter((x) => x.kind === "invalid").length} items counted as invalid moves the mean ` +
  `${m.toFixed(1)} -> ${mean(closed.filter((x) => x.kind === "fixed").map(duration)).toFixed(1)} days`);

metric (days)                items      mean   median     p90       max
time to remediate              121      17.0        6      57       116
open finding age (day 180)      21      59.8       37     148       176

30/121 items above the mean (17.0) carry 75% of the total days

grouping rule                    days 1-90 days 91-180   direction
counted in the half it opened         23.6         7.6       falls
counted in the half it closed         13.2        20.0       rises
open queue (day 90)               24 items   mean 30.4   median 19
open queue (day 180)              21 items   mean 59.8   median 37

source group       closed  mean remediation    open mean open finding age
pipeline               97              13.7       5                  10.0
non-pipeline           24              30.2      16                  75.3

14 items opened between days 91 and 180 are still open and never enter the metric; 9 of them come from non-pipeline sources. Removing the 13 items counted as invalid moves the mean 17.0 -> 18.8 days

The first line is the central flaw of the metric: the mean is 17.0 days, the median is 6 days. The mean describes no item; 30 of the 121 items are above the mean and carry 75% of the total waiting days. On the open side the distribution is also skewed, for a different reason: a long item never closes, so it does not enter the metric — it accumulates.

The second table is the lesson’s harshest number. The same 121 items, two grouping rules, two opposite directions: if an item is counted in the half it opened, the metric falls from 23.6 days to 7.6 days; if it is counted in the half it closed, it rises from 13.2 days to 20.0 days. Both are calculated correctly. The answer to whether the metric improved depends not on the data but on the grouping rule.

The bottom two lines describe the queue: while the item count falls from 24 to 21, the mean age rises from 30.4 to 59.8 and the median from 19 to 37 — the queue got smaller, and what is in it got older. The source group table gives the reason: pipeline items close in 13.7 days and leave five open items, non-pipeline items close in 30.2 days and sixteen of them wait at 75.3 days of age. The mean is a mixture of these two groups; when the mix ratio changes, the mean changes without any group getting faster. The 14 items opened in the second half never enter the metric; when the 13 invalid items are removed, the mean rises to 18.8 days.

Ways of Improving the Metric That Break the Measure

AC19. All three interventions change only the calculation rule or a record field; none of them touches a line of code. The acceptance threshold is 90 days and is fictional. AC20. Reclassification writes the item into a lower severity class and widens its target duration.

// metrics/intervention.mjs — three interventions that improve the metric are run on the same set.
import { FINDINGS, PERIOD, TARGET, closed, gen } from "./finding.mjs";

const mean = (a) => a.reduce((t, x) => t + x, 0) / a.length;
const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(p * (a.length - 1))];
const duration = (f) => f.closedAt - f.openedAt;
const openItems = FINDINGS.filter((f) => !f.closedAt), ages = openItems.map((f) => PERIOD - f.openedAt);

// 1) The clock starts at verification, not at opening; an item without a stamp never enters the metric.
const stamped = closed.filter((f) => f.verifiedAt !== null);
const A = mean(closed.map(duration)), B = mean(stamped.map(duration));
const C = mean(stamped.map((f) => f.closedAt - f.verifiedAt));

// 2) Open items older than ninety days are closed as accepted risk (the threshold is fictional).
const accepted = openItems.filter((f) => PERIOD - f.openedAt > 90);
const remaining = openItems.filter((f) => PERIOD - f.openedAt <= 90).map((f) => PERIOD - f.openedAt);
const D = mean([...closed.map(duration), ...accepted.map((f) => PERIOD - f.openedAt)]);

// 3) An item that missed its target is written into a lower severity class; its target duration widens.
const LOWER = { critical: "high", high: "medium", medium: "low", low: "low" };
const newTarget = (f) => duration(f) > f.target ? TARGET[LOWER[f.severity]] : f.target;
const before = closed.filter((f) => duration(f) <= f.target).length;
const after = closed.filter((f) => duration(f) <= newTarget(f)).length;

const missing = closed.length - stamped.length, g = (v) => v.toFixed(1);
const y = gen([34, 9, 22, 11, 11]);
console.log(y("intervention", "items", "metric", "before", "after"));
console.log(y("1 clock starts at verification", missing, "mean remediation", g(A), g(C)));
console.log(y("  a) unstamped items drop out", missing, "mean remediation", g(A), g(B)));
console.log(y("  b) clock is rolled back", stamped.length, "mean remediation", g(B), g(C)));
console.log(y("2 aged items closed as accepted", accepted.length, "mean open finding age",
  g(mean(ages)), g(mean(remaining))));
console.log(y("  same intervention", accepted.length, "mean remediation", g(A), g(D)));
console.log(y("3 missed items go one class down", after - before, "within-target rate",
  `${g(100 * before / closed.length)}%`, `${g(100 * after / closed.length)}%`));

console.log(`\nmedian open finding age ${percentile(ages, .5)} -> ${percentile(remaining, .5)}, oldest open item ` +
  `${Math.max(...ages)} -> ${Math.max(...remaining)} days`);
console.log(`of the ${accepted.length} items closed as accepted, ` +
  `${accepted.filter((f) => f.severity === "critical").length} are critical severity; ` +
  `additional items actually fixed across the three interventions: 0`);

intervention                          items                metric     before      after
1 clock starts at verification           16      mean remediation       17.0       12.6
  a) unstamped items drop out            16      mean remediation       17.0       14.6
  b) clock is rolled back               105      mean remediation       14.6       12.6
2 aged items closed as accepted           7 mean open finding age       59.8       23.4
  same intervention                       7      mean remediation       17.0       23.3
3 missed items go one class down         15    within-target rate      81.8%      94.2%

median open finding age 37 -> 16, oldest open item 176 -> 87 days
of the 7 items closed as accepted, 2 are critical severity; additional items actually fixed across the three interventions: 0

The first intervention brings the mean from 17.0 down to 12.6 days; the real information is in how the drop breaks down. Starting the clock at verification gains two days, from 14.6 to 12.6. The big part comes from the 16 items without a verification stamp leaving the base of the metric, and those items are from the slowest group. What improved is the narrowing of the measurement’s scope.

The second intervention sets the two metrics against each other. When the seven items older than ninety days are converted to accepted risk, the open finding age falls from 59.8 to 23.4 days, the median from 37 to 16, and the oldest item from 176 days to 87 days. If the same closures are counted in the remediation time, the mean rises to 23.3 days; if they are not counted, it does not move at all and both metrics seem to have improved at once. Two of the seven items are critical. The third intervention is the cheapest: when the 15 items that missed their target are written into a lower class, the within-target rate rises from 81.8% to 94.2%.

The sum of the three interventions is zero additional items actually fixed. The rule follows from this: a metric that can be improved without touching the thing it measures is not a decision tool but a report line. Two more numbers are written next to the metric — the number of items that do not enter the calculation and the date the calculation rule last changed.

Summary

• Mean time to remediate is 17.0 days and the median is 6 days; the 30 items above the mean carry 75% of the total waiting days. The mean hides the queue.
• Open age is calculated from a separate base: 21 items, a mean of 59.8 days, the oldest 176. While the queue falls from 24 items to 21, the mean age rises from 30.4 to 59.8.
• The same set gives opposite directions under two grouping rules: it falls from 23.6 to 7.6 or rises from 13.2 to 20.0; a metric whose rule is not written down cannot be read.
• Of eight questions, five are answered fully, one partially, and two are unanswered; all 25 items missing the verification stamp come from the two slowest sources.
• The three interventions improve the metrics and fix zero items: the mean goes to 12.6 days, the open age to 23.4 days, and the within-target rate to 94.2%; two of the seven items converted to accepted risk are critical.

Next Step

Both metrics were calculated and shown to be breakable. What remains is the item the metrics hide: the queue that does not close. Sixteen open items come from non-pipeline sources, are 75.3 days old on average, and are not tied to a plan.

The next lesson takes up this queue as security debt and runs the planned closure: in what order and with what capacity the debt items close, how long the duration of an accepted risk is, and where the point that will reassess the item when that duration expires is written.