--- title: 'Application Security Metrics' source: 'https://academia.sh/en/courses/application-detection-and-compliance/application-security-metrics' course: 'Detection, Response and Compliance' language: en updated: '2026-10-07T04:31:05+00:00' license: 'CC BY-SA 4.0' objectives: - 'Calculates mean time to remediate and open finding age on a finding set and separates the queue the mean hides with the median, p90, and maximum' - 'Runs the same data under two grouping rules and shows that the direction of the metric changes with the rule' - 'Runs the three interventions that improve the metric and writes down the number of items all three actually fixed' --- # Application Security Metrics Of the 142 findings, the 121 that closed have a mean time to remediate of 17.0 days and a median of 6 days: the 30 items above the mean carry 75% of the total days. The same set gives opposite directions under two grouping rules — counted in the half it opened it falls from 23.6 to 7.6, counted in the half it closed it rises from 13.2 to 20.0 — and while the open queue falls from 24 items to 21, its mean age rises from 30.4 days to 59.8 days. Of eight questions, five are answered fully, one partially, and two are unanswered. Three interventions that improve the metric fix zero items: the mean goes to 12.6 days, the open finding age to 23.4 days, and the within-target rate to 94.2%. The previous lesson measured where evidence is produced and separated the items into those that answer the whole period, the last ninety days, or a single day; what was measured was the **existence** of evidence. Some of the audit items, however, ask not for existence but for a **number**: was the finding closed within its target time, and how long has the item that was not closed been open. This lesson calculates those two numbers — **mean time to remediate** and **open finding age** — at the scale of one application team. The Governance, Risk and Compliance course covered the same metrics at the program level; that queue is the input here. The question asked is narrow: what the metrics **say**, what they **do not say**, and how many questions what they do not say leaves unanswered. ## Finding Set and Question Set **AC14.** The finding set is 142 items, comes from five sources, belongs to a 180-day period, and is a **model** of the subscriber portal team of the fictional regional measurement network; the generator is in this block, and the seed is visible. **AC15.** An item can have three timestamps: **opened** exists on all of them, **verification** is never written in some sources, and **closure** exists only on an item that closed. **AC16.** The target durations of the severity classes are **data** (7, 21, 45, and 90 days) and are fictional. **AC17.** Midway through the period the priority rule changed: items from the pipeline were moved up, items from outside the pipeline were held back — this is an assumption of the model. ```js // metrics/finding.mjs — the finding set is a MODEL: fictional regional measurement network, a single 180-day period. // The generator is written here and the seed is visible; the set comes out the same on every run. export const gen = (B) => (...h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); export const PERIOD = 180, SEED = 20250, PIPELINE_SOURCES = ["dependency", "static", "image"]; let s = SEED; const rnd = () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; // source | findings | base days | triage days | probability the verification stamp is missing const SOURCES = [["dependency", 44, 16, 1, 0], ["static", 34, 26, 2, 0], ["image", 24, 22, 2, 0], ["audit", 16, 52, 7, .5], ["report", 24, 70, 11, .7]]; export const TARGET = { critical: 7, high: 21, medium: 45, low: 90 }; const FACTOR = { critical: .4, high: .7, medium: 1, low: 1.4 }; const ORDER = { pipeline: "medium low medium high low medium critical high".split(" "), external: "high medium critical high low medium high critical".split(" ") }; export const FINDINGS = []; for (const [source, total, base, triage, missing] of SOURCES) { for (let i = 0; i < total; i++) { const inPipeline = PIPELINE_SOURCES.includes(source); const severity = ORDER[inPipeline ? "pipeline" : "external"][Math.floor(rnd() * 8)]; const openedAt = 1 + Math.floor(rnd() * PERIOD); // The duration distribution is skewed: most items take a short time, a few take very long. let duration = base * FACTOR[severity] * (0.15 + 4 * Math.pow(rnd(), 3.5)); // The priority rule changed midway through the period: pipeline items moved up, the others held back. duration *= inPipeline ? (openedAt > 90 ? .45 : 1) : (openedAt > 90 ? 3.2 : 1.8); const kind = inPipeline && rnd() < .12 ? "invalid" : "fixed"; duration = Math.max(1, Math.round(kind === "invalid" ? duration * .15 : duration)); const closedAt = openedAt + duration <= PERIOD ? openedAt + duration : null; const d = rnd() < missing ? null : openedAt + Math.round(triage * (.4 + 1.2 * rnd())); FINDINGS.push({ source, severity, target: TARGET[severity], openedAt, closedAt, kind, verifiedAt: d === null ? null : Math.min(d, closedAt ?? PERIOD) }); } } export const closed = FINDINGS.filter((f) => f.closedAt !== null); export const unstamped = FINDINGS.filter((f) => f.verifiedAt === null); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model: ${FINDINGS.length} findings, ${SOURCES.length} sources, ${PERIOD} days, seed ${SEED}`); console.log(`${closed.length} items closed, ${FINDINGS.length - closed.length} items still open`); console.log(`verification stamp missing on ${unstamped.length} items: ` + SOURCES.map(([k]) => `${k} ${FINDINGS.filter((f) => f.source === k && !f.verifiedAt).length}/` + `${FINDINGS.filter((f) => f.source === k).length}`).join(", ")); // Questions to be asked at the end of the period, the record field each one needs, and the number of items carrying that field. const K = closed.length, D = FINDINGS.length - unstamped.length; const QUESTIONS = [["mean time to remediate", "opened+closed", K], ["within-target rate", "opened+closed", K], ["age of an open item", "opened", FINDINGS.length], ["how much of the duration was triage", "verification", D], ["how many of the closed were fixed", "closure kind", K], ["how many times an item was reopened", "state transition", 0], ["which change made the fix", "change id", 0], ["why the metric changed", "opened+source", K]]; const answer = (t) => t === 0 ? "none" : t >= K ? "full" : "partial"; const z = gen([36, 20, 10, 8]); console.log("\n" + z("question", "record field needed", "carrying", "answer")); for (const [m, d, t] of QUESTIONS) console.log(z(m, d, t, answer(t))); const n = (c) => QUESTIONS.filter(([, , t]) => answer(t) === c).length; console.log(`\nof ${QUESTIONS.length} questions, ${n("full")} are answered fully, ${n("partial")} partially, ` + `${n("none")} are unanswered`); } ``` ``` model: 142 findings, 5 sources, 180 days, seed 20250 121 items closed, 21 items still open verification stamp missing on 25 items: dependency 0/44, static 0/34, image 0/24, audit 10/16, report 15/24 question record field needed carrying answer mean time to remediate opened+closed 121 full within-target rate opened+closed 121 full age of an open item opened 142 full how much of the duration was triage verification 117 partial how many of the closed were fixed closure kind 121 full how many times an item was reopened state transition 0 none which change made the fix change id 0 none why the metric changed opened+source 121 full of 8 questions, 5 are answered fully, 1 partially, 2 are unanswered ``` The third line answers the question the previous lesson left behind. The 25 items missing the verification stamp are not randomly chosen: it is missing on none of the 102 items from the pipeline, on 10 of those from the audit log, and on 15 of those from the external report intake. **The place where the stamp is missing is the same as the place where the duration is longest.** The two unanswered questions are of the same class: **they are the questions that audit the metric.** A reopened item appears in the metric not as one long duration but as two short ones; if the change that closed it is unknown, it cannot be verified that the closure was a fix. The cost of making them answerable — assuming four state transitions per item — is 568 additional records and a link to be built between the finding and the change. ## Two Metrics, One Set **AC18.** Mean time to remediate is calculated only from **closed** items; an open item never enters the metric. Open age, on the other hand, is an **instantaneous** measurement. The base of the two metrics is therefore not the same set, and the two can move in opposite directions in the same period. ```js // metrics/metrics.mjs — two metrics are calculated from the same set: mean time to remediate and open finding age. import { FINDINGS, PERIOD, PIPELINE_SOURCES, closed, gen } from "./finding.mjs"; const mean = (a) => a.reduce((t, x) => t + x, 0) / a.length; const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(p * (a.length - 1))]; const duration = (f) => f.closedAt - f.openedAt; const openAt = (t) => FINDINGS.filter((f) => f.openedAt <= t && (!f.closedAt || f.closedAt > t)); const age = (t) => openAt(t).map((f) => t - f.openedAt); const y = gen([26, 8, 10, 9, 8, 10]); console.log(y("metric (days)", "items", "mean", "median", "p90", "max")); for (const [name, a] of [["time to remediate", closed.map(duration)], ["open finding age (day 180)", age(PERIOD)]]) console.log(y(name, a.length, mean(a).toFixed(1), percentile(a, .5), percentile(a, .9), Math.max(...a))); const d = closed.map(duration), m = mean(d), above = d.filter((x) => x > m); console.log(`\n${above.length}/${d.length} items above the mean (${m.toFixed(1)}) carry ` + `${(100 * above.reduce((t, x) => t + x, 0) / d.reduce((t, x) => t + x, 0)).toFixed(0)}% of the total days`); // The same set under two grouping rules, then a snapshot of the open queue at two measurement points. const z = gen([30, 12, 12, 12]); console.log("\n" + z("grouping rule", "days 1-90", "days 91-180", "direction")); for (const [name, f] of [["counted in the half it opened", (x) => x.openedAt], ["counted in the half it closed", (x) => x.closedAt]]) { const a = mean(closed.filter((x) => f(x) <= 90).map(duration)); const c = mean(closed.filter((x) => f(x) > 90).map(duration)); console.log(z(name, a.toFixed(1), c.toFixed(1), c < a ? "falls" : "rises")); } for (const t of [90, PERIOD]) console.log(z(`open queue (day ${t})`, `${age(t).length} items`, `mean ${mean(age(t)).toFixed(1)}`, `median ${percentile(age(t), .5)}`)); const w = gen([16, 9, 18, 8, 22]); console.log("\n" + w("source group", "closed", "mean remediation", "open", "mean open finding age")); for (const [name, f] of [["pipeline", (x) => PIPELINE_SOURCES.includes(x.source)], ["non-pipeline", (x) => !PIPELINE_SOURCES.includes(x.source)]]) { const kk = closed.filter(f), aa = openAt(PERIOD).filter(f); console.log(w(name, kk.length, mean(kk.map(duration)).toFixed(1), aa.length, mean(aa.map((x) => PERIOD - x.openedAt)).toFixed(1))); } const h2 = FINDINGS.filter((x) => x.openedAt > 90 && !x.closedAt); console.log(`\n${h2.length} items opened between days 91 and 180 are still open and never enter the metric; ` + `${h2.filter((x) => !PIPELINE_SOURCES.includes(x.source)).length} of them come from non-pipeline sources. ` + `Removing the ${FINDINGS.filter((x) => x.kind === "invalid").length} items counted as invalid moves the mean ` + `${m.toFixed(1)} -> ${mean(closed.filter((x) => x.kind === "fixed").map(duration)).toFixed(1)} days`); ``` ``` metric (days) items mean median p90 max time to remediate 121 17.0 6 57 116 open finding age (day 180) 21 59.8 37 148 176 30/121 items above the mean (17.0) carry 75% of the total days grouping rule days 1-90 days 91-180 direction counted in the half it opened 23.6 7.6 falls counted in the half it closed 13.2 20.0 rises open queue (day 90) 24 items mean 30.4 median 19 open queue (day 180) 21 items mean 59.8 median 37 source group closed mean remediation open mean open finding age pipeline 97 13.7 5 10.0 non-pipeline 24 30.2 16 75.3 14 items opened between days 91 and 180 are still open and never enter the metric; 9 of them come from non-pipeline sources. Removing the 13 items counted as invalid moves the mean 17.0 -> 18.8 days ``` The first line is the central flaw of the metric: the mean is **17.0** days, the median is **6** days. The mean describes no item; 30 of the 121 items are above the mean and carry **75%** of the total waiting days. On the open side the distribution is also skewed, for a different reason: a long item never closes, so it does not enter the metric — it **accumulates**. The second table is the lesson's harshest number. The same 121 items, two grouping rules, two opposite directions: if an item is counted in the half it opened, the metric **falls** from 23.6 days to 7.6 days; if it is counted in the half it closed, it **rises** from 13.2 days to 20.0 days. Both are calculated correctly. **The answer to whether the metric improved depends not on the data but on the grouping rule.** The bottom two lines describe the queue: while the item count falls from 24 to 21, the mean age rises from 30.4 to 59.8 and the median from 19 to 37 — the queue got smaller, and what is in it got older. The source group table gives the reason: pipeline items close in 13.7 days and leave five open items, non-pipeline items close in 30.2 days and sixteen of them wait at 75.3 days of age. The mean is a mixture of these two groups; **when the mix ratio changes, the mean changes without any group getting faster.** The 14 items opened in the second half never enter the metric; when the 13 invalid items are removed, the mean rises to 18.8 days. ## Ways of Improving the Metric That Break the Measure **AC19.** All three interventions change only the **calculation rule** or a record field; none of them touches a line of code. The acceptance threshold is 90 days and is fictional. **AC20.** Reclassification writes the item into a lower severity class and widens its target duration. ```js // metrics/intervention.mjs — three interventions that improve the metric are run on the same set. import { FINDINGS, PERIOD, TARGET, closed, gen } from "./finding.mjs"; const mean = (a) => a.reduce((t, x) => t + x, 0) / a.length; const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(p * (a.length - 1))]; const duration = (f) => f.closedAt - f.openedAt; const openItems = FINDINGS.filter((f) => !f.closedAt), ages = openItems.map((f) => PERIOD - f.openedAt); // 1) The clock starts at verification, not at opening; an item without a stamp never enters the metric. const stamped = closed.filter((f) => f.verifiedAt !== null); const A = mean(closed.map(duration)), B = mean(stamped.map(duration)); const C = mean(stamped.map((f) => f.closedAt - f.verifiedAt)); // 2) Open items older than ninety days are closed as accepted risk (the threshold is fictional). const accepted = openItems.filter((f) => PERIOD - f.openedAt > 90); const remaining = openItems.filter((f) => PERIOD - f.openedAt <= 90).map((f) => PERIOD - f.openedAt); const D = mean([...closed.map(duration), ...accepted.map((f) => PERIOD - f.openedAt)]); // 3) An item that missed its target is written into a lower severity class; its target duration widens. const LOWER = { critical: "high", high: "medium", medium: "low", low: "low" }; const newTarget = (f) => duration(f) > f.target ? TARGET[LOWER[f.severity]] : f.target; const before = closed.filter((f) => duration(f) <= f.target).length; const after = closed.filter((f) => duration(f) <= newTarget(f)).length; const missing = closed.length - stamped.length, g = (v) => v.toFixed(1); const y = gen([34, 9, 22, 11, 11]); console.log(y("intervention", "items", "metric", "before", "after")); console.log(y("1 clock starts at verification", missing, "mean remediation", g(A), g(C))); console.log(y(" a) unstamped items drop out", missing, "mean remediation", g(A), g(B))); console.log(y(" b) clock is rolled back", stamped.length, "mean remediation", g(B), g(C))); console.log(y("2 aged items closed as accepted", accepted.length, "mean open finding age", g(mean(ages)), g(mean(remaining)))); console.log(y(" same intervention", accepted.length, "mean remediation", g(A), g(D))); console.log(y("3 missed items go one class down", after - before, "within-target rate", `${g(100 * before / closed.length)}%`, `${g(100 * after / closed.length)}%`)); console.log(`\nmedian open finding age ${percentile(ages, .5)} -> ${percentile(remaining, .5)}, oldest open item ` + `${Math.max(...ages)} -> ${Math.max(...remaining)} days`); console.log(`of the ${accepted.length} items closed as accepted, ` + `${accepted.filter((f) => f.severity === "critical").length} are critical severity; ` + `additional items actually fixed across the three interventions: 0`); ``` ``` intervention items metric before after 1 clock starts at verification 16 mean remediation 17.0 12.6 a) unstamped items drop out 16 mean remediation 17.0 14.6 b) clock is rolled back 105 mean remediation 14.6 12.6 2 aged items closed as accepted 7 mean open finding age 59.8 23.4 same intervention 7 mean remediation 17.0 23.3 3 missed items go one class down 15 within-target rate 81.8% 94.2% median open finding age 37 -> 16, oldest open item 176 -> 87 days of the 7 items closed as accepted, 2 are critical severity; additional items actually fixed across the three interventions: 0 ``` The first intervention brings the mean from 17.0 down to 12.6 days; the real information is in how the drop breaks down. Starting the clock at verification gains two days, from 14.6 to 12.6. The big part comes from **the 16 items without a verification stamp leaving the base of the metric**, and those items are from the slowest group. What improved is the narrowing of the measurement's scope. The second intervention sets the two metrics against each other. When the seven items older than ninety days are converted to accepted risk, the open finding age falls from 59.8 to 23.4 days, the median from 37 to 16, and the oldest item from 176 days to 87 days. If the same closures are counted in the remediation time, the mean rises to 23.3 days; if they are not counted, it does not move at all and both metrics seem to have improved at once. Two of the seven items are critical. The third intervention is the cheapest: when the 15 items that missed their target are written into a lower class, the within-target rate rises from 81.8% to 94.2%. The sum of the three interventions is zero additional items actually fixed. The rule follows from this: **a metric that can be improved without touching the thing it measures is not a decision tool but a report line.** Two more numbers are written next to the metric — the number of items that **do not enter** the calculation and the date the calculation rule last changed. ## Summary - Mean time to remediate is 17.0 days and the median is 6 days; the 30 items above the mean carry 75% of the total waiting days. The mean hides the queue. - Open age is calculated from a separate base: 21 items, a mean of 59.8 days, the oldest 176. While the queue falls from 24 items to 21, the mean age rises from 30.4 to 59.8. - The same set gives opposite directions under two grouping rules: it falls from 23.6 to 7.6 or rises from 13.2 to 20.0; a metric whose rule is not written down cannot be read. - Of eight questions, five are answered fully, one partially, and two are unanswered; all 25 items missing the verification stamp come from the two slowest sources. - The three interventions improve the metrics and fix zero items: the mean goes to 12.6 days, the open age to 23.4 days, and the within-target rate to 94.2%; two of the seven items converted to accepted risk are critical. ## Next Step Both metrics were calculated and shown to be breakable. What remains is the item the metrics hide: the queue that does not close. Sixteen open items come from non-pipeline sources, are 75.3 days old on average, and are not tied to a plan. The next lesson takes up this queue as **security debt** and runs the planned closure: in what order and with what capacity the debt items close, how long the **duration** of an accepted risk is, and where the point that will reassess the item when that duration expires is written.