--- title: 'Finding Prioritization' source: 'https://academia.sh/en/courses/security-automation/finding-prioritization' course: 'Security Automation and the Supply Chain' language: en updated: '2026-09-26T03:02:40+00:00' license: 'CC BY-SA 4.0' objectives: - 'Ranks the same finding set three times by three criteria and splits the top-ten lists into three numbers: on all three, on two, on only one' - 'Measures the reversed-pair rate between rankings and the largest position shift' - 'Fills finite capacity with four rankings and counts two losses: items with no working path and findings that appear in no ranking' --- # Finding Prioritization Findings from every audit in the course are gathered into a single set and ranked three times by severity, exploitability, and exposure; the measure is not the ranking itself but the difference between them: how many pairs reverse order, how many items land in the top ten on all three, how far the most-shifted item moves, and what finite capacity leaves out under which ranking. The policy has run, and thirty flagged components remain. This course's previous ten measurements each left their own list too: scan findings, matching items, the questions the inventory could not answer, the provenance document's untestable fields. All of them are separate lists, and all of them land in the same queue. Since the queue is finite, an order has to come from somewhere. This lesson applies three criteria separately: the item's own weight, whether a working path exists, and how many things are affected. The Testing Process topic of the Security Tooling and Penetration Testing course measured the manual verification of a single finding; that scoring is not repeated here. The measure is not the ranking itself, it is **the difference between rankings.** **SC25.** The combined set is fictional: thirty-four findings are gathered into a single list and modeled with `node`; the seed `330941` is visible. **SC26.** Severity is given directly; exploitability is computed from three properties (is it called, is it externally reachable, does it require authorization), exposure from three properties (is it in production, how many services, how many endpoints). The tie-breaker is the finding number. ## Same Set, Three Rankings ```js // measurement-network/ranking.mjs — the same finding set ranked three times by three criteria import { mkdirSync, writeFileSync } from "node:fs"; import { join } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-sc"); mkdirSync(ROOT, { recursive: true }); const rng = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; const r = rng(330941); // seed is visible // Fictional combined set: thirty-four findings from the course's eleven measurements are gathered into a single list. const SOURCES = ["static", "running-app", "dependency", "secret-scan", "image-layer", "template", "provenance", "policy"]; const FINDINGS = Array.from({ length: 34 }, (_, i) => { const b = { id: i + 1, source: SOURCES[Math.floor(r() * SOURCES.length)], severity: 3 + Math.floor(r() * 8), // criterion 1: the item's own weight calledAtRuntime: r() < 0.62, externallyReachable: r() < 0.45, authRequired: r() < 0.55, inProduction: r() < 0.7, services: 1 + Math.floor(r() * 3), endpoints: 1 + Math.floor(r() * 40) }; // Criterion 2: does a working path exist in this codebase. Criterion 3: how many things are affected. b.exploitability = (b.calledAtRuntime ? 4 : 0) + (b.externallyReachable ? 4 : 0) + (b.authRequired ? 0 : 2); b.exposure = (b.inProduction ? 4 : 0) + b.services + (b.endpoints >= 25 ? 3 : b.endpoints >= 10 ? 2 : 1); return b; }); writeFileSync(join(ROOT, "findings.json"), JSON.stringify(FINDINGS)); // Three rankings. The tie-breaker is the finding number: the ranking stays the same run to run. const rank = (field) => [...FINDINGS].sort((a, b) => b[field] - a[field] || a.id - b.id); const CRITERIA = [["severity", "severity"], ["exploitability", "exploitability"], ["exposure", "exposure"]]; const RANKINGS = Object.fromEntries(CRITERIA.map(([name, field]) => [name, rank(field)])); const N = 10; const S = [6, 22, 22, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(S[i])).join("")); row(["rank", "severity", "exploitability", "exposure"]); const format = (b, field) => `#${String(b.id).padStart(2, "0")} ${b.source} ${b[field]}`; for (let i = 0; i < N; i++) row([i + 1, format(RANKINGS.severity[i], "severity"), format(RANKINGS.exploitability[i], "exploitability"), format(RANKINGS.exposure[i], "exposure")]); const topTen = Object.fromEntries(CRITERIA.map(([name]) => [name, new Set(RANKINGS[name].slice(0, N).map((b) => b.id))])); const sets = Object.values(topTen); const countIn = (id) => sets.filter((k) => k.has(id)).length; const union = [...new Set(sets.flatMap((k) => [...k]))]; console.log(""); console.log(`findings ${FINDINGS.length}; union of top-${N} lists ${union.length} items, ` + `intersection ${union.filter((id) => countIn(id) === 3).length} items; seed 330941`); console.log(`in the top ${N} on all three: ${union.filter((id) => countIn(id) === 3).length}, ` + `on two: ${union.filter((id) => countIn(id) === 2).length}, ` + `on only one: ${union.filter((id) => countIn(id) === 1).length}`); console.log(`the one item in the top ${N} on all three: ` + union.filter((id) => countIn(id) === 3).map((id) => `#${id}`).join(", ") + `; findings in no criterion's top ${N} ${FINDINGS.length - union.length}`); ``` ``` rank severity exploitability exposure 1 #06 provenance 10 #04 dependency 10 #09 provenance 10 2 #14 dependency 10 #06 provenance 10 #21 running-app 10 3 #20 static 10 #15 provenance 10 #25 provenance 10 4 #25 provenance 10 #22 image-layer 10 #27 dependency 10 5 #27 dependency 10 #24 secret-scan 10 #31 template 10 6 #34 template 10 #25 provenance 10 #05 policy 9 7 #07 policy 9 #03 template 8 #16 provenance 9 8 #08 provenance 9 #09 provenance 8 #18 running-app 9 9 #04 dependency 8 #13 policy 8 #22 image-layer 9 10 #05 policy 8 #16 provenance 8 #29 provenance 9 findings 34; union of top-10 lists 21 items, intersection 1 items; seed 330941 in the top 10 on all three: 1, on two: 7, on only one: 13 the one item in the top 10 on all three: #25; findings in no criterion's top 10 13 ``` **The three columns are three views of the same items, and only one item lands in the top ten on all three: #25.** The union of the three lists is 21 items — "the top ten" means picking ten out of twenty-one items. **13 items of the union** land in the top ten on only one criterion; the choice of criterion can put those items at the front of the queue or at the back of it. The queue's silent portion is in the same table: **13 of thirty-four findings** land in the top ten on no criterion at all. ## How Many Places the Ranking Changes The top-ten comparison only sees the top of the list. The real question is in **how many pairs** two rankings disagree with each other. ```js // measurement-network/rank-shift.mjs — how many places the ranking changes between three rankings import { readFileSync } from "node:fs"; import { join } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-sc"); const FINDINGS = JSON.parse(readFileSync(join(ROOT, "findings.json"))); const CRITERIA = [["severity", "severity"], ["exploitability", "exploitability"], ["exposure", "exposure"]]; const rank = (field) => [...FINDINGS].sort((a, b) => b[field] - a[field] || a.id - b.id); // Position: the item's place in that ranking (1 is the top). The tie-breaker is the finding number. const POSITION = Object.fromEntries(CRITERIA.map(([name, field]) => [name, new Map(rank(field).map((b, i) => [b.id, i + 1]))])); const topTen = Object.fromEntries(CRITERIA.map(([name]) => [name, new Set(rank(CRITERIA.find(([n]) => n === name)[1]).slice(0, 10).map((b) => b.id))])); const PAIRS = [["severity", "exploitability"], ["severity", "exposure"], ["exploitability", "exposure"]]; const S = [30, 14, 9, 14, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(S[i])).join("")); row(["comparison", "reversed", "rate", "top-10 shared", "largest shift"]); const totalPairs = (FINDINGS.length * (FINDINGS.length - 1)) / 2; for (const [x, y] of PAIRS) { let reversed = 0, maxShift = 0; for (let i = 0; i < FINDINGS.length; i++) for (let j = i + 1; j < FINDINGS.length; j++) { const [a, b] = [FINDINGS[i].id, FINDINGS[j].id]; const isLeftFirst = POSITION[x].get(a) < POSITION[x].get(b); if (isLeftFirst !== (POSITION[y].get(a) < POSITION[y].get(b))) reversed++; } for (const b of FINDINGS) maxShift = Math.max(maxShift, Math.abs(POSITION[x].get(b.id) - POSITION[y].get(b.id))); const shared = [...topTen[x]].filter((id) => topTen[y].has(id)).length; row([`${x} - ${y}`, `${reversed}/${totalPairs}`, `${Math.round((100 * reversed) / totalPairs)}%`, `${shared}/10`, `${maxShift} places`]); } // The five items that shift the most: the difference between the highest and lowest position across three rankings. const spreadOf = (b) => { const positions = CRITERIA.map(([name]) => POSITION[name].get(b.id)); return { b, positions, spread: Math.max(...positions) - Math.min(...positions) }; }; const T = [8, 16, 12, 12, 0]; const row2 = (s) => console.log(s.map((v, i) => String(v).padEnd(T[i])).join("")); console.log(""); row2(["finding", "source", "severity", "exploit.", "exposure"]); for (const { b, positions } of FINDINGS.map(spreadOf).sort((x, y) => y.spread - x.spread || x.b.id - y.b.id).slice(0, 5)) row2([`#${b.id}`, b.source, `${b.severity} (${positions[0]}.)`, `${b.exploitability} (${positions[1]}.)`, `${b.exposure} (${positions[2]}.)`]); console.log(""); const mostSpread = FINDINGS.map(spreadOf).sort((x, y) => y.spread - x.spread)[0]; console.log(`${FINDINGS.filter((b) => spreadOf(b).spread >= 15).length} of the thirty-four items shift ` + `more than 15 places across the three rankings; the most-spread is #${mostSpread.b.id} with ${mostSpread.spread} places`); ``` ``` comparison reversed rate top-10 shared largest shift severity - exploitability 295/561 53% 3/10 31 places severity - exposure 290/561 52% 3/10 29 places exploitability - exposure 277/561 49% 4/10 29 places finding source severity exploit. exposure #14 dependency 10 (2.) 0 (33.) 7 (25.) #15 provenance 5 (26.) 10 (3.) 3 (32.) #22 image-layer 3 (33.) 10 (4.) 9 (9.) #31 template 3 (34.) 6 (24.) 10 (5.) #9 provenance 4 (29.) 8 (8.) 10 (1.) 21 of the thirty-four items shift more than 15 places across the three rankings; the most-spread is #14 with 31 places ``` The reversed-pair rate falls **between 49% and 53%** across all three comparisons: on roughly half of the 561 pairs, the two criteria disagree with each other. **A list ranked by severity reverses roughly half of a list ranked by exploitability.** The criteria are not approximations of each other; they measure separate things. The second table shows this at the item level. **#14 is second by severity and thirty-third by exploitability** — it is one of the highest-weight items, and no working path leads to it. The opposite end is **#31**: thirty-fourth by severity, fifth by exposure. **21** of the items shift by more than fifteen places. ## Finite Capacity A ranking on its own is not the outcome; what produces the outcome is where the list gets cut off. **SC27.** The combined rank is the sum of an item's positions across the three rankings; there is no weighting. **SC28.** Capacity is twelve items; the fix time of 45 minutes per item is given as data, not measured. ```js // measurement-network/capacity.mjs — finite queue: which ranking takes what into the top twelve import { readFileSync } from "node:fs"; import { join } from "node:path"; const ROOT = join(process.env.TMPDIR ?? "/tmp", "measurement-network-sc"); const FINDINGS = JSON.parse(readFileSync(join(ROOT, "findings.json"))); const CRITERIA = [["severity", "severity"], ["exploitability", "exploitability"], ["exposure", "exposure"]]; const rank = (field) => [...FINDINGS].sort((a, b) => b[field] - a[field] || a.id - b.id); const POSITION = Object.fromEntries(CRITERIA.map(([name, field]) => [name, new Map(rank(field).map((b, i) => [b.id, i + 1]))])); // Combined rank: the sum of positions across the three rankings. No weighting, all three criteria count equally. const totalPosition = (b) => CRITERIA.reduce((n, [name]) => n + POSITION[name].get(b.id), 0); const combined = [...FINDINGS].sort((a, b) => totalPosition(a) - totalPosition(b) || a.id - b.id); const CAPACITY = 12, FIX_MIN = 45; // duration is given as DATA const LISTS = [...CRITERIA.map(([name, field]) => [name, rank(field)]), ["combined", combined]] .map(([name, list]) => [name, list.slice(0, CAPACITY)]); const avg = (set, field) => (set.reduce((n, b) => n + b[field], 0) / set.length).toFixed(1); const S = [20, 14, 14, 15, 0]; const row = (s) => console.log(s.map((v, i) => String(v).padEnd(S[i])).join("")); row(["ranking", "avg. severity", "avg. exploit.", "avg. exposure", "shared w/ combined"]); const combinedIds = new Set(LISTS.at(-1)[1].map((b) => b.id)); for (const [name, set] of LISTS) row([name, avg(set, "severity"), avg(set, "exploitability"), avg(set, "exposure"), `${set.filter((b) => combinedIds.has(b.id)).length}/${CAPACITY}`]); const inNoList = FINDINGS.filter((b) => LISTS.every(([, k]) => k.every((x) => x.id !== b.id))); const noExploitPath = LISTS[0][1].filter((b) => b.exploitability <= 2).length; const excluded = rank("exploitability").slice(0, CAPACITY).filter((b) => !combinedIds.has(b.id)).length; console.log(""); console.log(`capacity ${CAPACITY} items, ${FIX_MIN} min fix time per item (data): ` + `${CAPACITY * FIX_MIN} min; remaining in the queue ${FINDINGS.length - CAPACITY} items`); console.log(`${noExploitPath} of the top ${CAPACITY} ranked by severity alone have no working path; ` + `${excluded} items of the exploitability ranking fall outside the combined ranking`); console.log(`findings in no ranking's top ${CAPACITY} ${inNoList.length}/` + `${FINDINGS.length}; ${inNoList.filter((b) => b.inProduction).length} of them are in production`); ``` ``` ranking avg. severity avg. exploit. avg. exposure shared w/ combined severity 9.2 6.0 7.3 7/12 exploitability 6.8 9.0 7.5 8/12 exposure 6.1 6.3 9.3 7/12 combined 8.0 7.5 8.8 12/12 capacity 12 items, 45 min fix time per item (data): 540 min; remaining in the queue 22 items 2 of the top 12 ranked by severity alone have no working path; 4 items of the exploitability ranking fall outside the combined ranking findings in no ranking's top 12 9/34; 5 of them are in production ``` Each ranking picks the highest by its own criterion and falls to the average on the others: the severity ranking brings an average severity of 9.2, while its average exploitability stays at 6.0. The combined ranking is highest in no single column, but it is above eight on all three. **Combining the three criteria does not give the best list, it gives the least-bad list.** **2 items** of the top twelve ranked by severity alone **have no working path**: ninety minutes are spent, and zero paths are closed. The last line writes the queue's blind spot: **9 findings land in no ranking's top twelve, and 5 of them sit in production.** ## Summary - When the same 34 findings are ranked by three criteria, only one item lands in the top ten on all three; the union is 21 items, 13 of them on only one criterion. - The reversed-pair rate falls between 49% and 53%; 21 of 34 items shift by more than fifteen places. The item that is second by severity is thirty-third by exploitability. - The combined ranking is highest in no single column but above eight on all three; 2 items of the top twelve chosen by severity alone have no working path. - Capacity is 12 items and 540 minutes; 22 items remain in the queue, 9 findings land in no ranking's top twelve, and 5 of them sit in production. ## Course Wrap-Up | Lesson | Scanned / findings | True / false positive / missed | Time taken from pipeline | |---|---|---|---| | Static Application Security Testing | 720 points / 418 findings | 101 / 317 / 62 | 2,085 min, 61 wrongly blocked | | Dynamic Application Security Testing | 84 endpoints / 17 findings | 10 / 7 / 18 | 5,316 min, 443 min nightly | | Dependency Component Analysis | 210 components / 139 matches | 31 / 108 / 12 | 1,870 min, 57 blocked builds | | Secret Scanning in Repository History | 1,800 commits / 677 matches | 260 / 417 / 50 | 57 min scan, 1,668 min review | | Container Image Scanning | 348 components / 144 findings | 144 / not measured / 18 | 1,805 min, 49 blocked builds | | Infrastructure as Code Scanning | 420 items / 55 definite findings | 55 / 0 / 19 | 800 min, 34 blocked builds | | Software Bill of Materials | 73 records / 12 questions | 8 answered / 3 with missing fields / 4 unaskable | 820 s, 41 s in the sparse case | | Build Artifact Signing | 124 artifacts / 8 claims | 3 proven / 5 wrongly passing / 4 forged signatures | 120 artifacts rebuilt | | Provenance Information | 14 fields / 12 builds | 4 verifiable fields / 3 false marks / 7 untestable attestations | 480 s, 9,040 s with rebuilding | | Build Environment Isolation | 18 items / 90 access pairs | 15 required / 75 unneeded / 25 that cannot be closed off | 1,380 s, 30 runs | | Dependency Policy | 48 components / 46 flags | 10 real / 20 false marks / 3 unseen | 270 s, 96 min intervention | | Finding Prioritization | 34 findings / 3 rankings | 1 at the top on all three / 13 on only one / 9 in no list | 540 min, 22 items in the queue | The third column is filled in on all twelve rows; that is the course's rule: **an automated check's value is not the number of findings it produces, it is knowing what it misses and getting back the time it took from the pipeline.** A scan whose misses go unwritten is not an assurance, it is an illusion. Access-control gaps closed with zero findings in two separate audits, 50 secrets matched no pattern at all, 19 violations had no rule, and a maintenance transfer showed up in no field anywhere; unless these numbers are reported, "zero findings" and "never looked" cannot be told apart. The fourth column reads with the same rigor: the time taken from the pipeline is not always paid back. Searching for the provenance document's presence spent 480 seconds and let all four of the four broken requests through. For an audit to count as measured, all three columns have to be written. Code was scanned, components were inventoried, and build artifacts were signed. All of these audits share one limit: every one of them stood **before deployment**. Twenty-two findings stayed in the queue, nine of them entered no list at all, and they go to production exactly as they are. **M25/K04 Runtime Security and Secrets Management** asks what happens after that moment: what protects the application from the instant it starts running was never asked in this course.