Risk-Based Prioritization

The previous four lessons measured catching: checks were distributed across the stages, cost per stage was computed, requirements were converted into a testable form, and competence was spread across teams. All four stopped at the same place — they assumed the found item gets fixed. Shift left’s 250-hour gain sat on top of that assumption, and it was written there in plain terms that it was an upper bound.

This lesson removes that assumption. The same twenty-two developers will do the fixing, and their time is shared with feature work. Once the stack is larger than capacity, the question becomes “which item first,” and that question calls for a sorting criterion. The lesson runs four criteria under the same budget, compares closed items per unit of capacity, and finally separates the set that is left out no matter which criterion is chosen.

One point needs separating from the start: the ordering here is not a risk ordering. Risk ordering works with an event that has not happened and carries a probability term. The items in this stack have already been caught; a found item has no probability of existing, only a cost of being deferred. None of the criteria contain a probability term.

• DA22. The item set is the same as in the previous lessons. The two added fields are fix effort and deferred cost.
• DA23. Fix effort is in fictional developer-days and is a model value. Deferred cost is derived: if the item is not closed now, it carries forward to runtime and pays the undo difference in between.
• DA24. Sprint capacity is the total developer-days the teams set aside for security work; an item is either closed completely or not opened at all, there is no partial closing.
• DA25. The budget is spent in the criterion’s order, and an item that does not fit is skipped — the order does not stop there. This rule works in favor of small-effort items, and its effect shows up in the table.
• DA26. None of the criteria carry a probability or impact term. The orderings are built only from the birth stage, deferred cost, and effort.

Capacity and Deferred Cost

The model carries two new numbers: how many developer-days closing an item takes, and how many hours carrying it forward to runtime adds if it is not closed. The second one is not given directly; it is derived from the previous lesson’s undo cost.

// capacity.mjs — the same 23 items, now with fix effort and the multiplied cost of delay (model)
export const REDO = [0, 3, 8, 2, 6, 24], BASE = 2;
export const cost = (born, caught) => BASE + REDO.slice(born + 1, caught + 1).reduce((s, x) => s + x, 0);
// row: name | birth stage | visibility | fix effort (fictional developer-days)
export const ITEM = `
  no-data-classification        0 100000 5
  undefined-retention-period    0 100001 3
  unclear-personal-data-scope   0 100010 4
  unauthenticated-internal-call 1 011110 6
  weak-session-model            1 010011 4
  trust-boundary-free-flow      1 010000 7
  missing-role-in-authz-model   1 011001 5
  missing-authorization-check   2 001101 2
  embedded-secret-key           2 001110 3
  unvalidated-input             2 001001 2
  personal-data-in-logs         2 001001 1
  incomplete-error-path         2 001000 1
  weak-randomness                2 001000 2
  unbounded-query-size          2 000001 4
  outdated-dependency           3 000110 1
  unsigned-build-output         3 000110 3
  unnecessary-build-tool        3 000100 1
  default-password              4 000011 1
  overly-broad-network-rule     4 000011 2
  unencrypted-backup            4 000010 2
  open-admin-endpoint           4 000011 1
  race-under-scale              5 000001 6
  third-party-disclosed-flaw    5 000001 2`
  .trim().split('\n').map((r) => {
    const [name, d, g, e] = r.trim().split(/\s+/);
    const born = +d, caught = [...g].findIndex((c, i) => i >= born && c === '1');
    // Cost of delay: if the item is not closed now, it carries forward to runtime.
    return { name, born, caught, effort: +e, prevented: cost(born, 5) - cost(born, caught) };
  });
export const SPRINT = 22; // developer-days per sprint set aside for security work
export const TOTAL_EFFORT = ITEM.reduce((s, k) => s + k.effort, 0);
export const TOTAL_PREVENTED = ITEM.reduce((s, k) => s + k.prevented, 0);
// Sort criteria: none contains a probability term, the item has already been caught.
export const CRITERIA = {
  'born-first': (a, b) => a.born - b.born || a.effort - b.effort,
  'high-multiple': (a, b) => b.prevented - a.prevented || a.effort - b.effort,
  'low-effort': (a, b) => a.effort - b.effort || b.prevented - a.prevented,
  'per-unit': (a, b) => b.prevented / b.effort - a.prevented / a.effort,
};
export function distribute(name, budget) {
  const taken = [];
  let spent = 0;
  for (const k of [...ITEM].sort(CRITERIA[name])) if (spent + k.effort <= budget) { taken.push(k); spent += k.effort; }
  return { taken, spent, closed: taken.length,
    prevented: taken.reduce((s, k) => s + k.prevented, 0),
    outside: ITEM.filter((k) => !taken.includes(k)) };
}

// run1.mjs — the same sprint capacity is given to four sorting criteria
import { ITEM, SPRINT, TOTAL_EFFORT, TOTAL_PREVENTED, CRITERIA, distribute } from './capacity.mjs';

console.log(`${ITEM.length} items, ${TOTAL_EFFORT} developer-days effort, ` +
  `${TOTAL_PREVENTED} hours deferred cost; sprint capacity ${SPRINT} days ` +
  `(stack ${(TOTAL_EFFORT / SPRINT).toFixed(1)} sprints)`);
console.log('\ncriterion      closed   spent     prevented hrs  items per day     hours per day    outside');
for (const name of Object.keys(CRITERIA)) {
  const r = distribute(name, SPRINT);
  console.log(name.padEnd(15) + String(r.closed).padEnd(9) + String(r.spent).padEnd(10) +
    String(r.prevented).padEnd(15) + (r.closed / r.spent).toFixed(2).padEnd(18) +
    (r.prevented / r.spent).toFixed(2).padEnd(17) + r.outside.length);
}

console.log('\nclosed items as capacity grows (per criterion)');
console.log('capacity  ' + Object.keys(CRITERIA).map((a) => a.padEnd(14)).join(''));
for (const b of [11, 22, 33, 44, 68])
  console.log(String(b).padEnd(10) +
    Object.keys(CRITERIA).map((a) => {
      const r = distribute(a, b);
      return `${r.closed} items`.padEnd(14);
    }).join(''));

23 items, 68 developer-days effort, 667 hours deferred cost; sprint capacity 22 days (stack 3.1 sprints)

criterion      closed   spent     prevented hrs  items per day     hours per day    outside
born-first     6        22        241            0.27              10.95            17
high-multiple  6        22        241            0.27              10.95            17
low-effort     13       21        359            0.62              17.10            10
per-unit       13       22        391            0.59              17.77            10

closed items as capacity grows (per criterion)
capacity  born-first    high-multiple low-effort    per-unit
11        3 items       3 items       8 items       8 items
22        6 items       6 items       13 items      13 items
33        10 items      10 items      16 items      16 items
44        14 items      14 items      19 items      18 items
68        23 items      23 items      23 items      23 items

The first row gives the scale: 23 items call for 68 developer-days, the sprint capacity is 22 days, so the stack takes 3.1 sprints. This does not mean capacity is insufficient; it means the ordering matters across three sprints.

The first two rows being identical is not a coincidence, and it needs to be read correctly. Deferred cost is derived from the birth stage alone; that is why the “biggest multiplier” ordering is a renamed version of the born-first ordering and carries no independent information. A program that sorts the same measure under two different names believes it is choosing between two options, when it has only one.

The independent information sits in effort, and that is what produces the difference. With the same 22 days, born-first closes 6 items, low-effort closes 13 items; prevented hours per unit rise from 10.95 to 17.10. A difference of nearly double is obtained by changing a single column, and it requires no extra capacity.

The lower table shows that this holds at every budget, but it reverses somewhere: at 44 days the low-effort ordering closes 19 items, the per-unit ordering closes 18. The ratio ordering takes a high-ratio but high-effort item early, and that item later takes the place of two small items. The ordering best for per-unit is not the ordering best for item count; which one is wanted has to be stated in advance. At sixty-eight days all four criteria arrive at the same place — ordering is only a decision while capacity is scarce.

Cost Carried Across Three Sprints

A single-sprint view is incomplete, because the stack closes out over three sprints. The real measure is how many sprints the deferred cost stays open: at the end of each sprint, the hours carried by the still-open items are summed.

// run3.mjs — deferred cost carried across three sprints: what does ordering speed up
import { ITEM, SPRINT, TOTAL_PREVENTED, CRITERIA } from './capacity.mjs';

console.log('criterion      sprint 1 open   sprint 2 open   sprint 3 open   carried over 3 sprints');
for (const name of Object.keys(CRITERIA)) {
  const remaining = [...ITEM].sort(CRITERIA[name]);
  const trace = [];
  let carried = 0;
  for (let s = 0; s < 3; s++) {
    let spent = 0;
    for (const k of [...remaining]) if (spent + k.effort <= SPRINT) { spent += k.effort; remaining.splice(remaining.indexOf(k), 1); }
    const open = remaining.reduce((a, k) => a + k.prevented, 0);
    trace.push(open); carried += open;
  }
  console.log(name.padEnd(15) + trace.map((x) => String(x).padEnd(16)).join('') + carried);
}
console.log(`\nstack total ${TOTAL_PREVENTED} hours; the carried figure measures how many sprints ` +
  'the deferred cost stays open for');

criterion      sprint 1 open   sprint 2 open   sprint 3 open   carried over 3 sprints
born-first     426             158             0               584
high-multiple  426             158             0               584
low-effort     308             163             40              511
per-unit       276             80              0               356

stack total 667 hours; the carried figure measures how many sprints the deferred cost stays open for

Three criteria close the stack out in three sprints, one does not. The low-effort ordering was the criterion that closed the most items in the first sprint, but at the end of the third sprint it still carries a 40-hour gap: because it puts small items first, the high-effort items keep failing to fit into each sprint’s remaining budget and get pushed to the end. An ordering’s success in the first sprint does not say whether it finishes the stack.

The carried column reduces the difference between criteria to a single number. The born-first ordering carries 584 hour-sprints of open cost across three sprints, the per-unit ordering 356. The difference between them is 39 percent, and it is obtained without spending any extra capacity; only the order of the same work has changed. The cost remaining after the first sprint points the same way: 426 hours against 276 hours.

Whose Fate the Ordering Decides

The closed-item count gives the criterion’s value, but the real question is this: when the criterion changes, who is left out. Two criteria can leave out the same number of items, and they can leave out entirely different items.

// run2.mjs — how much does the left-out set change when the ordering changes
import { ITEM, SPRINT, CRITERIA, distribute } from './capacity.mjs';

const names = Object.keys(CRITERIA);
const outside = new Map(names.map((a) => [a, new Set(distribute(a, SPRINT).outside)]));
console.log('shared items in the left-out set between criterion pairs (22-day sprint)');
console.log('               ' + names.map((a) => a.padEnd(14)).join(''));
for (const a of names)
  console.log(a.padEnd(15) + names.map((b) => {
    const shared = [...outside.get(a)].filter((k) => outside.get(b).has(k)).length;
    return String(shared).padEnd(14);
  }).join(''));

const alwaysOut = ITEM.filter((k) => names.every((a) => outside.get(a).has(k)));
const neverOut = ITEM.filter((k) => names.every((a) => !outside.get(a).has(k)));
const switching = ITEM.length - alwaysOut.length - neverOut.length;
console.log(`\n${ITEM.length} items: inside in every criterion ${neverOut.length}, outside in ` +
  `every criterion ${alwaysOut.length}, switching by criterion ${switching}`);
console.log(`the deferred cost of items always left out is ` +
  `${alwaysOut.reduce((s, k) => s + k.prevented, 0)} hours, fix effort ` +
  `${alwaysOut.reduce((s, k) => s + k.effort, 0)} days`);

console.log('\nitems left out under every criterion');
console.log('item                           effort  prevented hrs  hrs/day');
for (const k of [...alwaysOut].sort((a, b) => b.prevented / b.effort - a.prevented / a.effort))
  console.log(k.name.padEnd(31) + String(k.effort).padEnd(8) + String(k.prevented).padEnd(15) +
    (k.prevented / k.effort).toFixed(1));

shared items in the left-out set between criterion pairs (22-day sprint)
               born-first    high-multiple low-effort    per-unit
born-first     17            17            6             6
high-multiple  17            17            6             6
low-effort     6             6             10            9
per-unit       6             6             9             10

23 items: inside in every criterion 2, outside in every criterion 5, switching by criterion 16
the deferred cost of items always left out is 110 hours, fix effort 26 days

items left out under every criterion
item                           effort  prevented hrs  hrs/day
unsigned-build-output          3       30             10.0
unauthenticated-internal-call  6       40             6.7
trust-boundary-free-flow       7       40             5.7
unbounded-query-size           4       0              0.0
race-under-scale               6       0              0.0

The intersection table carries this lesson’s hardest number. Of the 17 items the born-first ordering leaves out and the 10 items the low-effort ordering leaves out, only 6 are shared. Two programs look at the same stack, spend the same budget, and defer largely different items.

The bottom line sums this up in one sentence: of the 23 items, 2 are inside under every criterion, 5 are outside under every criterion, 16 switch sides depending on the criterion. The fate of 70 percent of the stack is decided not by the items’ own properties but by the choice of sorting criterion. A security program’s most decisive decision is therefore not deciding which item matters, but writing down which column it is sorted by; when it is not written down, the decision still gets made, just somewhere nobody sees it.

Left Out Under Every Criterion

The last list shows the residue that is independent of the criterion choice: five items, 110 hours of deferred cost, 26 days of fix effort. These five items are outside for two separate reasons, and each requires a separate decision.

Three of them have high effort. Fixing the trust-boundary-free data flow takes seven days, fixing the unauthenticated internal call takes six days; both prevent forty hours each, but their per-hour ratio sits in the middle of the list, and they cannot get into a 22-day budget under any ordering. The right response to these is not to force the ordering, it is to split the effort: if a single seven-day item can be broken into two three-day steps, it gets onto the list; if it cannot be split, it needs a separate budget. A stack measured with a single sprint’s capacity structurally keeps out high-effort items that cannot be split.

The remaining two are different, and their prevented hours are zero. The race under scale and the unbounded query size are the class set apart in the second lesson: born early, visible only at runtime. These do not fall to the bottom of the ordering, they never enter it, because closing them early prevents nothing. The right decision for these two items is not prioritization; it is building a way to see them cheaply at runtime, and that work is outside this budget. Not keeping zero-value rows separate in a stack list corrupts the ordering itself.

Summary

• The 23-item stack carries 68 developer-days of effort and 667 hours of deferred cost; with 22-day sprint capacity the stack takes 3.1 sprints, and the ordering produces a decision across all three sprints.
• The born-first ordering and the biggest-multiplier ordering give the same result, because deferred cost is derived from the birth stage alone; the two names are a single criterion.
• With the same 22 days, the born-first ordering closes 6 items, the low-effort ordering closes 13; prevented hours per unit rise from 10.95 to 17.10, and this requires no extra capacity.
• The ordering best for per-unit is not the ordering best for item count: at 44 days the low-effort ordering closes 19 items, the ratio ordering 18. The cost carried across three sprints is 584 hour-sprints for the born-first ordering, 356 for the per-unit ordering, and the low-effort ordering, despite being best in the first sprint, leaves a 40-hour gap at the end of the third.
• Of the 23 items, 2 are inside under every criterion, 5 are outside under every criterion, 16 switch sides depending on the criterion; the criterion choice decides the fate of 70 percent of the stack.
• The 5 items left out under every criterion carry 110 hours: three have high effort and cannot get in unless split, two have zero prevented hours and should never enter the ordering at all.

Next Step

This topic turned security from a single stage into a decision with a different price at every stage. The gate’s queue was counted, the per-stage fix cost was derived from the work that gets undone, requirements were split into testability classes, competence’s coverage and depth were measured, and finally limited capacity was distributed under four separate criteria. All of these measurements had a single input: that we already have a caught set of items. Twenty-three items were given from the start, and where they came from was never asked.

The stages that carried the largest share of the gain were design and requirements — more than half of the prevented hours came from there. But which item gets found in a design session is too decisive to be left to the eye of whoever finds it. The third lesson noted that it was not measured where the five abuse cases came from; the fourth lesson noted that the difficulty level a champion could see stood as a ceiling. Both point to the same gap: finding which item is born at the design stage has a systematic method of its own, and that method was never explained. The next topic, Threat Modeling, builds that method: the system’s data flow is written as a graph, the flows that cross a trust boundary are counted, threats are generated by applying a systematic classification to the graph’s elements, and how many of the generated threats are meaningful, how many turn into a countermeasure, is measured.