Impact Assessment

The previous lesson produced 61 evidenced findings, 9 pending candidates, and a list left unvalidated. If this set is going into a remediation queue, it has to be put in order. The order can be built from technical criteria: is the flaw reachable from outside or only from the inside, how many items does it affect, how much privilege does it grant. These three criteria give a single number, and that number produces a ranking.

This lesson shows that this ranking is not the only ranking. The same finding carries a different weight depending on which business the asset it sits on carries, and the answer to “which business matters” changes from one organization to another. Business impact is the finding’s technical weight multiplied by the business the asset carries. What is being measured is not which of the two scorings is correct, but how many positions apart the two orderings put the same list.

• TP19. The finding set is the previous lesson’s 61 evidenced findings, and each finding is tied to one of five asset classes. Both the set and the weights are fictional.
• TP20. The technical score comes from three components: access level (weight 0.45), impact breadth (0.30), and privilege gained (0.25). The result is scaled to a range of 0 to 10.
• TP21. Business weight is an asset class’s value in that context. Two contexts are defined: revenue-and-subscriber-data priority and field-continuity priority.
• TP22. The business score is the product of the technical score and the business weight. The same finding carries two separate business scores across the two contexts, while its technical score stays the same in both.
• TP23. The difference between two rankings is written with four measures: the number of findings that stay in the same position, the mean rank shift, the top-ten list’s shared members, and the rate of pairwise comparisons that invert.
• TP24. The remediation budget is 40 person-days, and the entire set totals 286 person-days. The queue pulls from the ordering and skips any finding that does not fit the budget.

One List, Three Rankings

// scoring.mjs — the same evidenced finding set is scored by technical weight and in two
// separate business contexts. THIS IS A MODEL; the finding set and the weights are fictional.
export const SEED = 20260222, N = 61;
const makeRng = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = makeRng(SEED);

// The previous lesson's 61 evidenced findings. Each finding is tied to an asset class.
export const ASSET = ["subscriber data", "billing", "measurement accuracy", "field operations",
  "internal management"];
const ACCESS = { external: 1.0, credentialed: 0.6, internal: 0.3 };
export const FINDING = Array.from({ length: N }, (_, i) => {
  const e = ["external", "credentialed", "internal"][Math.floor(r() * 3)];
  const breadth = Math.round(r() * 18) / 20, privilege = Math.round(r() * 20) / 20;
  return { id: i + 1, asset: ASSET[Math.floor(r() * ASSET.length)], access: e,
    breadth, privilege, days: 1 + Math.floor(r() * 8),   // remediation cost (person-days)
    technical: +(10 * (0.45 * ACCESS[e] + 0.30 * breadth + 0.25 * privilege)).toFixed(2) };
});

// Business weight: an asset class's weight in that context. The same finding, with the same
// technical score, carries a different business impact in each of the two contexts.
export const CONTEXT = {
  "revenue and subscriber data": { "subscriber data": 1.0, "billing": 0.9, "measurement accuracy": 0.6,
    "field operations": 0.35, "internal management": 0.3 },
  "field continuity": { "subscriber data": 0.4, "billing": 0.45, "measurement accuracy": 0.85,
    "field operations": 1.0, "internal management": 0.5 },
};
export const businessScore = (b, name) => +(b.technical * CONTEXT[name][b.asset]).toFixed(2);

// Ranking: descending by score; the finding id breaks ties.
export const rank = (key) => [...FINDING]
  .sort((a, b) => key(b) - key(a) || a.id - b.id).map((b) => b.id);
export const position = (s) => new Map(s.map((id, i) => [id, i + 1]));

// The difference between two rankings: how many stay in the same spot, the mean shift, the
// largest shift, the top-ten overlap, and the inverted-pair rate.
export function diff(s1, s2) {
  const p1 = position(s1), p2 = position(s2);
  const shift = s1.map((id) => Math.abs(p1.get(id) - p2.get(id)));
  const top10 = new Set(s1.slice(0, 10));
  let inverted = 0, pairs = 0;
  for (let i = 0; i < s1.length; i++) for (let j = i + 1; j < s1.length; j++) {
    pairs++;
    if ((p1.get(s1[i]) - p1.get(s1[j])) * (p2.get(s1[i]) - p2.get(s1[j])) < 0) inverted++;
  }
  return { same: shift.filter((k) => k === 0).length, largest: Math.max(...shift),
    mean: shift.reduce((a, k) => a + k, 0) / shift.length,
    top10Common: s2.slice(0, 10).filter((id) => top10.has(id)).length,
    inverted: +(100 * inverted / pairs).toFixed(1) };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const NAMES = Object.keys(CONTEXT);
  const y1 = gen([23, 8, 14, 30, 21]);
  console.log(`model (seed ${SEED}): ${N} evidenced findings, ${ASSET.length} asset classes`);
  console.log("\n" + y1(["asset class", "findings", "avg technical", ...NAMES]));
  for (const v of ASSET) {
    const l = FINDING.filter((b) => b.asset === v);
    const ot = l.reduce((a, b) => a + b.technical, 0) / l.length;
    console.log(y1([v, l.length, ot.toFixed(2), ...NAMES.map((name) => CONTEXT[name][v].toFixed(2))]));
  }

  const sT = rank((b) => b.technical), sA = rank((b) => businessScore(b, NAMES[0]));
  const sB = rank((b) => businessScore(b, NAMES[1]));
  const y2 = gen([29, 10, 12, 12, 14, 16]);
  console.log("\n" + y2(["comparison", "same rank", "mean shift", "largest", "top-10 common",
    "inverted pair %"]));
  for (const [ad, x, y] of [["technical - revenue context", sT, sA],
    ["technical - field context", sT, sB], ["revenue - field context", sA, sB]]) {
    const f = diff(x, y);
    console.log(y2([ad, f.same, f.mean.toFixed(2), f.largest, `${f.top10Common}/10`, f.inverted]));
  }

  const pT = position(sT), pA = position(sA), pB = position(sB);
  const y3 = gen([8, 18, 11, 8, 9, 9, 9]);
  console.log("\n" + y3(["finding", "asset", "technical", "rank T", "rank A", "rank B", "A-B diff"]));
  const focus = [...FINDING].sort((a, b) => Math.abs(pA.get(b.id) - pB.get(b.id)) -
    Math.abs(pA.get(a.id) - pB.get(a.id))).slice(0, 6);
  for (const b of focus) {
    console.log(y3([b.id, b.asset, b.technical.toFixed(2), pT.get(b.id), pA.get(b.id),
      pB.get(b.id), pA.get(b.id) - pB.get(b.id)]));
  }
}

model (seed 20260222): 61 evidenced findings, 5 asset classes

asset class            findings avg technical   revenue and subscriber data     field continuity
subscriber data              16          5.06                          1.00                 0.40
billing                      13          5.37                          0.90                 0.45
measurement accuracy         14          5.77                          0.60                 0.85
field operations              5          5.24                          0.35                 1.00
internal management          13          5.94                          0.30                 0.50

comparison                    same rank  mean shift     largest top-10 common inverted pair %
technical - revenue context           2       14.79          39          5/10            31.3
technical - field context             1       10.00          30          2/10            20.6
revenue - field context               0       22.23          43          0/10            41.6

finding              asset  technical  rank T   rank A   rank B A-B diff
6         field operations       6.40      21       47        4       43
17        field operations       7.05      17       42        2       40
16        field operations       3.60      51       58       21       37
33        field operations       7.50      10       38        1       37
34         subscriber data       5.22      37       13       46      -33
31         subscriber data       5.28      33       12       44      -32

The last two columns of the first table are this lesson’s actual subject. The same asset class carries a weight ranging from 0.30 to 1.00 across the two contexts, and the weights are not related to the average technical score at all. The most striking row is internal management: its average technical score, 5.94, is the highest on the list, yet it carries the list’s lowest weight (0.30) in the revenue context and sits in the middle (0.50) in the field context. Items that rise to the top of the technical ranking can fall to the bottom on the business side.

The second table is the numerical counterpart of this. When the two business contexts are compared, none of the 61 findings stay in the same position, the mean rank shift is 22.23, and the largest shift is 43 positions. The top-ten lists share not a single finding in common. 41.6 percent of the pairwise comparisons invert, meaning that when you take two findings and ask “which one matters more,” the two contexts give a different answer to almost every other question.

The technical ranking’s distance to the two contexts is not equal, either: it shares 5 top-ten findings with the revenue context and 2 with the field context; the inverted-pair rate is 31.3 percent against 20.6 percent. A ranking presented without a context is not actually context-free; it has silently picked one — and the report does not say which one it picked.

This raises the question of who is supposed to fill in the weight table. The test team can derive the technical score from the data it measures itself, but it cannot measure how valuable an asset class is to the organization; that number comes from the business side. The rule that follows from this is a formal one: the report should not be a single ranked list — the technical score, the asset class, and the weight table must be given together. If the weights are written down, the ranking can be reproduced; if they are not, the list is a decision whose reasoning does not show.

The third table shows individual items one by one. Finding 33 sits in first place in the field context; it is 10th in the technical ranking and 38th in the revenue context. Finding 6 is 4th in the field context and 47th in the revenue context — a gap of 43 positions. These are not a scoring error; they are two different answers to two different questions.

The Cost of Ranking Under a Budget

// budget.mjs — under a limited remediation budget, what each ranking closes.
import { FINDING, CONTEXT, businessScore, rank } from "./scoring.mjs";

const BUDGET = 40;                                   // remediation budget (person-days, fictional)
const NAMES = Object.keys(CONTEXT);
const record = new Map(FINDING.map((b) => [b.id, b]));
const total = (name) => FINDING.reduce((a, b) => a + businessScore(b, name), 0);

// Pull from the ranking until the budget runs out; a finding that does not fit is skipped.
function queue(rankedIds) {
  const selected = []; let days = 0;
  for (const id of rankedIds) {
    const b = record.get(id);
    if (days + b.days <= BUDGET) { selected.push(b); days += b.days; }
  }
  return { selected, days };
}

const PATH = [["technical weight", rank((b) => b.technical)],
  ["revenue context", rank((b) => businessScore(b, NAMES[0]))],
  ["field context", rank((b) => businessScore(b, NAMES[1]))],
  ["revenue efficiency", rank((b) => businessScore(b, NAMES[0]) / b.days)],
  ["field efficiency", rank((b) => businessScore(b, NAMES[1]) / b.days)]];

const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = gen([21, 12, 7, 32, 20]);
console.log(`budget ${BUDGET} person-days; all ${FINDING.length} findings together ` +
  `total ${FINDING.reduce((a, b) => a + b.days, 0)} person-days`);
console.log(yz(["ranking", "remediated", "days", `${NAMES[0]} %`, `${NAMES[1]} %`]));
const result = {};
for (const [name, s] of PATH) {
  const { selected, days } = queue(s);
  const closed = NAMES.map((x) =>
    100 * selected.reduce((a, b) => a + businessScore(b, x), 0) / total(x));
  result[name] = closed;
  console.log(yz([name, selected.length, days, closed[0].toFixed(1), closed[1].toFixed(1)]));
}

// The cost of ranking on the wrong axis: for each context, the closed business impact gap
// between that context's own ranking and the technical ranking.
const y2 = gen([30, 14, 20, 14]);
console.log("\n" + y2(["context", "by technical", "by own ranking", "lost points"]));
for (const [i, name] of NAMES.entries()) {
  const t = result["technical weight"][i], k = result[i === 0 ? "revenue context" : "field context"][i];
  console.log(y2([name, `${t.toFixed(1)}%`, `${k.toFixed(1)}%`, `${(k - t).toFixed(1)} points`]));
}
const topItem = [rank((b) => businessScore(b, NAMES[0]))[0], rank((b) => businessScore(b, NAMES[1]))[0]];
const posT = new Map(rank((b) => b.technical).map((id, i) => [id, i + 1]));
for (const [i, id] of topItem.entries()) {
  const b = record.get(id);
  console.log(`the finding ranked first in the ${NAMES[i]} context is finding ${id} (${b.asset}, ` +
    `technical ${b.technical.toFixed(2)}) — rank ${posT.get(id)} in the technical ranking`);
}

budget 40 person-days; all 61 findings together total 286 person-days
ranking                remediated   days   revenue and subscriber data %  field continuity %
technical weight               12     40                            27.0                24.6
revenue context                 9     40                            28.8                14.1
field context                   9     40                            14.5                28.8
revenue efficiency             18     40                            40.8                31.3
field efficiency               20     40                            33.9                39.4

context                         by technical      by own ranking   lost points
revenue and subscriber data            27.0%               28.8%    1.8 points
field continuity                       24.6%               28.8%    4.2 points
the finding ranked first in the revenue and subscriber data context is finding 10 (subscriber data, technical 9.57) — rank 1 in the technical ranking
the finding ranked first in the field continuity context is finding 33 (field operations, technical 7.50) — rank 10 in the technical ranking

The budget covers only one-seventh of the 61 findings, and the table shows how much work the ranking does under this constraint. The first three rows spend the same 40 days in three different ways. The field context closes 28.8 percent of its own impact using its own ranking, but only 14.1 percent using the revenue ranking: ranking on the wrong axis means spending the same budget at half the efficiency.

The cost of the technical ranking is not equal across the two contexts, either. It falls behind its own ranking by 1.8 points in the revenue context and by 4.2 points in the field context. The technical weight looks “neutral,” but that neutrality costs one context nearly two and a half times as much as the other.

The last two rows are more interesting on their own. The ranking that divides the business score by the remediation day closes 18 and 20 findings in the same 40 days, and beats the other three rankings in both contexts — 40.8 percent in the revenue context, 39.4 percent in the field context. The reason shows up in the table: ranking by pure business impact puts the budget into nine expensive items, while dividing by cost fits more than twice as many items into the same budget. A report’s “critical”-tagged top ten makes this comparison impossible if the remediation cost is not written down.

Another thing the last two rows show is that the rankings do not all start at the same place: the revenue context’s top item is also first in the technical ranking, while the field context’s top item is tenth in the technical ranking. When a report imposes a single order, this second item falling nine positions behind is not a decision — it is a side effect.

Summary

• When the same 61 findings are scored in two business contexts, not a single one stays in the same position; the mean rank shift is 22.23, the largest shift is 43 positions, and the top-ten lists share no common members.
• 41.6 percent of the pairwise comparisons invert between the two contexts; the technical ranking’s distance to the two contexts is not equal either (5/10 and 2/10 shared), meaning a context-free ranking has silently picked a context.
• Business weight is not related to the technical score: the internal management class has the highest average technical score at 5.94, yet carries the list’s lowest weight in the revenue context.
• Under a 40-person-day budget, the field context closes 28.8 percent of its impact with its own ranking and only 14.1 percent with the revenue ranking; ranking on the wrong axis spends the same budget at half the efficiency.
• The technical ranking falls behind by 1.8 points in the revenue context and by 4.2 points in the field context: the cost of the seemingly neutral axis varies twofold by context.
• The ranking that divides the business score by the remediation day closes 18 to 20 findings on the same budget and beats the other rankings in both contexts; a list that does not record the remediation cost makes this comparison impossible.

Next Step

The findings are evidenced and put in order along two separate axes. What comes next goes to a remediation team, and that team will read the finding for the first time. Having evidence on hand does not mean that evidence can be reproduced on the other side: if the precondition, the input, and the environment it was observed under are not written down, verifying the fix is impossible too. The next lesson measures the report in terms of reproducibility: how many findings can be reproduced step by step, how many stay environment-dependent, and how many items of the remediation recommendation are directly actionable.