Vulnerability Validation

The previous lesson compared the three assessment types and found the same residue in all of them: 62 candidates in black box, 36 in gray box, 19 in white box did not correspond to a real flaw. Those numbers were produced by a single noise multiplier. In reality, which candidate gets eliminated is determined not by a multiplier but by what signal the candidate rests on.

This lesson runs a candidate list through validation rules and counts how many candidates each rule removes and how many real findings it takes down with them. A false positive is a candidate that does not correspond to a real flaw; validation is the step that turns a candidate into evidence. The validation step does not stay on paper — it actually touches the target: taking this step without written authorization and a defined scope is both illegal and unprofessional.

• TP13. The candidate list is 240 raw items and rests on the 20 in-scope items from the previous two lessons. Both the list and the flaw set are fictional; no action is taken against any target.
• TP14. A slot is one chain step of one item. 20 items and 4 steps give 80 slots; a portion of the slots actually have a flaw. This set is the model’s ground truth, and the assessment does not know it.
• TP15. Each candidate carries a signal class: version info, response difference, configuration, or behavior. Weak classes scatter most of their candidates onto clean slots.
• TP16. A finding is a real flaw at a slot; more than one candidate can point to the same finding. Lost is the number of real findings with no remaining candidate.
• TP17. Validation has three costs: time, the risk score added to the target, and the trace left on the defense side. All three vary per candidate and by signal class.
• TP18. The chain model is taken as input from the previous course: if a step finds no flaw, the item’s subsequent steps can be treated as unreachable.

The Elimination Pipeline

// elimination.mjs — runs the candidate list through validation rules and counts false positives.
// THIS IS A MODEL; the candidate list is fictional, no target is touched.
export const SEED = 20260214, ITEM = 20;
const makeRng = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = makeRng(SEED);

// Signal class: what evidence the candidate rests on and that evidence's real-flaw rate.
// Version info is the weakest signal, because a fix can be backported without a version bump.
export const CLASS = [
  { name: "version info", share: 0.46, real: 0.17, time: 0.15, risk: 0.0, trace: 2 },
  { name: "response diff", share: 0.24, real: 0.44, time: 0.35, risk: 0.1, trace: 5 },
  { name: "configuration", share: 0.18, real: 0.71, time: 0.50, risk: 0.0, trace: 1 },
  { name: "behavior", share: 0.12, real: 0.88, time: 0.90, risk: 0.6, trace: 9 },
];
const pickClass = (u) => { let t = 0;
  for (const s of CLASS) { t += s.share; if (u < t) return s; } return CLASS.at(-1); };

// Slot: one chain step of an item. 20 items x 4 steps = 80 slots; a portion of them
// actually have a flaw. This set is the model's ground truth, the assessment does not know it.
const SLOT = [];
for (let k = 0; k < ITEM; k++) for (let z = 1; z <= 4; z++) SLOT.push({ k, z, flaw: r() < 0.35 });
const FLAW_SLOT = SLOT.filter((y) => y.flaw), CLEAN_SLOT = SLOT.filter((y) => !y.flaw);
const pick = (l) => l[Math.floor(r() * l.length)];

// 240 raw candidates. Weak signal classes scatter most of the candidates onto clean slots.
export const CANDIDATE = Array.from({ length: 240 }, (_, i) => {
  const s = pickClass(r()), y = r() < s.real ? pick(FLAW_SLOT) : pick(CLEAN_SLOT);
  return { id: i, item: y.k, step: y.z, class: s.name, real: y.flaw, impact: r() < 0.08 };
});

// A finding is a real flaw at one item's chain step; multiple candidates can point to the
// same finding. Lost is the number of real findings with no remaining candidate.
export const FINDING = new Set(CANDIDATE.filter((a) => a.real).map((a) => `${a.item}|${a.step}`));
const lost = (l) => FINDING.size -
  new Set(l.filter((a) => a.real).map((a) => `${a.item}|${a.step}`)).size;

export function pipeline() {
  const stage = [];
  const pass = (name, list, keep) => {
    const remaining = [], eliminated = [];
    for (const a of list) (keep(a) ? remaining : eliminated).push(a);
    stage.push({ name, in: list.length, eliminated: eliminated.length, remaining: remaining.length,
      lost: lost(remaining) });
    return remaining;
  };
  let l = CANDIDATE;
  // K1: multiple candidates from the same signal, on the same item's same chain step, are deduplicated.
  const seen = new Set();
  l = pass("deduplication", l, (a) => {
    const k = `${a.item}|${a.class}|${a.step}`;
    if (seen.has(k)) return false;
    seen.add(k); return true;
  });
  // K2: a candidate that rests only on version info is eliminated unless the same item's
  // same chain step also has a behavior or response-diff signal. A version match is not evidence.
  const supported = new Set(l.filter((a) => a.class === "behavior" || a.class === "response diff")
    .map((a) => `${a.item}|${a.step}`));
  l = pass("version support", l,
    (a) => a.class !== "version info" || supported.has(`${a.item}|${a.step}`));
  // K3: a candidate whose evidence requires an action that would affect the service is not
  // validated, it stays pending.
  const pending = l.filter((a) => a.impact);
  l = pass("impact boundary", l, (a) => !a.impact);
  // K4: the remaining candidates are validated; the ones that are not real are eliminated false positives.
  const toValidate = l;
  l = pass("validation", l, (a) => a.real);
  return { stage, evidence: l, pending, toValidate };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const { stage, evidence, pending } = pipeline();
  console.log(`model (seed ${SEED}): ${CANDIDATE.length} raw candidates, ${ITEM} items, ` +
    `${CANDIDATE.filter((a) => a.real).length} of them real flaws`);
  const y1 = gen([16, 9, 9, 11, 13]);
  console.log("\n" + y1(["rule", "in", "elim.", "remaining", "lost finding"]));
  for (const a of stage) console.log(y1([a.name, a.in, a.eliminated, a.remaining, a.lost]));
  console.log(y1(["final result", "", `${pending.length} pending`, `${evidence.length} evidence`, ""]));

  const y2 = gen([16, 8, 9, 10, 9, 8]);
  console.log("\n" + y2(["signal class", "cand.", "real", "real %", "evid.", "elim."]));
  for (const s of CLASS) {
    const l = CANDIDATE.filter((a) => a.class === s.name);
    const k = evidence.filter((a) => a.class === s.name).length;
    console.log(y2([s.name, l.length, l.filter((a) => a.real).length,
      `${(100 * l.filter((a) => a.real).length / l.length).toFixed(1)}%`, k, l.length - k]));
  }
  console.log(`\nof ${CANDIDATE.length} candidates, ${CANDIDATE.length - evidence.length - pending.length} ` +
    `are eliminated, ${evidence.length} become evidence, ${pending.length} stay pending`);
  console.log(`of the ${FINDING.size} real findings, ${stage.at(-1).lost} are lost during elimination; ` +
    `the version support rule takes ${stage[1].lost - stage[0].lost}, the impact boundary rule takes ` +
    `${stage[2].lost - stage[1].lost}`);
}

model (seed 20260214): 240 raw candidates, 20 items, 114 of them real flaws

rule                   in    elim.  remaining lost finding
deduplication         240      101        139            0
version support       139       23        116            1
impact boundary       116        9        107            2
validation            107       46         61            2
final result             9 pending61 evidence

signal class       cand.     real    real %    evid.   elim.
version info         104       17     16.3%        7      97
response diff         60       33     55.0%       18      42
configuration         37       30     81.1%       17      20
behavior              39       34     87.2%       19      20

of 240 candidates, 170 are eliminated, 61 become evidence, 9 stay pending
of the 29 real findings, 2 are lost during elimination; the version support rule takes 1, the impact boundary rule takes 1

The pipeline’s four rules do four very different jobs. Deduplication alone removes 101 candidates — 42 percent of the list — and drops no findings. This is the pipeline’s cheapest rule: duplicate candidates produced by the same signal, on the same item’s same chain step, collapse into a single item. Nearly half of the “240 findings” written on a report is a counting error that this one operation alone could erase.

The version support rule eliminates 23 candidates and, in exchange, takes down 1 real finding. The impact boundary sets 9 candidates aside as pending, dropping one more finding. The actual validation step eliminates 46 candidates — these are the pipeline’s real false positives. What remains is 61 evidenced findings: 25.4 percent of the 240 raw candidates.

The fourth column writes down the pipeline’s own error. 2 of the 29 real findings are lost during elimination — that is, 6.9 percent. The value of an elimination pipeline cannot be measured only by how many candidates it removes; every rule is a threshold, and a threshold cuts both noise and signal. If these two numbers are not reported together, an aggressive elimination pipeline looks high-quality even while it narrows the scope.

The second table shows where the candidates come from and explains why the pipeline is built in this order. Version info, with 104 candidates, is the largest source in the list, but only 16.3 percent of it is real and it leaves only 7 evidenced findings: 43 percent of the candidates, 11.5 percent of the evidence. The reason is not a measurement flaw — when a fix is backported, the flaw closes without the version string changing, so a version match does not carry evidence. The behavior class, meanwhile, with 39 candidates the second-smallest source, gives the largest contribution at 19 evidenced findings: observed behavior is measurement, not inference.

Validation Order and Added Risk

// sequence.mjs — validation order: if the chain's precondition step is validated first,
// candidates in later steps are decided without ever being run.
import { CLASS, pipeline } from "./elimination.mjs";

const info = (name) => CLASS.find((s) => s.name === name);
const LIST = pipeline().toValidate;                 // candidates that passed the elimination rules

const total = (l) => l.reduce((o, a) => {
  const s = info(a.class);
  return { count: o.count + 1, hours: o.hours + s.time, risk: o.risk + s.risk, trace: o.trace + s.trace };
}, { count: 0, hours: 0, risk: 0, trace: 0 });

// Cut steps: if a chain step gets a negative result, the rest of the item is treated as
// unreachable. Empty set = every candidate is validated one by one.
function run(cutSteps) {
  const ran = [], skipped = [];
  for (const k of new Set(LIST.map((a) => a.item))) {
    const itemCandidates = LIST.filter((a) => a.item === k);
    let cut = false;
    for (let z = 1; z <= 4; z++) {
      const stepCandidates = itemCandidates.filter((a) => a.step === z);
      if (cut) { skipped.push(...stepCandidates); continue; }
      ran.push(...stepCandidates);
      if (cutSteps.has(z) && stepCandidates.length && !stepCandidates.some((a) => a.real)) cut = true;
    }
  }
  return { ...total(ran), evidence: ran.filter((a) => a.real).length, skipped,
    missed: skipped.filter((a) => a.real).length, ran };
}

const PATH = [["unordered", new Set()], ["precondition only", new Set([1])],
  ["full chain", new Set([1, 2, 3, 4])]];
const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = gen([18, 12, 8, 10, 16, 8, 8, 10]);
console.log(`${LIST.length} candidates enter the validation pipeline; the risk score is a ` +
  `validation step's weight of affecting the service`);
console.log(yz(["validation order", "validated", "evid.", "skipped", "skipped real",
  "hours", "risk", "trace"]));
const result = {};
for (const [name, cutSteps] of PATH) {
  const o = run(cutSteps); result[name] = o;
  console.log(yz([name, o.count, o.evidence, o.skipped.length, o.missed, o.hours.toFixed(1),
    o.risk.toFixed(1), o.trace]));
}

const y2 = gen([16, 12, 14, 14, 12]);
console.log("\n" + y2(["signal class", "risk/cand.", "in pipeline", "in full chain",
  "not run"]));
for (const s of CLASS) {
  const a = LIST.filter((x) => x.class === s.name).length;
  const b = result["full chain"].ran.filter((x) => x.class === s.name).length;
  console.log(y2([s.name, s.risk.toFixed(1), a, b, a - b]));
}
const [d, o1, o2] = PATH.map(([name]) => result[name]);
console.log(`\nthe precondition cut skips ${o1.skipped.length} candidates: it brings risk down ` +
  `from ${d.risk.toFixed(1)} to ${o1.risk.toFixed(1)}, trace from ${d.trace} to ${o1.trace}`);
console.log(`the full chain cut skips ${o2.skipped.length} candidates and lowers risk by ` +
  `${(100 * (1 - o2.risk / d.risk)).toFixed(1)}%; the cost is that ${o2.missed} real flaws whose ` +
  `precondition is treated as closed stay unevidenced`);

107 candidates enter the validation pipeline; the risk score is a validation step's weight of affecting the service
validation order     validated   evid.   skipped    skipped real   hours    risk     trace
unordered                  107      61         0               0    48.9    17.8       461
precondition only           89      53        18               8    41.3    15.2       383
full chain                  74      45        33              16    33.3    11.9       313

signal class      risk/cand.   in pipeline in full chain     not run
version info             0.0            26            19           7
response diff            0.1            34            23          11
configuration            0.0            23            16           7
behavior                 0.6            24            16           8

the precondition cut skips 18 candidates: it brings risk down from 17.8 to 15.2, trace from 461 to 383
the full chain cut skips 33 candidates and lowers risk by 33.1%; the cost is that 16 real flaws whose precondition is treated as closed stay unevidenced

The three rows handle the same 107 candidates in three different orders, and each one buys something different. Unordered validation gives the most evidence — 61 — but it is the most expensive: 48.9 hours, a risk score of 17.8, and 461 records on the defense side. The full-chain order brings the number of validated candidates down to 74; risk drops 33.1 percent, trace falls to 313 records, and 15.6 hours are saved.

Where the gain comes from is written in the second table. Almost all of the risk score sits in the behavior class; version and configuration candidates add zero risk to the target, because both are observation. The full-chain order never runs 8 of the 24 behavior candidates, and most of the risk drop comes from those eight runs not happening. The same logic works in the trace column: a behavior candidate produces 9 records, while a configuration candidate produces 1.

The cost is in the fourth column: in the full-chain order, 33 candidates are never run at all, and 16 of them are real flaws. Whether these 16 items count as a loss depends on how accurate the chain model is. If the model is correct, the precondition leading to those flaws is closed and an attacker cannot reach them either — in that case, the 16 runs would have been unnecessary risk. If the model is incomplete, that is, if there is a second path to the same place, then 16 real flaws stay unevidenced. This is why the middle row exists: cutting only at the precondition step brings risk down from 17.8 to 15.2 while keeping the skipped real flaws at 8.

Whichever order is chosen, one rule stays fixed on the reporting side. A candidate that was not run is not deleted from the list; it stays on record as “not validated because its precondition was treated as closed,” with a note on which step produced which result. Otherwise, the report shows a scope wider than it actually covered.

Summary

• 240 raw candidates pass through four rules and come down to 61 evidenced findings; deduplication alone removes 101 candidates — 42 percent of the list — without dropping a single finding.
• The elimination pipeline also produces its own error: 2 of the 29 real findings are lost during the rules, which is why the eliminated-candidate count and the dropped-finding count must be reported together.
• 43 percent of the candidates come from version info, but only 16.3 percent of that class is real and it gives only 11.5 percent of the evidence; when a fix is backported, the version string does not change.
• The behavior class is the second-smallest source, yet gives the most evidence — 19 out of 39 candidates — because observed behavior is measurement, not inference.
• When the validation order is tied to the chain, the validated candidates drop from 107 to 74, risk from 17.8 to 11.9, and trace from 461 to 313 records; most of the drop comes from the 8 behavior candidates that are never run.
• The cost of the full-chain order is that 16 real flaws whose precondition is treated as closed stay unevidenced; these items are not removed from the report, they are written down together with the reason they were not validated.

Next Step

What remains on hand is 61 evidenced findings, 9 pending candidates, and a list left unvalidated. If these three sets are going into a remediation queue, they have to be put in order, and the criterion for that order has not been written yet. Technical weight produces one order: ease of access, breadth of impact, difficulty of the fix. The same list, looked at from the business side, produces a different order, because the business carried by one item is not the same as another’s. The next lesson scores the same set of findings in two separate business contexts and counts how many positions the two rankings diverge by.