False Positives and False Negatives

The previous lesson’s four metrics ran with fixed thresholds, and the result was that 39 of 43 flags turned out legitimate. Tightening the thresholds is the first fix that comes to mind: making the coefficient of variation 0.05 instead of 0.15, requiring an exit ratio of two hundred instead of twenty. This lesson runs that fix as a sweep and draws both curves on the same axis.

The result is not whether a single threshold is right. What gets counted is the band where the two distributions overlap: for the items in that band, no threshold can reduce both false positive and missed at once. This is the measurement the rest of the course builds on. The score distribution below is a model; the scores come from fictional attributes, not a real measurement.

• DI23. The candidate set is 900 items: 880 legitimate work, 20 fictional items. The label is the model’s ground truth and does not enter the scoring.
• DI24. The score comes out as a weighted sum of four attributes: regularity, exit ratio, destination novelty and off-hours. The weights are model data.
• DI25. Both distributions have a tail: some legit items score high, some fictional items score low. This is not a flaw in the model — it is the lesson’s subject.
• DI26. The overlap band is the range between the lowest fictional item score and the highest legit score. Items outside the band are classified correctly at every threshold.
• DI27. The cost is two-sided and both sides are counted with model data: 9 minutes of review per alert, 6 extra items a missed item reaches by day’s end, and 144 daily analyst-minutes allotted to this rule.

Score Distribution and Threshold Sweep

// threshold.mjs — candidate items are scored, the threshold is swept and the overlap band is counted. THIS IS A MODEL.
export const SEED = 20260405;
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = generator(SEED);

// Four attributes, each between 0 and 1. Weights are model data.
export const WEIGHT = { regularity: 0.30, exitRatio: 0.30, destNovelty: 0.25, offHours: 0.15 };
const low = () => r() ** 3;              // legit item: mostly low, has a tail
const high = () => 1 - r() ** 3;         // fictional item: mostly high, has a tail

export const computeScore = (o) => Math.round(100 *
  Object.entries(WEIGHT).reduce((a, [k, w]) => a + w * o[k], 0));

export const CANDIDATE = [];
for (let k = 0; k < 880; k++)
  CANDIDATE.push({ regularity: low(), exitRatio: low(), destNovelty: low(),
    offHours: low(), label: "legit" });
for (let k = 0; k < 20; k++)
  CANDIDATE.push({ regularity: high(), exitRatio: high(), destNovelty: high(),
    offHours: high(), label: "item" });
for (const o of CANDIDATE) o.score = computeScore(o);
CANDIDATE.sort((a, b) => a.score - b.score);

export const legit = CANDIDATE.filter((o) => o.label === "legit");
export const item = CANDIDATE.filter((o) => o.label === "item");
export const sweep = (threshold) => ({
  threshold,
  falsePositive: legit.filter((o) => o.score >= threshold).length,
  missed: item.filter((o) => o.score < threshold).length,
  alert: CANDIDATE.filter((o) => o.score >= threshold).length,
});

if (import.meta.url === `file://${process.argv[1]}`) {
  const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const yz = fmt([9, 9, 15, 13, 13, 12]);
  console.log(`model (seed ${SEED}): ${CANDIDATE.length} candidate items, ${item.length} of them ` +
    `fictional; score comes out as a weighted sum of four attributes`);
  console.log(`legit score range ${legit[0].score}-${legit[legit.length - 1].score}, ` +
    `item score range ${item[0].score}-${item[item.length - 1].score}`);
  const yd = fmt([16, 10, 14]);
  console.log();
  console.log(yd(["score range", "legit", "item"]));
  for (let a = 0; a < 100; a += 20) {
    const ic = (l) => l.filter((o) => o.score >= a && o.score < a + 20).length;
    console.log(yd([`${a}-${a + 19}`, ic(legit), ic(item)]));
  }
  console.log();
  console.log(yz(["threshold", "alert", "false positive", "missed", "total error", "precision"]));
  for (const e of [10, 20, 30, 40, 50, 60, 70, 80]) {
    const t = sweep(e);
    console.log(yz([e, t.alert, t.falsePositive, t.missed,
      t.falsePositive + t.missed,
      t.alert ? `${Math.round(100 * (t.alert - t.falsePositive) / t.alert)}%` : "-"]));
  }
  // Overlap band: between the lowest fictional item score and the highest legit score.
  const lo = item[0].score, hi = legit[legit.length - 1].score;
  const bandLegit = legit.filter((o) => o.score >= lo && o.score <= hi).length;
  const bandItem = item.filter((o) => o.score >= lo && o.score <= hi).length;
  console.log(`\noverlap band ${lo}-${hi}: this band holds ${bandLegit} legit and ${bandItem} ` +
    `fictional items; no threshold can separate the two at once`);
  const best = [...Array(101).keys()].map(sweep)
    .reduce((a, b) => (b.falsePositive + b.missed < a.falsePositive + a.missed ? b : a));
  console.log(`threshold minimizing total error ${best.threshold}: false positive ` +
    `${best.falsePositive}, missed ${best.missed}; no threshold gives zero-zero`);
}

model (seed 20260405): 900 candidate items, 20 of them fictional; score comes out as a weighted sum of four attributes
legit score range 0-88, item score range 37-97

score range          legit          item
0-19                   357             0
20-39                  399             1
40-59                  108             2
60-79                   15             8
80-99                    1             9

threshold    alert false positive       missed  total error   precision
10             766            746            0          746          3%
20             543            523            0          523          4%
30             302            282            0          282          7%
40             143            124            1          125         13%
50              74             56            2           58         24%
60              33             16            3           19         52%
70              20              4            4            8         80%
80              10              1           11           12         90%

overlap band 37-88: this band holds 157 legit and 17 fictional items; no threshold can separate the two at once
threshold minimizing total error 70: false positive 4, missed 4; no threshold gives zero-zero

The distribution table gives the lesson’s whole subject in five rows. 756 of the legit items are below 40, and 19 of the fictional items are above 40; that means the score does useful work. But the third and fourth rows show the two distributions sharing the same bins: in the 40–59 range, 108 legit items sit alongside 2 fictional items, and in the 60–79 range, 15 legit items sit alongside 8 fictional items. In these rows, the score produces no separation.

The threshold sweep writes the same fact in the language of decisions, and there is no flat region in between. At threshold 40, 124 false positives and 1 missed; at threshold 80, 1 false positive and 11 missed. Lowering the threshold reduces missed and grows the review load; raising it lightens the review and grows missed. The transition in between is continuous; there is no point that drives both numbers to zero at once.

The reason is written in the second line: legit scores fall between 0 and 88, fictional item scores fall between 37 and 97. These two ranges overlap between 37–88, and that band holds 157 legit items and 17 fictional items. Wherever a threshold lands inside the band, it misses the fictional items below it and flags the legit items above it. The 726 items outside the band are classified correctly at every threshold; the measured uncertainty is only as large as the band.

The precision column corrects another misconception. For threshold 10, precision comes out to 3 percent: 98 percent of alerts are legitimate work. At threshold 80, precision rises to 90 percent, but missed jumps to 11. Precision alone is not a quality measure, because it is undefined for a rule that produces no alerts at all, and any rule that produces few alerts shows high precision.

Budget Threshold and Narrowing the Band

The threshold that minimizes total error is 70, and there 4 false positives and 4 missed remain. But this choice is not made in practice, because threshold 70 produces 20 alerts and every alert demands analyst time. Capacity, not accuracy, determines the operating point.

// point.mjs — the threshold's two-sided cost in minutes, then the band with a fifth attribute.
import { CANDIDATE, sweep } from "./threshold.mjs";

// Model data (not a measurement): review minutes, the number of extra items a missed item
// reaches by day's end, and the daily analyst-minutes allotted to this rule.
const REVIEW = 9, EXTRA_ITEM = 6, BUDGET = 144;
const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = fmt([9, 8, 13, 12, 15, 13]);
console.log(`daily budget ${BUDGET} analyst-minutes, review per alert ${REVIEW} minutes; ` +
  `each missed item reaches ${EXTRA_ITEM} extra items by day's end (model data)`);
console.log();
console.log(yz(["threshold", "alert", "review min", "missed", "item reached", "budget"]));
for (const e of [40, 50, 60, 70, 80, 90]) {
  const t = sweep(e);
  console.log(yz([e, t.alert, t.alert * REVIEW, t.missed, t.missed * EXTRA_ITEM,
    t.alert * REVIEW <= BUDGET ? "fits" : "does not fit"]));
}
const fitting = [...Array(101).keys()].map(sweep).filter((t) => t.alert * REVIEW <= BUDGET);
const pick = fitting[0];
console.log(`\nlowest threshold that fits the budget ${pick.threshold}: ${pick.alert} alerts, ` +
  `${pick.missed} missed items, ${pick.missed * EXTRA_ITEM} items reached`);

// Fifth attribute: whether a matching record exists at the identity source. Discriminating but not perfect.
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r2 = generator(880517);
const W2 = { regularity: 0.21, exitRatio: 0.21, destNovelty: 0.18,
  offHours: 0.10, noIdentityMatch: 0.30 };
const CANDIDATE2 = CANDIDATE.map((o) => ({ ...o,
  noIdentityMatch: (o.label === "item" ? r2() < 0.8 : r2() < 0.03) ? 1 : 0 }));
for (const o of CANDIDATE2) o.score = Math.round(100 *
  Object.entries(W2).reduce((a, [k, w]) => a + w * o[k], 0));

const band = (list) => {
  const m = list.filter((o) => o.label === "legit"), k = list.filter((o) => o.label === "item");
  const lo = Math.min(...k.map((o) => o.score)), hi = Math.max(...m.map((o) => o.score));
  const count = (l) => l.filter((o) => o.score >= lo && o.score <= hi).length;
  // Operating point: the LOWEST threshold that fits the budget.
  const best = [...Array(101).keys()].map((e) => ({ e, n: list.filter((o) => o.score >= e).length,
    fp: m.filter((o) => o.score >= e).length, mi: k.filter((o) => o.score < e).length }))
    .find((x) => x.n * REVIEW <= BUDGET);
  return { lo, hi, m: count(m), k: count(k), best };
};
const yb = fmt([18, 9, 16, 16, 17, 13]);
console.log();
console.log(yb(["scoring", "band", "legit in band", "item in band", "budget threshold",
  "fp / missed"]));
for (const [name, list] of [["four attributes", CANDIDATE], ["five attributes", CANDIDATE2]]) {
  const b = band(list);
  console.log(yb([name, `${b.lo}-${b.hi}`, b.m, b.k, b.best.e, `${b.best.fp} / ${b.best.mi}`]));
}
console.log(`\nthe fifth attribute lowers the item count in the band but does not close it; ` +
  `no threshold gives zero false positive and zero missed in either scoring`);

daily budget 144 analyst-minutes, review per alert 9 minutes; each missed item reaches 6 extra items by day's end (model data)

threshold   alert   review min      missed   item reached       budget
40            143         1287           1              6 does not fit
50             74          666           2             12 does not fit
60             33          297           3             18 does not fit
70             20          180           4             24 does not fit
80             10           90          11             66         fits
90              3           27          17            102         fits

lowest threshold that fits the budget 72: 16 alerts, 7 missed items, 42 items reached

scoring                band   legit in band    item in band budget threshold  fp / missed
four attributes       37-88             157              17               72        3 / 7
five attributes       49-75              15               7               66        1 / 5

the fifth attribute lowers the item count in the band but does not close it; no threshold gives zero false positive and zero missed in either scoring

The budget column takes the threshold choice out of our hands. The threshold 70 that minimizes total error demands 180 minutes of review and does not fit the 144-minute budget. The lowest threshold that fits is 72; there, 16 alerts are reviewed, 7 fictional items get missed, and per the model data, those 7 items reach 42 extra items by day’s end. What chooses the threshold is not accuracy — it is analyst-minutes.

The two-sided cost has different units, and that is the real difficulty that makes comparison hard. The left side is minutes, paid directly: 1,287 minutes at threshold 40, more than a nine-person day. The right side is items, paid with a delay: 17 missed items at threshold 90, 102 items reached. Adding the two on the same axis is only possible once someone writes down how many minutes an item is worth, and that number is an organizational decision, not a technical one.

The last table shows the one real way out: narrowing the band. The fifth attribute — the item having no matching record at the identity source — brings the band from 37–88 down to 49–75, the legit count in the band from 157 to 15, and the fictional item count in the band from 17 to 7. The budget threshold drops from 72 to 66, and at that point false positive falls from 3 to 1, missed falls from 7 to 5. Both numbers improve at the same time, because what changes is not the threshold — it is the score itself.

The fifth attribute has a cost too, and that cost was measured in the second lesson. To say “no matching record at the identity source,” the mapping has to be attempted first; in a five-minute window, that mapping resolved part of the records and left the rest ambiguous. For an item that stays ambiguous, this attribute cannot be filled in. So the improvement that narrows the band rests on a limitation of the pipeline measured earlier, and as that limitation grows, the improvement shrinks.

Yet the band does not close. The fifth attribute is present in 80 percent of fictional items and also present in 3 percent of legit items; the remaining overlap is a direct result of these two proportions. No threshold gives zero false positive and zero missed in either scoring, and adding more attributes narrows the band, it does not close it.

Summary

• 900 candidate items’ scores overlap in the 37–88 band; that band holds 157 legit and 17 fictional items and no threshold can separate the two at once. The 726 items outside the band are classified correctly at every threshold.
• At threshold 40, 124 false positives and 1 missed; at threshold 80, 1 false positive and 11 missed; even at the threshold 70 that minimizes total error, 4 and 4 remain.
• Precision alone is not a quality measure: any rule that produces few alerts shows high precision — at threshold 80, precision is 90 percent while missed is 11.
• Capacity, not accuracy, picks the operating point: the lowest threshold that fits 144 analyst-minutes is 72, where 7 fictional items get missed, reaching 42 extra items.
• The fifth attribute narrows the band to 49–75, the legit count to 15 and the item count to 7; at the budget threshold, false positive falls from 3 to 1 and missed falls from 7 to 5, but a zero-zero threshold still does not exist.

Next Step

What narrowed the band was not a threshold adjustment but a new attribute — that is, a change to the scoring itself. This is not tuning a rule, it is rewriting it, and that is a routine part of the job. The next lesson treats that work as a cycle: it runs two versions of a rule set against the same technique list and compares the number of techniques each covers. What is measured there is not how many rules were written, but how many cells of the tactic-technique matrix are covered by at least one rule, how many are never covered, and in how many of the cells marked as covered the rule could never actually fire — because that cell’s data source is somewhere the pipeline never collects from.