Detection Engineering

In the previous lesson, what narrowed the band was not a threshold adjustment but a new attribute — that is, a change to the scoring itself. This is not tuning a rule, it is rewriting it, and that is a routine part of the job. Detection engineering is the name for that job: write a rule, run it, measure what it misses, fix it.

The cycle’s measure cannot be how many rules were written, because the rule count gives neither coverage nor accuracy. This lesson builds the measure on a classification matrix. The matrix is treated by its structure: columns are tactics, rows are techniques; the techniques and their traces here are fictional and do not correspond to the content of a real classification effort.

• DI28. The matrix has 8 tactic columns and 26 technique rows. Each technique row is labeled with which source its trace lands on.
• DI29. A technique is in one of three states: its trace lands on a collected source, its trace lands on an uncollected source, or it leaves no trace in any source. In the last two cases, a rule can be written but it can never fire.
• DI30. A rule covers one or more techniques. Every rule has three numbers: write-and-tune minutes, sensitivity, and daily false positive. All three are model data.
• DI31. A covered technique is one that at least one rule targets. Firable coverage is the subset of that whose source is collected. The gap between the two is coverage that stays on paper.
• DI32. The sample set consists of fictional examples of the techniques and is identical for both versions; the comparison’s validity depends on this.

Coverage in the Matrix and Two Versions

// coverage.mjs — two rule-set versions' coverage over a tactic-technique matrix. THIS IS A MODEL.
// The matrix is treated by its STRUCTURE: columns are tactics, rows are techniques. The techniques are fictional.
export const TACTIC = [["initial access", 3], ["execution", 4], ["persistence", 4],
  ["privilege escalation", 3], ["credential access", 4], ["discovery", 3],
  ["lateral movement", 3], ["exfiltration", 2]];

// Where each technique's trace lands: I identity, E endpoint, F network flow,
// M management plane, S UNcollected server application log, - no trace in any source.
const TRACE = "IES EFEM E-EF IMS I-EM FFE S-I FF".replace(/ /g, "").split("");
export const TECHNIQUE = [];
for (const [tactic, n] of TACTIC)
  for (let k = 0; k < n; k++) {
    const i = TECHNIQUE.length;
    TECHNIQUE.push({ id: `t${String(i + 1).padStart(2, "0")}`, tactic, trace: TRACE[i] });
  }
// Sources collected in the pipeline (model data). Source S exists but is not collected.
export const COLLECTED = new Set(["I", "E", "M", "F"]);

// Rule: [name, covered technique indexes, write+tune minutes, sensitivity, daily false positive]
export const VERSION1 = [["s1", [0, 2], 90, 0.8, 3], ["s2", [6, 8], 120, 0.7, 5],
  ["s3", [12, 16], 150, 0.6, 4], ["s4", [20, 23], 90, 0.75, 2],
  ["s5", [1, 13], 180, 0.5, 9], ["s6", [4], 240, 0.4, 7]];
export const VERSION2 = [["s1", [0, 2], 30, 0.85, 2], ["s2", [6, 8], 40, 0.8, 2],
  ["s3", [12, 16], 40, 0.7, 2], ["s4", [20, 23], 20, 0.8, 1],
  ["s5", [1, 13, 17], 60, 0.6, 3], ["s7", [3, 9], 150, 0.7, 3],
  ["s8", [5, 10, 14], 210, 0.65, 4], ["s9", [18, 21], 180, 0.55, 3],
  ["s10", [7, 11], 120, 0.5, 2], ["s11", [24, 25], 90, 0.6, 2],
  ["s12", [15, 19], 150, 0.45, 3]];

export function coverage(rules) {
  const covered = new Set(rules.flatMap(([, l]) => l));
  const count = (f) => [...covered].filter((i) => f(TECHNIQUE[i].trace)).length;
  return { covered, firable: new Set([...covered].filter((i) => COLLECTED.has(TECHNIQUE[i].trace))),
    uncollected: count((z) => z === "S"), traceless: count((z) => z === "-"),
    none: TECHNIQUE.length - covered.size,
    minutes: rules.reduce((a, k) => a + k[2], 0),
    fp: rules.reduce((a, k) => a + k[4], 0) };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const yt = fmt([26, 9, 10, 12, 12]);
  const k1 = coverage(VERSION1), k2 = coverage(VERSION2);
  console.log(`matrix: ${TACTIC.length} tactic columns, ${TECHNIQUE.length} technique rows; ` +
    `${TECHNIQUE.filter((t) => t.trace === "S").length} techniques' source is not collected, ` +
    `${TECHNIQUE.filter((t) => t.trace === "-").length} techniques have no trace in any source`);
  console.log();
  console.log(yt(["tactic", "technique", "version 1", "version 2", "no trace"]));
  for (const [tactic] of TACTIC) {
    const l = TECHNIQUE.filter((t) => t.tactic === tactic);
    const count = (s) => l.filter((t) => s.has(TECHNIQUE.indexOf(t))).length;
    console.log(yt([tactic, l.length, count(k1.covered), count(k2.covered),
      l.filter((t) => !COLLECTED.has(t.trace)).length]));
  }
  const yk = fmt([10, 8, 12, 15, 14, 10, 16, 11]);
  console.log();
  console.log(yk(["version", "rules", "covered", "firable", "no source",
    "no trace", "never covered", "write min"]));
  for (const [name, k, l] of [["version 1", k1, VERSION1], ["version 2", k2, VERSION2]])
    console.log(yk([name, l.length, `${k.covered.size}/${TECHNIQUE.length}`,
      `${k.firable.size}/${TECHNIQUE.length}`, k.uncollected, k.traceless, k.none, k.minutes]));
}

matrix: 8 tactic columns, 26 technique rows; 3 techniques' source is not collected, 3 techniques have no trace in any source

tactic                    technique version 1   version 2    no trace
initial access                    3         3           3           1
execution                         4         2           3           0
persistence                       4         1           4           1
privilege escalation              3         2           3           1
credential access                 4         1           4           1
discovery                         3         1           3           0
lateral movement                  3         1           2           2
exfiltration                      2         0           2           0

version      rules     covered        firable     no source  no trace   never covered  write min
version 1        6       11/26           8/26             2         1              15        870
version 2       11       24/26          19/26             3         2               2       1090

The tactic table gives the first finding: version 1’s coverage is not evenly distributed. All three initial-access rows are covered; none of the exfiltration rows are. This is detection writing’s natural tendency — the easy rule gets written on the source whose fields are most distinguishing, and that source is usually near the start of the chain. The result shows up not in a coverage map but in missed items. The exfiltration row staying empty does not come from that tactic being seen as unimportant — it comes from its trace sitting in network flow data, and from the difficulty, measured in the fourth lesson, of writing a distinguishing rule on that source.

The gap between the two columns in the lower table is the lesson’s central concept. Version 1 covers eleven techniques but only eight are firable: three techniques have a rule and it never triggers. Two of those have a source, but the pipeline does not collect it; one technique leaves no trace in any source. The sentence “we cover eleven of twenty-six techniques” is a false sentence when this distinction goes unwritten.

Version 2 raises coverage to 24, firable coverage to 19, and brings never-covered down from 15 to 2. Of the two remaining techniques, one cannot be covered because it leaves no trace in any source; the other could have been covered, but a rule from version 1 got removed in the cycle. Writing work rose from 870 minutes to 1,090 minutes; that number includes tuning old rules alongside writing new ones, and the first four rules’ minutes in the second version are noticeably lower, because tuning is cheaper than writing.

Running the Versions Against the Same Sample Set

// cycle.mjs — two rule versions are run against a fictional sample set; the reason for each miss is separated.
import { TECHNIQUE, COLLECTED, VERSION1, VERSION2, coverage } from "./coverage.mjs";

const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const REVIEW = 9;
// Fictional sample set: 0-4 samples per technique. The sample count is DATA.
const SAMPLE = [];
{
  const r = generator(20260406);
  for (const [i] of TECHNIQUE.entries())
    for (let k = 0, n = Math.floor(r() * 5); k < n; k++) SAMPLE.push({ technique: i, z: r() });
}

function run(rules) {
  const k = coverage(rules);
  const result = { caught: 0, noRule: 0, noSource: 0, sensitivity: 0 };
  const firing = new Set(), share = new Map(rules.map(([name]) => [name, 0]));
  for (const o of SAMPLE) {
    const covering = rules.filter(([, l]) => l.includes(o.technique));
    if (covering.length === 0) { result.noRule++; continue; }
    if (!COLLECTED.has(TECHNIQUE[o.technique].trace)) { result.noSource++; continue; }
    const top = Math.max(...covering.map(([, , , d]) => d));
    if (o.z < top) {
      result.caught++;
      covering.forEach(([name]) => { firing.add(name); share.set(name, share.get(name) + 1); });
    } else result.sensitivity++;
  }
  const dead = rules.filter(([name]) => !firing.has(name)).map(([name, l]) => {
    const ns = l.filter((i) => !COLLECTED.has(TECHNIQUE[i].trace)).length;
    const smp = SAMPLE.filter((o) => l.includes(o.technique)).length;
    return `${name} (${l.length} technique, ${ns} with no source, ${smp} samples)`;
  });
  return { ...result, fp: k.fp, minutes: k.minutes, dead, share,
    review: (result.caught + k.fp) * REVIEW };
}

const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = fmt([10, 8, 11, 11, 12, 13, 13, 13]);
console.log(`fictional sample set: ${SAMPLE.length} samples, spread over ${TECHNIQUE.length} techniques`);
console.log();
console.log(yz(["version", "sample", "caught", "no rule", "no source", "sensitivity",
  "false pos.", "review min"]));
const [a, b] = [VERSION1, VERSION2].map(run);
for (const [name, s] of [["version 1", a], ["version 2", b]])
  console.log(yz([name, SAMPLE.length, s.caught, s.noRule, s.noSource, s.sensitivity,
    s.fp, s.review]));
console.log(`\nrule that never fires — version 1: ${a.dead.join(", ") || "none"}; ` +
  `version 2: ${b.dead.join(", ") || "none"}`);
console.log(`version 2's extra ${b.minutes - a.minutes} minutes of writing work ` +
  `catches ${b.caught - a.caught} extra samples: ` +
  `${Math.round((b.minutes - a.minutes) / (b.caught - a.caught))} minutes per sample`);
console.log(`daily false positive ${a.fp} -> ${b.fp}, daily review ` +
  `${a.review} -> ${b.review} minutes`);

const yp = fmt([8, 9, 13, 12, 12, 11, 17]);
console.log("\nversion 2 by rule:");
console.log(yp(["rule", "technique", "no source", "caught", "daily fp", "write min",
  "min per sample"]));
for (const [name, l, dk, , fp] of VERSION2) {
  const n = b.share.get(name);
  console.log(yp([name, l.length, l.filter((i) => !COLLECTED.has(TECHNIQUE[i].trace)).length, n, fp, dk,
    n ? Math.round(dk / n) : "-"]));
}

fictional sample set: 51 samples, spread over 26 techniques

version     sample     caught    no rule   no source  sensitivity   false pos.   review min
version 1       51         10         32           4            5           30          360
version 2       51         30          3          11            7           27          513

rule that never fires — version 1: s6 (1 technique, 0 with no source, 1 samples); version 2: s9 (2 technique, 1 with no source, 3 samples)
version 2's extra 220 minutes of writing work catches 20 extra samples: 11 minutes per sample
daily false positive 30 -> 27, daily review 360 -> 513 minutes

version 2 by rule:
rule    technique    no source      caught    daily fp  write min   min per sample
s1              2            1           1           2         30               30
s2              2            1           4           2         40               10
s3              2            0           3           2         40               13
s4              2            0           3           1         20                7
s5              3            1           4           3         60               15
s7              2            0           5           3        150               30
s8              3            0           1           4        210              210
s9              2            1           0           3        180                -
s10             2            0           3           2        120               40
s11             2            0           5           2         90               18
s12             2            1           1           3        150              150

The three reasons for a miss sit in separate columns, and it is exactly this separation that shows what the cycle is doing. In version 1, 32 of 41 misses are from no rule; in version 2, that number drops to 3. Against that, no-source rises from 4 to 11. This rise is not a regression: as rules get written, the reason for a miss shifts from a missing rule to a missing source. Once that second column runs dry, the remaining number cannot be reduced by writing rules; it needs a collection decision.

The sensitivity column holds the third reason: there is a rule, there is a source, but the sample did not cross the rule’s threshold. This number rises from 5 to 7, because version 2 covers more techniques and the new rules have low sensitivity. Growing a rule set changes the distribution of misses; it does not shrink every reason at once. All three columns correspond to a separate piece of work: the first calls for writing a rule, the second for a collection decision, the third for the threshold-and-attribute work from the previous lesson. Rolling the same number up into a single “missed” column leaves it unclear which work needs to be done.

The dead-rule row is maintenance’s hidden line item. In version 1, s6 never fired, and that rule had cost 240 minutes of writing work; in version 2, s9 never fired, and one of the two techniques it covers already has an uncollected source. A dead rule shows up as covered in a coverage table, never shows up in an alert list, and gets reviewed all over again at every version upgrade.

The cycle’s bill is this: an extra 220 minutes of writing work catches 20 extra samples, 11 minutes per sample. Daily false positive drops from 30 to 27 — it falls even though the rule count nearly doubled, because version 1’s noisiest rule got removed. But daily review load rises from 360 minutes to 513 minutes: a correct detection has a review cost too, and a good rule set does not lighten the shift — it fills it.

The final table, broken down by rule, opens up what the average hides. The minutes-per-sample column ranges from 7 to 210: s4 catches three samples with twenty minutes of tuning, s8 catches one sample with two hundred ten minutes of writing work. Both are single rows in the coverage table and look identical there. Taking coverage as the measure is not enough, then; how many samples a covered cell brings in is a separate number.

In the same table, the no source column is greater than zero for five rules. These rules cover two techniques but can only fire on one; they contribute two to the coverage count and one to the firable count. The most extreme example is s9: two techniques, one on an uncollected source, zero catches all day, 180 minutes of writing work. Whether a rule is good or bad can only be said after it is run, and running it is this cycle’s real step.

Summary

• Covered technique and firable coverage are counted separately: version 1 covers eleven techniques but only eight are firable, three have a rule and it never triggers.
• Coverage is not evenly distributed across tactics; version 1 covers all three initial-access rows while covering none of the exfiltration rows.
• Version 2 raises coverage to 24, firable coverage to 19, and brings never-covered down from 15 to 2; one of the remaining two cannot be covered because it leaves no trace in any source.
• As rules get written, the reason for a miss shifts: no-rule drops from 32 to 3 while no-source rises from 4 to 11, and that number cannot be reduced by writing rules.
• An extra 220 minutes of writing work comes to 11 minutes per sample; daily false positive drops from 30 to 27 while review load rises from 360 minutes to 513 minutes.
• Rule-by-rule efficiency ranges from 7 to 210 minutes, and this gap does not show up in the coverage table; s9 covers two techniques but never fires, and its 180 minutes of writing work go unrewarded.

Next Step

After version 2, 21 of 51 samples are still missed, and 11 of those sit where no rule can ever reach: techniques whose source is not collected or that leave no trace at all. Writing rules does not reduce that number, because a rule can only ask the field it can query. The next lesson takes a different route: it starts not from a rule but from a hypothesis. A hypothesis asks the data set a question, and the answer is not an alert — it is a count of eliminated items. The measure there is how many items get eliminated per hypothesis, how many hypotheses return zero results, and how many of the eliminated items turn into a new rule.