Threat Detection

The previous lesson closed the public endpoint and cut the three non-legitimate access records written into the table. Those three could be cut because they were in the table. An event that comes from inside the private network, carries valid credentials, and performs a permitted operation shows up in no access table; access control counts it as legitimate, because by definition it is legitimate. Threat detection targets this gap: it asks whether the activity that authorization let through is unusual.

The shape of the question is therefore different. An access decision is a binary predicate with a defined correct answer; detection is a guess, and it is measured with three numbers: caught, missed, false alarm. This lesson runs detection rules against a known event set, derives the three numbers, then counts two costs — the charge for collecting signal, and the behavior a false alarm produces in the team.

Event Set, Signal Sources, and Rules

The example course is the fictional regional measurement network. CS37 — known event set: thirty events, eleven malicious-tagged. The nineteen benign ones are deliberately chosen as hard look-alikes; each shares one surface with a malicious event, so the false-alarm rate will run higher than in an easy set. CS38 — three signal sources: audit log (operation, operation count, hour), network flow (outbound volume, previously unseen destination, unusual port), and identity behavior (new location, failed attempt, first-seen subject). CS39 — nine detection rules: each rule looks at a single source, three rules per source. CS40 — collection charge: daily volume per source and unit per GB (fictional unit). CS41 — false-alarm threshold: a rule producing three or more false alarms gets closed. CS42 — collection cap: 60 GB a day (fictional unit).

The file below builds the set, the rules, and the two costs. It is a process-internal model; no real event, signal stream, or cost exists.

// detection/rule.mjs — running detection rules against a known event set. Process-internal model:
// the regional measurement network (fictional); no real event, signal stream, or cost exists.

const parseRows = (s) => s.trim().split("\n").map((l) => l.trim().split("|"));
const row = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));

// Event: name|malicious|D op|D count|D hour|N egress MB|N new dest|N odd port|
//        I new location|I failed attempt|I new subject
const EVENT = parseRows(`
  bulk-archive-delete|1|delete|40|3|20|0|0|0|0|0
  policy-expansion|1|policy-change|2|4|5|0|0|1|0|0
  nightly-role-chain|1|assume-role|9|2|10|0|0|1|2|0
  store-outbound-copy|1|read|60|14|900|1|0|0|0|0
  leak-to-new-destination|1|read|12|11|120|1|0|0|0|0
  key-exfiltration|1|key-create|3|1|8|0|0|1|0|1
  brute-force-login|1|read|4|22|3|0|0|1|18|0
  quiet-side-channel|1|read|6|13|40|0|1|0|0|0
  shared-account-takeover|1|assume-role|2|10|15|0|0|1|1|1
  logging-shutdown|1|policy-change|1|13|4|0|0|0|0|0
  slow-exfiltration|1|read|9|15|45|0|0|0|0|0
  nightly-backup-restore|0|delete|30|3|400|0|0|0|0|0
  bulk-cleanup|0|delete|25|11|10|0|0|0|0|0
  release-rollout|0|policy-change|1|10|6|0|0|0|0|0
  measurement-ingest|0|write|80|9|650|0|0|0|0|0
  nightly-report-pull|0|read|20|4|300|1|0|0|0|0
  new-team-member|0|read|5|9|12|0|0|1|1|1
  login-from-travel|0|read|7|8|9|0|0|1|2|0
  report-from-shared-network|0|read|11|16|80|1|0|0|0|0
  edge-upload|0|write|14|20|140|1|0|0|0|0
  staging-run|0|assume-role|6|13|20|0|0|0|0|0
  tool-update|0|write|3|2|18|0|1|0|0|0
  password-reset|0|read|2|9|2|0|0|0|11|0
  maintenance-window|0|delete|22|1|30|0|0|0|0|0
  backup-verification|0|read|18|5|520|0|0|0|0|0
  setup-job|0|key-create|2|14|7|0|0|0|0|1
  catalog-refresh|0|read|15|10|60|0|0|0|0|0
  staging-data-delete|0|delete|8|14|5|0|0|0|0|0
  log-archiving|0|write|19|12|210|0|0|0|0|0
  scheduled-job|0|read|10|7|25|0|0|0|0|0`)
  .map(([name, m, di, ds, dh, ac, ay, ap, kk, kb, ko]) => ({ name, malicious: m === "1",
    auditOp: di, auditCount: +ds, auditHour: +dh, netEgress: +ac, netNewDest: +ay, netOddPort: +ap,
    idNewLocation: +kk, idFailedAttempt: +kb, idNewSubject: +ko }));

// Detection rule: id|signal source|predicate. Every rule looks at a single source.
const RULE = {
  T1: ["audit log", (o) => o.auditOp === "policy-change" && o.auditHour < 6],
  T2: ["audit log", (o) => o.auditOp === "delete" && o.auditCount >= 20],
  T3: ["audit log", (o) => o.auditOp === "assume-role" && o.auditCount >= 5],
  T4: ["network flow", (o) => o.netEgress >= 500],
  T5: ["network flow", (o) => o.netNewDest === 1 && o.netEgress >= 50],
  T6: ["network flow", (o) => o.netOddPort === 1],
  T7: ["identity behavior", (o) => o.idNewLocation === 1 && o.idFailedAttempt >= 1],
  T8: ["identity behavior", (o) => o.idFailedAttempt >= 10],
  T9: ["identity behavior", (o) => o.idNewSubject === 1 && o.idNewLocation === 1],
};
const SOURCE = ["audit log", "network flow", "identity behavior"];
const RULES = Object.keys(RULE);

const alert = (rules) => EVENT.map((o) => ({ o,
  rule: rules.filter((t) => RULE[t][1](o)) }));

const summary = (rules) => {
  const a = alert(rules);
  const malicious = a.filter((x) => x.o.malicious);
  return {
    caught: malicious.filter((x) => x.rule.length > 0).length,
    missed: malicious.filter((x) => x.rule.length === 0).length,
    falseAlarm: a.filter((x) => !x.o.malicious && x.rule.length > 0).length,
    quiet: a.filter((x) => !x.o.malicious && x.rule.length === 0).length,
  };
};

const maliciousCount = EVENT.filter((o) => o.malicious).length;
console.log(`${EVENT.length} known events (${maliciousCount} malicious-tagged), ` +
  `${RULES.length} detection rules, ${SOURCE.length} signal sources`);

// Rule by rule
const K = [-6, -20, 9, 11, 14];
console.log();
row(K, "rule", "signal source", "alerts", "caught", "false alarm");
for (const t of RULES) {
  const a = alert([t]).filter((x) => x.rule.length > 0);
  const y = a.filter((x) => x.o.malicious).length;
  row(K, t, RULE[t][0], a.length, y, a.length - y);
}
const T = summary(RULES);
row(K, "total", "", T.caught + T.falseAlarm, `${T.caught}/${maliciousCount}`, T.falseAlarm);

// Signal source by signal source: source alone and source removed
const S = [-20, 13, 14, 14, 19];
console.log();
row(S, "signal source", "alone", "false alarm", "removed", "only this source");
for (const s of SOURCE) {
  const own = RULES.filter((t) => RULE[t][0] === s);
  const others = RULES.filter((t) => RULE[t][0] !== s);
  const alone = summary(own), without = summary(others);
  row(S, s, `${alone.caught}/${maliciousCount}`, alone.falseAlarm, `${without.caught}/${maliciousCount}`,
    T.caught - without.caught);
}

// Events caught by only a single source are named
console.log();
for (const x of alert(RULES).filter((x) => x.o.malicious && x.rule.length > 0)) {
  const sources = [...new Set(x.rule.map((t) => RULE[t][0]))];
  if (sources.length === 1) console.log(`"${sources[0]}" only: ${x.o.name}`);
}
for (const x of alert(RULES).filter((x) => x.o.malicious && x.rule.length === 0))
  console.log(`no source catches: ${x.o.name}`);

// Collection cost (fictional unit): source|daily GB|unit per GB
const COST = Object.fromEntries(parseRows(`
  audit log|12|0.5
  network flow|40|0.3
  identity behavior|3|0.8`).map(([s, gb, u]) => [s, { gb: +gb, unit: +u }]));
const U = [-20, 10, 12, 13, 18];
console.log();
row(U, "signal source", "GB/day", "unit/GB", "unit/mo", "per only-this");
let totalMonth = 0, totalGb = 0;
for (const s of SOURCE) {
  const month = COST[s].gb * COST[s].unit * 30;
  const others = RULES.filter((t) => RULE[t][0] !== s);
  const only = T.caught - summary(others).caught;
  totalMonth += month; totalGb += COST[s].gb;
  row(U, s, COST[s].gb, COST[s].unit.toFixed(1), month.toFixed(0),
    only ? (month / only).toFixed(0) : "-");
}
row(U, "total", totalGb, "", totalMonth.toFixed(0), "");

// The behavior a false alarm produces: a rule over the threshold gets closed (same measure as M20/K04)
const THRESHOLD = 3;
const closedRules = RULES.filter((t) => {
  const a = alert([t]).filter((x) => x.rule.length > 0);
  return a.length - a.filter((x) => x.o.malicious).length >= THRESHOLD;
});
const after = summary(RULES.filter((t) => !closedRules.includes(t)));
console.log(`\nfalse-alarm threshold ${THRESHOLD}: closed rules ${closedRules.join(", ")} ` +
  `(${closedRules.length}/${RULES.length}); after closing, caught ` +
  `${T.caught} -> ${after.caught}, missed ${T.missed} -> ${after.missed}, ` +
  `false alarm ${T.falseAlarm} -> ${after.falseAlarm}`);
const open = RULES.filter((t) => !closedRules.includes(t));
const newlyMissed = alert(RULES).filter((x) => x.o.malicious && x.rule.length > 0
  && !x.rule.some((t) => open.includes(t))).map((x) => x.o.name);
console.log(`added to missed after closing: ${newlyMissed.join(", ")}; ` +
  `the cost of the sources the closed rules watch (${[...new Set(closedRules.map((t) =>
    RULE[t][0]))].join(", ")}) keeps being paid`);

// Limit and remaining (fictional): daily collection cap
const LIMIT = 60;
console.log(`collection cap ${LIMIT} GB/day; used ${totalGb}, remaining ${LIMIT - totalGb} -> ` +
  `${LIMIT - totalGb} GB/day of room for a fourth source, network flow doubling would exceed the cap`);

30 known events (11 malicious-tagged), 9 detection rules, 3 signal sources

rule  signal source          alerts     caught   false alarm
T1    audit log                   1          1             0
T2    audit log                   4          1             3
T3    audit log                   2          1             1
T4    network flow                3          1             2
T5    network flow                5          2             3
T6    network flow                2          1             1
T7    identity behavior           5          3             2
T8    identity behavior           2          1             1
T9    identity behavior           3          2             1
total                            22       9/11            13

signal source               alone   false alarm       removed   only this source
audit log                    3/11             4          7/11                  2
network flow                 3/11             6          6/11                  3
identity behavior            4/11             3          6/11                  3

"audit log" only: bulk-archive-delete
"audit log" only: policy-expansion
"network flow" only: store-outbound-copy
"network flow" only: leak-to-new-destination
"identity behavior" only: key-exfiltration
"identity behavior" only: brute-force-login
"network flow" only: quiet-side-channel
"identity behavior" only: shared-account-takeover
no source catches: logging-shutdown
no source catches: slow-exfiltration

signal source           GB/day     unit/GB      unit/mo     per only-this
audit log                   12         0.5          180                90
network flow                40         0.3          360               120
identity behavior            3         0.8           72                24
total                       55                      612                  

false-alarm threshold 3: closed rules T2, T5 (2/9); after closing, caught 9 -> 7, missed 2 -> 4, false alarm 13 -> 7
added to missed after closing: bulk-archive-delete, leak-to-new-destination; the cost of the sources the closed rules watch (audit log, network flow) keeps being paid
collection cap 60 GB/day; used 55, remaining 5 -> 5 GB/day of room for a fourth source, network flow doubling would exceed the cap

The numbers are of the measurement class; their inputs are the assumptions above. Nine rules produce twenty-two alerts over thirty events: nine of eleven malicious events are caught, two get away, thirteen benign events get flagged. The false-alarm count being larger than the caught count comes from how this set was chosen — the benign events were chosen to share surfaces with the malicious ones — but the direction of the ratio is real: detection rules do not run without flagging legitimate work.

Rule by rule, the distribution is not even. T1 produces a single alert and has no false ones; it is a narrowly written rule and sees only one event in exchange. T7 gets three of five alerts right. T2 and T5 are the noisiest pair with three false alarms each, and these two numbers come back in the lesson’s last section.

Signal Sources, Separately

The real distinction sits in the source table, and it separates two different questions. The first column says what the source catches alone: audit log three, network flow three, identity behavior four. The three add up to ten, but the combined catch is nine; there is overlap. The third column is more useful — what remains once the source is removed. If the audit log is removed, caught drops from nine to seven; if either network flow or identity behavior is removed, it drops to six.

The last column is the number that carries the decision: the event caught only by this source. Two in the audit log, three in network flow, three in identity behavior. Eight malicious events rest on a single source’s shoulders, and they get named. Only the audit log sees the bulk archive delete and the policy expansion; neither leaves a trace on the network, because no data goes out. Only network flow sees the outbound copy to the store, the leak flowing to a new destination, and the quiet side channel; all three are permitted reads by an authorized subject, and they look ordinary in the audit log. Only identity behavior sees the key exfiltration, the brute-force login, and the shared-account takeover. Of the nine caught events, only one — the nightly role chain — shows up in two sources at once.

Both missed events sit under the tables, named, and the reasons they got away resemble each other. The logging shutdown is a single policy change made during business hours; T1 could have caught it, but its predicate also checks the hour and it does not fire outside the night. The slow exfiltration is a 45 MB read; well under T4’s threshold, and the destination is not new for T5. Both sit on the other side of the rule set’s threshold, not its scope — meaning they can be caught by loosening it. Removing the hour condition from T1 catches the logging shutdown and turns the release rollout into a false alarm; dropping T4’s threshold to 40 MB catches the slow exfiltration and raises that rule’s false alarms from two to eight. Loosening the threshold moves a missed event to caught, but for every event it moves, it writes more than one item into the false-alarm column; the next section counts what those items do.

The sentence this leads to is this: the three sources are not backups for each other. Cutting a source because of its cost turns the events that source alone saw directly into missed events. Which source can be cut is decided by looking at this column, not at its cost.

Detection’s Two Costs

The first cost is the charge table. The three sources produce 55 GB a day and cost 612 fictional units a month. The line items are not equal: network flow takes three quarters of the volume and 360 units of the charge, identity behavior takes three GB for 72 units. The last column makes the two comparable — the monthly charge per event caught only by that source: identity behavior 24 units, audit log 90, network flow 120. Network flow is the most expensive source and the only thing that alone sees three events; both are true at once, and the decision is made between these two numbers.

The second cost is not paid in currency. When the false-alarm threshold is set to three, T2 and T5 close; both had produced three false alarms each. After closing, caught goes from nine to seven, missed goes from two to four, false alarm drops from thirteen to seven. The two events added to missed are named: the bulk archive delete and the leak to a new destination. Both were events only the closed rule saw. This measure has the same shape as the one M20/K04 used for fitness functions — every setting that raises catch rate also raises false alarms, and the behavior a false alarm produces is the rule getting closed — the relationship is stated in one sentence and moved past.

This second cost needs underlining: when a rule closes, collection does not stop. The audit log and network flow charge keep being paid exactly as before; what gets cut is not the charge, it is the visibility the charge bought. The decision to close is also made by the team reading the alerts, so detection’s real scope is not the written rule set — it is the rule set left open, and the gap between the two shows up on no bill.

Little remains on the limit side: the daily collection cap is 60 GB, used is 55, remaining is 5. There is room for 5 GB for a fourth signal source, and if network flow doubles, the cap is exceeded. Adding a source and an existing source’s volume growing both run into the same limit.

Summary

• Nine rules produce twenty-two alerts over thirty events: nine of eleven malicious events are caught, two leave no trace in any source, thirteen benign events get flagged.
• Eight of the nine caught events are caught by only a single signal source; only one shows up in two sources at once. The three sources are not backups for each other.
• The cost of removing a source is in the table: if the audit log is removed, caught drops from nine to seven; if either network flow or identity behavior is removed, it drops to six.
• The collection charge is 612 fictional units a month; per event caught only by that source, identity behavior costs 24, audit log 90, network flow 120 units.
• At a false-alarm threshold of three, two rules close: caught goes from 9 to 7, missed goes from 2 to 4, false alarm drops from 13 to 7; the charge for the closed rules’ sources keeps being paid.
• The collection cap is 60 GB/day, used is 55, remaining is 5; a fourth source or network flow’s growth runs into the same limit.

Next Step

Detection rules ran, caught and missed were counted, and the closing of two rules was recorded too. All of these numbers look inward: the team’s own measure, its own threshold, its own decision. When someone outside says “show that access is restricted and unusual activity is monitored,” what is being asked for is not a measure but evidence, and part of that evidence is not even in the tenant’s hands. The next lesson takes compliance requirements by type and separates, for every requirement, where the evidence comes from — the provider’s document or the tenant’s own record — and counts in which requirements the evidence stays in the gap.