--- title: 'Security Information and Event Management' source: 'https://academia.sh/en/courses/security-operations/security-information-and-event-management' course: 'Security Operations and Incident Response' language: en updated: '2026-09-26T03:02:56+00:00' license: 'CC BY-SA 4.0' objectives: - 'Writes a three-condition correlation rule as a state machine and runs it against an event stream' - "Counts the rule's two outputs separately: how many raw events it merges into a single alert, and which campaign it never sees" - 'Measures the chain from record to decision step by step, and records the candidates eliminated and minutes added at each step' --- # Security Information and Event Management A three-condition correlation rule runs against a fictional 3,083-record event stream: the rule produces 10 alerts, and these alerts merge 161 raw events, 16.1 events per alert. Three of five fictional campaigns are caught; one stays below the threshold, one never appears because its third step exists only in network flow. The funnel narrows from 3,083 records to 12 candidates, from 12 candidates to 10 alerts, from 10 alerts to 3 decisions, and takes 43 minutes from the first record landing. In the threshold, window and lag sweep, no setting brings in all five of the five campaigns; the best row is 4/5 and 1 false positive. The previous lesson clustered records with a single criterion: the same actor and a time window. That criterion produced 184 clusters, and only 21 of them corresponded to an event the model actually knew about. A security information and event management system does not cluster this way; it writes a **condition sequence** — which field from which source, in which order, after which count. This lesson runs such a rule over a normalized stream and produces two numbers: how many raw events the rule merges into a single alert, and what it never sees outside what it merges. The stream below is a **model**, and events enter only **as records**; nowhere does it write how an attack step is carried out. **Repeated password attempts against a set of accounts, without written authorization and a defined scope, is both illegal and unprofessional**; what is measured here is the record the attempt leaves behind. - **DI13.** The stream covers one day: 120 personnel accounts, 6 service identities and four event types. A service identity cannot change permissions, so that record **never exists** for it. - **DI14.** Five fictional campaigns are given as record sequences. The label is the model's ground truth and is **never written to the stream**; the rule cannot see it. - **DI15.** Correlation rule R1 has three conditions: at least `E` failed records at the same actor within `P` minutes, then a success, then a permission change within `G` minutes. - **DI16.** An alert's **evidence** is the set of raw events the rule held on to while producing it. The merge measure is the size of this set. - **DI17.** The chain's fixed lags are model data: transmission and normalization take 3 minutes, the alert queue 14 minutes, analyst review 9 minutes per alert. ## A Correlation Rule's State Machine ```js // flow.mjs — fictional event stream and a correlation rule's state machine. IS A MODEL. // Events enter as RECORDS; not the attack step but the record the step leaves behind is modeled. export const SEED = 20260403; const generator = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = generator(SEED); const randInt = (n) => Math.floor(r() * n); const ACCOUNT = Array.from({ length: 120 }, (_, i) => `sh-${301 + i}`); const SERVICE = Array.from({ length: 6 }, (_, i) => `sv-${11 + i}`); const raw = []; // {minute, actor, type, label} — label is the model's ground truth const add = (t, a, type, e) => raw.push({ t: Math.round(t), actor: a, type, label: e }); const times = (n, f) => { for (let k = 0; k < n; k++) f(); }; // Background: routine daily work. A service identity cannot change permissions, so that record // never exists for it. for (const h of ACCOUNT) { times(2 + randInt(6), () => add(randInt(1440), h, "identity.success", "routine")); times(randInt(9), () => add(randInt(1440), h, "identity.failure", "routine")); times(randInt(4), () => add(randInt(1440), h, "permission.change", "routine")); times(4 + randInt(14), () => add(randInt(1440), h, "flow.outbound", "routine")); } for (const h of SERVICE) { times(6 + randInt(10), () => add(randInt(1440), h, "identity.success", "routine")); times(randInt(20), () => add(randInt(1440), h, "identity.failure", "routine")); times(20 + randInt(40), () => add(randInt(1440), h, "flow.outbound", "routine")); } // Dense password reset: a legitimate burst that crosses the threshold, followed by a login and // permission work. for (const h of ACCOUNT.slice(0, 9)) { let t = 420 + randInt(600); times(6 + randInt(5), () => add(t += 1 + r() * 3, h, "identity.failure", "reset")); add(t += 12, h, "identity.success", "reset"); if (r() < 0.55) add(t + 5 + randInt(30), h, "permission.change", "reset"); } // Five fictional campaigns: [name, actor, failure-record count, third-step type, lag min]. // Only the RECORD sequence is given; the method is not described. const CAMPAIGN = [["k1", ACCOUNT[8], 40, "permission.change", 25], ["k2", ACCOUNT[21], 22, "permission.change", 48], ["k3", ACCOUNT[55], 61, "permission.change", 9], ["k4", SERVICE[2], 33, "flow.outbound", 20], // third step only in network flow ["k5", ACCOUNT[96], 4, "permission.change", 35]]; // failure count below threshold for (const [name, h, n, third, lag] of CAMPAIGN) { let t = 60 + randInt(1200); times(n, () => add(t += 0.4 + r() * 1.2, h, "identity.failure", name)); add(t += 3, h, "identity.success", name); add(t + lag, h, third, name); } raw.sort((a, b) => a.t - b.t); export const EVENT = raw.map(({ t, actor, type }) => ({ t, actor, type })); export const LABEL = raw.map((o) => o.label); // the pipeline never SEES this export const NAME = CAMPAIGN.map(([a]) => a); // Correlation rule R1 (state machine): at the same actor, >=E failed records within P // minutes, then a success, then a permission change within G minutes. export function R1({ threshold = 5, window = 30, lag = 60 } = {}) { const state = new Map(), alerts = [], candidates = new Set(), passed = new Set(); for (const [i, o] of EVENT.entries()) { const d = state.get(o.actor) || { failed: [], open: null, evidence: [] }; state.set(o.actor, d); if (o.type === "identity.failure") { d.failed = d.failed.filter((x) => EVENT[x].t > o.t - window).concat(i); if (d.failed.length >= threshold) candidates.add(o.actor); } else if (o.type === "identity.success" && d.failed.length >= threshold) { d.open = o.t; d.evidence = [...d.failed, i]; passed.add(o.actor); } else if (o.type === "permission.change" && d.open !== null && o.t - d.open <= lag) { alerts.push({ actor: o.actor, t: o.t, evidence: [...d.evidence, i] }); d.open = null; d.failed = []; } } return { alerts, candidates: candidates.size, passed: passed.size }; } export const catches = (alerts, name) => alerts.find((x) => x.evidence.some((i) => LABEL[i] === name)); export const falsePositives = (alerts) => alerts.filter((x) => !x.evidence.some((i) => NAME.includes(LABEL[i]))); if (import.meta.url === `file://${process.argv[1]}`) { const s = R1(), merged = s.alerts.reduce((a, u) => a + u.evidence.length, 0); console.log(`model (seed ${SEED}): ${EVENT.length} events, ${ACCOUNT.length} personnel and ` + `${SERVICE.length} service accounts, ${CAMPAIGN.length} fictional campaigns (as records)`); console.log(`R1 (threshold 5 / window 30 min / lag 60 min): ${s.alerts.length} alerts; ` + `these alerts merge ${merged} raw events, per alert ` + `${(merged / s.alerts.length).toFixed(1)}`); const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = fmt([11, 9, 18, 11, 15, 32]); console.log(); console.log(yz(["campaign", "records", "third step", "caught", "merged events", "reason not seen"])); for (const [name, , n, third] of CAMPAIGN) { const u = catches(s.alerts, name); console.log(yz([name, n + 2, third, u ? "yes" : "no", u ? u.evidence.length : 0, u ? "-" : third === "flow.outbound" ? "third step only in network flow" : "failure count below threshold"])); } console.log(`\ncandidate actors ${s.candidates}, passed second condition ${s.passed}, alerts ` + `${s.alerts.length}; ${falsePositives(s.alerts).length} of these contain no campaign ` + `record`); } ``` ``` model (seed 20260403): 3083 events, 120 personnel and 6 service accounts, 5 fictional campaigns (as records) R1 (threshold 5 / window 30 min / lag 60 min): 10 alerts; these alerts merge 161 raw events, per alert 16.1 campaign records third step caught merged events reason not seen k1 42 permission.change yes 32 - k2 24 permission.change yes 24 - k3 63 permission.change yes 32 - k4 35 flow.outbound no 0 third step only in network flow k5 6 permission.change no 0 failure count below threshold candidate actors 12, passed second condition 12, alerts 10; 7 of these contain no campaign record ``` The merge ratio is the rule's most visible work: 10 alerts gather 161 raw events, 16.1 records per alert. The analyst faces not 161 lines but 10 items, each with an evidence set underneath it. This is the number that replaces the previous lesson's 184 clusters, and it is the rule's **reason for existing**. The third column shows a detail: k3 has 63 records, but only 32 events merge into the alert. The thirty-minute window drops older failures, because the counter is a sliding window. The evidence set is not the whole event, but **the part the rule saw**; the rest is searched by hand during review. The bottom two rows are the lesson's real result. k5 stays below the threshold: four failed records do not cross the threshold of five, and the rule never wakes up. k4 is invisible for a different reason — its third step is not a permission change but an outbound network connection, and the rule's third condition never asks about that type. A permission-change record **never occurs** for a service identity; the rule cannot fire for that actor, by definition. These two rows connect directly to the previous lesson's measure. The rule can only ask about fields that exist in the schema; k4's third record fits the shared schema, but the rule's condition sequence has no place for that type. The gap does not come from the source not being collected — it comes from **which field the rule asks about** — and this distinction is decisive in counting detection coverage. The last line gives the cost: 7 of the 10 alerts contain no campaign record at all. Most of these are the dense password-reset cluster — a legitimate burst of failures that crosses the threshold, followed by a login, followed by routine permission work. The rule's condition sequence cannot tell this sequence apart from a campaign, because both leave the same three records in the same order. ## From Record to Decision: Funnel and Minutes ```js // funnel.mjs — the chain from record to decision: how many candidates are eliminated and how // many minutes are added at each step. import { EVENT, R1, NAME, catches, falsePositives } from "./flow.mjs"; // The chain's fixed lags are model DATA, not a measurement (minutes). const TRANSMISSION = 3, QUEUE = 14, REVIEW = 9; const s = R1(), fp = falsePositives(s.alerts); // The average time the rule has to wait for the third step. const wait = s.alerts.reduce((a, u) => a + (u.t - EVENT[u.evidence[u.evidence.length - 2]].t), 0) / s.alerts.length; const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yh = fmt([48, 12, 11, 13]); console.log(`${EVENT.length} raw events, ${s.alerts.length} alerts, ` + `${s.alerts.length - fp.length} of these contain a campaign record`); console.log(); console.log(yh(["step", "remaining", "eliminated", "added min"])); const step = [["raw records", EVENT.length, 0], ["transmission and normalization", EVENT.length, TRANSMISSION], ["condition 1: 5 failures / 30 min", s.candidates, 0], ["condition 2: then a success", s.passed, 0], ["condition 3: a permission change within 60 min", s.alerts.length, Math.round(wait)], ["alert queue", s.alerts.length, QUEUE], ["analyst review", s.alerts.length - fp.length, REVIEW]]; let previous = EVENT.length, total = 0; for (const [name, remaining, mins] of step) { total += mins; console.log(yh([name, remaining, previous - remaining, mins])); previous = remaining; } console.log(`total: ${EVENT.length} records to ${s.alerts.length - fp.length} decisions, ` + `${total} minutes after the first record lands`); const yz = fmt([10, 10, 10, 8, 8, 12, 13, 13]); console.log(); console.log(yz(["threshold", "window", "lag", "cand.", "alerts", "caught", "false pos.", "review min"])); for (const [threshold, window, lag] of [[5, 30, 60], [3, 30, 60], [8, 30, 60], [5, 5, 60], [3, 5, 60], [5, 30, 15], [5, 30, 240]]) { const k = R1({ threshold, window, lag }); const y = NAME.filter((a) => catches(k.alerts, a)).length; const n = falsePositives(k.alerts).length; console.log(yz([threshold, `${window} min`, `${lag} min`, k.candidates, k.alerts.length, `${y}/${NAME.length}`, n, n * REVIEW])); } ``` ``` 3083 raw events, 10 alerts, 3 of these contain a campaign record step remaining eliminated added min raw records 3083 0 0 transmission and normalization 3083 0 3 condition 1: 5 failures / 30 min 12 3071 0 condition 2: then a success 12 0 0 condition 3: a permission change within 60 min 10 2 17 alert queue 10 0 14 analyst review 3 7 9 total: 3083 records to 3 decisions, 43 minutes after the first record lands threshold window lag cand. alerts caught false pos. review min 5 30 min 60 min 12 10 3/5 7 63 3 30 min 60 min 18 11 4/5 7 63 8 30 min 60 min 10 8 3/5 5 45 5 5 min 60 min 4 2 2/5 0 0 3 5 min 60 min 12 5 4/5 1 9 5 30 min 15 min 12 6 1/5 5 45 5 30 min 240 min 12 11 3/5 8 72 ``` The funnel's first row does all the eliminating in a single step: the first condition narrows 3,083 events down to 12 candidates. The second condition **eliminates nothing** — all twelve actors that crossed the threshold eventually log in. What keeps the rule narrow is not, as one might think, the three conditions coming one after another — it is the first condition; the second condition only marks the start of the chain. The minutes column shows a less obvious cost. The 17 minutes the third condition adds is not a processing delay — the rule **has to wait**, because whether the third record arrives is only known once the lag window closes. The total of 43 minutes is this rule's time to detect, and 17 of it comes from the rule's own definition, 14 from the queue, 9 from review. Transmission and normalization account for only 3 minutes; that is the most any effort to speed up the pipeline could gain. The bottom table draws the parameters' boundary. Pulling the threshold down to three brings in k5, and caught becomes 4/5, but the candidate count rises from 12 to 18. Narrowing the window to five minutes zeroes out the false positive and drops the review load from 63 minutes to zero — in exchange, k2 disappears, because its records are spread across the thirty-minute window. The best row is the combination of threshold 3 and window 5: 4/5 and a single false positive. No row gives 5/5. k4 stays invisible under every setting, because what is missing is not the threshold or the window — it is the **source**: the third step's record exists only in network flow data, and R1 never asks about that type. A parameter sweep cannot step **outside a rule's own definition**. ## Summary - The three-condition correlation rule narrows 3,083 raw events down to 10 alerts, and these alerts merge 161 events; 16.1 records per alert reach the analyst as a single item. - The evidence set is not the whole event: only 32 of k3's 63 records enter the alert, because the counter is a sliding window and drops the older ones. - Three of the five campaigns are caught; k5 stays below the threshold, and k4 never appears — by the rule's own definition — because it leaves its third step only in network flow. - 7 of the 10 alerts contain no campaign record; the dense password-reset cluster leaves the same three records in the same order as a campaign, so the condition sequence cannot tell them apart. - 43 minutes pass from record to decision, and of that, 17 comes from the rule waiting for its third step, 14 from the queue, 9 from review; transmission and normalization account for 3 minutes. ## Next Step k4's third step existed in no identity or management record; it existed only as an outbound connection, in network flow data. That source is structurally different from the other two: a flow record does not write the connection's **content**, only who talked to whom, for how long, and how many bytes. The next lesson turns this constraint into a measurement — which question can be answered without content, which cannot, and how the pattern readable from metadata in an encrypted connection separates out a number of items along an axis the other sources never see at all.