Intrusion Detection and Prevention Systems

The previous lesson wrote two hundred thirteen thousand connections into a record. Keeping a record is not the same as seeing: nothing among the period’s records says which one is worth looking at. This lesson’s subject is the layer that takes on that judgment — and that layer itself does something countable, because every “worth looking at” decision produces one correct call and one wrong one.

Three numbers are measured: how many of the period’s bad events are caught, how many legitimate events are wrongly flagged alongside the caught ones, and how much legitimate work stops when detection switches to prevention — that is, when connections start being cut instead of merely recorded. A fourth is added to these: whether the sensor sits on the network or on the host, because the two see different event sets over the same period. No attack step is written in this lesson; the event classes are the indicators the defense sees.

• PD18. The fourteen event classes, the period counts, and the seven signatures belong to the fictional regional measurement network. No real event is produced; what is measured is the model built with node.
• PD19. The true class of each event was marked by hand. In reality this class is known only after investigation; the model assumes it is known so that false positives can be counted.
• PD20. An event class matches either a single signature or none. In reality an event can touch more than one signature; that enlarges the alert count without changing the direction of the ratios.
• PD21. Visibility was modeled with three values: from the network only, from the host only, or from both. An encrypted flow was treated as open on the host side and closed on the network side.
• PD22. Threshold, frequency window, and a learning phase were not modeled. A signature either alerts or it does not.
• PD23. In prevention mode, every event that alerts is assumed to be stopped; there is no latency or partial block.

The Event Set and Two Sensor Types

The model holds five things for each event class: its name, its true class, which locations can see it, its count for the period, and the signature it matches. A sensor’s alerting depends on two conditions — being able to see the event, and a signature matching. Because the two can be counted separately, missed events can be split into two causes.

// event.mjs — period event set, signature list, and two sensor types; a model, not a real event
// [name, true class, visibility, period count, matching signature]
export const EVENT = [
  ['heavy attempts against the management gate', 'bad', 'both', 12000, 'heavy-attempt'],
  ['outbound to an unknown server name', 'bad', 'network', 40, 'unknown-name'],
  ['new privileged process on host', 'bad', 'host', 6, 'new-process'],
  ['small outbound at regular intervals', 'bad', 'both', 2880, null],
  ['local file permission change', 'bad', 'host', 14, 'permission-change'],
  ['known payload inside encrypted flow', 'bad', 'host', 3, 'known-payload'],
  ['scan-like sequence in the inside zone', 'bad', 'network', 220, 'scan-sequence'],
  ['quarter-end batch query', 'legit', 'both', 12, 'scan-sequence'],
  ['backup transfer', 'legit', 'both', 12, 'large-volume'],
  ['patch rollout', 'legit', 'both', 90, 'new-process'],
  ['field-endpoint reconnection wave', 'legit', 'network', 5100, 'heavy-attempt'],
  ['metric pull', 'legit', 'both', 259200, null],
  ['branch interface requests', 'legit', 'network', 184000, null],
  ['time-source query', 'legit', 'network', 43200, null],
];
export const SENSOR = { network: ['network', 'both'], host: ['host', 'both'], both: ['network', 'host', 'both'] };
export const SIGNATURE = [...new Set(EVENT.map((o) => o[4]).filter(Boolean))];
export const sees = (s, o) => SENSOR[s].includes(o[2]);
export const alerts = (s, o, closed = []) => sees(s, o) && o[4] !== null && !closed.includes(o[4]);
export const sum = (l) => l.reduce((a, o) => a + o[3], 0);
export function count(s, closed = []) {
  const u = (f) => EVENT.filter((o) => f(o));
  const caught = u((o) => o[1] === 'bad' && alerts(s, o, closed));
  const missed = u((o) => o[1] === 'bad' && !alerts(s, o, closed));
  const falsePositive = u((o) => o[1] === 'legit' && alerts(s, o, closed));
  return { caught, missed, falsePositive, alert: sum(caught) + sum(falsePositive) };
}

The signature set has seven entries and none belongs to a product; each is the name of an indicator. The period’s total event count exceeds five hundred six thousand, the large majority of which comes from three classes, and those three classes have no signature at all. Measurement therefore starts with class, not volume: which class is visible from which location, and which class a signature touches.

Network-Based versus Host-Based

// run1.mjs — classes seen by network-based and host-based sensors, caught and missed events
import { EVENT, SENSOR, SIGNATURE, sees, alerts, sum, count } from './event.mjs';

const BAD = EVENT.filter((o) => o[1] === 'bad');
console.log(`${EVENT.length} event classes, ${BAD.length} bad / ${EVENT.length - BAD.length} legitimate; ` +
  `${sum(EVENT)} events in the period, ${SIGNATURE.length} signatures\n`);
console.log(`${'event class'.padEnd(43)} ${'class'.padStart(6)} ${'count'.padStart(7)} ${'signature'.padStart(18)} ` +
  `${'network'.padStart(8)} ${'host'.padStart(8)}`);
for (const o of EVENT)
  console.log(`${o[0].padEnd(43)} ${o[1].padStart(6)} ${String(o[3]).padStart(7)} ${String(o[4] ?? '-').padStart(18)} ` +
    `${(sees('network', o) ? (alerts('network', o) ? 'alert' : 'sees') : '-').padStart(8)} ` +
    `${(sees('host', o) ? (alerts('host', o) ? 'alert' : 'sees') : '-').padStart(8)}`);
console.log(`\n${'sensor'.padEnd(11)} ${'seen classes'.padStart(13)} ${'caught bad'.padStart(14)} ` +
  `${'missed bad'.padStart(10)} ${'false-positive classes'.padStart(23)} ${'period alerts'.padStart(14)}`);
for (const s of Object.keys(SENSOR)) {
  const r = count(s), seen = EVENT.filter((o) => sees(s, o)).length;
  console.log(`${s.padEnd(11)} ${String(seen).padStart(13)} ${`${r.caught.length}/${BAD.length}`.padStart(14)} ` +
    `${String(r.missed.length).padStart(10)} ${String(r.falsePositive.length).padStart(23)} ${String(r.alert).padStart(14)}`);
}
const both = count('both');
console.log(`\nbad classes missed with both sensors together:`);
for (const o of both.missed) console.log(`  ${o[0].padEnd(43)} ${String(o[3]).padStart(6)}  ${o[4] === null ? 'no signature' : 'not seen'}`);

14 event classes, 7 bad / 7 legitimate; 506777 events in the period, 7 signatures

event class                                  class   count          signature  network     host
heavy attempts against the management gate     bad   12000      heavy-attempt    alert    alert
outbound to an unknown server name             bad      40       unknown-name    alert        -
new privileged process on host                 bad       6        new-process        -    alert
small outbound at regular intervals            bad    2880                  -     sees     sees
local file permission change                   bad      14  permission-change        -    alert
known payload inside encrypted flow            bad       3      known-payload        -    alert
scan-like sequence in the inside zone          bad     220      scan-sequence    alert        -
quarter-end batch query                      legit      12      scan-sequence    alert    alert
backup transfer                              legit      12       large-volume    alert    alert
patch rollout                                legit      90        new-process    alert    alert
field-endpoint reconnection wave             legit    5100      heavy-attempt    alert        -
metric pull                                  legit  259200                  -     sees     sees
branch interface requests                    legit  184000                  -     sees        -
time-source query                            legit   43200                  -     sees        -

sensor       seen classes     caught bad missed bad  false-positive classes  period alerts
network                11            3/7          4                       4          17474
host                    9            4/7          3                       3          12137
both                   14            6/7          1                       4          17497

bad classes missed with both sensors together:
  small outbound at regular intervals           2880  no signature

The two sensor types see different sets over the same period, and their intersection is not complete. The network-based sensor sees eleven classes, the host-based sensor nine; their union is fourteen, meaning every class is visible from at least one. Where they diverge is readable row by row. A privileged process opened on the host and a local file permission change are never visible from the network, because these events produce no packet. The known payload inside an encrypted flow is likewise invisible from the network — the loss measured in the first lesson shows up here on the detection side. The scan-like connection sequence in the inside zone, by contrast, is invisible from the host: each machine sees only the single connection that falls to it, and seeing the sequence requires looking at every machine’s connections at the same time.

The caught column is the sum of this difference. The network-based sensor catches three of the seven bad classes, the host-based sensor four; together they catch six. A single class escapes both, and the reason is not visibility: both sensors see the small outbound at regular intervals, but no signature matches. This event breaks no rule; what is anomalous is its regularity, and regularity is caught by a baseline, not a signature.

The false-positive column must be read in the opposite direction. The network-based sensor produces false alerts in four classes, the host-based sensor in three; running both together keeps the false-positive class count at four but raises the alert count from seventeen thousand four hundred seventy-four to seventeen thousand four hundred ninety-seven. When two sensors flag the same event twice, the caught class count does not grow — the queue does. The gain from setting up both — the caught class count rising from three or four to six — is not free: two separate rule sets, two separate queues, and an item to be deployed on every machine on the host side.

Detection, Prevention, and Closing a Signature

// run2.mjs — the same set first in detection then in prevention mode; both sides of closing a signature
import { EVENT, SIGNATURE, alerts, sum, count } from './event.mjs';

console.log(`${'signature'.padEnd(18)} ${'alerted bad'.padStart(13)} ${'alerted legit'.padStart(14)} ${'noise share'.padStart(12)}`);
for (const i of SIGNATURE) {
  const e = EVENT.filter((o) => o[4] === i && alerts('both', o));
  const [k, m] = [sum(e.filter((o) => o[1] === 'bad')), sum(e.filter((o) => o[1] === 'legit'))];
  console.log(`${i.padEnd(18)} ${String(k).padStart(13)} ${String(m).padStart(14)} ${`${(m * 100 / (k + m)).toFixed(1)}%`.padStart(12)}`);
}
console.log(`\n${'signature set'.padEnd(26)} ${'alerts'.padStart(6)} ${'noise'.padStart(7)} ${'caught bad'.padStart(14)} ` +
  `${'missed bad'.padStart(10)} ${'legit cut in prevention'.padStart(24)}`);
for (const [name, closed] of [['all open', []], ['large-volume closed', ['large-volume']],
  ['+ heavy-attempt closed', ['large-volume', 'heavy-attempt']]]) {
  const r = count('both', closed);
  console.log(`${name.padEnd(26)} ${String(r.alert).padStart(6)} ${`${(sum(r.falsePositive) * 100 / r.alert).toFixed(1)}%`.padStart(7)} ` +
    `${String(sum(r.caught)).padStart(14)} ${String(sum(r.missed)).padStart(10)} ${String(sum(r.falsePositive)).padStart(24)}`);
}
const r = count('both');
console.log('\nlegitimate work cut once switched to prevention mode:');
for (const o of r.falsePositive) console.log(`  ${o[0].padEnd(34)} ${String(o[3]).padStart(5)}  signature: ${o[4]}`);
console.log(`\n${'measure'.padEnd(30)} ${'detection mode'.padStart(15)} ${'prevention mode'.padStart(16)}`);
for (const [name, t, o] of [['bad events stopped', 0, sum(r.caught)], ['legitimate events cut', 0, sum(r.falsePositive)],
  ['alerts needing manual review', r.alert, r.alert], ['bad events missed', sum(r.missed), sum(r.missed)]])
  console.log(`${name.padEnd(30)} ${String(t).padStart(15)} ${String(o).padStart(16)}`);

signature            alerted bad  alerted legit  noise share
heavy-attempt              12000           5100        29.8%
unknown-name                  40              0         0.0%
new-process                    6             90        93.8%
permission-change             14              0         0.0%
known-payload                  3              0         0.0%
scan-sequence                220             12         5.2%
large-volume                   0             12       100.0%

signature set              alerts   noise     caught bad missed bad  legit cut in prevention
all open                    17497   29.8%          12283       2880                     5214
large-volume closed         17485   29.8%          12283       2880                     5202
+ heavy-attempt closed        385   26.5%            283      14880                      102

legitimate work cut once switched to prevention mode:
  quarter-end batch query               12  signature: scan-sequence
  backup transfer                       12  signature: large-volume
  patch rollout                         90  signature: new-process
  field-endpoint reconnection wave    5100  signature: heavy-attempt

measure                         detection mode  prevention mode
bad events stopped                           0            12283
legitimate events cut                        0             5214
alerts needing manual review             17497            17497
bad events missed                         2880             2880

The upper table weighs each signature individually, and there are two extremes. The large-volume transfer signature caught no bad event in the period and flagged twelve legitimate backup transfers: a noise share of one hundred percent. The known-payload and permission-change signatures produced no false positive at all. The difference lies in how narrowly the signature is written; a signature that measures volume looks at the same thing the work itself produces, while a file permission changing in one specific way never occurs anywhere in that work.

The lower table runs the outcome of that weighing. Closing the signature whose noise share is one hundred percent drops twelve alerts and costs no detection — this is a gain with no cost, and it is rare. The second step shows the real trade-off. The heavy-attempt signature produces almost all of the period’s false positives on its own — five thousand one hundred of the five thousand two hundred fourteen; closing it brings the alert count down from seventeen thousand to three hundred eighty-five. The same move drops caught bad events from twelve thousand to two hundred eighty-three and raises missed bad events from two thousand eight hundred eighty to fourteen thousand eight hundred eighty. The noise share, meanwhile, falls from thirty percent to twenty-six percent: the ratio of noise barely changes, because both the numerator and the denominator came from the same signature. A system that decides by noise ratio would mistake this move for an improvement.

The final table gives the only real difference between detection and prevention. The two modes produce the same alerts and miss the same events; where they diverge is the first two rows. Prevention mode stops twelve thousand two hundred eighty-three bad events and, in exchange, cuts five thousand two hundred fourteen legitimate events. Which of the four cut classes these are is written in the list below, and what matters more than the count is the class itself: ninety patch rollouts and twelve backup transfers get cut. When a prevention system stops a patch rollout, it does not merely protect the network for that period; by leaving machines unpatched, it produces the next period’s open item itself. Cutting five thousand one hundred field-endpoint reconnection attempts, in turn, means a loss of measurement data.

One of the cut classes shows the narrowest form of this trade-off. Patch rollout and the privileged process opened on the host trigger the same signature, and that signature’s noise share is ninety-four percent: ninety of ninety-six flags are wrong. Leaving such a signature active in prevention mode means cutting ninety patch rollouts to stop six real events. Closing it, on the other hand, makes those six events fully invisible. A third option does not exist in the model, and in reality it is not cheap either: making the signature context-aware, that is, knowing what started the process. That information does not exist at the network layer at all.

In detection mode none of these costs exist; in their place is seventeen thousand four hundred ninety-seven alerts forming a queue that must be reviewed by hand. The choice between the two modes is not a choice about accuracy — the same signature set, the same errors. What is chosen is where the error gets written: into cut legitimate work, or into a queue no one will ever look through to the end.

Summary

• The network-based sensor sees eleven of fourteen classes, the host-based sensor nine. A privileged process opened on the host and a file permission change are never visible from the network; the scan-like sequence in the inside zone is not visible from the host, because each machine sees only its own single connection.
• Together the two catch six of seven bad classes. Both see the one class that escapes; no signature matches — the only anomaly is its regularity.
• The noise share of the seven signatures ranges from 0% to 100%. The large-volume signature catches no bad event in the period and flags 12 legitimate transfers; closing it is a gain with no cost.
• Closing the heavy-attempt signature brings alerts down from 17,497 to 385, but drops caught bad events from 12,283 to 283 and raises missed events from 2,880 to 14,880. The noise ratio falls from 29.8% to 26.5%: a system that decides by ratio mistakes this move for an improvement.
• Prevention mode stops 12,283 bad events and cuts 5,214 legitimate events across 4 classes; among those cut are 90 patch rollouts. Both modes miss the same events; what is chosen is whether the error gets written into cut work or into an alert queue no one can review.

Next Step

Every bit of noise in this lesson came from one place: the signatures were looking at a space where both legitimate work and bad events pass through. The way to lower the false-positive rate is not to write better signatures, but to look at a place legitimate work never touches at all. That is the next lesson’s question: if an item is placed that no legitimate connection in the period should ever touch, what is the false-positive rate of every interaction that reaches it, on which day of the period does that signal arrive, and how many items remain that the same system never sees at all.