Application-Specific Detection Scenarios

The previous lesson measured the side that queries the log after the incident: how many of the sixteen questions had an answer in the schema. The same records have a second job — announcing the incident that gave rise to the question while it is still going on. At the application level this job does not resemble the one at the network level. The abusing party makes requests with a valid session, to valid endpoints, and often with a positive authorization decision; what is abnormal is not the protocol but a number that a business rule measures.

This lesson runs the detection scenarios built on those numbers over a fictional event stream. The architecture of the detection pipeline, the normalization of events, and the correlation rule were measured in the Security Operations and Incident Response course; they are not repeated here, they become inputs. The only thing measured is the two costs of choosing a threshold: how many abuse sessions are missed and how many times someone is woken up for nothing.

• DR7. The event stream is fictional: 900 sessions in 30 days in the subscriber portal, four roles, and four abuse classes. The numbers come from a generator whose seed is visible.
• DR8. A detection scenario is a threshold rule built on a single application metric. The metrics are read from the business logic: distinct subscriber records read in a session, the number of denied authorization decisions, distinct accounts tried from the same client address, and records read outside the account’s authorized region.
• DR9. An alert counts as real only if the session is an abuse session; otherwise it is a false positive. Business hours are taken as 08–18; a false positive outside this range is an off-hours page and is counted separately.
• DR10. 8% of normal sessions produce a heavy tail, such as an “end-of-day batch job”. This tail is the source of the overlap that no rule can remove.
• DR11. The out-of-region read metric depends on a field that is not in the previous lesson’s schema. If the field is not written, the scenario cannot be built; this is the detection-side counterpart of log design.

Threshold Scan on a Single Metric

// event-stream.mjs — generates an application event stream and runs the business-logic
// detection scenarios threshold by threshold. A MODEL: the sessions and the abuse are fictional.
export const SEED = 20260418, SESSION_COUNT = 900, DAYS = 30;
const rng = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = rng(SEED);
// A session with v < 0.08 produces a normal but heavy tail, like an "end-of-day batch job".
const count = (mean, u, v = 1) => Math.round(mean * (0.35 + 1.3 * u) * (v < 0.08 ? 3.5 : 1));

// Role = the account's normal workload. An integration account is high-volume even when normal.
const ROLES = [
  { name: "subscriber representative", share: 0.42, reads: 12, denials: 2.4, address: 1.4, region: 0.3 },
  { name: "field technician", share: 0.28, reads: 6, denials: 3.0, address: 1.2, region: 0.5 },
  { name: "integration account", share: 0.12, reads: 140, denials: 1.0, address: 2.2, region: 2.5 },
  { name: "subscriber", share: 0.18, reads: 1, denials: 1.6, address: 1.6, region: 0.0 },
];
const pickRole = (u) => { let t = 0;
  for (const x of ROLES) { t += x.share; if (u < t) return x; } return ROLES.at(-1); };

// Abuse class = which metric of the session it pushes outside normal.
export const ABUSE = [
  { name: "bulk read", n: 6, apply: (o, u) => { o.reads = count(300, u); } },
  { name: "authorization probing", n: 4, apply: (o, u) => { o.denials = count(18, u); } },
  { name: "out-of-region leak", n: 5, apply: (o, u) => { o.region = count(12, u); } },
  { name: "account takeover", n: 3, apply: (o, u) => { o.address = count(6, u); } },
];

export const SESSIONS = Array.from({ length: SESSION_COUNT }, (_, i) => {
  const role = pickRole(r()), hour = Math.floor(r() * 24), tail = r();
  return { i, role: role.name, hour, abuse: null, reads: count(role.reads, r(), tail),
    denials: count(role.denials, r(), tail), address: count(role.address, r(), tail),
    region: count(role.region, r(), tail) };
});
for (const k of ABUSE) for (let n = 0; n < k.n; n++) {
  const o = SESSIONS[Math.floor(r() * SESSION_COUNT)];
  o.abuse = k.name; k.apply(o, r());
}
export const ABUSE_TOTAL = SESSIONS.filter((o) => o.abuse).length;

// Scenario = a threshold rule built on a single application metric.
export const SCENARIOS = [
  { name: "distinct record reads", metric: "reads", target: "bulk read",
    thresholds: [40, 80, 150, 260] },
  { name: "denied authorization", metric: "denials", target: "authorization probing",
    thresholds: [3, 6, 12, 25] },
  { name: "accounts per address", metric: "address", target: "account takeover",
    thresholds: [2, 3, 5, 7] },
  { name: "out-of-region reads", metric: "region", target: "out-of-region leak",
    thresholds: [2, 4, 8, 12] },
];
const offHours = (o) => o.hour < 8 || o.hour >= 18;

export function run(s, t) {
  const alerts = SESSIONS.filter((o) => o[s.metric] >= t);
  const falseAlerts = alerts.filter((o) => !o.abuse);
  return { threshold: t, alerts: alerts.length, real: alerts.filter((o) => o.abuse).length,
    targetCaught: alerts.filter((o) => o.abuse === s.target).length,
    targetTotal: ABUSE.find((k) => k.name === s.target).n,
    falseAlerts: falseAlerts.length, pages: falseAlerts.filter(offHours).length };
}

export const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`model (seed ${SEED}): ${SESSION_COUNT} sessions, ${DAYS} days, ` +
    `${ABUSE_TOTAL} abuse sessions`);
  console.log("role distribution: " + ROLES.map((x) =>
    `${x.name} ${SESSIONS.filter((o) => o.role === x.name).length}`).join(", "));
  const y = gen([23, 11, 8, 10, 8, 11, 12]);
  console.log("\n" + y(["scenario", "threshold", "alerts", "missed", "real", "false+",
    "pages"]));
  for (const s of SCENARIOS) for (const t of s.thresholds) {
    const o = run(s, t);
    console.log(y([s.name, t, o.alerts, o.targetTotal - o.targetCaught, o.real, o.falseAlerts,
      o.pages]));
  }
}

model (seed 20260418): 900 sessions, 30 days, 18 abuse sessions
role distribution: subscriber representative 388, field technician 229, integration account 120, subscriber 163

scenario                 threshold  alerts    missed    real     false+       pages
distinct record reads           40     149         0       7        142          83
distinct record reads           80     110         0       7        103          60
distinct record reads          150      58         1       6         52          29
distinct record reads          260      10         2       4          6           5
denied authorization             3     377         0      13        364         211
denied authorization             6      49         0       5         44          28
denied authorization            12      17         0       4         13           6
denied authorization            25       0         4       0          0           0
accounts per address             2     467         0      12        455         261
accounts per address             3     115         0       6        109          65
accounts per address             5      43         1       4         39          25
accounts per address             7      20         2       2         18          14
out-of-region reads              2     127         0       5        122          68
out-of-region reads              4      41         0       5         36          21
out-of-region reads              8       7         2       3          4           3
out-of-region reads             12       3         4       1          2           1

All four scenarios draw the same shape: as the threshold rises, the missed count grows and the false positives fall. What is interesting is where this curve stops. The denied-authorization scenario’s threshold of 3 produces 377 alerts, and 364 of them are false; 211 arrive outside business hours, which is 211 night pages in 30 days. The same scenario’s threshold of 25 gives zero false positives and misses all four of the four abuse sessions at once. At one end there is an alert stream nobody reads, at the other a rule that never alerts; both are two settings of the same rule.

The distinct record reads scenario shows why the overlap does not come from a tuning error. Integration accounts read 140 distinct records on average in their normal work; an abuse session doing bulk reads reads 300 on average. The two distributions overlap. When the threshold is pulled up to 260, the false positives fall to 6 but two of the six bulk-read sessions are missed. What produces the overlap is not the quality of the tool but normal itself: there is a role that legitimately runs high on the same metric, and taking that role out of the rule means leaving the easiest place for abuse to hide outside the rule.

The out-of-region read scenario stands apart in this table. Its metric is not a volume but a rule violation: the account touching a record outside its authorized region. For this reason, at a threshold of 8 it catches three leak sessions with only 4 false positives; in the same stream the accounts-per-address metric pays 39 false positives for four real alerts. The difference comes not from the tightness of the rule but from the choice of metric: a metric based on a business rule separates normal from abnormal by definition, not in the tail of a distribution. It still depends on the tail — at a threshold of 12, four of the five sessions are missed — but it gives the same separating power far more cheaply.

The heavy tail spreads the same fact across all roles. 8% of sessions do an end-of-day job that is normal but three and a half times as intense. At the accounts-per-address scenario’s threshold of 2, 467 sessions raise an alert — more than half of the stream. This threshold is not “sensitive”; it stands at a point where the metric has no separating power and turns the rule’s output into a random sample of the population.

Choosing the Thresholds Together

Scenarios are not tuned one by one. The output of the four rules goes to the person on call together; the cost also arises together.

// thresholds.mjs — the thresholds of the four scenarios are chosen together: the threshold
// combination that catches the most abuse sessions under an alert budget is searched for. A MODEL.
import { SESSIONS, SCENARIOS, ABUSE, ABUSE_TOTAL, gen } from "./event-stream.mjs";

const offHours = (o) => o.hour < 8 || o.hour >= 18;
function combo(thresholds, scenarios = SCENARIOS) {
  const alerts = SESSIONS.filter((o) => scenarios.some((s, j) => o[s.metric] >= thresholds[j]));
  const falseAlerts = alerts.filter((o) => !o.abuse);
  return { alerts: alerts.length, caught: alerts.filter((o) => o.abuse).length,
    falseAlerts: falseAlerts.length, pages: falseAlerts.filter(offHours).length,
    byClass: ABUSE.map((k) => alerts.filter((o) => o.abuse === k.name).length) };
}
function best(budget, scenarios = SCENARIOS) {
  const lists = scenarios.map((s) => s.thresholds);
  let top = null;
  const walk = (j, choice) => {
    if (j === lists.length) {
      const o = combo(choice, scenarios);
      if (o.falseAlerts > budget) return;
      if (!top || o.caught > top.o.caught ||
        (o.caught === top.o.caught && o.falseAlerts < top.o.falseAlerts)) top = { choice: [...choice], o };
      return;
    }
    for (const t of lists[j]) walk(j + 1, [...choice, t]);
  };
  walk(0, []);
  return top;
}

const y = gen([9, 19, 8, 12, 12, 10, 12, 15]);
console.log(y(["budget", "chosen thresholds", "alerts", "caught", "missed", "false+",
  "pages", "false/real"]));
for (const b of [25, 50, 100, 180, 400]) {
  const e = best(b);
  console.log(y([b, e.choice.join(" "), e.o.alerts, `${e.o.caught}/${ABUSE_TOTAL}`,
    ABUSE_TOTAL - e.o.caught, e.o.falseAlerts, e.o.pages,
    (e.o.falseAlerts / e.o.caught).toFixed(1)]));
}
const full = best(100), byClass = ABUSE.map((k, i) => `${k.name} ${full.o.byClass[i]}/${k.n}`);
console.log(`\nclass breakdown of the combination at budget 100: ${byClass.join(", ")}`);

// if the region field is not written, the fourth scenario cannot be built at all.
const three = SCENARIOS.slice(0, 3), withoutRegion = best(100, three);
console.log(`without the region field at the same budget: caught ${withoutRegion.o.caught}/${ABUSE_TOTAL}, ` +
  `from the out-of-region leak ${withoutRegion.o.byClass[2]}/${ABUSE[2].n}`);

// The field's cost from the previous lesson's model: 42e6 events/day, application audit log share 0.18,
// 9.55 GB daily and a 600 GB budget (the schema with the version field added).
const DAILY_EVENTS = 42e6, SHARE = 0.18, BASE = 9.55248e9, BUDGET = 600e9, REGION_BYTES = 14;
const next = BASE + DAILY_EVENTS * SHARE * REGION_BYTES;
console.log(`region field ${REGION_BYTES} bytes: daily ${(BASE / 1e9).toFixed(2)} GB -> ` +
  `${(next / 1e9).toFixed(2)} GB (${((next / BASE - 1) * 100).toFixed(1)}%), ` +
  `retention ${Math.floor(BUDGET / BASE)} -> ${Math.floor(BUDGET / next)} days`);

budget     chosen thresholds  alerts      caught      missed    false+       pages     false/real
25                260 25 7 8      29        8/18          10        21          17            2.6
50                260 12 5 8      56       13/18           5        43          27            3.3
100               260 12 5 4      87       15/18           3        72          43            4.8
180                80 12 3 4     191       18/18           0       173         101            9.6
400                80 12 3 4     191       18/18           0       173         101            9.6

class breakdown of the combination at budget 100: bulk read 4/6, authorization probing 4/4, out-of-region leak 5/5, account takeover 2/3
without the region field at the same budget: caught 13/18, from the out-of-region leak 2/5
region field 14 bytes: daily 9.55 GB -> 9.66 GB (1.1%), retention 62 -> 62 days

The table writes that choosing a threshold is not a setting but a purchase. There is a combination that catches all eighteen sessions, and its price is known: 173 false positives and 101 night pages in 30 days, which is 3.4 times each night on average. There are 9.6 false positives per caught session. In the same stream, the combination with a budget of 100 catches 15 of the 18 and brings the pages down to 43; three sessions are missed. Which one is right cannot be read from this table — what can be read is that both decisions are made in exchange for a number. When the budget is raised from 180 to 400 the row does not change; the separating power of the metrics has run out, and accepting more false positives no longer gets anything caught.

The class breakdown brings a second distinction. The combination at a budget of 100 catches all four authorization probing sessions and all five out-of-region leaks; the three it misses come from the bulk read and account takeover classes — that is, from the metrics where normal is widest. The missed session count is a total; which class it escaped from is separate information, and the decision is made only with that detail.

The last two lines tie this lesson to the previous one. The out-of-region read scenario depends on a field that is not in the previous lesson’s schema. When the field is not written, the scenario cannot be built at any threshold: the number caught at the same budget falls from 15 to 13, and of the five leak sessions only two are seen, because they happen to come out high on other metrics. A field absent from the log removes not a scenario’s threshold but its existence. The field’s logging cost is small — a 1.1% increase in daily volume, a loss of less than one day in retention. The real cost lies elsewhere: the rule requires the account’s list of authorized regions; this list is staff assignment data and enters the log set as a fourth personal data item.

Summary

• Application-specific detection looks not at a protocol anomaly but at a number measured by a business rule: distinct records read, denied authorization, accounts per address, out-of-region reads.
• Every threshold gives the same trade-off, and both ends are useless: a denied-authorization threshold of 3 gives 364 false positives and 211 pages, while 25 gives zero false positives and misses all four of the four sessions at once.
• The source of the overlap is not the quality of the tool but normal itself: integration accounts are legitimately high-volume, and 8% of sessions produce a heavy tail.
• Thresholds are chosen together and the price arises together: all 18 sessions are caught with 173 false positives and 101 pages in 30 days, 15 of them with 72 false positives and 43 pages.
• The class of a missed session is information as much as its count: the three sessions that escape come from the two metrics where normal is widest.
• A field absent from the log removes not the scenario’s threshold but its existence: when the region field is not written the number caught falls from 15 to 13, and the cost is a 1.1% volume increase and a fourth personal data item.

Next Step

At the end of this lesson there is an alert in hand, and the alert is right: a session has read subscriber records without authorization. The questions to be asked in the first minute of the alert are different from the earlier ones, because the answer now feeds not a report but a decision — will the account be shut down, whose data has been seen, where did the intrusion begin. The next lesson runs the response steps in this application breach scenario and measures a single thing: which step needs which record, how many minutes the step is prolonged if that record does not exist, and which step cannot be completed at all.