Threat Hunting

After the previous lesson’s second version, 21 of 51 samples were still missed, and 11 of those sat where no rule could ever reach. Writing rules does not reduce that number, because a rule can only ask the field it can query and only search for the form it was written to match. Threat hunting starts from a different place: not from a rule, but from a hypothesis.

A hypothesis does not produce an alert; it asks a question, and the answer is a count of eliminated items. This lesson runs six hypotheses against a fictional telemetry set and produces three numbers: how many items get eliminated per hypothesis, how many hypotheses return zero results, and how many of the eliminated ones can convert into a rule. In the set below, two fictional items are embedded only as records; nowhere does it write out how an attack step was carried out. Running this kind of search on a system outside your own environment, without written authorization and a defined scope, is both illegal and unprofessional.

DI33. The telemetry set covers one day: 12,910 records, 192 hosts, 90 personnel, and 8 service identities. A record carries a process-execution event and its associated outbound transfer.

DI34. The baseline is each path’s usual parent process. In three percent of records, the parent process falls outside that match; this is not a flaw in the model, it is the counterpart of noise in a real environment.

DI35. A hypothesis consists of sequential filters. Each filter narrows the set, and how many items it eliminates is counted separately.

DI36. The two fictional items are embedded only as records, and their labels never enter the filters. Both are behaviors that the previous lesson’s rule set does not cover.

DI37. Cost has two components: search minutes per hypothesis and 2 review minutes per remaining item. A hypothesis can convert into a rule if its remaining item count stays under the daily ten-alert limit.

Six Hypotheses, Six Funnels

// hunt.mjs — fictional telemetry set and the items six hypotheses eliminate step by step. THIS IS A MODEL.
export const SEED = 20260407;
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = generator(SEED);
const randInt = (n) => Math.floor(r() * n);
const pick = (d) => d[randInt(d.length)];

const HOST = [...Array.from({ length: 180 }, (_, i) => `uc-${101 + i}`),
              ...Array.from({ length: 12 }, (_, i) => `tp-${11 + i}`)];
const ACCOUNT = Array.from({ length: 90 }, (_, i) => `sh-${201 + i}`);
const SERVICE = Array.from({ length: 8 }, (_, i) => `sv-${11 + i}`);
// Baseline: each path's usual parent process (model data).
const PAIR = [["/usr/bin/measurement-client", "/sbin/service-manager"],
  ["/usr/bin/archive", "/usr/bin/scheduler"],
  ["/opt/measurement/collector", "/opt/measurement/verifier"],
  ["/usr/bin/report", "/usr/bin/scheduler"],
  ["/opt/portal/client", "/usr/bin/session"],
  ["/usr/bin/backup", "/usr/bin/scheduler"],
  ["/usr/bin/shell", "/usr/bin/session"]];
export const BASELINE = new Set(PAIR.map(([y, u]) => `${u}|${y}`));
const COLLECTOR_PATH = ["/opt/measurement/collector", "/usr/bin/archive", "/usr/bin/backup"];

// Record: a process-execution event with its associated outbound transfer (fictional).
export const RECORD = [];
const add = (o, label) => RECORD.push({ ...o, label });
for (let k = 0; k < 12000; k++) {
  const host = pick(HOST);
  const path = host.startsWith("tp-") ? pick(COLLECTOR_PATH) : pick(PAIR)[0];
  // Three percent of records: the parent process matches unusually: real-world noise.
  const parent = r() < 0.03 ? pick(PAIR)[1] : PAIR.find(([y]) => y === path)[1];
  add({ host, account: r() < 0.06 ? pick(SERVICE) : pick(ACCOUNT), path, parent,
    hour: 7 + randInt(12), signed: r() < 0.97, external: r() < 0.22,
    bytes: 1000 + randInt(200000) }, "routine");
}
// Night runs: scheduled legit jobs.
for (let k = 0; k < 900; k++) {
  const path = pick(["/usr/bin/backup", "/usr/bin/archive"]);
  add({ host: pick(HOST), account: pick(SERVICE), path, parent: "/usr/bin/scheduler",
    hour: r() < 0.5 ? randInt(6) : 20 + randInt(4), signed: true, external: r() < 0.1,
    bytes: 2e6 + randInt(4e8) }, "night-run");
}
// Two fictional items, given only as RECORD entries.
for (let k = 0; k < 4; k++)
  add({ host: "uc-137", account: "sh-244", path: "/usr/bin/shell", parent: "/usr/bin/session",
    hour: 2 + k, signed: false, external: true, bytes: 40000 + randInt(20000) }, "a1");
for (let k = 0; k < 6; k++)
  add({ host: `tp-${11 + k}`, account: "sv-13", path: "/usr/bin/report",
    parent: "/opt/measurement/verifier", hour: 13, signed: true, external: false,
    bytes: 8000 + randInt(4000) }, "a2");

// Hypothesis: [name, steps (label + filter), search-and-review minutes]
const manyHosts = (l) => {
  const m = new Map();
  for (const o of l) {
    const a = `${o.parent}|${o.path}`;
    (m.get(a) || m.set(a, new Set()).get(a)).add(o.host);
  }
  return l.filter((o) => m.get(`${o.parent}|${o.path}`).size >= 5);
};
export const HYPOTHESIS = [
  ["h1 unsigned binary, off-hours, external target", [
    ["unsigned", (l) => l.filter((o) => !o.signed)],
    ["off-hours", (l) => l.filter((o) => o.hour < 7 || o.hour >= 19)],
    ["external target", (l) => l.filter((o) => o.external)]], 75],
  ["h2 shell with a service identity", [
    ["service identity", (l) => l.filter((o) => o.account.startsWith("sv-"))],
    ["shell path", (l) => l.filter((o) => o.path.endsWith("shell"))]], 40],
  ["h3 off-baseline parent-path, 5+ hosts", [
    ["off-baseline pair", (l) => l.filter((o) => !BASELINE.has(`${o.parent}|${o.path}`))],
    ["5+ hosts", manyHosts]], 90],
  ["h4 unexpected path on a collector", [
    ["collector host", (l) => l.filter((o) => o.host.startsWith("tp-"))],
    ["off-baseline path", (l) => l.filter((o) => !COLLECTOR_PATH.includes(o.path))]], 60],
  ["h5 large night-time outbound transfer", [
    ["off-hours", (l) => l.filter((o) => o.hour < 7 || o.hour >= 19)],
    ["over 10 MB", (l) => l.filter((o) => o.bytes > 1e7)],
    ["external target", (l) => l.filter((o) => o.external)]], 55],
  ["h6 unsigned binary from the scheduler", [
    ["unsigned", (l) => l.filter((o) => !o.signed)],
    ["scheduler parent", (l) => l.filter((o) => o.parent === "/usr/bin/scheduler")]], 45],
];

export function run([, steps]) {
  let l = RECORD, funnel = [RECORD.length];
  for (const [, f] of steps) { l = f(l); funnel.push(l.length); }
  return { remaining: l, funnel,
    item: [...new Set(l.map((o) => o.label))].filter((e) => e.startsWith("a")) };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const yz = fmt([48, 8, 12, 10, 10, 12]);
  console.log(`model (seed ${SEED}): ${RECORD.length} records, ` +
    `${new Set(RECORD.map((o) => o.host)).size} hosts; two fictional items embedded as records`);
  console.log();
  console.log(yz(["hypothesis", "step", "eliminated", "remaining", "minute", "item"]));
  for (const h of HYPOTHESIS) {
    const s = run(h);
    console.log(yz([h[0], h[1].length, RECORD.length - s.remaining.length, s.remaining.length, h[2],
      s.item.join(",") || "-"]));
  }
  const h1 = HYPOTHESIS[0], s1 = run(h1);
  console.log(`\n${h1[0]} — step by step funnel:`);
  const yh = fmt([22, 10, 11]);
  console.log(yh(["step", "remaining", "eliminated"]));
  console.log(yh(["start", s1.funnel[0], "-"]));
  for (const [i, [name]] of h1[1].entries())
    console.log(yh([name, s1.funnel[i + 1], s1.funnel[i] - s1.funnel[i + 1]]));
}

model (seed 20260407): 12910 records, 192 hosts; two fictional items embedded as records

hypothesis                                          step  eliminated remaining    minute        item
h1 unsigned binary, off-hours, external target         3       12906         4        75          a1
h2 shell with a service identity                       2       12800       110        40           -
h3 off-baseline parent-path, 5+ hosts                  2       12627       283        90          a2
h4 unexpected path on a collector                      2       12904         6        60          a2
h5 large night-time outbound transfer                  3       12816        94        55           -
h6 unsigned binary from the scheduler                  2       12744       166        45           -

h1 unsigned binary, off-hours, external target — step by step funnel:
step                   remaining eliminated
start                      12910          -
unsigned                     378      12532
off-hours                      4        374
external target                4          0

The funnel table shows hunting’s real work. h1’s first filter eliminates 12,532 records and leaves 378; the second filter eliminates 374 more. The third filter eliminates nothing — all four remaining records were already going to an external target. This says the hypothesis’s third condition is unnecessary on this data set and can be dropped when it is converted into a rule. Running a hypothesis produces not just the items found but also which condition actually did the work.

Two of the six hypotheses give a usable result: h1 leaves four items, h4 leaves six, and both can be reviewed by hand. h3 finds the same fictional item, but buried inside 283 items; a hypothesis being correct does not make it useful, because if there is no time to review the rest, a finding counts as not found. h3 and h4 find the same fictional item by different routes, and the only difference between them is what the first filter relies on: h4 knows that collector machines have a narrow list of paths, h3 does not.

Three hypotheses find no fictional item at all. h2 searches for “shell execution with a service identity” and finds 110 records; this means the hypothesis was falsified — that behavior is routine in this fictional environment. h6’s 166 records are unsigned binaries running from the scheduler, and they say the same thing.

The number of eliminated items alone does not give a hypothesis’s quality. h6 and h1 use the same first filter and leave the same 378 records; where they part ways is the second filter. h1 asks about the time and leaves 4 items, h6 asks about the parent process and leaves 166. Both hypotheses look at the same data at the same cost; what makes the difference is how rare the second question is in the environment, and that cannot be known while writing the hypothesis — it can only be measured after running it.

Cost, the Value of a Zero Result, and Converting to a Rule

// cost.mjs — cost per hypothesis, the value of a zero result, and the measure for converting to a rule.
import { RECORD, HYPOTHESIS, run } from "./hunt.mjs";

// Model data: review minutes per remaining item and a rule's daily alert limit.
const ITEM_MIN = 2, RULE_LIMIT = 10;
const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = fmt([48, 11, 10, 11, 13, 8, 15]);
const s = HYPOTHESIS.map((h) => ({ h, ...run(h) }));
console.log(yz(["hypothesis", "eliminated", "remaining", "search min", "review min", "item", "rule fit"]));
let totalMin = 0, found = new Set();
for (const x of s) {
  const rev = x.remaining.length * ITEM_MIN;
  totalMin += x.h[2] + rev;
  x.item.forEach((k) => found.add(k));
  console.log(yz([x.h[0], RECORD.length - x.remaining.length, x.remaining.length, x.h[2], rev,
    x.item.join(",") || "-", x.remaining.length <= RULE_LIMIT ? "yes" : "no"]));
}
const empty = s.filter((x) => x.item.length === 0);
console.log(`\ntotal ${totalMin} minutes of hunting, ${found.size} fictional items found: ` +
  `${Math.round(totalMin / found.size)} minutes per item`);
console.log(`${empty.length} hypotheses found no item at all, but ` +
  `${empty.reduce((a, x) => a + x.remaining.length, 0)} items get written to the baseline; ` +
  `these items will not be eliminated by the same hypothesis again`);

// Converting to a rule: a hypothesis that stays under the daily alert limit becomes a rule.
const candidate = s.filter((x) => x.remaining.length <= RULE_LIMIT);
const yk = fmt([48, 14, 19, 13]);
console.log();
console.log(yk(["rule-candidate hypothesis", "daily alert", "monthly review min", "item type"]));
for (const x of candidate)
  console.log(yk([x.h[0], x.remaining.length, x.remaining.length * ITEM_MIN * 30,
    x.item.join(",")]));
console.log(`\n${s.length} hypotheses, ${candidate.length} convert into a rule; ` +
  `${s.length - candidate.length} exceed the daily alert limit or return empty`);
console.log(`in hunting, ${Math.round(totalMin / found.size)} minutes per item, ` +
  `in rule writing, 11 minutes per sample; hunting finds the expensive one`);

hypothesis                                       eliminated remaining search min   review min    item       rule fit
h1 unsigned binary, off-hours, external target        12906         4         75            8      a1            yes
h2 shell with a service identity                      12800       110         40          220       -             no
h3 off-baseline parent-path, 5+ hosts                 12627       283         90          566      a2             no
h4 unexpected path on a collector                     12904         6         60           12      a2            yes
h5 large night-time outbound transfer                 12816        94         55          188       -             no
h6 unsigned binary from the scheduler                 12744       166         45          332       -             no

total 1691 minutes of hunting, 2 fictional items found: 846 minutes per item
3 hypotheses found no item at all, but 370 items get written to the baseline; these items will not be eliminated by the same hypothesis again

rule-candidate hypothesis                          daily alert monthly review min    item type
h1 unsigned binary, off-hours, external target               4                240           a1
h4 unexpected path on a collector                            6                360           a2

6 hypotheses, 2 convert into a rule; 4 exceed the daily alert limit or return empty
in hunting, 846 minutes per item, in rule writing, 11 minutes per sample; hunting finds the expensive one

The cost table’s real information is in the review column. h3’s search cost is 90 minutes, but reviewing the 283 items it leaves takes 566 minutes — the hypothesis’s total bill is 656 minutes, and 86 percent of that comes from the filter’s failure to narrow enough. The way to make a hypothesis cheaper is not searching faster, it is asking a narrower question.

A total of 1,691 minutes finds two fictional items: 846 minutes per item. In the previous lesson, writing a rule cost 11 minutes per sample. The two numbers cannot be compared directly, and here is the direction the comparison points: a rule catches, cheaply, what is already been seen; hunting finds, expensively, what was never written in a form any rule could ask about. Both fictional items are behaviors that the previous lesson’s rule set does not cover.

The three hypotheses that return zero results do not count as wasted. They write 370 items to the baseline: a service identity running a shell, an unsigned binary coming from the scheduler, and a large night-time transfer are all routine in this environment, and they do not need to be eliminated by the same hypothesis again. Hunting’s output is not findings — it is a map of the environment, and the map grows with every run.

The rule-candidate table closes the cycle. h1 would generate 4 alerts a day, h4 would generate 6; both stay under the ten-alert limit, and their monthly review loads are 240 and 360 minutes. The remaining four hypotheses cannot convert into a rule: h3 would generate 283 alerts a day, and the others would become a rule that contains no items at all. Whether a hypothesis converts into a rule depends not on its correctness but on its volume.

Summary

• A hypothesis produces not an alert but a count of eliminated items: h1 eliminates 12,906 records in three steps and leaves 4 items, h4 leaves 6 items in two steps, and both can be reviewed by hand.
• h1’s third filter eliminates nothing; running it produces, along with the items found, which condition did no work, and that condition can be dropped when it is converted into a rule.
• h3 finds the correct fictional item but leaves it buried inside 283 items; if there is no time to review the rest, a finding counts as not found.
• The three hypotheses that return zero results write 370 items to the baseline; this proves those behaviors are routine, and they do not need to be eliminated by the same hypothesis again.
• A total of 1,691 minutes of search finds two fictional items, 846 minutes per item; writing a rule cost 11 minutes per sample, and a rule only catches what has already been seen.
• Two of the six hypotheses convert into a rule because they stay under the daily ten-alert limit; the measure for converting is not correctness, it is volume.

Next Step

All six of this lesson’s hypotheses came from the environment’s own data: the baseline, the usual parent process, the usual hour. None of them used information about “what has been seen elsewhere.” That information comes from outside, and it comes in two separate forms: individual indicators — an address, a hash, a name — and behavior definitions. The two look like the same thing, but their lifespans differ. The next lesson measures that difference: how many days an indicator stays valid, how many months a behavior definition does; how the match rate changes as the feed grows, and where the storage cost comes from.