Threat Intelligence

The previous lesson’s six hypotheses all came from the environment’s own data: the baseline, the usual parent process, the usual hour. None of them used information about “what has been seen elsewhere.” That information comes from outside, and it comes in two separate forms — individual indicators and behavior definitions.

When the two sit side by side on a list, they look like the same thing; both say “look at this.” Where they part ways is lifespan. An indicator is a specific value, and it becomes useless once that value changes; a behavior definition describes a method, and it stays valid as long as the method does not change. This lesson counts that difference in days and months. The feed below is a model; no provider, source, or campaign name appears in it, and the lifespan values are data.

• DI38. The feed carries 900 indicators: 420 addresses, 380 file hashes, 100 names. Each indicator has a day it was added and a lifespan.
• DI39. Lifespan is drawn from an exponential distribution; the averages are 6, 3, and 21 days by type. These numbers are model data.
• DI40. When an address indicator’s lifespan expires, the value can be reassigned elsewhere. The model sets this probability at 0.55 for addresses, 0.10 for names, and 0 for file hashes.
• DI41. A behavior definition is 12 items, and its lifespan is measured in months. Reassignment does not apply, because what is described is a method, not a value.
• DI42. Sixty fictional contact events are each tied to an indicator and a day. Fifteen percent of the contacts have no behavior definition at all.

An Indicator’s Lifespan

// feed.mjs — the lifespan of an indicator feed and its matching in a 90-day window. THIS IS A MODEL.
// The feed has NO provider, source, or campaign name; the types are fictional.
export const SEED = 20260408;
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = generator(SEED);
const randInt = (n) => Math.floor(r() * n);
export const WINDOW_DAYS = 90;

// [type, item count, average life in days, probability of reassignment once life expires]
const TYPE = [["address", 420, 6, 0.55], ["file hash", 380, 3, 0], ["name", 100, 21, 0.1]];
export const INDICATOR = [];
for (const [type, n, life, reassign] of TYPE)
  for (let k = 0; k < n; k++)
    INDICATOR.push({ type, added: randInt(WINDOW_DAYS), life: Math.max(1, Math.round(-life * Math.log(1 - r()))),
      reassigned: r() < reassign });

// Behavior definition: describes a method, not a single value. Its life is measured in months.
export const BEHAVIOR = Array.from({ length: 12 }, (_, i) => ({
  id: `d${i + 1}`, added: randInt(WINDOW_DAYS), life: 120 + randInt(420), scope: 30 + randInt(60) }));

// Fictional contact events: each is tied to one indicator and one day.
export const CONTACT = Array.from({ length: 60 }, () => ({
  day: randInt(WINDOW_DAYS), indicator: randInt(INDICATOR.length),
  behavior: r() < 0.85 ? randInt(BEHAVIOR.length) : -1 }));   // 15 percent have no behavior definition

export const valid = (x, day) => day >= x.added && day <= x.added + x.life;
const median = (v) => v.slice().sort((a, b) => a - b)[Math.floor(v.length / 2)];

if (import.meta.url === `file://${process.argv[1]}`) {
  const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const yt = fmt([22, 9, 13, 14, 16]);
  console.log(`model (seed ${SEED}): ${INDICATOR.length} indicators, ${BEHAVIOR.length} behavior ` +
    `definitions, ${WINDOW_DAYS}-day window, ${CONTACT.length} fictional contacts`);
  console.log();
  console.log(yt(["type", "item", "median life", "avg. life", "reassigned"]));
  for (const [type] of TYPE) {
    const l = INDICATOR.filter((g) => g.type === type);
    console.log(yt([type, l.length, `${median(l.map((g) => g.life))} day`,
      `${(l.reduce((a, g) => a + g.life, 0) / l.length).toFixed(1)} day`,
      l.filter((g) => g.reassigned).length]));
  }
  console.log(yt(["behavior definition", BEHAVIOR.length, `${median(BEHAVIOR.map((d) => d.life))} day`,
    `${(BEHAVIOR.reduce((a, d) => a + d.life, 0) / BEHAVIOR.length).toFixed(1)} day`, 0]));

  const yl = fmt([16, 10, 16, 10, 13]);
  console.log();
  console.log(yl(["life range", "address", "file hash", "name", "total share"]));
  for (const [lo, hi] of [[1, 2], [3, 7], [8, 14], [15, 30], [31, 9999]]) {
    const inRange = (l) => l.filter((g) => g.life >= lo && g.life <= hi).length;
    const all = inRange(INDICATOR);
    console.log(yl([hi > 999 ? `${lo}+ day` : `${lo}-${hi} day`,
      inRange(INDICATOR.filter((g) => g.type === "address")),
      inRange(INDICATOR.filter((g) => g.type === "file hash")),
      inRange(INDICATOR.filter((g) => g.type === "name")),
      `${(100 * all / INDICATOR.length).toFixed(1)}%`]));
  }

  const yg = fmt([8, 12, 15, 12, 18]);
  console.log();
  console.log(yg(["day", "added", "valid ind.", "dead ind.", "valid behavior"]));
  for (const d of [0, 15, 30, 45, 60, 89]) {
    const va = INDICATOR.filter((x) => valid(x, d)).length;
    const ad = INDICATOR.filter((x) => x.added <= d).length;
    console.log(yg([d, ad, va, ad - va, BEHAVIOR.filter((x) => valid(x, d)).length]));
  }
  const caught = CONTACT.filter((t) => valid(INDICATOR[t.indicator], t.day));
  const early = CONTACT.filter((t) => t.day < INDICATOR[t.indicator].added);
  console.log(`\n${caught.length} of ${CONTACT.length} contacts match the indicator list; ` +
    `${early.length} have an indicator not yet on the list that day, ` +
    `${CONTACT.length - caught.length - early.length} have an indicator whose life has expired`);
}

model (seed 20260408): 900 indicators, 12 behavior definitions, 90-day window, 60 fictional contacts

type                       item  median life     avg. life      reassigned
address                     420        5 day       6.6 day             239
file hash                   380        2 day       3.0 day               0
name                        100       15 day      22.5 day              11
behavior definition          12      358 day     317.4 day               0

life range         address       file hash      name  total share
1-2 day                126             220        15        40.1%
3-7 day                156             129        16        33.4%
8-14 day                96              31        18        16.1%
15-30 day               38               0        28         7.3%
31+ day                  4               0        23         3.0%

day            added     valid ind.   dead ind.    valid behavior
0                  9              9           0                 0
15               142             58          84                 3
30               285             77         208                 4
45               455             77         378                 6
60               622             73         549                 8
89               900             72         828                12

5 of 60 contacts match the indicator list; 28 have an indicator not yet on the list that day, 27 have an indicator whose life has expired

The first table gives the lifespan gap directly: a file hash’s median life is two days, an address’s is five, a name’s is fifteen. A behavior definition’s median life is 358 days — on the same axis, twenty-four to one hundred eighty times longer depending on indicator type. This does not mean a behavior definition is better; it means the two items cannot be managed on the same list with the same procedure.

The lifespan distribution table opens up what the average hides. Forty percent of the feed dies within one or two days, and the bulk of that share is file hashes: 220 of 380 hashes live shorter than two days, and none exceed fifteen. Names sit at the opposite end — 51 of 100 names live longer than fifteen days. If all three types are carried on the same list, picking a single retention period for the whole list mismanages at least two of the three types.

This shape of the distribution also feeds into an operational decision: since 73 percent of items live shorter than a week, pulling the feed once a day means adding a significant share of items to the list at a moment when they are never valid. Unless the feed’s pull frequency is on the same scale as the item’s lifespan, the list grows but coverage does not.

The second table shows how the list behaves. By the ninetieth day, all 900 indicators have been added, but only 72 are valid; 828 are dead. The list grows linearly, while the valid share flattens out between 72 and 77 after the thirtieth day. An indicator feed’s size is not a number about its strength; strength is the count of items still valid on that list on that day.

The last line gives the first result: only 5 of 60 contacts match the indicator list. The two causes of missing it split almost evenly — in 28 contacts the indicator is not yet on the list that day, in 27 its life has expired. The first number measures the feed’s latency, the second measures the indicator’s short lifespan, and they call for two separate fixes: the first wants a faster feed, the second wants a longer-lived measure.

Retention Period and Comparison with Behavior

// life.mjs — retention-window scan, the false positives from dead indicators, and comparison with behavior.
import { INDICATOR, BEHAVIOR, CONTACT, WINDOW_DAYS, valid } from "./feed.mjs";

const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const REVIEW_MIN = 9, MAINT_ITEM = 0.2, MAINT_BEHAVIOR = 30;

// An indicator stays on the list from added..added+S days.
// If it stays on the list after its life expires and the value has been reassigned, it matches legit traffic.
function scan(S) {
  const r = generator(4413571);
  let dead = 0, list = 0;
  for (const g of INDICATOR) {
    list += Math.min(S, WINDOW_DAYS - g.added);
    for (let d = g.added + g.life + 1; d <= Math.min(g.added + S, WINDOW_DAYS - 1); d++)
      if (g.reassigned && r() < 0.02) dead++;
  }
  const caught = CONTACT.filter((t) => {
    const g = INDICATOR[t.indicator];
    return t.day >= g.added && t.day <= g.added + Math.min(S, g.life);
  }).length;
  return { list: Math.round(list / WINDOW_DAYS), caught, dead,
    maint: Math.round(INDICATOR.length * MAINT_ITEM) };
}

const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const yz = fmt([24, 14, 16, 16, 14, 13]);
console.log(yz(["feed / retention", "list size", "caught contact", "dead match",
  "review min", "maint min"]));
for (const S of [3, 7, 14, 30, 90]) {
  const t = scan(S);
  console.log(yz([`indicator, ${S} day`, t.list, `${t.caught}/${CONTACT.length}`, t.dead,
    (t.caught + t.dead) * REVIEW_MIN, t.maint]));
}
const behaviorCaught = CONTACT.filter((t) => t.behavior >= 0 && valid(BEHAVIOR[t.behavior], t.day)).length;
const noDef = CONTACT.filter((t) => t.behavior < 0).length;
console.log(yz(["behavior definition", BEHAVIOR.length, `${behaviorCaught}/${CONTACT.length}`, 0,
  behaviorCaught * REVIEW_MIN, BEHAVIOR.length * MAINT_BEHAVIOR]));

const end = [3, 7, 14, 30, 90].map(scan);
console.log(`\nraising indicator-list retention from 3 to 90 days moves caught ` +
  `${end[0].caught} -> ${end[4].caught}, dead match ${end[0].dead} -> ${end[4].dead}`);
console.log(`of ${CONTACT.length} contacts, ${noDef} have no behavior definition at all; ` +
  `${CONTACT.length - behaviorCaught - noDef} have a definition but it was not yet added on the contact day`);
const perItem = (n, k) => (n / k * 1000).toFixed(1);
console.log(`catch per item (per mille): indicator ` +
  `${perItem(end[4].caught, INDICATOR.length)}, behavior ` +
  `${perItem(behaviorCaught, BEHAVIOR.length)}`);

feed / retention             list size  caught contact      dead match    review min    maint min
indicator, 3 day                    30            2/60               2            36          180
indicator, 7 day                    68            4/60              13           153          180
indicator, 14 day                  131            4/60              45           441          180
indicator, 30 day                  255            5/60             101           954          180
indicator, 90 day                  451            5/60             190          1755          180
behavior definition                 12           27/60               0           243          360

raising indicator-list retention from 3 to 90 days moves caught 2 -> 5, dead match 2 -> 190
of 60 contacts, 10 have no behavior definition at all; 23 have a definition but it was not yet added on the contact day
catch per item (per mille): indicator 5.6, behavior 2250.0

The retention scan repeats the fifth lesson’s curve on a different axis. Raising retention from three days to ninety moves caught contacts from 2 to 5; in the same step, dead matches rise from 2 to 190 and review load from 36 minutes to 1,755 minutes. Three extra catches cost 188 extra false matches, and almost all of it — 185 of the 190 dead matches — comes from expired address indicators colliding with reassigned values. This line says: an indicator list’s most expensive part is the part that has never been cleaned out.

Between thirty and ninety days, caught contacts stay flat while dead matches climb from 101 to 190. Extending retention in that range gains nothing and nearly doubles the cost. A feed’s retention period should be read off the feed’s own lifespan distribution; if most items die within a week, keeping the list for thirty days does not pay off. The list-size column says the same thing in a different unit: at three-day retention the average list is 30 items, at ninety days it is 451, and for that fifteen-fold growth, caught contacts only rise from 2 to 5.

The behavior row is the measure’s real result. Twelve items catch 27 contacts, produce no dead matches, and cost 243 minutes of review — a 900-item indicator list caught 5 contacts and cost 1,755 minutes. Catch per item is 5.6 per mille against 2,250 per mille. Maintenance minutes run higher on the behavior side (360 against 180), because reviewing a behavior definition takes far longer than deleting a value from a list; but that maintenance is for twelve items, not nine hundred.

This comparison does not mean the two feeds are interchangeable. An indicator, when it matches, is an exact match: a value either is or is not on the list, and review finishes in a few minutes. Because a behavior definition is a description, matching it also calls for interpretation, and this lesson’s model takes the review time as equal for both — in reality, reviewing a behavior match takes longer. What the measure says is this: the same maintenance minute covers more contacts on the behavior side, and the indicator side only pays off when the feed’s latency is shorter than the lifespan.

The behavior side has its own misses too, and the last line writes them out: 10 of 60 contacts have no behavior definition at all, and 23 have a definition that had not been added yet on the day of contact. The second number is the counterpart of the indicator side’s latency, and it is larger here because behavior definitions are written more slowly. A behavior definition is long-lived, but it is born late.

Summary

• An indicator’s lifespan is measured in days, a behavior definition’s in months: median 2-15 days against 358 days; the two cannot be managed on the same list with the same procedure.
• Of 900 indicators added by the ninetieth day, only 72 are valid; the list grows linearly while the valid share flattens out after the thirtieth day.
• Only 5 of 60 contacts match the indicator list; 28 have an indicator not yet added, 27 have one whose life has expired, and the two causes call for two separate fixes.
• Raising retention from 3 to 90 days moves caught contacts from 2 to 5 while raising dead matches from 2 to 190 and review time from 36 minutes to 1,755 minutes; between 30 and 90 days, caught contacts do not rise at all.
• Twelve behavior definitions catch 27 contacts and produce no dead matches; catch per item is 5.6 per mille against 2,250 per mille.
• A behavior definition is born late: 23 of 60 contacts have a definition that was not yet added on the day of contact, and 10 have no definition at all.

Next Step

Across eight lessons, every step of the pipeline has been counted: sources were merged, rules were written, a threshold was chosen, a hypothesis was run, and information coming from outside was separated on the lifespan axis. Every number that came out of it points to the same place — analyst minutes. In this eighth lesson, ninety-day retention cost 1,755 minutes; in the sixth lesson, the second version cost 513 minutes a day; and in the first lesson, daily capacity was 1,440 minutes. None of these lessons dealt with what happens after an alert turns out to be correct. The next lesson closes the topic and measures automating repeated response with two numbers together: minutes saved and the cost of a wrong automated action.