Alerting Rules

A reordered dashboard only works when it is looked at. If nobody is looking at three in the morning, the panel ordering changes nothing; someone has to be paged. Which metric the paging rule should be written from — the component’s internal metric or the metric the member sees — was measured in a separate lesson and is not repeated here. The question here is the rule’s form: three different rule types can be written on the same metric, and the three do not catch the same incidents.

The threshold rule compares the value against a fixed bound. The rate rule looks at how fast the value changes compared with one window earlier. The anomaly rule looks for a move outside a band learned from the past. All three are run over the regional measurement network’s (fiction) two ten-day metric series and eight known incidents.

Ten Days, Eight Incidents

// alert/series.mjs — MODEL of two minute-by-minute metric series over ten days and eight known incidents.
// MD19: request count carries a daily pattern (low at night, peak in the afternoon); error rate is
// patternless. MD20: both series have harmless short spikes (once every five thousand minutes, 2-4 min).
// MD21: the eight incidents are known incidents; their windows and target values are written below.
export const SEED = 20260804, DAYS = 10, MINUTES = DAYS * 1440;
export const HOUR_PATTERN = [40, 32, 28, 26, 30, 48, 78, 118, 150, 168, 176, 172,
  165, 170, 178, 174, 160, 138, 120, 104, 92, 78, 62, 50];
// [day, hour, duration min, name, [[series, target value, shape]]] — shape: "spike" or "ramp"
export const INCIDENTS = [
  [2, 11, 20, "sudden error spike", [["error", 6.0, "spike"]]],
  [3, 9, 300, "slow error ramp", [["error", 1.4, "ramp"]]],
  [4, 3, 90, "overnight traffic loss", [["request", 8, "spike"]]],
  [5, 14, 4, "short deep outage", [["error", 9.0, "spike"]]],
  [6, 8, 600, "steady drift", [["error", 0.9, "spike"]]],
  [7, 16, 45, "post-release increase", [["error", 2.6, "spike"]]],
  [8, 1, 300, "overnight request ramp", [["request", 110, "ramp"]]],
  [9, 6, 360, "gradual traffic decline", [["request", 0.45, "ramp"]]],
];
export const window = (i) => [INCIDENTS[i][0] * 1440 + INCIDENTS[i][1] * 60, INCIDENTS[i][2]];

export function series() {
  let s = SEED;
  const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32);
  const S = { request: new Float64Array(MINUTES), error: new Float64Array(MINUTES) };
  const spike = { request: 0, error: 0 };
  for (let t = 0; t < MINUTES; t += 1) {
    const hour = Math.floor(t / 60) % 24, base = HOUR_PATTERN[hour];
    S.request[t] = base * (0.94 + r() * 0.12);
    S.error[t] = 0.2 + (r() * 2 - 1) * 0.08;
    for (const name of ["request", "error"]) {
      if (spike[name] > 0) { spike[name] -= 1; S[name][t] *= name === "error" ? 4.2 : 1.35; }
      else if (r() < 1 / 5000) spike[name] = 2 + Math.floor(r() * 3);
    }
    for (let i = 0; i < INCIDENTS.length; i += 1) {
      const [start, dur] = window(i);
      if (t < start || t >= start + dur) continue;
      for (const [name, target, shape] of INCIDENTS[i][4]) {
        const p = shape === "ramp" ? (t - start + 1) / dur : Math.min(1, (t - start + 1) / 3);
        const eventBase = name === "request" ? HOUR_PATTERN[hour] : 0.2;
        const targetValue = name === "request" && target < 1 ? eventBase * target : target;
        S[name][t] = eventBase + (targetValue - eventBase) * p;
      }
    }
  }
  return S;
}

if (import.meta.filename === process.argv[1]) {
  const S = series();
  const avg = (a, b, name) => { let x = 0; for (let t = a; t < b; t += 1) x += S[name][t]; return x / (b - a); };
  console.log(`seed ${SEED}, ${DAYS} days, ${MINUTES} minutes, 2 metrics, ${INCIDENTS.length} known incidents`);
  console.log(`baseline: overnight error ${avg(0, 1440, "error").toFixed(2)}, request ` +
    `${avg(180, 300, "request").toFixed(0)}/min (03-05), afternoon request ` +
    `${avg(840, 960, "request").toFixed(0)}/min (14-16)\n`);
  console.log(`${"incident".padEnd(24)}${"day".padStart(5)}${"duration".padStart(10)}` +
    `${"peak value in incident".padStart(25)}`);
  INCIDENTS.forEach(([day, , dur, name, targets], i) => {
    const [b] = window(i), a = targets[0][0];
    let peak = S[a][b];
    for (let t = b; t < b + dur; t += 1) peak = a === "error" || targets[0][1] > 100
      ? Math.max(peak, S[a][t]) : Math.min(peak, S[a][t]);
    console.log(`${name.padEnd(24)}${String(day).padStart(5)}${`${dur} min`.padStart(10)}` +
      `${`${a} ${peak.toFixed(a === "error" ? 2 : 0)}`.padStart(25)}`);
  });
}

seed 20260804, 10 days, 14400 minutes, 2 metrics, 8 known incidents
baseline: overnight error 0.20, request 28/min (03-05), afternoon request 176/min (14-16)

incident                  day  duration   peak value in incident
sudden error spike          2    20 min               error 6.00
slow error ramp             3   300 min               error 1.40
overnight traffic loss      4    90 min                request 8
short deep outage           5     4 min               error 9.00
steady drift                6   600 min               error 0.90
post-release increase       7    45 min               error 2.60
overnight request ramp      8   300 min              request 110
gradual traffic decline     9   360 min               request 71

The incident set was deliberately chosen from different shapes: a twenty-minute sudden spike, a slow ramp spread over five hours, a four-minute deep outage, a steady drift lasting ten hours, and two of them on a seasonal metric — an overnight drop in traffic and an overnight ramp in requests. The last one, the gradual traffic decline, never looks low in absolute terms: it drops relative to the afternoon baseline of 176, but the number never falls below 71.

Three Sweeps

Each type is swept with a single parameter. The selection rule is written in the code: among the parameters that do not exceed a budget of at most twelve false alarms over ten days, the one catching the most incidents is chosen.

// alert/rule.mjs — three rule types are run over the same two series of the same ten days. MD22: a rule condition
// produces an alert once it holds for K rounds (minutes) in a row, and does not produce again until the condition
// drops; evaluation runs one round per minute. MD23: the threshold is written as a multiple of the median; the
// rate rule looks at a two-times rise or a halving compared with W minutes earlier; the anomaly rule uses the band
// from the previous L days for the same minute-of-day (mean plus four deviations, band floor 15% of the mean).
// MD24: there is a budget of at most 12 false alarms over ten days, and for each type the parameter with the most
// caught incidents among those not exceeding this budget is chosen.
import { series, INCIDENTS, window, MINUTES } from "./series.mjs";
const S = series(), METRICS = ["error", "request"], STREAK = { threshold: 5, rate: 3, anomaly: 10 }, BUDGET = 12;
const median = Object.fromEntries(METRICS.map((a) =>
  [a, [...S[a]].sort((x, y) => x - y)[Math.floor(MINUTES / 2)]]));

const conditions = {
  threshold: (k) => (a, t) => (a === "error" ? S[a][t] > k * median[a]
    : S[a][t] > k * median[a] || S[a][t] < median[a] / k),
  rate: (w) => (a, t) => { if (t < w) return false;
    const o = S[a][t] / Math.max(1e-9, S[a][t - w]); return o > 2 || o < 0.5; },
  anomaly: (l) => (a, t) => { const g = Math.min(l, Math.floor(t / 1440));
    if (g < 1) return false;
    const past = []; for (let j = 1; j <= g; j += 1) past.push(S[a][t - j * 1440]);
    const m = past.reduce((x, y) => x + y, 0) / g;
    const sd = Math.sqrt(past.reduce((x, y) => x + (y - m) ** 2, 0) / g);
    return Math.abs(S[a][t] - m) > 4 * Math.max(sd, 0.15 * m); },
};

function run(kind, p) {
  const alerts = [];
  const condition = conditions[kind](p);
  for (const a of METRICS) {
    let streak = 0, active = false;
    for (let t = 0; t < MINUTES; t += 1) {
      if (condition(a, t)) { streak += 1; if (streak >= STREAK[kind] && !active) { active = true; alerts.push([t - STREAK[kind] + 1, a]); } }
      else { streak = 0; active = false; }
    }
  }
  const insideEvent = (t) => INCIDENTS.findIndex((o, i) => t >= window(i)[0] && t < window(i)[0] + o[2]);
  const caught = new Map();
  for (const [t] of alerts) { const i = insideEvent(t);
    if (i >= 0 && !caught.has(i)) caught.set(i, t - window(i)[0]); }
  const falseAlarms = alerts.filter(([t]) => insideEvent(t) < 0).length;
  const rounds = [...caught.values()];
  return { p, alerts: alerts.length, caught, missed: INCIDENTS.length - caught.size, falseAlarms,
    round: rounds.length ? rounds.reduce((a, b) => a + b, 0) / rounds.length : null };
}

const SWEEP = { threshold: [2, 3, 5, 8], rate: [10, 30, 60, 180], anomaly: [1, 3, 7] };
const HEADING = { threshold: "threshold (x median)", rate: "rate window (min)", anomaly: "learning window (days)" };
const chosen = {};
for (const kind of Object.keys(SWEEP)) {
  console.log(`${HEADING[kind].padEnd(25)}${"alerts".padStart(8)}${"caught".padStart(11)}` +
    `${"missed".padStart(8)}${"false alarms".padStart(14)}${"detection round".padStart(18)}`);
  const rows = SWEEP[kind].map((p) => run(kind, p));
  for (const d of rows) console.log(`${String(d.p).padEnd(25)}${String(d.alerts).padStart(8)}` +
    `${`${d.caught.size}/${INCIDENTS.length}`.padStart(11)}${String(d.missed).padStart(8)}` +
    `${String(d.falseAlarms).padStart(14)}${(d.round === null ? "-" : `${d.round.toFixed(0)} rounds`).padStart(18)}`);
  const fits = rows.filter((d) => d.falseAlarms <= BUDGET);
  chosen[kind] = (fits.length ? fits : rows).slice()
    .sort((a, b) => b.caught.size - a.caught.size || a.falseAlarms - b.falseAlarms)[0];
  console.log(`chosen: ${chosen[kind].p}\n`);
}

const KINDS = Object.keys(SWEEP);
console.log(`${"incident".padEnd(25)}${KINDS.map((t) => t.padStart(12)).join("")}  alone`);
const alone = [];
INCIDENTS.forEach((o, i) => {
  const h = KINDS.map((t) => chosen[t].caught.get(i));
  const count = h.filter((x) => x !== undefined).length;
  if (count === 1) alone.push([o[3], KINDS[h.findIndex((x) => x !== undefined)]]);
  console.log(`${o[3].padEnd(25)}${h.map((x) => (x === undefined ? "-" : `${x} rounds`).padStart(12)).join("")}` +
    `  ${count === 1 ? KINDS[h.findIndex((x) => x !== undefined)] : count === 0 ? "none" : ""}`);
});
const union = new Set(KINDS.flatMap((t) => [...chosen[t].caught.keys()]));
console.log(`\nall three together ${union.size}/${INCIDENTS.length} incidents, total false alarms ` +
  `${KINDS.reduce((a, t) => a + chosen[t].falseAlarms, 0)}, total alerts ${KINDS.reduce((a, t) => a + chosen[t].alerts, 0)}`);
console.log(`caught by only one type ${alone.length} incidents: ` +
  `${alone.map(([a, t]) => `${a} (${t})`).join(", ")}`);
const SETS = [["threshold"], ["rate"], ["anomaly"], ["threshold", "anomaly"], ["threshold", "rate"],
  ["rate", "anomaly"], ["threshold", "rate", "anomaly"]];
console.log(`\n${"rule set".padEnd(24)}${"incidents covered".padStart(19)}${"false alarms".padStart(14)}` +
  `${"alerts".padStart(8)}`);
for (const k of SETS) {
  const covered = new Set(k.flatMap((t) => [...chosen[t].caught.keys()]));
  console.log(`${k.join("+").padEnd(24)}${`${covered.size}/${INCIDENTS.length}`.padStart(19)}` +
    `${String(k.reduce((a, t) => a + chosen[t].falseAlarms, 0)).padStart(14)}` +
    `${String(k.reduce((a, t) => a + chosen[t].alerts, 0)).padStart(8)}`);
}

threshold (x median)       alerts     caught  missed  false alarms   detection round
2                              15        4/8       4            11         13 rounds
3                              14        5/8       3             9         21 rounds
5                               4        4/8       4             0         52 rounds
8                               3        3/8       5             0          1 rounds
chosen: 3

rate window (min)          alerts     caught  missed  false alarms   detection round
10                             18        5/8       3            13          1 rounds
30                             18        5/8       3            13          0 rounds
60                             19        6/8       2            12          8 rounds
180                            74        8/8       0            57         29 rounds
chosen: 60

learning window (days)     alerts     caught  missed  false alarms   detection round
1                              12        6/8       2             6         21 rounds
3                               8        6/8       2             0         22 rounds
7                               9        6/8       2             0         22 rounds
chosen: 3

incident                    threshold        rate     anomaly  alone
sudden error spike           0 rounds    0 rounds    0 rounds  
slow error ramp            104 rounds   48 rounds   58 rounds  
overnight traffic loss              -    1 rounds    2 rounds  
short deep outage                   -    0 rounds           -  rate
steady drift                 1 rounds    1 rounds    0 rounds  
post-release increase        0 rounds    0 rounds    2 rounds  
overnight request ramp       0 rounds           -   68 rounds  
gradual traffic decline             -           -           -  none

all three together 7/8 incidents, total false alarms 21, total alerts 41
caught by only one type 1 incidents: short deep outage (rate)

rule set                  incidents covered  false alarms  alerts
threshold                               5/8             9      14
rate                                    6/8            12      19
anomaly                                 6/8             0       8
threshold+anomaly                       6/8             9      22
threshold+rate                          7/8            21      33
rate+anomaly                            7/8            12      27
threshold+rate+anomaly                  7/8            21      41

Each sweep shows its own trade-off. As the threshold tightens, false alarms drop to zero (0 false alarms at five times the median) but caught incidents drop too, and the detection round rises to 52. The eight-times threshold has an average detection round of 1; that number does not come from the rule being fast, it comes from the rule catching only fast incidents — the average is taken only over the incidents it caught.

In the rate rule, coverage grows as the window grows: the 180-minute window catches all eight of the eight incidents. Its cost is 57 false alarms, because on the seasonal metric the morning rise also exceeds twice its value from three hours earlier. The window choice here is a compromise between incident shape and the daily pattern.

In the anomaly rule, when the learning window is one day, the band rests on a single day, so it stays narrow and produces six false alarms. At three days, false alarms drop to zero and coverage does not fall. Seven days is no better than three; the return on extending the learning window runs out after the third day.

Complementarity

The incident table shows the three types behaving differently on the same incidents. All three catch the sudden spike at round zero. On the slow ramp, threshold fires at round 104, rate at round 48, anomaly at round 58 — the threshold rule arrives toward the end of the incident, because the value only crosses the threshold that late. Only the rate rule catches the four-minute deep outage: the incident goes above the threshold but ends before the five-minute streak condition fills, while the rate rule’s three-round condition does fill.

The rule-set table carries the real decision. Anomaly alone catches six of the eight incidents with zero false alarms. Adding the threshold rule to this set does not change coverage — all five incidents the threshold catches are already caught by anomaly — but it adds nine false alarms. Adding the rate rule brings coverage from six to seven: the one incident gained is the four-minute deep outage, at a cost of twelve false alarms. Catching one more incident costs twelve false alarms, and that is the subject of the next lesson.

The eighth incident is not caught by any of the three types. The gradual traffic decline never falls below the absolute threshold, its hour-over-hour change never reaches double, and it does not deviate enough to cross the learned band’s lower bound. The cost of the unanswered question here is a six-hour decline going unnoticed; seeing this incident does not come from changing the rule type but from changing the metric — a ratio normalized against the daily pattern instead of an absolute request count.

Summary

• Three rule types catch different incidents on the same metric: threshold looks at the value, rate at the speed of change, anomaly at a learned band.
• The parameter sweep showed a trade-off in every type: as the threshold tightens, false alarms drop to zero but the detection round rises to 52; as the rate window grows to 180 minutes, coverage reaches 8/8 but false alarms jump to 57.
• Extending the anomaly rule’s learning window from one day to three brought false alarms from 6 to 0; extending it to seven days had no further payoff.
• Only the rate rule caught the four-minute deep outage; threshold and anomaly missed it because their streak condition is longer than the incident itself.
• Anomaly alone catches 6/8 incidents with zero false alarms; adding threshold brings nine false alarms without changing coverage, adding rate brings twelve false alarms in exchange for one incident.
• The eighth incident was not caught by any type: what is missing is not the rule type but the metric itself.

Next Step

When this lesson’s three rules were run together, they produced 41 alerts over ten days, and 21 of them matched no incident. The numbers look small on paper; but these alerts do not go to a list, they go to a person. When the same incident trips more than one rule at once, that person is alerted three times for the same thing, closes the fourth alert of the night without reading it, and when a real incident sits among the alerts closed unread, nobody notices. The next lesson applies grouping, suppression, and dependency suppression to the alerts produced over a period and counts alerts per person, the actionable-alert ratio, and the real incidents suppression misses.