Alert Design

In the previous lesson’s remaining-budget table, the decision needed to be made on day 12. If no one is looking at the table that day, the number has no value; someone has to be paged. Which rule the page comes from is still an open question. Does the rule watch the notification queue’s depth, the loan service’s memory usage, or the error rate the member sees directly?

This lesson does not choose thresholds. The threshold-and-window search was done in the Performance Anti-Patterns and Monitoring course; the question here is different: the rule’s source. Two separate rule sets are run across the same seven days, and three numbers are compared — alerts produced, rules firing together for the same event, events that produce no alert at all.

The Rule’s Source

A cause-based rule reads a component’s internal metric: memory usage, disk occupancy, cache hit rate, connection pool utilization, garbage collection duration. These metrics come from inside the processes, and their count grows with the number of services.

A symptom-based rule reads what the member sees: error rate, latency, traffic deviation, notification delivery rate. These metrics look the same from outside the system, and their count is independent of the number of services.

The distinction is not arbitrary: an alert pages a person, and the person paged has to have work to do. The loan service’s memory sitting at 91% does not, by itself, define any work; 4% of loan requests coming back failed does. This is what the distinction between a dashboard and an alert answering different questions looks like once it lands in how a rule is written.

Sixteen Rules and Six Events

The generator below writes seven days as per-minute samples. Twelve component metrics and four user metrics are generated at the same minutes; each metric’s base, noise, and threshold sit on their own row. Six events move some of these metrics toward a target value.

// alert/metric.mjs — a 7-day per-minute metric sample. Twelve component metrics (cause) and
// four user metrics (symptom) are generated at the same minutes. Rule format: [name, base,
// noise, direction, threshold]; direction "over" means above the threshold is bad, "under"
// means below it is bad.
export const MIN = 7 * 1440, SEED = 20260731;
export const CAUSE = [
  ["catalog.cache_hit_rate", 92, 2, "under", 70], ["catalog.cpu", 38, 6, "over", 85],
  ["loan.memory", 58, 3, "over", 90], ["loan.gc_ms", 25, 6, "over", 120],
  ["loan.restarts", 0, 0, "over", 0], ["membership.pool_usage", 44, 6, "over", 90],
  ["membership.dependency_ms", 40, 8, "over", 300], ["notification.queue", 12, 8, "over", 400],
  ["notification.active_workers", 8, 1, "under", 3], ["fee.disk", 70, 1, "over", 95],
  ["fee.file_handles", 900, 60, "over", 1800], ["general.retries", 3, 2, "over", 30],
];
export const SYMPTOM = [
  ["error_rate", 0.2, 0.15, "over", 1], ["p95_latency_ms", 180, 25, "over", 400],
  ["traffic_deviation", 0, 6, "over", 45], ["delivery_rate", 99.5, 0.3, "under", 97],
];
// SD1 (assumption): six events [day, hour, duration min, name, [[metric, value during event], ...]]
export const EVENT = [
  [1, 10, 90, "cache collapse", [["catalog.cache_hit_rate", 35], ["catalog.cpu", 91],
    ["p95_latency_ms", 520]]],
  [2, 8, 240, "memory leak", [["loan.memory", 96], ["loan.gc_ms", 190],
    ["loan.restarts", 2]]],
  [3, 0, 300, "disk fill-up", [["fee.disk", 98], ["fee.file_handles", 2100]]],
  [4, 14, 120, "dependency slowdown", [["membership.dependency_ms", 680],
    ["membership.pool_usage", 96], ["general.retries", 48], ["p95_latency_ms", 690],
    ["error_rate", 1.8]]],
  [5, 9, 180, "silent delivery loss", [["delivery_rate", 3]]],
  [6, 11, 60, "post-release error increase", [["error_rate", 4.6]]],
];
export const eventWindow = (i) => [EVENT[i][0] * 1440 + EVENT[i][1] * 60, EVENT[i][2]];

// SD2 (assumption): a component metric is read from a single process, so it spikes for
// 3-7 minutes every 1500 minutes; a user metric is averaged over thousands of requests,
// so it spikes every 6000 minutes.
export function series() {
  let s = SEED;
  const rand = (n) => Math.floor(((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
  const out = {};
  for (const [name, base, noise, dir, threshold] of [...CAUSE, ...SYMPTOM]) {
    const isCause = CAUSE.some((k) => k[0] === name);
    const v = new Float64Array(MIN);
    let spike = 0;
    for (let t = 0; t < MIN; t += 1) {
      let val = base + (noise ? (rand(2001) - 1000) / 1000 * noise : 0);
      if (spike > 0) { spike -= 1; val = dir === "over" ? threshold * 1.15 + 1 : threshold * 0.85; }
      else if (rand(isCause ? 1500 : 6000) === 0) spike = 3 + rand(5);
      for (let i = 0; i < EVENT.length; i += 1) {
        const [start, dur] = eventWindow(i), target = EVENT[i][4].find((h) => h[0] === name);
        if (!target || t < start || t >= start + dur) continue;
        const progress = Math.min(1, (t - start + 1) / Math.max(1, dur * 0.2));
        val = base + (target[1] - base) * progress;
      }
      v[t] = val;
    }
    out[name] = v;
  }
  return out;
}

if (import.meta.filename === process.argv[1]) {
  const S = series();
  console.log(`seed ${SEED}, ${MIN} minutes, ${CAUSE.length} component metrics, ` +
    `${SYMPTOM.length} user metrics, ${EVENT.length} events`);
  const [b, dur] = eventWindow(0);
  console.log(`sample — cache collapse window (minute ${b}, ${dur} min): ` +
    `cache hit ${S["catalog.cache_hit_rate"][b + 30].toFixed(1)}, ` +
    `cpu ${S["catalog.cpu"][b + 30].toFixed(1)}, ` +
    `p95 ${S["p95_latency_ms"][b + 30].toFixed(0)} ms, ` +
    `error rate ${S["error_rate"][b + 30].toFixed(2)}`);
}

seed 20260731, 10080 minutes, 12 component metrics, 4 user metrics, 6 events
sample — cache collapse window (minute 2040, 90 min): cache hit 35.0, cpu 91.0, p95 520 ms, error rate 0.07

Running Two Rule Sets Through the Same Week

A rule fires an alert once its condition holds for three consecutive minutes, and does not repeat until the condition clears. The two sets run independently over the same metrics.

// alert/rule.mjs — the same seven days are run through two rule sets: twelve cause-based
// rules and four symptom-based rules. A rule fires an alert once its condition holds for 3
// consecutive minutes, and does not fire again until the condition clears.
import { series, CAUSE, SYMPTOM, EVENT, MIN, eventWindow } from "./metric.mjs";
const S = series();
const crosses = (k, v) => (k[3] === "over" ? v > k[4] : v < k[4]);

function alerts(set) {
  const m = new Map();
  for (const k of set) {
    const v = S[k[0]], mins = [];
    let run = 0, active = false;
    for (let t = 0; t < MIN; t += 1) {
      if (crosses(k, v[t])) { run += 1; if (run >= 3 && !active) { active = true; mins.push(t - 2); } }
      else { run = 0; active = false; }
    }
    m.set(k[0], mins);
  }
  return m;
}
const SET = [["cause-based", CAUSE, alerts(CAUSE)], ["symptom-based", SYMPTOM, alerts(SYMPTOM)]];

console.log(`${"event".padEnd(29)}${"duration".padStart(9)}` +
  `${SET.map(([name]) => `${name} rules`.padStart(24) + "first alert".padStart(13)).join("")}`);
const caught = SET.map(() => new Set());
EVENT.forEach(([, , dr, name], i) => {
  const [start] = eventWindow(i);
  const s = SET.map(([, , m], j) => {
    const c = [...m.values()].map((mins) => mins.filter((t) => t >= start && t < start + dr));
    const rules = c.filter((x) => x.length > 0).length;
    const first = Math.min(...c.flat().map((t) => t - start), Infinity);
    if (rules > 0) caught[j].add(i);
    return [rules, first];
  });
  console.log(`${name.padEnd(29)}${(dr + " min").padStart(9)}` +
    `${s.map(([rules, first]) => String(rules).padStart(24) +
      (first === Infinity ? "-" : `${first} min`).padStart(13)).join("")}`);
});

const duringAnyEvent = (t) => EVENT.some((o, i) => t >= eventWindow(i)[0] && t < eventWindow(i)[0] + o[2]);
console.log(`\n${"set".padEnd(16)}${"rules".padStart(7)}${"alerts".padStart(7)}` +
  `${"off-event".padStart(11)}${"caught".padStart(11)}${"per event".padStart(13)}  missed events`);
SET.forEach(([name, set, m], j) => {
  const all = [...m.values()].flat();
  const off = all.filter((t) => !duringAnyEvent(t)).length;
  const on = all.length - off;
  console.log(`${name.padEnd(16)}${String(set.length).padStart(7)}${String(all.length).padStart(7)}` +
    `${String(off).padStart(11)}${`${caught[j].size}/${EVENT.length}`.padStart(11)}` +
    `${(on / caught[j].size).toFixed(1).padStart(13)}  ` +
    `${EVENT.filter((_, i) => !caught[j].has(i)).map((o) => o[3]).join(", ") || "-"}`);
});

const together = new Set([...caught[0], ...caught[1]]);
console.log(`\nboth together: ${CAUSE.length + SYMPTOM.length} rules, ` +
  `${together.size}/${EVENT.length} events`);
const noisiest = EVENT.map((o, i) => {
  const [start] = eventWindow(i);
  return [o[3], [...SET[0][2].values()].filter((mins) =>
    mins.some((t) => t >= start && t < start + o[2])).length];
}).sort((a, b) => b[1] - a[1])[0];
console.log(`noisiest event: ${noisiest[0]} — ${noisiest[1]} cause-based alerts at once in one event`);
const busiest = [...SET[0][2]].sort((a, b) => b[1].length - a[1].length)[0];
console.log(`busiest single rule: ${busiest[0]} — ${busiest[1].length} alerts in 7 days`);

event                         duration       cause-based rules  first alert     symptom-based rules  first alert
cache collapse                  90 min                       2        6 min                       1       11 min
memory leak                    240 min                       5        0 min                       0            -
disk fill-up                   300 min                       3       45 min                       0            -
dependency slowdown            120 min                       3        9 min                       2       10 min
silent delivery loss           180 min                       0            -                       1        0 min
post-release error increase     60 min                       0            -                       1        2 min

set               rules alerts  off-event     caught    per event  missed events
cause-based          12     81         67        4/6          3.5  silent delivery loss, post-release error increase
symptom-based         4      9          4        4/6          1.3  memory leak, disk fill-up

both together: 16 rules, 6/6 events
noisiest event: memory leak — 5 cause-based alerts at once in one event
busiest single rule: notification.queue — 10 alerts in 7 days

Three Numbers

Alerts produced. The twelve cause-based rules produced 81 alerts over seven days, the four symptom-based rules produced 9. Three times the rules, nine times the alerts. 67 of those 81 do not line up with any event: 83% of the cause-based alerts turn out empty. In the symptom-based set, the same ratio is 4 of 9. The source of the difference is written in SD2: a component metric is read from a single process and produces short spikes, a user metric is an average of thousands of requests and stays smooth. One single rule — the notification queue — produced 10 alerts over seven days.

Rules firing together for the same event. The cause-based set averages 3.5 alerts per event, the symptom-based set 1.3. During the memory leak, five rules fired at once: three were the event itself (memory, garbage collection, restarts), two were unrelated spikes that happened to land inside that four-hour window. When the person paged opens five alerts, they cannot tell which one is the root cause; the count of alerts carries no information, because all of them came from the same event.

Events that produce no alert at all. Both sets caught four of the six events, but missed a different two. The cause-based rules could not see the silent delivery loss or the post-release error increase: in both, the component metrics were normal and what was broken was the work itself. The symptom-based rules could not see the memory leak or the disk fill-up: in both, the member had not noticed anything yet, because the resource had not run out yet.

The Rule That Wakes You, the Metric That Diagnoses

The timing column separates the two sets’ roles. In the cache collapse, the cause-based rule fired at minute 6, the symptom-based rule at minute 11; in the dependency slowdown, the gap narrows to one minute. Cause-based metrics fire a little earlier, but the cost of those five minutes is 67 empty alerts.

The pattern that emerges is this: the twelve component metrics are not deleted, they stop being alert rules. The rules that wake someone up are written from symptoms — four rules, the four metrics the member sees. The component metrics stay on the dashboard, and the person paged looks there after receiving the alert. An alert says “something must be done”; a dashboard says “where to look”. The union of the two sets covers all six of the six events, but it does that not with 16 alert rules, but with 4 alert rules and 12 dashboard panels.

The two events the symptom-based set misses are this pattern’s real gap, and changing a threshold does not close it. With the disk at 98% full, the member still sees nothing; the symptom only appears once the disk actually fills up, and by then reversing course is expensive. For resources that run out, the rule is written not on a threshold but on time to exhaustion: the sentence “at this fill rate, the disk fills up within four hours” is a symptom, because it defines work for a person to do. The same sentence can be written for the memory leak; both are read from a cause-based metric but answer a symptom-based question.

Summary

• An alert rule’s source is one of two kinds: a component’s internal metric (cause) or a metric the member sees (symptom); the test for the distinction is whether the person paged has work to do.
• The twelve cause-based rules produced 81 alerts over seven days and 67 lined up with no event; the four symptom-based rules produced 9 alerts and 4 turned out empty.
• The cause-based set averages 3.5 alerts per event: during the memory leak, five rules fired at once, three were the event itself, two were unrelated spikes.
• Both sets caught four of six events but missed a different two: the cause-based rules missed events where the work itself was broken, the symptom-based rules missed exhaustions that had not yet reached the member.
• The cause-based rules fired five minutes earlier in the cache collapse; the cost of those five minutes is 67 empty alerts.
• The fix is not to delete the component metrics but to stop them from being alert rules: four alert rules and twelve dashboard panels cover all six of the six events.

Next Step

Every rule set up in this lesson pages a person, and the person responds within minutes. Some decisions are never put to a person at all. Whether one of the loan system’s processes is alive and whether it should keep receiving traffic is decided by a machine, within seconds and thousands of times a day. That decision looks at two separate questions — is the process dead, and can the process take work right now — and when the two questions are wired to a single endpoint, the answer given is not just wrong, it builds a self-feeding loop. The next lesson separates the two endpoints in code and measures the cost of not separating them in restart count and dropped request count.