Alert Fatigue

The previous lesson’s three rules produced 41 alerts over ten days, and 21 of them matched no incident. A real system does not have three rules; every service has several rules, and all of them route to the same person. This lesson builds a thirty-day alert stream from the ground up, applies three filters in sequence, and measures the same three numbers at every stage: how many notifications reach the person, how many of them turn into work, and how many real incidents get lost before reaching that person.

The stream arises from the regional measurement network’s (fiction) six services and fourteen rules; a shared data store sits below the services, and some incidents spread upward from there.

Alert fatigue is not a state of mind, it is a measurable threshold: as the number of alerts a person receives in twenty-four hours grows, response time lengthens, and past a certain count, alerts get closed unread. The model in this lesson takes that curve as a given assumption and counts its consequences.

A Thirty-Day Stream

// fatigue/stream.mjs — MODEL of a thirty-day alert stream. MD25: six services and the dependency
// between them are fiction; there are four services under the data store's edge. MD26: twenty real
// incidents are known incidents; 1-3 rules fire in the incident's own service, 1-2 derived rules
// fire in each service depending on it. MD27: three of the fourteen rules are noisy and produce
// alerts independent of any incident.
export const SEED = 20260805, DAYS = 30, ONCALL = 2;
export const DEPENDENCY = { gateway: ["read", "billing"], read: ["store"], validate: ["store"],
  billing: ["workorder", "store"], workorder: ["store"], store: [] };
export const SERVICES = Object.keys(DEPENDENCY);
export const RULES = SERVICES.flatMap((s) => [`${s}.error_rate`, `${s}.latency`])
  .concat(["store.connection", "store.disk"]);
export const NOISY = ["gateway.latency", "store.connection", "workorder.latency"];
// The services standing above a service: when that service breaks, these fire too.
export const upstreams = (s) => SERVICES.filter((u) => DEPENDENCY[u].includes(s));

export function stream() {
  let s = SEED;
  const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32);
  const minutes = DAYS * 1440, incidents = [], alerts = [];
  for (let i = 0; i < 20; i += 1) {
    const t = Math.floor(r() * minutes), service = SERVICES[Math.floor(r() * SERVICES.length)];
    incidents.push({ id: i, t, service, duration: 30 + Math.floor(r() * 150) });
  }
  incidents.sort((a, b) => a.t - b.t).forEach((o, i) => { o.id = i; });
  for (const o of incidents) {
    const own = RULES.filter((k) => k.startsWith(`${o.service}.`));
    for (const k of own.slice(0, 1 + Math.floor(r() * own.length)))
      alerts.push({ t: o.t + Math.floor(r() * 8), rule: k, service: o.service, incident: o.id, derived: false });
    for (const u of upstreams(o.service)) {                    // cascade: upstream services fire too
      const uRules = RULES.filter((k) => k.startsWith(`${u}.`));
      for (const k of uRules.slice(0, 1 + Math.floor(r() * 2)))
        alerts.push({ t: o.t + 2 + Math.floor(r() * 12), rule: k, service: u, incident: o.id, derived: true });
    }
  }
  for (const k of RULES) {                                      // noise independent of any incident
    const n = NOISY.includes(k) ? 40 + Math.floor(r() * 30) : 2 + Math.floor(r() * 6);
    for (let i = 0; i < n; i += 1)
      alerts.push({ t: Math.floor(r() * minutes), rule: k, service: k.split(".")[0], incident: null, derived: false });
  }
  alerts.sort((a, b) => a.t - b.t);
  return { incidents, alerts };
}

if (import.meta.filename === process.argv[1]) {
  const { incidents, alerts } = stream();
  const real = alerts.filter((u) => u.incident !== null);
  console.log(`seed ${SEED}, ${DAYS} days, ${RULES.length} rules, ${SERVICES.length} services, ` +
    `${incidents.length} real incidents, ${alerts.length} raw alerts`);
  console.log(`incident-linked alerts ${real.length} (${real.filter((u) => u.derived).length} derived), ` +
    `incident-independent ${alerts.length - real.length}`);
  console.log(`${(alerts.length / DAYS).toFixed(1)} alerts/day, ${(alerts.length / DAYS / ONCALL).toFixed(1)} per on-call\n`);
  const count = new Map();
  for (const u of alerts) count.set(u.rule, (count.get(u.rule) || 0) + 1);
  console.log(`${"noisiest rule".padEnd(22)}${"alerts".padStart(8)}${"incident-linked".padStart(17)}`);
  [...count].sort((a, b) => b[1] - a[1]).slice(0, 5).forEach(([k, n]) =>
    console.log(`${k.padEnd(22)}${String(n).padStart(8)}` +
      `${String(alerts.filter((u) => u.rule === k && u.incident !== null).length).padStart(17)}`));
}

seed 20260805, 30 days, 14 rules, 6 services, 20 real incidents, 268 raw alerts
incident-linked alerts 56 (26 derived), incident-independent 212
8.9 alerts/day, 4.5 per on-call

noisiest rule           alerts  incident-linked
workorder.latency           70                2
gateway.latency             58                3
store.connection            40                0
gateway.error_rate          16                9
billing.error_rate          13                9

268 alerts were produced over thirty days; 8.9 per day. Of these, 212 are tied to no incident, and 56 arose from twenty real incidents — 26 of the 56 are derived, that is, an alert born not in the incident’s own service but in an upstream service depending on it. Because there are four services below the data store’s edge, every incident that touches the store produces alerts from up to four services.

The noisiest-rule list shows a second separation: the workorder latency rule produced 70 alerts over thirty days, and only two of them matched an incident. The gateway error-rate rule produced 16 alerts, nine of them incident-linked. What is counted is not the rule count, it is the empty alerts per rule.

Three Filters

The filters are applied in sequence: nearby alerts from the same service are grouped into a single notification, a rule producing too many alerts in a short time is temporarily silenced, and if a service’s dependency is already alerting, that service’s alert is suppressed.

The “actionable notification” in this measurement has a narrow definition: a notification counts as actionable only if it is tied to a real incident and is the first read notification for that incident. A second alert for the same incident does not define work, because the work has already started; an alert with no incident does not define work either. This definition shows a low ratio, but it counts the work the person actually does.

// fatigue/filter.mjs — three filters are applied to the raw alert stream in sequence, and the same
// measurements are taken again at each stage. MD28: response time base is 4 min; it grows 18% for
// every alert the on-call received in the last 24 hours; the ninth and later alerts in the last 24
// hours are closed unread. MD29: grouping window 15 min (same service), silencing threshold 3
// alerts in the last 24 hours (same rule), dependency suppression window 45 min. MD30: on-call is
// shared between two people on a fifteen-day rotation.
import { stream, DEPENDENCY, DAYS, ONCALL, SEED } from "./stream.mjs";
const { incidents, alerts } = stream();
const BASE = 4, INCREASE = 0.18, UNREAD = 9, WINDOW = { group: 15, silence: 3, dependency: 45 };

const fatigueThreshold = Math.ceil(1 + 1 / INCREASE);   // rank at which response time doubles

const group = (U) => U.filter((u, i) => !U.some((v, j) =>
  j < i && v.service === u.service && u.t - v.t <= WINDOW.group));
const silence = (U) => U.filter((u) =>
  U.filter((v) => v.rule === u.rule && v.t <= u.t && u.t - v.t < 1440).length <= WINDOW.silence);
// Suppression is causal: if a dependency already alerted, the service above it is suppressed.
const suppress = (U) => U.filter((u) => !U.some((v) =>
  DEPENDENCY[u.service].includes(v.service) && v.t <= u.t && u.t - v.t <= WINDOW.dependency));

function measure(U) {
  const seen = new Set(), responseTimes = [], ranks = [], delivered = new Set(U.map((u) => u.incident));
  let actioned = 0, unread = 0;   // responseTimes: response time of the first read notification of a seen incident
  for (let i = 0; i < U.length; i += 1) {
    const u = U[i];
    const rank = U.filter((v, j) => j <= i && u.t - v.t < 1440).length;
    ranks.push(rank);
    if (rank >= UNREAD) { unread += 1; continue; }        // closed unread
    if (u.incident !== null && !seen.has(u.incident)) {
      seen.add(u.incident); actioned += 1; responseTimes.push(BASE * (1 + INCREASE * (rank - 1))); }
  }
  const avg = (x) => (x.length ? x.reduce((a, b) => a + b, 0) / x.length : 0);
  const undelivered = incidents.filter((o) => !delivered.has(o.id)).length;
  return { notifications: U.length, perDay: U.length / DAYS, actioned: actioned / U.length * 100,
    covered: seen.size, undelivered, unseen: incidents.length - seen.size - undelivered,
    responseTime: avg(responseTimes), unread, fatigued: ranks.filter((x) => x >= fatigueThreshold).length / U.length * 100 };
}

const STAGES = [["raw stream", alerts]];
STAGES.push(["grouping", group(STAGES[0][1])]);
STAGES.push(["+ silencing", silence(STAGES[1][1])]);
STAGES.push(["+ dependency suppression", suppress(STAGES[2][1])]);
console.log(`fatigue threshold: ${fatigueThreshold}. response time after that alert is double the base ` +
  `(${BASE} min -> ${(BASE * (1 + INCREASE * (fatigueThreshold - 1))).toFixed(1)} min)\n`);
console.log(`${"stage".padEnd(26)}${"notifications".padStart(15)}${"day".padStart(6)}${"actioned".padStart(11)}` +
  `${"covered".padStart(9)}${"undelivered".padStart(13)}${"unread".padStart(9)}` +
  `${"incident response".padStart(20)}${"above threshold".padStart(17)}`);
const results = STAGES.map(([name, U]) => {
  const d = measure(U);
  console.log(`${name.padEnd(26)}${String(d.notifications).padStart(15)}${d.perDay.toFixed(1).padStart(6)}` +
    `${`${d.actioned.toFixed(1)}%`.padStart(11)}${`${d.covered}/${incidents.length}`.padStart(9)}` +
    `${String(d.undelivered).padStart(13)}${String(d.unseen).padStart(9)}` +
    `${`${d.responseTime.toFixed(1)} min`.padStart(20)}${`${d.fatigued.toFixed(0)}%`.padStart(17)}`);
  return d;
});
const [first, last] = [results[0], results[3]];
console.log(`\nper on-call day (${DAYS / ONCALL} days each): raw stream ` +
  `${(first.notifications / ONCALL).toFixed(0)} notifications, after filters ${(last.notifications / ONCALL).toFixed(0)}`);
console.log(`alerts closed unread ${first.unread} -> ${last.unread}`);
const remaining = incidents.filter((o) => !STAGES[3][1].some((u) => u.incident === o.id));
console.log(`real incidents the filters do not deliver ${remaining.length}: ` +
  `${remaining.map((o) => `incident ${o.id} (${o.service})`).join(", ") || "none"}`);
for (const [name, before, after] of [["grouping", STAGES[0][1], STAGES[1][1]],
  ["silencing", STAGES[1][1], STAGES[2][1]]]) {
  const k = before.filter((u) => !after.includes(u));
  console.log(`${name} removed ${k.length} notifications, ${k.filter((u) => u.incident !== null).length} incident-linked`);
}
console.log(`\n${"suppression window".padEnd(21)}${"removed".padStart(9)}${"derived".padStart(9)}` +
  `${"independent incident".padStart(23)}${"noise".padStart(7)}${"undelivered incident".padStart(23)}`);
for (const w of [15, 45, 120, 360]) {
  WINDOW.dependency = w;
  const B = suppress(STAGES[2][1]), removed = STAGES[2][1].filter((u) => !B.includes(u));
  const undelivered = incidents.filter((o) => !B.some((u) => u.incident === o.id)).length;
  console.log(`${`${w} min`.padEnd(21)}${String(removed.length).padStart(9)}` +
    `${String(removed.filter((u) => u.derived).length).padStart(9)}` +
    `${String(removed.filter((u) => u.incident !== null && !u.derived).length).padStart(23)}` +
    `${String(removed.filter((u) => u.incident === null).length).padStart(7)}` +
    `${String(undelivered).padStart(23)}`);
}

fatigue threshold: 7. response time after that alert is double the base (4 min -> 8.3 min)

stage                       notifications   day   actioned  covered  undelivered   unread   incident response  above threshold
raw stream                            268   8.9       3.0%     8/20            0       12             7.8 min              80%
grouping                              247   8.2       3.2%     8/20            0       12             7.6 min              77%
+ silencing                           208   6.9       5.8%    12/20            0        8             7.8 min              64%
+ dependency suppression              191   6.4       6.8%    13/20            0        7             7.8 min              59%

per on-call day (15 days each): raw stream 134 notifications, after filters 96
alerts closed unread 152 -> 40
real incidents the filters do not deliver 0: none
grouping removed 21 notifications, 18 incident-linked
silencing removed 39 notifications, 1 incident-linked

suppression window     removed  derived   independent incident  noise   undelivered incident
15 min                      15       14                      0      1                      0
45 min                      17       14                      0      3                      0
120 min                     25       15                      3      7                      3
360 min                     46       16                      6     24                      6

The Number Behind Fatigue

The fatigue threshold falls out of the arithmetic: after the seventh alert, response time reaches double the base, from 4 minutes to 8.3 minutes. In the raw stream, 80% of notifications arrive at a rank above this threshold. That is, four-fifths of the alerts reaching the on-call arrive at a moment when that person has already received six alerts in the last twenty-four hours.

The third column carries the outcome. In the raw stream, only eight of the twenty real incidents are seen; twelve stay buried among the alerts closed unread. No alert was lost, no rule was written wrong, undelivered incidents are zero — the incidents were only too far back in the queue.

The filters bring this number from eight to thirteen. The source of the gain is this: the filters add no new information, they deliver the same information in fewer notifications; as the noise falls away, the real incident’s notification moves toward the front of the queue. The actionable-notification ratio rises from 3.0% to 6.8%, that is, it doubles — and still 93% of notifications turn into no work.

The contribution of the three filters is not equal. Grouping removed 21 notifications, and 18 of them were incident-linked: several rules of the same service fire at once for the same incident, and grouping collects them into a single notification. The removed count is small, because the noise is not concentrated in one service, it is spread across fourteen rules. Silencing is the opposite: it removed 39 notifications, and only one was incident-linked. This is the step that brings the seen-incident count from eight to twelve — once the daily share of the three noisy rules is cut, the on-call’s queue shortens, and the real incidents left behind drop under the threshold.

The response time of the seen incidents, though, barely changes: 7.6 minutes at grouping and 7.8 minutes at the other three stages. The reason is that the filters bring down the count, not the rank itself — the newly seen incidents also arrive at high ranks. The gain is not in response time, it is in the number of incidents seen.

The Cost of Suppression

Dependency suppression is the most dangerous filter, because it deletes information. The window sweep shows the cost. At the fifteen-minute window, 14 of the 15 removed notifications are derived, one is noise; no independent incident’s own alert is deleted. At forty-five minutes the table is the same: the derived gain stops at 14, because derived alerts are already born within the incident’s first forty-five minutes.

Widening the window past this point only does harm. At one hundred twenty minutes, three independent incidents’ own alerts are suppressed and three real incidents receive no notification at all; at three hundred sixty minutes this count rises to six. The suppression window is chosen against how long the dependency’s effect lasts; going longer leaves the upstream service’s own incident in the dependency’s shadow.

The real distinction here is this: grouping and silencing delay or collect a notification, while dependency suppression deletes it. The first two fail visibly, the third fails invisibly — a suppressed alert appears in no list at all.

Summary

• Over the thirty-day stream, 212 of 268 alerts were tied to no incident; a single noisy rule produced 70 alerts, and only two of them matched an incident.
• The fatigue threshold is the seventh alert: past that rank, response time rises from 4 minutes to 8.3 minutes, and 80% of notifications in the raw stream arrive above this threshold.
• In the raw stream, eight of the twenty real incidents were seen; twelve were delivered yet closed unread — the loss was not in collection, it was in the queue.
• The three filters brought notifications from 268 to 191, from 8.9 to 6.4 per day; seen incidents rose from eight to thirteen, the actionable-notification ratio from 3.0% to 6.8%. What really raised the seen-incident count was silencing: only one of the 39 notifications it removed was incident-linked.
• The response time of seen incidents stayed in the 7.6-7.8 minute band across all four stages: the filters bring down the count, not the rank, and the gain is in the number of incidents seen.
• Dependency suppression’s gain saturates at forty-five minutes; at a one-hundred-twenty-minute window three real incidents, and at a three-hundred-sixty-minute window six real incidents, receive no notification at all.

Next Step

Throughout this lesson, every signal was born from real traffic: a request arrived, a rule fired, a person was alerted. There are also cases real traffic never sees. Between three and five in the morning, billing queries almost never arrive; a flow that breaks at that hour produces no alert until nine in the morning, because there is no request to give rise to one. A flow used a few times a year can be broken for months. An external dependency’s own failure, in turn, is visible only once someone sends it a request. The next lesson sets up periodic polling from the outside, sweeps the polling frequency to measure detection delay together with polling cost, and names the case where polling passes while the real user flow is broken.