Dashboards and Alerts

The previous two lessons decided what to collect, in what form, and at what cost. The collected number still tells no one anything. It becomes a decision one of two ways: it is plotted somewhere and read by someone already looking, or it crosses a threshold and calls someone who is not. Both paths use the same number, and they do not work with the same threshold.

This lesson separates the two, then tests the calling path’s one parameter — the threshold — with a scan, and ties where it comes from to the Introduction to System Design course’s monthly outage budget.

Two Separate Questions

A dashboard answers the question of someone already looking. Its question is: is this normal right now, and if not, in which direction? This is why every panel is written next to its expected range; a line chart without a range only shows a shape going up or down and carries no decision.

An alert calls someone who is not looking. Its question is different: should something be done now? One rule follows from this — every alert is tied to an action. If there is no work to do in response, the collected metric goes into the dashboard, not an alert. This distinction is not about tool choice; it is about producing separate answers to two separate questions.

The first screen’s content follows from the previous lesson’s category list: one metric per category, together with the load measured. A performance panel cannot be read without a utilization panel beside it, because the same response time does not say the same thing at 513.89 requests/s and at midnight. A panel’s expected range and an alert threshold are not the same number: the range answers “is this normal,” the threshold answers “should I wake someone up,” and the second is always wider.

A Threshold Is a Matter of Scanning

A threshold is measured by two errors. A false alert is one triggered when no real event is present, and its cost is direct: each one takes a person’s attention, and piling up lowers every alert’s credibility. A missed alert is one that does not trigger during a real event, and its cost is paid out of the budget.

The setup below is a model: for each of a month’s 43,200 minutes, an error count is produced with seeded integer arithmetic, six real events are placed on a known schedule (IZ6), and noise is defined by a separate assumption (IZ7). The model does not predict a real failure distribution; what it tests is what different threshold and window pairs do on the same data.

// alert/scan.mjs — MODEL of an alert threshold scan. For each minute of a month, an error count
// is produced with seeded integer arithmetic; for each threshold-window pair, false alerts,
// missed events, and detection delay are counted.
const MINUTES = 43_200;            // K01: a 30-day month
const REQUESTS_PER_MIN = 30_833;   // K01: 513.89 requests/s x 60
const SEED = 20260731;
let s = SEED;
const rand = (n) => Math.floor((((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32)) * n);

// IZ6 (assumption): six real events in a month [start minute, duration, severity (per thousand)]
const EVENT = [[3000, 240, 8], [8000, 30, 15], [15000, 3, 30], [21000, 20, 120],
  [30000, 4, 800], [38000, 45, 400]];

// IZ7 (assumption): noise model — baseline error and a one-minute spike every 240 minutes
const error = new Array(MINUTES);
for (let t = 0; t < MINUTES; t += 1) {
  let n = rand(40);
  if (rand(240) === 0) n += rand(1000);
  error[t] = n;
}
for (const [start, duration, severity] of EVENT)
  for (let t = start; t < start + duration; t += 1) error[t] += Math.round((REQUESTS_PER_MIN * severity) / 1000);

const ratio = error.map((h) => h / REQUESTS_PER_MIN);
const eventAt = (t) => EVENT.findIndex(([start, duration]) => t >= start && t < start + duration);
const lostMinutes = EVENT.reduce((a, [, duration, severity]) => a + (duration * severity) / 1000, 0);

console.log(`seed ${SEED}, ${MINUTES} minutes, ${EVENT.length} real events, ` +
  `total ${EVENT.reduce((a, o) => a + o[1], 0)} minutes of degradation`);
console.log(`what the events eat from the budget: ${lostMinutes.toFixed(2)} minute-equivalents ` +
  `(K01's failure share is 28.2 minutes)`);
const noise = ratio.filter((_, t) => eventAt(t) < 0);
console.log(`noise: highest minute ${(noise.reduce((a, c) => (c > a ? c : a), 0) * 100).toFixed(2)}%, ` +
  `minutes exceeding the 0.5% threshold ${noise.filter((o) => o >= 0.005).length}, ` +
  `minutes exceeding the 2% threshold ${noise.filter((o) => o >= 0.02).length}`);

function scan(threshold, window) {
  const r = { alerts: 0, falseAlerts: 0, caught: new Set(), delay: [] };
  let active = false;
  for (let t = window - 1; t < MINUTES; t += 1) {
    let all = true;
    for (let k = 0; k < window; k += 1) if (ratio[t - k] < threshold) all = false;
    if (all === false) { active = false; continue; }
    if (active) continue;
    active = true;
    r.alerts += 1;
    const i = eventAt(t);
    if (i < 0) r.falseAlerts += 1;
    else if (r.caught.has(i) === false) {
      r.caught.add(i);
      r.delay.push(t - EVENT[i][0]);
    }
  }
  return r;
}

const THRESHOLD = [0.005, 0.01, 0.02, 0.05];
const WINDOW = [1, 3, 5];
console.log(`\n${"threshold".padStart(11)}${"window".padStart(9)}${"alerts".padStart(7)}` +
  `${"false alerts".padStart(14)}${"events caught".padStart(16)}${"missed".padStart(11)}` +
  `${"avg delay".padStart(14)}${"budget lost (min)".padStart(19)}`);
for (const e of THRESHOLD)
  for (const w of WINDOW) {
    const r = scan(e, w);
    const missedBudget = EVENT.filter((_, i) => r.caught.has(i) === false)
      .reduce((a, [, duration, severity]) => a + (duration * severity) / 1000, 0);
    const avg = r.delay.length ? r.delay.reduce((a, c) => a + c, 0) / r.delay.length : 0;
    console.log(`${((e * 100).toFixed(1) + "%").padStart(11)}${(w + " min").padStart(9)}` +
      `${String(r.alerts).padStart(7)}${String(r.falseAlerts).padStart(14)}` +
      `${`${r.caught.size}/${EVENT.length}`.padStart(16)}` +
      `${String(EVENT.length - r.caught.size).padStart(11)}` +
      `${(avg.toFixed(1) + " min").padStart(14)}${missedBudget.toFixed(2).padStart(19)}`);
  }

seed 20260731, 43200 minutes, 6 real events, total 342 minutes of degradation
what the events eat from the budget: 26.06 minute-equivalents (K01's failure share is 28.2 minutes)
noise: highest minute 3.34%, minutes exceeding the 0.5% threshold 149, minutes exceeding the 2% threshold 69

  threshold   window alerts  false alerts   events caught     missed     avg delay  budget lost (min)
       0.5%    1 min    155           149             6/6          0       0.0 min               0.00
       0.5%    3 min      6             0             6/6          0       2.0 min               0.00
       0.5%    5 min      4             0             4/6          2       4.0 min               3.29
       1.0%    1 min    131           125             6/6          0       2.0 min               0.00
       1.0%    3 min      5             0             5/6          1       2.0 min               1.92
       1.0%    5 min      3             0             3/6          3       4.0 min               5.21
       2.0%    1 min     74            69             5/6          1       2.4 min               0.45
       2.0%    3 min      4             0             4/6          2       2.0 min               2.37
       2.0%    5 min      2             0             2/6          4       4.0 min               5.66
       5.0%    1 min      3             0             3/6          3       0.0 min               2.46
       5.0%    3 min      3             0             3/6          3       2.0 min               2.46
       5.0%    5 min      2             0             2/6          4       4.0 min               5.66

How to Shrink Both Errors at Once

The table is read on two axes, and the two are not the same thing.

Raising the threshold cuts both false alerts and real events. At the 0.5 percent threshold and a one-minute window, 155 alerts trigger in a month, 149 of them false — five times a day, for nothing. When the threshold is raised to 5 percent, false alerts drop to zero, but events caught fall from 6 to 3 and the missed events eat 2.46 minutes from the budget. This is the cost of cutting false alerts with the threshold: a threshold that rises above the noise also rises above the mild events.

Widening the window cuts only short-lived noise. At the same 0.5 percent threshold, when the window is extended to three minutes, false alerts drop from 149 to 0, all six events are still caught, and the budget lost stays at 0.00 minutes. The cost is only a two-minute detection delay. The reason is written into the model: noise lasts one minute, while real events last at least three. The window is chosen longer than the noise’s duration and shorter than the shortest real event.

The five-minute window shows the limit: at the same threshold, false alerts are again 0, but the two events lasting three and four minutes are no longer seen, and 3.29 minutes leak from the budget. Missing begins the moment the window passes the shortest event.

One row also shows a misleading success. At the 1.0 percent threshold and a one-minute window, all six of six events appear to be caught, yet the first event’s 0.8 percent severity stays below the threshold. It is caught because a noise spike happens to land inside that event’s 240 minutes. This is coincidence, not diagnosis, and the scan table reads correctly only once the reason is known.

Where the Threshold Comes From

The scan finds the best pair, but it does not say which number the threshold should be; 0.5 percent and 2 percent both look defensible. The number’s source is not the metric itself but the service’s commitment. The Introduction to System Design course established the monthly budget: the 99.9 percent target is 43.2 minutes, of which 15.0 minutes go to planned work and 28.2 minutes to failure. The percentage arithmetic is not repeated here; these three numbers are inputs. Dividing the failure share by a month gives a continuously permitted error rate, and the observed rate divided by that rate is the burn rate.

// alert/budget.mjs — deriving the alert threshold from K01's outage budget: burn rate, the
// threshold's error-rate equivalent, and the budget spent by the time the alert triggers. All of
// it is arithmetic.
const MONTH_MIN = 43_200;         // K01: a 30-day month (minutes)
const MONTHLY_BUDGET = 43.2;      // K01: monthly outage budget for 99.9% (minutes)
const FAILURE_SHARE = 28.2;       // K01: the budget's failure share (minutes)
const allowed = FAILURE_SHARE / MONTH_MIN;         // request-based permitted error rate
const b = (x, n = 2) => x.toFixed(n);

console.log(`monthly budget ${MONTHLY_BUDGET} min, failure share ${FAILURE_SHARE} min, month ${MONTH_MIN} min`);
console.log(`the continuously permitted error rate = ${b(allowed * 100, 4)}% (burn rate 1.0)`);

const WINDOW = [["5 min", 5], ["1 hr", 60], ["6 hr", 360]];
console.log(`\n${"burn rate".padStart(13)}${"error rate threshold".padStart(22)}` +
  `${"share exhausted in".padStart(20)}` +
  `${WINDOW.map(([a]) => `${a} window`.padStart(16)).join("")}`);
for (const rate of [1, 3, 6, 14.4]) {
  const errRate = allowed * rate;
  const row = WINDOW.map(([, min]) =>
    `${b((100 * min * errRate) / FAILURE_SHARE)}%`.padStart(16)).join("");
  console.log(`${b(rate, 1).padStart(13)}${(b(errRate * 100, 3) + "%").padStart(22)}` +
    `${(b(30 / rate, 2) + " days").padStart(20)}${row}`);
}
console.log(`window columns: how much of the failure share has been spent when the alert triggers`);

const SCAN = [0.005, 0.01, 0.02, 0.05];
console.log(`\nthe previous scan's thresholds in budget terms:`);
for (const e of SCAN)
  console.log(`  ${b(e * 100, 1)}% threshold -> burn rate ${b(e / allowed, 1)}, ` +
    `at this rate the failure share runs out in ${b(FAILURE_SHARE / (e * 1440), 2)} days`);

// IZ6 (assumption): the previous block's six events [duration (min), severity (per thousand)]
const EVENT = [[240, 8], [30, 15], [3, 30], [20, 120], [4, 800], [45, 400]];
console.log(`\n${"event".padStart(5)}${"duration".padStart(10)}${"severity".padStart(9)}` +
  `${"burn rate".padStart(14)}${"consumed (min)".padStart(16)}${"share consumed".padStart(15)}` +
  `${"exhausted at this rate".padStart(23)}`);
let total = 0;
EVENT.forEach(([duration, severity], i) => {
  const rate = severity / 1000, consumed = duration * rate;
  total += consumed;
  console.log(`${String(i + 1).padStart(5)}${(duration + " min").padStart(10)}` +
    `${(b(rate * 100, 1) + "%").padStart(9)}${b(rate / allowed, 1).padStart(14)}` +
    `${b(consumed).padStart(16)}${(b((100 * consumed) / FAILURE_SHARE, 1) + "%").padStart(15)}` +
    `${(b(FAILURE_SHARE / (rate * 1440), 2) + " days").padStart(23)}`);
});
console.log(`total ${b(total)} min, ${b((100 * total) / FAILURE_SHARE, 1)}% of the failure share; ` +
  `remaining share ${b(FAILURE_SHARE - total)} min`);

monthly budget 43.2 min, failure share 28.2 min, month 43200 min
the continuously permitted error rate = 0.0653% (burn rate 1.0)

    burn rate  error rate threshold  share exhausted in    5 min window     1 hr window     6 hr window
          1.0                0.065%          30.00 days           0.01%           0.14%           0.83%
          3.0                0.196%          10.00 days           0.03%           0.42%           2.50%
          6.0                0.392%           5.00 days           0.07%           0.83%           5.00%
         14.4                0.940%           2.08 days           0.17%           2.00%          12.00%
window columns: how much of the failure share has been spent when the alert triggers

the previous scan's thresholds in budget terms:
  0.5% threshold -> burn rate 7.7, at this rate the failure share runs out in 3.92 days
  1.0% threshold -> burn rate 15.3, at this rate the failure share runs out in 1.96 days
  2.0% threshold -> burn rate 30.6, at this rate the failure share runs out in 0.98 days
  5.0% threshold -> burn rate 76.6, at this rate the failure share runs out in 0.39 days

event  duration severity     burn rate  consumed (min) share consumed exhausted at this rate
    1   240 min     0.8%          12.3            1.92           6.8%              2.45 days
    2    30 min     1.5%          23.0            0.45           1.6%              1.31 days
    3     3 min     3.0%          46.0            0.09           0.3%              0.65 days
    4    20 min    12.0%         183.8            2.40           8.5%              0.16 days
    5     4 min    80.0%        1225.5            3.20          11.3%              0.02 days
    6    45 min    40.0%         612.8           18.00          63.8%              0.05 days
total 26.06 min, 92.4% of the failure share; remaining share 2.14 min

The first table converts the threshold into a measure of time. A burn rate of 1.0 finishes the budget exactly at month’s end; 14.4 finishes it within two days. Choosing a threshold means answering the question “at what rate do I want to be woken up,” and the window column shows its cost: waiting six hours at a burn rate of 14.4 means 12 percent of the failure share is already spent by the time the alert arrives. A short window wakes someone early and is exposed to noise; a long window is quiet and expensive. This is the trade-off measured in the scan, written out in terms of the budget.

The second block converts the scan’s four thresholds into the same language and removes the arbitrariness: a 0.5 percent threshold means a burn rate of 7.7, and at that rate the failure share runs out in 3.92 days. A 5 percent threshold is a rate of 76.6 and 0.39 days — waiting for this threshold lets the month’s budget be exhausted in nine hours. Once the threshold is written this way, the disagreement stops being one of preference.

The third table gives the cost of missing, line by line. The six events ate 92.4 percent of the failure share, leaving 2.14 minutes. The most expensive event is not the most severe one but the one whose severity times duration is largest: the 40 percent event lasting forty-five minutes alone took 63.8 percent of the share. By contrast, the 0.8 percent-severity event — the one missed by every threshold of 1 percent and above in the scan — ate 6.8 percent of the share. An event not producing an alert does not mean it was cheap; a mild, long degradation can eat more budget than a short, sharp event. This is why a budget-based alert setup wants not a single threshold but at least two: one with a short window and a high rate, another with a long window and a low rate.

Summary

• A dashboard answers “is this normal” for someone looking; an alert answers “should I do something now” for someone not looking; a panel’s expected range and an alert threshold are not the same number.
• Every alert is tied to an action; if there is no work to do in response, the metric goes into the dashboard.
• Raising the threshold changes both errors at once: at 0.5 percent and a one-minute window, 149 of 155 monthly alerts are false; at 5 percent, false alerts drop to 0, but three events are missed and 2.46 minutes leave the budget.
• Widening the window cuts only short-lived noise: at 0.5 percent and a three-minute window, false alerts drop from 149 to 0, all six of six events are caught, and the cost is a 2.0-minute detection delay. If the window passes the shortest real event (5 minutes), two events are missed and 3.29 minutes leave the budget.
• The threshold is derived from the budget: a 28.2-minute failure share equals a continuous 0.0653 percent error rate, and the observed rate divided by that rate is the burn rate; a 0.5 percent threshold means a rate of 7.7, a share that runs out in 3.92 days.
• A missed event’s cost is duration times severity: the 45-minute, 40 percent event eats 63.8 percent of the share, the 240-minute, 0.8 percent event eats 6.8 percent.

Next Step

Everything measured so far was a number that arose on its own, under the system’s real traffic. These numbers only describe what has happened: they say what happened under today’s load, not what will happen under tomorrow’s. The alert threshold is tuned to today’s traffic; which metric will cross its threshold first when traffic doubles, at what point the system will saturate, and whether it can recover after a sudden spike cannot be read from the collected data, because that condition has never occurred. The next lesson fills this gap: instead of waiting for load, it generates it. Three separate tests ask three separate questions, and each one surfaces a different symptom.