Golden Signals

The previous lesson built all three indicator definitions on a single bad-request measure: status code or duration. In the loan system, that is not the only way a request can go bad. The catalog service can slow down without ever returning an error, the notification queue can fill up while running error-free, and the fee service receiving no requests at all can be a failure signal on its own. All of this is written in the same telemetry, but an indicator that looks at a single ratio cannot see part of it.

This lesson computes four signals from the same records, runs them across six separate event classes, and counts what each signal alone fails to catch.

Four Signals, Four Separate Questions

• Latency: how long a request took. Its question is “is it slow”.
• Traffic: how many requests arrive per unit time. Its question is “what is the load”.
• Errors: how many requests came back failed. Its question is “is it broken”.
• Saturation: how much of the system’s scarcest resource is full. Its question is “how much room is left”.

The four are chosen together because the first three describe what has already happened, and the fourth describes what is about to happen. This signal is the same concept as saturation in the Introduction to System Design course.

The first three come directly from the request log. Saturation does not: what the scarce resource is depends on the application, and it must be measured explicitly in code. In the split loan system, that resource is the notification queue; every time a notification request is written to the queue, its own record also captures the queue’s depth at that moment. Queue depth is a gauge, not a counter: it rises and falls.

The Seven-Day Log and Six Event Classes

// signal/record.mjs — the loan system's 7-day request log and notification queue gauge.
// Record: [minute, service, status, duration_ms, queue_depth, delivery(-1: not a notification)].
export const MIN = 7 * 1440, SEED = 20260731, LIMIT = 500, DRAIN = 25;
export const SERVICE = ["catalog", "membership", "loan", "notification", "fee"];
const BASE = [40, 25, 60, 30, 45];
// SD1 (assumption): 100 requests/minute between 08-22, 30 outside it; day 0 is event-free (base day).
// SD2 (assumption): six event classes [day, hour, duration min, name].
export const EVENT = [
  [1, 10, 60, "error burst"], [2, 14, 90, "silent slowdown"],
  [3, 3, 120, "traffic collapse"], [4, 15, 120, "queue saturation"],
  [5, 12, 30, "traffic spike"], [6, 9, 180, "silent delivery loss"],
];
export const eventWindow = (i) => [EVENT[i][0] * 1440 + EVENT[i][1] * 60, EVENT[i][2]];

export function records() {
  let s = SEED;
  const rand = (n) => Math.floor(((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
  const out = [];
  let queue = 0;
  for (let t = 0; t < MIN; t += 1) {
    const hour = Math.floor(t / 60) % 24;
    const within = (i) => t >= eventWindow(i)[0] && t < eventWindow(i)[0] + EVENT[i][2];
    let n = hour >= 8 && hour < 22 ? 100 : 30;
    if (within(2)) n = Math.round(n * 0.1);
    if (within(4)) n = Math.round(n * 2.5);
    queue = Math.max(0, queue - (within(3) ? 14 : DRAIN));
    for (let i = 0; i < n; i += 1) {
      const sv = rand(5);
      let status = 200, duration = BASE[sv] + rand(60), delivery = -1;
      if (rand(1000) < 1) status = 500;
      if (within(0) && rand(1000) < 80) status = 500;
      if (within(1) && sv === 0) duration += 700 + rand(400);
      if (sv === 3 && status === 200) {
        if (queue >= LIMIT) status = 503;
        else { queue += 1; delivery = rand(1000) < (within(5) ? 970 : 4) ? 0 : 1; }
      }
      out.push([t, SERVICE[sv], status, duration, queue, delivery]);
    }
  }
  return out;
}

if (import.meta.filename === process.argv[1]) {
  const R = records();
  console.log(`seed ${SEED}, ${MIN} minutes, ${R.length} requests, queue limit ${LIMIT}`);
  console.log(`event classes: ${EVENT.map((o) => o[3]).join(", ")}`);
}

seed 20260731, 10080 minutes, 715260 requests, queue limit 500
event classes: error burst, silent slowdown, traffic collapse, queue saturation, traffic spike, silent delivery loss

Six event classes were chosen separately: one moves only the status code, one moves only the duration, one moves only the request count, one moves only the queue. In the last one, notification requests return 200, their durations do not change, the queue drains normally — but the notification sent does not reach the member.

Computing the Four

Each signal is turned into a decision with a threshold. The latency and traffic thresholds are derived from the event-free day 0; the error and saturation thresholds are fixed. For a signal to count as having caught an event, it must trigger for three consecutive minutes within the window; a single-minute trigger does not count.

// signal/golden.mjs — the four golden signals computed minute by minute from the same
// records, and a count of which event class each signal alone fails to catch. Catch
// criterion: trigger for 3 consecutive minutes within the window.
import { records, MIN, EVENT, eventWindow, LIMIT } from "./record.mjs";
const R = records();
const count = new Int32Array(MIN), errC = new Int32Array(MIN), queueD = new Int32Array(MIN);
const ms = Array.from({ length: MIN }, () => []);
for (const [t, , status, duration, queue] of R) {
  count[t] += 1; ms[t].push(duration); queueD[t] = Math.max(queueD[t], queue);
  if (status >= 500) errC[t] += 1;
}
const p95 = (a) => [...a].sort((x, z) => x - z)[Math.floor(a.length * 0.95)];
const lat = new Int32Array(MIN);
for (let t = 0; t < MIN; t += 1) lat[t] = count[t] ? p95(ms[t]) : 0;

// SD3 (assumption): the latency and traffic thresholds are derived from day 0, the error and
// saturation thresholds are fixed.
const P95 = p95(R.filter((r) => r[0] < 1440).map((r) => r[3])), hourlyBase = new Float64Array(24);
for (let t = 0; t < 1440; t += 1) hourlyBase[Math.floor(t / 60)] += count[t] / 60;
const CHECK = {
  latency: (t) => lat[t] > 2 * P95,
  traffic: (t) => count[t] < 0.5 * hourlyBase[Math.floor(t / 60) % 24] || count[t] > 2 * hourlyBase[Math.floor(t / 60) % 24],
  error: (t) => count[t] > 0 && errC[t] / count[t] > 0.01,
  saturation: (t) => queueD[t] / LIMIT > 0.85,
};
const NAMES = Object.keys(CHECK);
const firstThree = (name, start, dur) => {
  for (let t = start, run = 0; t < start + dur; t += 1)
    if ((run = CHECK[name](t) ? run + 1 : 0) === 3) return t - start - 2;
  return -1;
};
console.log(`base p95 ${P95} ms, latency threshold ${2 * P95} ms, error threshold %1, ` +
  `saturation threshold ${0.85 * LIMIT}/${LIMIT}\n`);
console.log(`${"event class".padEnd(22)}${"duration".padStart(9)}` +
  `${NAMES.map((a) => a.padStart(14)).join("")}${"caught".padStart(11)}`);
const caught = NAMES.map(() => new Set());
EVENT.forEach(([, , dr, name], i) => {
  const [start] = eventWindow(i);
  const s = NAMES.map((a) => {
    let n = 0;
    for (let t = start; t < start + dr; t += 1) if (CHECK[a](t)) n += 1;
    return [n, firstThree(a, start, dr)];
  });
  s.forEach(([, k], j) => { if (k >= 0) caught[j].add(i); });
  console.log(`${name.padEnd(22)}${(dr + " min").padStart(9)}` +
    `${s.map(([n, k]) => (k >= 0 ? `${n} min @ ${k}` : "-").padStart(14)).join("")}` +
    `${String(s.filter(([, k]) => k >= 0).length).padStart(11)}`);
});
const duringAnyEvent = (t) => EVENT.some((o, i) => t >= eventWindow(i)[0] && t < eventWindow(i)[0] + o[2]);
console.log(`\n${"signal".padEnd(12)}${"caught".padStart(11)}${"off-event trigger".padStart(18)}` +
  `  missed event classes`);
NAMES.forEach((a, j) => {
  let d = 0;
  for (let t = 0; t < MIN; t += 1) if (CHECK[a](t) && !duringAnyEvent(t)) d += 1;
  console.log(`${a.padEnd(12)}${`${caught[j].size}/${EVENT.length}`.padStart(11)}${(d + " min").padStart(18)}` +
    `  ${EVENT.filter((_, i) => !caught[j].has(i)).map((o) => o[3]).join(", ")}`);
});
const all = new Set(caught.flatMap((s) => [...s]));
console.log(`\nall four together: ${all.size}/${EVENT.length} event classes; ` +
  `missed: ${EVENT.filter((_, i) => !all.has(i)).map((o) => o[3]).join(", ")}`);
const deliveryRate = (a, b) => {
  const rows = R.filter((r) => r[5] >= 0 && r[0] >= a && r[0] < b);
  return `%${((100 * rows.filter((r) => r[5] === 1).length) / rows.length).toFixed(1)}`;
};
const [b5, d5] = [eventWindow(5)[0], EVENT[5][2]];
console.log(`delivery rate: before event ${deliveryRate(b5 - d5, b5)}, during event ${deliveryRate(b5, b5 + d5)}`);
let low = 0, off = 0, firstQueueTrig = -1;
for (let t = 0; t < MIN; t += 1) if (!duringAnyEvent(t)) {
  if (CHECK.error(t)) { off += 1; if (count[t] <= 40) low += 1; }
  if (CHECK.saturation(t) && firstQueueTrig < 0) firstQueueTrig = t;
}
const [eb, ed] = [eventWindow(3)[0], EVENT[3][2]];
console.log(`of the error signal's ${off} off-event triggers, ${low} fell in minutes under 40 ` +
  `requests/min; saturation's first off-event trigger was minute ${firstQueueTrig}, the event ended ` +
  `at minute ${eb + ed}`);
console.log(`during queue saturation: saturation caught it at minute ${firstThree("saturation", eb, ed)}, ` +
  `error caught it at minute ${firstThree("error", eb, ed)}`);

base p95 105 ms, latency threshold 210 ms, error threshold %1, saturation threshold 425/500

event class            duration       latency       traffic         error    saturation     caught
error burst              60 min             -             -    60 min @ 0             -          1
silent slowdown          90 min    90 min @ 0             -             -             -          1
traffic collapse        120 min             -   120 min @ 0             -             -          1
queue saturation        120 min             -             -   43 min @ 72   59 min @ 61          2
traffic spike            30 min             -    30 min @ 0   12 min @ 18   15 min @ 15          3
silent delivery loss    180 min             -             -             -             -          0

signal           caught off-event trigger  missed event classes
latency             1/6             0 min  error burst, traffic collapse, queue saturation, traffic spike, silent delivery loss
traffic             2/6             0 min  error burst, silent slowdown, queue saturation, silent delivery loss
error               3/6           162 min  silent slowdown, traffic collapse, silent delivery loss
saturation          2/6            23 min  error burst, silent slowdown, traffic collapse, silent delivery loss

all four together: 5/6 event classes; missed: silent delivery loss
delivery rate: before event %99.7, during event %3.0
of the error signal's 162 off-event triggers, 136 fell in minutes under 40 requests/min; saturation's first off-event trigger was minute 6780, the event ended at minute 6780
during queue saturation: saturation caught it at minute 61, error caught it at minute 72

Each Signal’s Blind Spot

No signal sees more than four of the six events. Latency catches one in six, traffic and saturation catch two in six, error catches three in six. These numbers have nothing to do with the quality of the signals: each one misses a different event class because it answers a different question.

The most instructive row is the traffic collapse. For two hours, the request count drops to a tenth of normal; the error rate is zero throughout, latency sits at its base value, the queue is empty. An indicator that looks only at the error rate sees one hundred percent success in those two hours, because every one of the few requests that did arrive came back successful. This is exactly the picture a failed upstream producer or a broken route produces: the ratio is clean, because the denominator has collapsed.

The queue saturation row shows the signal’s time value. The notification consumer slows down, and the queue grows by six notifications a minute. Saturation clears its three-minute threshold at minute 61; the error signal also triggers once the queue hits its limit and notification requests start getting rejected, but that happens at minute 72. The 11 minutes in between is the window in which the incident can be addressed before it reaches the member. If saturation is not measured, those eleven minutes are lost and the incident is first seen through dropped requests.

The off-event triggers show two separate things. Of the error signal’s 162 triggers, 136 fall in minutes receiving fewer than 40 requests: with a denominator of 30 in night traffic, a single 5xx makes 3.3% and clears the 1% threshold. A ratio-based signal gets noisier as its denominator shrinks — the previous lesson’s denominator discussion comes back here as alert noise. Saturation’s 23 triggers, on the other hand, are not wrong: the first one starts at exactly minute 6780, right where the queue saturation event ends, and measures how long the queue takes to drain. A signal’s window does not have to match the event’s window.

What the Four Together Cannot See

The four signals together cover five of the six event classes. The sixth one that slips through is silent delivery loss: notification requests return 200, their durations do not change, the queue drains normally, no threshold moves. Yet the delivery field inside those same records says 99.7% before the event and 3.0% during it.

The distinction here is a lasting one: the four golden signals measure transport, not correctness. A request being accepted, answered quickly, and not consuming resources does not mean the work got done. In the loan system, the work being done means “the overdue notification reached the member”, and only a business-level counter can say that. The four signals are placed on every service the same way; a fifth counter has to be thought through separately for each service, and for that reason it is often never written at all.

Summary

• The four golden signals ask four separate questions: latency asks “is it slow”, traffic asks “what is the load”, errors ask “is it broken”, saturation asks “how much room is left”; saturation is the same concept as in the Introduction to System Design course.
• The first three come directly from the request log; saturation does not, and the scarce resource must be measured in code — here, every notification request writes the queue depth into its own record.
• Across the six event classes, no signal alone caught more than three: latency 1/6, traffic 2/6, saturation 2/6, error 3/6.
• In the traffic collapse, the error rate is zero and the indicator looks flawless; a ratio-based signal goes blind when its denominator collapses and gets noisy when its denominator shrinks (136 of 162 off-event triggers fall in minutes under 40 requests/min).
• Saturation triggered at minute 61 while the queue was filling; error only triggered at minute 72, once requests started getting rejected — an 11-minute early warning.
• The four together cover five of the six classes; the one that slips through is silent delivery loss (the delivery rate dropped from 99.7% to 3.0%), because the golden signals measure transport, not correctness.

Next Step

The four signals are now computed minute by minute and can be compared against an objective. What happens once it drops below that objective is still not written down. The loan system might get two releases a week, or ten; every release leaves a measurable degradation in the records, and every minute spent below the objective is paid for from somewhere. The next lesson computes that payment from telemetry: it derives a budget from the same request records, counts how fast two different release rates consume that budget, and shows what changes in the code once the budget runs out.