Synthetic Monitoring

Every signal produced up to this point had one thing in common: all of them were born from real traffic. A request arrived, a span record was written, a metric bucket filled, a rule fired. If the first link of this chain breaks, none of the rest works at all — if no request arrives, there is nothing to measure either.

This lesson models fourteen days of the regional measurement network’s (fiction) traffic across five flows and measures ten faults two separate ways: a rule born from real traffic, and periodic polling from the outside. Three things are counted: the fault real traffic never sees at all, the probe’s detection delay together with its cost, and the case where the probe passes while the user is affected.

Real Traffic’s Blind Spots

// synthetic/traffic.mjs — MODEL of fourteen days of real traffic and ten faults. MD31: the per-minute request
// count of five flows is fiction; the billing query is busy by day and idle at night, the annual closing flow
// is called a few times a month. MD32: a fault is one of three shapes — "full" breaks every request, "partial"
// breaks only one subscriber group, "data" breaks the path that does not return from cache.
export const SEED = 20260806, DAYS = 14, MINUTES = DAYS * 1440;
export const HOUR_PATTERN = [2, 1, 1, 1, 1, 4, 14, 32, 46, 52, 55, 53, 48, 51, 54, 50, 44, 34, 26, 18, 12, 8, 5, 3];
// [name, daytime multiplier, runs off-hours, share affected by partial breakage]
export const FLOWS = [["billing_query", 1, true, 0.18], ["payment_intake", 0.35, true, 0.22],
  ["meter_change", 0.012, false, 0.30], ["annual_closing", 0.00006, false, 0.5],
  ["external_verification", 0.12, true, 0.25]];
// [flow, day, hour, duration min, type, name]
export const FAULTS = [
  [0, 2, 14, 40, "full", "daytime billing outage"], [0, 4, 3, 120, "full", "overnight billing outage"],
  [0, 6, 10, 90, "partial", "error in one subscriber group"], [0, 9, 11, 60, "data", "non-cached path broken"],
  [1, 3, 20, 75, "full", "evening payment outage"], [1, 7, 2, 180, "full", "overnight payment outage"],
  [2, 5, 22, 600, "full", "weekend meter change"], [3, 8, 4, 1440, "full", "annual closing flow"],
  [4, 10, 1, 240, "full", "external dependency fault"], [4, 12, 9, 50, "partial", "partial external verification"],
];
export const window = (i) => [FAULTS[i][1] * 1440 + FAULTS[i][2] * 60, FAULTS[i][3]];

export function traffic() {
  let s = SEED;
  const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32);
  const T = FLOWS.map(() => ({ request: new Float64Array(MINUTES), error: new Float64Array(MINUTES) }));
  for (let t = 0; t < MINUTES; t += 1) {
    const hour = Math.floor(t / 60) % 24, day = Math.floor(t / 1440) % 7;
    for (let a = 0; a < FLOWS.length; a += 1) {
      const [, multiplier, allDay] = FLOWS[a];
      const workHours = allDay || (hour >= 8 && hour < 18 && day < 5);
      const n = HOUR_PATTERN[hour] * multiplier * (workHours ? 1 : 0.02) * (0.85 + r() * 0.3);
      T[a].request[t] = Math.max(0, Math.floor(n + (r() < n % 1 ? 1 : 0)));
      let broken = 0;
      for (let i = 0; i < FAULTS.length; i += 1) {
        const [flow, , , dur, type] = FAULTS[i], [start] = window(i);
        if (flow !== a || t < start || t >= start + dur) continue;
        broken = Math.max(broken, type === "full" ? 1 : type === "partial" ? FLOWS[a][3] : 0.45);
      }
      T[a].error[t] = Math.round(T[a].request[t] * broken);
    }
  }
  return T;
}

if (import.meta.filename === process.argv[1]) {
  const T = traffic();
  const sum = (a, f, b, e) => { let x = 0; for (let t = b; t < e; t += 1) x += T[a][f][t]; return x; };
  console.log(`seed ${SEED}, ${DAYS} days, ${FLOWS.length} flows, ${FAULTS.length} faults\n`);
  console.log(`${"flow".padEnd(24)}${"daily requests".padStart(16)}${"overnight 03-05/min".padStart(22)}` +
    `${"idle minutes".padStart(15)}`);
  FLOWS.forEach(([name], a) => {
    let idle = 0;
    for (let t = 0; t < MINUTES; t += 1) if (T[a].request[t] === 0) idle += 1;
    console.log(`${name.padEnd(24)}${(sum(a, "request", 0, MINUTES) / DAYS).toFixed(0).padStart(16)}` +
      `${((sum(a, "request", 180, 300) + sum(a, "request", 1620, 1740)) / 240).toFixed(2).padStart(22)}` +
      `${`${(idle / MINUTES * 100).toFixed(0)}%`.padStart(15)}`);
  });
}

seed 20260806, 14 days, 5 flows, 10 faults

flow                      daily requests   overnight 03-05/min   idle minutes
billing_query                      36899                  1.02             1%
payment_intake                     12936                  0.35            12%
meter_change                         251                  0.00            83%
annual_closing                         2                  0.00           100%
external_verification               4438                  0.11            24%

The table shows three separate blind spots. Idle hour: the billing query gets 36,899 requests a day, but between three and five in the morning it drops to 1.02 requests per minute. Over a five-minute window that comes to five requests, and five requests is not enough to compute an error rate; the rule does not fire, because the denominator itself is not there. Idle flow: 83% of the meter-change flow’s minutes, and 100% of the annual-closing flow’s minutes, are idle — the annual closing flow gets two requests a day. A flow like this can stay broken for months and no metric shows it. External dependency: the external verification endpoint is visible only when we send it a request; that endpoint’s own fault does not show up in our metrics during the hours we do not call it.

Probe and Traffic Side by Side

The probe sends a request to every flow at set intervals and checks the response. Three choices define this request: where it is sent from (from the direction the user comes from, from outside the system), which flow is run (not a single endpoint, but an end-to-end piece of work), and what it leaves behind — when a write-performing flow is probed, a record stays in the system every time, so the probe either picks a read-only path or deletes what it leaves behind itself. The comparison below measures the same ten faults two ways.

// synthetic/probe.mjs — the rule born from real traffic and periodic polling from the outside are compared
// on the same ten faults. MD33: the real-traffic rule does not fire without seeing at least 20 requests in
// a five-minute window; it alerts if the error rate exceeds 5% for three minutes. MD34: the probe sends
// one request per flow every P minutes, two consecutive failed probes produce an alert. MD35: the probe
// runs with a fixed meter id and a single account; this is why the probe passes on "partial" and "data"
// type faults. MD36: every probe to the external verification endpoint is billed.
import { traffic, FLOWS, FAULTS, window, MINUTES, DAYS } from "./traffic.mjs";
const T = traffic(), MIN = 20, RATE = 0.05, STREAK = 3;

function realDetect(i) {
  const flow = FAULTS[i][0], dur = FAULTS[i][3], [start] = window(i);
  let streak = 0;
  for (let t = start; t < start + dur; t += 1) {
    let n = 0, h = 0;
    for (let u = Math.max(0, t - 4); u <= t; u += 1) { n += T[flow].request[u]; h += T[flow].error[u]; }
    if (n >= MIN && h / n > RATE) { streak += 1; if (streak >= STREAK) return t - start; } else streak = 0;
  }
  return null;
}
// The probe only fails on "full" faults; because it runs with fixed input it never sees the other two types (MD35).
const probeFails = (i, t) => {
  const [, , , dur, type] = FAULTS[i], [start] = window(i);
  return type === "full" && t >= start && t < start + dur;
};
function syntheticDetect(i, P) {
  const [, , , dur] = FAULTS[i], [start] = window(i);
  let streak = 0;
  for (let t = Math.ceil(start / P) * P; t < start + dur; t += P) {
    if (probeFails(i, t)) { streak += 1; if (streak >= 2) return t - start; } else streak = 0;
  }
  return null;
}

const P0 = 5;
console.log(`${"fault".padEnd(31)}${"flow".padEnd(24)}${"type".padStart(9)}` +
  `${"real traffic".padStart(15)}${"probe (5 min)".padStart(16)}`);
const fmt = (x) => (x === null ? "-" : `${x} min`);
FAULTS.forEach(([flow, , , , type, name], i) =>
  console.log(`${name.padEnd(31)}${FLOWS[flow][0].padEnd(24)}${type.padStart(9)}` +
    `${fmt(realDetect(i)).padStart(15)}${fmt(syntheticDetect(i, P0)).padStart(16)}`));
const real = FAULTS.map((_, i) => realDetect(i)), synthetic = FAULTS.map((_, i) => syntheticDetect(i, P0));
const missed = FAULTS.filter((_, i) => real[i] === null);
console.log(`\nfaults real traffic never saw ${missed.length}: ` +
  `${missed.map((a) => a[5]).join(", ")}`);
const falseConfidence = FAULTS.filter((_, i) => synthetic[i] === null && FAULTS[i][4] !== "full");
console.log(`faults where the probe passes but the user is affected ${falseConfidence.length}: ` +
  `${falseConfidence.map((a) => `${a[5]} (${a[4]})`).join(", ")}`);
const both = FAULTS.filter((_, i) => real[i] === null && synthetic[i] === null);
console.log(`real traffic ${real.filter((x) => x !== null).length}/${FAULTS.length}, probe ` +
  `${synthetic.filter((x) => x !== null).length}/${FAULTS.length}, both together ` +
  `${FAULTS.length - both.length}/${FAULTS.length}`);

const dailyRequestTotal = T.reduce((x, a) => x + [...a.request].reduce((p, q) => p + q, 0), 0) / DAYS;
console.log(`\n${"probe interval".padEnd(17)}${"caught".padStart(11)}${"avg delay".padStart(14)}` +
  `${"probes/day".padStart(13)}${"billed/day".padStart(13)}${"synthetic share".padStart(18)}` +
  `${"sparse flow".padStart(15)}`);
for (const P of [1, 5, 15, 60]) {
  const hits = FAULTS.map((_, i) => syntheticDetect(i, P)).filter((x) => x !== null);
  const dailyProbes = FLOWS.length * (1440 / P), sparse = [...T[3].request].reduce((p, q) => p + q, 0) / DAYS;
  console.log(`${`${P} min`.padEnd(17)}${`${hits.length}/${FAULTS.length}`.padStart(11)}` +
    `${`${(hits.reduce((a, b) => a + b, 0) / hits.length).toFixed(1)} min`.padStart(14)}` +
    `${String(dailyProbes).padStart(13)}${String(1440 / P).padStart(13)}` +
    `${`${(dailyProbes / (dailyRequestTotal + dailyProbes) * 100).toFixed(2)}%`.padStart(18)}` +
    `${`${((1440 / P) / (sparse + 1440 / P) * 100).toFixed(1)}%`.padStart(15)}`);
}

fault                          flow                         type   real traffic   probe (5 min)
daytime billing outage         billing_query                full          2 min           5 min
overnight billing outage       billing_query                full              -           5 min
error in one subscriber group  billing_query             partial          3 min               -
non-cached path broken         billing_query                data          2 min               -
evening payment outage         payment_intake               full          2 min           5 min
overnight payment outage       payment_intake               full              -           5 min
weekend meter change           meter_change                 full              -           5 min
annual closing flow            annual_closing               full              -           5 min
external dependency fault      external_verification        full              -           5 min
partial external verification  external_verification     partial          2 min               -

faults real traffic never saw 5: overnight billing outage, overnight payment outage, weekend meter change, annual closing flow, external dependency fault
faults where the probe passes but the user is affected 3: error in one subscriber group (partial), non-cached path broken (data), partial external verification (partial)
real traffic 5/10, probe 7/10, both together 10/10

probe interval        caught     avg delay   probes/day   billed/day   synthetic share    sparse flow
1 min                   7/10       1.0 min         7200         1440            11.66%          99.9%
5 min                   7/10       5.0 min         1440          288             2.57%          99.4%
15 min                  7/10      15.0 min          480           96             0.87%          98.3%
60 min                  6/10      60.0 min          120           24             0.22%          93.6%

Real traffic caught five of the ten faults, and it was fast when it did: it saw the daytime billing outage at minute 2, the probe saw the same fault at minute 5. During daytime hours, real traffic beats the probe, because a sample of fifty requests a minute is bigger than any single probe.

The five faults it missed fit on one list too: two overnight outages, the weekend meter-change breakage, the annual closing flow, and the external dependency’s own fault. What the five have in common is that not enough requests reached that flow during the fault. The probe caught all five of these, because the request the probe sends does not depend on the hour or the season.

Frequency and Cost

The frequency sweep holds two curves in the same table. At a one-minute interval, detection delay drops to 1 minute, but 7,200 probes are sent a day, and 1,440 of them are billed external calls. A heavier cost sits in the two right-hand columns: synthetic requests climb to 11.66% of total traffic, and to 99.9% on the sparse flow. Everything seen on that flow’s dashboard is now our own probing; the real usage measure disappears.

At a sixty-minute interval, cost drops to 0.22%, but caught faults fall from seven to six: the forty-minute daytime outage closes before it reaches two consecutive probes. The probe interval must be smaller than the shortest fault meant to be seen — the two-consecutive-failed-probes condition doubles the interval.

The fifteen-minute interval holds the balance in this sweep: seven faults, 15 minutes of delay, 480 probes a day, 96 billed calls, and 0.87% of total traffic. The choice is a comparison of cost per probe against detection delay, and it can be made separately per flow: a flow left idle at night gets probed often, a flow busy by day gets probed rarely, because real traffic already sees the latter.

False Confidence

There are three faults where the probe passes while the user is affected, and two of them have a clear name. Fixed-input probe: the probe runs with a single meter id and a single account; in a breakage affecting one subscriber group, that account is not affected, so the probe passes. The same shape repeats in the external verification’s partial fault. Probe answered from cache: because the probe always requests the same record, the response comes from cache; while the data path behind the cache is broken, the probe stays green.

The common root of the two cases is this: the probe measures one path through the system, not the system itself. A green probe dashboard does not say “everything works,” it says “the path we tried works.” That is why the probe does not replace real traffic; the two together cover ten of ten faults, neither one covers them alone.

Summary

• A rule born from real traffic does not fire where no request arrives: on a flow that drops to one request a minute at night, a five-minute window is not enough to compute an error rate.
• Real traffic never saw five of the ten faults; what they all had in common was that not enough requests reached that flow during the fault (two overnight outages, a weekend breakage, a flow used a few times a year, an external dependency’s own fault).
• The probe caught all five of these too but missed three faults, and it lagged behind real traffic on two faults during daytime traffic; together the two cover ten of ten faults.
• The frequency sweep showed two costs together: a one-minute interval brings delay down to 1 minute but costs 7,200 probes a day, 1,440 billed calls, and 11.66% of total traffic; on the sparse flow, 99.9% of the metrics become synthetic.
• Because the probe interval alerts on two consecutive failed probes, it must be chosen at less than half the shortest fault meant to be seen; at sixty minutes, the forty-minute outage is missed.
• False confidence has two names: a fixed-input probe does not show a breakage affecting one subscriber group, and a probe answered from cache does not show a broken data path.

Next Step

In this topic, the signal passed through three stages: a request’s path across services became visible as a trace tree, became mergeable through a shared convention, was tied to a question on a dashboard, was filtered through rules, and was confirmed by polling from the outside. The same thing always sits at the end of the chain: the signal wakes up a person. That person’s arrangement, though, was never addressed at all. Who wakes up, how many nights a month they wake up, where they read what to do once they are awake, who they hand the work to in the morning, and whether the item opened after an incident closes so the same incident does not happen again — none of this sits in a metric bucket or on a panel. The next topic continues from where the signal ends and the person begins.