Instrumentation Design

The previous lesson placed fifteen metrics into five categories and tested the list’s completeness with coverage. The test was one-sided: every metric was evaluated only by the diagnosis it enables, and none of them had its cost counted. Yet every metric is produced, carried, stored, and added to a code path; the code path it is added to runs on every one of the 513.89 requests at the peak edge.

This lesson makes the decision two-sided. The answer to whether a metric is worth collecting comes from comparing its cost against the diagnosis it enables. Both are converted to a number.

The Metric’s Three Cost Items

Instrumentation is the calls added to code to produce a metric. Its cost has three items. Records produced: how many metric records are born per unit of time. Bytes stored: how much space those records take up, and for how long. Code path added: how many instrumentation calls run per request.

The third item differs from the first two and cannot be counted directly — incrementing a counter is not the same work as building an event record and queuing it. In this lesson, code path is represented by records produced per request: a form that produces one record per request runs more code path than one that produces one record per second. The computation below’s first column is therefore both the records item and the code path item.

Monthly volume is written in GB-months: one gigabyte held for one month. No currency is used, because a price depends on a provider and a date; a resource unit does not. This rule holds for the rest of the course.

// monitoring/cost.mjs — the cost of five aggregation forms: records produced, bytes stored, and
// the ratio to the system's own write rate. Inputs are K01's figures and this lesson's assumptions.
const PEAK_EDGE = 513.89;      // K01: peak edge requests/s
const PEAK_WRITE = 97.22;      // K01: peak write requests/s (the system's own persistent record)
const EVENT_BYTES = 180;       // IZ2 (assumption): one raw metric record (name, value, time, three tags)
const BUCKET_BYTES = 40;       // IZ3 (assumption): one aggregated bucket record
const BUCKETS = 20;            // IZ4 (assumption): histogram bucket count
const RETENTION = 30;          // IZ5 (assumption): raw record retention (days)
const b = (x, n = 2) => x.toFixed(n);

const FORM = [
  ["raw event", PEAK_EDGE, EVENT_BYTES],
  ["1/10 sample", PEAK_EDGE / 10, EVENT_BYTES],
  ["1/100 sample", PEAK_EDGE / 100, EVENT_BYTES],
  ["per-second histogram", BUCKETS, BUCKET_BYTES],
  ["per-second counter+sum", 2, BUCKET_BYTES],
];

console.log(`${"aggregation form".padEnd(23)}${"records/s".padStart(10)}${"bytes/s".padStart(10)}` +
  `${"daily GB".padStart(11)}${"GB-month (30 days)".padStart(20)}${"system write ratio".padStart(23)}`);
for (const [name, rate, bytes] of FORM) {
  const dailyGB = (rate * bytes * 86_400) / 1e9;
  console.log(`${name.padEnd(23)}${b(rate).padStart(10)}${b(rate * bytes, 0).padStart(10)}` +
    `${b(dailyGB, 3).padStart(11)}${b(dailyGB * RETENTION, 2).padStart(20)}` +
    `${(b(rate / PEAK_WRITE) + "x").padStart(23)}`);
}

const raw = PEAK_EDGE * EVENT_BYTES * 86_400 / 1e9;
const hist = BUCKETS * BUCKET_BYTES * 86_400 / 1e9;
console.log(`\nraw event / per-second histogram = ${b(raw / hist, 0)}x volume`);
console.log(`sampling grows with request rate, aggregation does not: if the request rate rises to 2x, ` +
  `raw becomes ${b(2 * raw, 3)} GB/day, histogram stays ${b(hist, 3)} GB/day`);
console.log(`if 10 metrics are collected in the same form, volume rises to 10x: raw becomes ${b(10 * raw, 2)} GB/day`);

aggregation form        records/s   bytes/s   daily GB  GB-month (30 days)     system write ratio
raw event                  513.89     92500      7.992              239.76                  5.29x
1/10 sample                 51.39      9250      0.799               23.98                  0.53x
1/100 sample                 5.14       925      0.080                2.40                  0.05x
per-second histogram        20.00       800      0.069                2.07                  0.21x
per-second counter+sum       2.00        80      0.007                0.21                  0.02x

raw event / per-second histogram = 116x volume
sampling grows with request rate, aggregation does not: if the request rate rises to 2x, raw becomes 15.984 GB/day, histogram stays 0.069 GB/day
if 10 metrics are collected in the same form, volume rises to 10x: raw becomes 79.92 GB/day

The last column is the measure’s harshest form. Dropping one metric record onto every request produces 5.29 times as many records as the system’s own persistent write: the shipment tracking service writes 97.22 state events per second while monitoring writes 513.89 records. The previous lesson’s list had fifteen metrics; if all of them were collected this way, the monitoring load would stop being a small line item next to the system’s own load. The last line says this directly: ten metrics in raw-event form produce 79.92 GB per day.

The two forms depend on two separate kinds of growth, and the difference decides the design. Sampling grows with request rate, because what is sampled is requests; when the request rate doubles, raw volume rises from 7.992 to 15.984 GB. Aggregation does not grow with request rate, because the number of records produced is set by the bucket count and the aggregation interval; at the same doubling, the histogram stays at 0.069 GB. In a scaling system, this is the question of whether the monitoring cost grows along with the scale.

Gains Are Measured in Loss

Every form that lowers the cost also loses something, and the decision cannot be made until that loss is named. To measure the loss, all five forms are applied to a single population at once. The setup below is a model: latencies are produced with integer arithmetic and a visible seed, so the run repeats independent of machine. The model does not predict a real distribution; what it shows is what the forms do to the same data.

The quantity measured is tail latency, the slowest end of the distribution — it has nothing to do with the queue as a line of waiting jobs; here, the tail is the distribution’s tail. Percentiles were defined in the Introduction to System Design course and are not redefined here.

// monitoring/loss.mjs — information-loss MODEL of sampling and aggregation. Latencies are
// produced with integer arithmetic (seed visible), so the run repeats independent of machine.
const N = 200_000;              // model population: 400 seconds x 500 requests/second
const WINDOW = 500;             // requests per second (close to K01's 513.89 peak rate)
const SEED = 20260730;
let s = SEED;
const rand = (n) => Math.floor((((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32)) * n);

// Latencies are integers in units of 0.1 ms: body, mid-tail, far-tail, farthest-tail.
const LAYER = [[900, 30, 30], [90, 60, 140], [9, 200, 700], [1, 900, 3100]];
const data = [];
for (const [share, base, width] of LAYER)
  for (let i = 0; i < (N * share) / 1000; i += 1) data.push(base + rand(width));
for (let i = data.length - 1; i > 0; i -= 1) {          // seeded shuffle
  const j = rand(i + 1);
  [data[i], data[j]] = [data[j], data[i]];
}

const percentile = (arr, p) => {
  const d = [...arr].sort((a, b) => a - b);
  return d[Math.min(d.length - 1, Math.floor((p / 100) * d.length))];
};
const ms = (x) => (x / 10).toFixed(1);
const maxOf = (d) => d.reduce((a, c) => (c > a ? c : a), 0);

const EDGE = [30, 40, 50, 60, 80, 100, 130, 170, 220, 290, 380, 500, 650, 850, 1100, 1500,
  2000, 2600, 3400, Infinity];
const bucket = new Array(EDGE.length).fill(0);
for (const v of data) bucket[EDGE.findIndex((k) => v < k)] += 1;
const bucketPercentile = (p) => {
  let total = 0;
  const target = (p / 100) * data.length;
  for (let i = 0; i < bucket.length; i += 1) {
    total += bucket[i];
    if (total >= target) return EDGE[i];
  }
  return Infinity;
};
const bucketMax = EDGE[bucket.findLastIndex((k) => k > 0)];

const averages = [];
for (let i = 0; i < data.length; i += WINDOW)
  averages.push(data.slice(i, i + WINDOW).reduce((a, c) => a + c, 0) / WINDOW);

const everyKth = (k) => data.filter((_, i) => i % k === 0);
const exact = { p50: percentile(data, 50), p99: percentile(data, 99), p999: percentile(data, 99.9),
  max: maxOf(data) };

console.log(`seed ${SEED}, ${N} requests, ${data.length / WINDOW} seconds, ${EDGE.length} buckets`);
console.log(`true value: p50 ${ms(exact.p50)} ms, p99 ${ms(exact.p99)} ms, ` +
  `p99.9 ${ms(exact.p999)} ms, maximum ${ms(exact.max)} ms`);
console.log(`\n${"aggregation form".padEnd(23)}${"records stored".padStart(15)}${"p50".padStart(8)}` +
  `${"p99".padStart(8)}${"p99.9".padStart(9)}${"maximum".padStart(10)}${"p99 error".padStart(12)}`);

const print = (name, records, p50, p99, p999, max) =>
  console.log(`${name.padEnd(23)}${records.toLocaleString("en-US").padStart(15)}${ms(p50).padStart(8)}` +
    `${ms(p99).padStart(8)}${ms(p999).padStart(9)}${ms(max).padStart(10)}` +
    `${(((p99 - exact.p99) / exact.p99) * 100).toFixed(1).padStart(11)}%`);

print("raw event", data.length, exact.p50, exact.p99, exact.p999, exact.max);
for (const k of [10, 100]) {
  const o = everyKth(k);
  print(`1/${k} sample`, o.length, percentile(o, 50), percentile(o, 99), percentile(o, 99.9),
    maxOf(o));
}
print("per-second histogram", (data.length / WINDOW) * EDGE.length, bucketPercentile(50),
  bucketPercentile(99), bucketPercentile(99.9), bucketMax);
print("per-second counter+sum", (data.length / WINDOW) * 2, percentile(averages, 50),
  percentile(averages, 99), percentile(averages, 99.9), maxOf(averages));

const overallAvg = data.reduce((a, c) => a + c, 0) / data.length;
const avgOfAvg = averages.reduce((a, c) => a + c, 0) / averages.length;
console.log(`\nthe average is preserved exactly in both forms: raw ${ms(overallAvg)} ms, ` +
  `the average of per-second averages ${ms(avgOfAvg)} ms`);
const slow = data.filter((v) => v >= 200).length;
console.log(`requests over 200 ms: ${slow} in raw data, ${everyKth(100).filter((v) => v >= 200).length} ` +
  `in the 1/100 sample (expected ${(slow / 100).toFixed(1)})`);

seed 20260730, 200000 requests, 400 seconds, 20 buckets
true value: p50 4.6 ms, p99 20.0 ms, p99.9 90.3 ms, maximum 399.5 ms

aggregation form        records stored     p50     p99    p99.9   maximum   p99 error
raw event                      200,000     4.6    20.0     90.3     399.5        0.0%
1/10 sample                     20,000     4.6    20.4     91.3     397.6        2.0%
1/100 sample                     2,000     4.6    38.6    351.8     397.6       93.0%
per-second histogram             8,000     5.0    22.0    110.0  Infinity       10.0%
per-second counter+sum             800     5.8     7.9      8.8       8.8      -60.5%

the average is preserved exactly in both forms: raw 5.9 ms, the average of per-second averages 5.9 ms
requests over 200 ms: 2000 in raw data, 22 in the 1/100 sample (expected 20.0)

Same Cost, Different Loss

Reading the two tables side by side gives the lesson’s main result. The 1/100 sample costs 2.40 GB-months, the per-second histogram 2.07 GB-months — their costs are close. Their losses are not: the sample’s p99 error is 93.0 percent, the histogram’s 10.0 percent. In the sample, p99.9 becomes entirely meaningless, because at 2,000 records that percentile rests on only two records; it reads 351.8 ms instead of the true 90.3 ms. When a form exists that loses far less information at the same cost, choosing the form that loses more is not a decision — it is an oversight.

What sampling loses stands out plainly in the last line: the raw data has 2,000 requests over 200 ms, and the 1/100 sample sees 22 of them. The number is close to the expected 20.0, so the sample is not biased — what is lost is precision. The more extreme a percentile, the fewer records carry it, and sampling erases the most extreme information first. The 1/10 sample still holds 20,000 records and its p99 error is 2.0 percent; this shows that the loss does not grow linearly with the sampling ratio.

Aggregation’s loss takes a different form. The histogram counts every request, drops none, but rounds the value to the bucket boundary: p99 reads 22.0 instead of 20.0, and the error comes from the bucket width. The maximum column, meanwhile, shows Infinity — because the top bucket has no upper bound, the histogram cannot bound the maximum latency. Bringing it back means storing one more number per second: the maximum. One number costs 40 bytes.

The per-second counter and sum, meanwhile, erase the distribution entirely. The average is preserved exactly in both forms (5.9 ms), but the per-second averages’ p99 is 7.9 ms and the largest value it sees is 8.8 ms; the true maximum is 399.5 ms. When five hundred requests are averaged, one slow request is swallowed by four hundred ninety-nine fast ones. A metric that reports average latency does not measure tail latency; this form’s cheapness comes from precisely that information being thrown away.

The Decision Rule

Three results give the rule for the per-metric decision. First: an aggregation form is chosen not by its cost but by the information it preserves per unit of cost. Second: the form is chosen by the diagnosis the metric will enable. The histogram answers “how slow” cheaply but cannot answer “which request was slow,” because it has no record identity; the sample does the opposite — it holds few but complete records, giving a request’s trace along with its tags. Of the previous lesson’s ten diagnoses, the ones separated by metric composition want complete records; those that only need to see a threshold crossed are satisfied with a histogram.

Third is the scaling rule: raw events and samples grow with request rate, aggregation does not. A sample that costs 2.40 GB-months today at 513.89 requests/s costs 4.80 GB-months once traffic doubles; under the same condition, the histogram stays at 2.07 GB-months. A monitoring design cannot be made independent of the system’s scaling plan.

Summary

• A metric’s cost has three items: records produced, bytes stored, and code path added; the third is represented by records produced per request. Volume is written in GB-months, not currency.
• A single metric that drops a record onto every request produces 5.29 times as many records as the system’s own persistent write (513.89 against 97.22 records/s); ten metrics collected the same way come to 79.92 GB per day.
• Sampling grows with request rate, aggregation does not: when the request rate doubles, raw volume rises from 7.992 to 15.984 GB/day, the histogram stays at 0.069 GB/day.
• The 1/100 sample and the per-second histogram are close in cost (2.40 and 2.07 GB-months), but their p99 errors are 93.0 percent and 10.0 percent; the form that loses less at the same cost is chosen.
• Sampling erases the most extreme information first (22 of 2,000 slow requests are seen); aggregation rounds the value to the bucket boundary and, because the top bucket is unbounded, cannot bound the maximum value at all.
• Storing an average erases the distribution: the largest value the per-second averages see is 8.8 ms while the true maximum is 399.5 ms; the average itself is preserved as 5.9 ms in both forms.

Next Step

These two lessons decided what to collect and in what form. The collected number still tells no one anything: it will be plotted somewhere, or it will wake someone up. The two are not the same thing, and they do not work with the same threshold. A plot answers the question of someone already looking at it; an alert calls someone who is not looking to look, and it has a cost when it calls wrongly. The next lesson separates the two, ties where the alert threshold comes from to the Introduction to System Design course’s monthly 43.2-minute outage budget, and tests the threshold choice with two numbers: how many false alerts it produces and how many real events it misses.