Metric Types

The previous two lessons matured the log: the record split into fields, and a request became traceable across four services. But every query still starts by reading every record. In the first lesson the metric sat outside that cost, because the question it could answer was already chosen when the metric was written. This lesson looks at that choice: the same fact is measured with three metric types, and what question each type opens up and what question it forecloses is counted.

The fact being measured is a single one: the count of open loans. Every request opens a loan, and in between there are batch returns and occasional batch-loan days. The same number is measured with three types. A counter only ever increases. A gauge holds the current value. A histogram distributes values into buckets. All three feed off the same fact, and all three answer a different question.

Instrumentation’s own cost — the overhead producing a metric adds to a request — was measured in the Performance Anti-Patterns and Monitoring course and is not repeated here. The cost here is memory and accuracy: how many fields a type holds, and how far off its answer really is.

Three Types

// metrics.mjs — three metric types; all three are hand-written, nothing comes from outside
export class Counter {                     // only increases; the past is folded into a single number
  constructor() { this.value = 0; }
  add(n = 1) { this.value += n; }
  read() { return this.value; }
}
export class Gauge {                       // holds the current value; the previous value is lost
  constructor() { this.value = 0; }
  set(v) { this.value = v; }
  read() { return this.value; }
}
export class Histogram {                   // bucket counts; the distribution is preserved approximately
  constructor(bounds) { this.bounds = bounds; this.bucket = new Array(bounds.length + 1).fill(0); this.count = 0; }
  observe(v) { let i = 0; while (i < this.bounds.length && v > this.bounds[i]) i += 1; this.bucket[i] += 1; this.count += 1; }
  size() { return this.bucket.length; }
  percentile(p) {                          // linear interpolation inside the bucket that crosses the target
    const target = p * this.count; let accumulated = 0;
    for (let i = 0; i < this.bucket.length; i += 1) {
      if (accumulated + this.bucket[i] >= target) {
        const lower = i === 0 ? 0 : this.bounds[i - 1];
        const upper = i < this.bounds.length ? this.bounds[i] : lower * 2;
        return lower + ((target - accumulated) / this.bucket[i]) * (upper - lower);
      }
      accumulated += this.bucket[i];
    }
    return this.bounds[this.bounds.length - 1];
  }
}
export const equalBounds = (k, max) =>     // k buckets: the [0, max] range is split evenly
  Array.from({ length: k - 1 }, (_, i) => Math.round(((i + 1) * max) / (k - 1)));

The three classes’ total state is this: the counter is one number, the gauge is one number, the histogram is as many numbers as it has buckets. The types differ not in their behavior but in what they forget. The counter folds every observation into the total and forgets the observation. The gauge holds the latest observation and forgets the one before it. The histogram holds which bucket an observation fell into and forgets the value itself.

TL3 — the pattern by which the open-loan count changes is independent of the run: return and batch-loan decisions come from a generator with a visible seed. Rationale: what is being compared is the behavior of the metric types, not the speed of the environment; if the same observation sequence is not produced on every run, the error margins cannot be compared.

Same Fact, Three Types

// service3.mjs — loan service; measures the same fact (open loan count) with three types; long-lived process
import { createServer } from "node:http";
import { Counter, Gauge, Histogram, equalBounds } from "./metrics.mjs";
const opened = new Counter(), open = new Gauge(), distribution = new Histogram(equalBounds(8, 400));
let seed = 20260731;                       // visible seed: the return pattern is independent of the run
const next = () => (seed = (seed * 1103515245 + 12345) % 2147483648);
let value = 0, peak = 0;                   // peak: kept to show what the gauge misses
const sample = [];                         // full sample: kept only for comparison

createServer((req, res) => {
  if (req.url === "/metrics") {
    res.end(JSON.stringify({ opened: opened.read(), open: open.read(),
      bucket: distribution.bucket, bounds: distribution.bounds, peak }));
    return;
  }
  if (req.url === "/samples") { res.end(JSON.stringify(sample)); return; }
  value += 1; opened.add();                          // every request opens a loan
  if (next() % 60 === 0) value = Math.max(0, value - Math.ceil(value / 2));  // batch return
  if (next() % 211 === 0) value += 180;               // batch-loan day
  peak = Math.max(peak, value);
  open.set(value); distribution.observe(value); sample.push(value);
  res.end("done");
}).listen(9301);

The service also keeps two things purely for comparison: the true peak value and the full sample. Neither would be kept in a real system — precisely because of the cost this lesson measures. They stay here so that the answer each of the three types gives can be compared against the truth.

Measurement

The query tool does two things. First it lines up the three types side by side and prints the question each one answers and cannot answer. Then it re-summarizes the same observation sequence with histograms of four, eight, sixteen, thirty-two, and sixty-four buckets, and works out each one’s memory cost and p95 error.

// query3.mjs — the question the three types answer, and the cost of the histogram's bucket count
import { Histogram, equalBounds } from "./metrics.mjs";
const m = await (await fetch("http://127.0.0.1:9301/metrics")).json();
const sample = await (await fetch("http://127.0.0.1:9301/samples")).json();
const sorted = [...sample].sort((a, b) => a - b);
const actual = (p) => sorted[Math.floor(p * sorted.length)];
const h = new Histogram(m.bounds); h.bucket = m.bucket; h.count = sample.length;

console.log(`${"type".padEnd(11)}${"size".padStart(5)}  ${"question answered".padEnd(36)}question left unanswered`);
console.log(`${"counter".padEnd(11)}${"1".padStart(5)}  ${`loans opened: ${m.opened}`.padEnd(36)}loans open right now`);
console.log(`${"gauge".padEnd(11)}${"1".padStart(5)}  ${`loans open right now: ${m.open}`.padEnd(36)}what the past peak was`);
console.log(`${"histogram".padEnd(11)}${String(m.bucket.length).padStart(5)}  ` +
  `${`95% of observations under ${Math.round(h.percentile(0.95))}`.padEnd(36)}which observation was the peak`);
console.log(`actual p95 ${actual(0.95)}, actual peak ${m.peak}; ` +
  `full sample ${sample.length} values x 8 bytes = ${sample.length * 8} bytes`);

console.log(`\n${"buckets".padStart(8)}${"memory".padStart(10)}${"p95 estimate".padStart(14)}${"error".padStart(7)}${"relative".padStart(10)}`);
for (const k of [4, 8, 16, 32, 64]) {
  const hh = new Histogram(equalBounds(k, 400));
  for (const v of sample) hh.observe(v);
  const t = hh.percentile(0.95), error = Math.abs(t - actual(0.95));
  console.log(`${String(k).padStart(8)}${`${hh.size() * 8} bytes`.padStart(10)}${t.toFixed(1).padStart(14)}` +
    `${error.toFixed(1).padStart(7)}${`%${((100 * error) / actual(0.95)).toFixed(1)}`.padStart(10)}`);
}

cat > run.mjs <<'EOF'
const READ_EVERY = 100;                            // the gauge is read once every 100 requests
let highest = 0, reads = 0;
for (let i = 1; i <= 2000; i += 1) {
  await fetch("http://127.0.0.1:9301/work");
  if (i % READ_EVERY !== 0) continue;
  const m = await (await fetch("http://127.0.0.1:9301/metrics")).json();
  highest = Math.max(highest, m.open); reads += 1;
}
const m = await (await fetch("http://127.0.0.1:9301/metrics")).json();
console.log(`gauge: 2000 observations, ${reads} reads, highest seen ${highest}, ` +
  `actual peak ${m.peak}; observations missed between reads ${2000 - reads}`);
EOF
node service3.mjs &
sleep 1

node run.mjs
echo
node query3.mjs
pkill -f "node service3.mjs"

gauge: 2000 observations, 20 reads, highest seen 199, actual peak 293; observations missed between reads 1980

type        size  question answered                   question left unanswered
counter        1  loans opened: 2000                  loans open right now
gauge          1  loans open right now: 26            what the past peak was
histogram      8  95% of observations under 159       which observation was the peak
actual p95 159, actual peak 293; full sample 2000 values x 8 bytes = 16000 bytes

 buckets    memory  p95 estimate  error  relative
       4  32 bytes         209.1   50.1     %31.5
       8  64 bytes         158.5    0.5      %0.3
      16 128 bytes         157.1    1.9      %1.2
      32 256 bytes         161.5    2.5      %1.6
      64 512 bytes         158.5    0.5      %0.3

Each Type Forecloses One Question

The top table gives the division of labor between the types. The counter says in a single field that two thousand loans were opened, but because it does not know how many closed, it cannot answer “how many loans are open right now” — an increase that cannot be taken back is the counter’s definition. The gauge answers that question in a single field: twenty-six. The histogram answers what neither of the other two can: ninety-five percent of observations stayed under 159.

The gauge’s cost sits in the first line, and it is this lesson’s sharpest number. Two thousand observations were produced, the gauge was read twenty times, and the highest value read was 199 — the actual peak was 293. The nineteen hundred and eighty observations that fell between reads were never seen at all, and the peak was reported thirty-two percent short. Because what a gauge holds is “the value right now,” whatever movement happens between reads is lost no matter how often it is read. When a gauge is asked whether a threshold was crossed, a “no” answer can mean the threshold was never crossed, or it can mean the read never landed on the moment it was crossed.

The bottom table ties the histogram’s bucket count to its cost. Four buckets take thirty-two bytes and give a p95 of 209: a deviation of roughly thirty-one and a half percent from the true 159. At eight buckets the error drops to half a unit, 0.3 percent. What comes after this is surprising, and it is the lesson’s second finding: error does not shrink evenly with the bucket count. It is 1.9 at sixteen buckets, 2.5 at thirty-two, and back down to 0.5 at sixty-four. The reason is plain — the error depends on where the bucket boundaries happen to fall relative to the distribution. The ninety-fifth percentile sits around 159; if a boundary happens to fall close to it, the interpolation covers a short distance and the error shrinks, and if the boundary is far off, the inside of the bucket is assumed linear and the error grows. Raising the bucket count lowers the expected error, but it gives no guarantee for any single percentile.

The comparison line is the last one: the full sample takes 16,000 bytes and gives the p95 exactly. The eight-bucket histogram takes 64 bytes and gives the same answer with half a unit of error — two hundred fifty times less memory. This is the metric’s real job: to hold an approximate answer at a fixed cost. A fixed cost is the opposite of the previous lesson’s log volume, which grows with the request count.

Summary

• The same fact was measured with three types: the counter gave 2000 opens, the gauge gave 26 open loans, the histogram gave 95% of observations under 159; none answered the others’ question.
• The types differ in what they forget: the counter forgets the observation, the gauge forgets the previous value, the histogram forgets the value itself.
• The gauge saw 20 of 2000 observations; the peak read was 199, the actual peak was 293 — the 1980 observations between reads were lost, and the peak was reported short.
• The histogram’s bucket count grows memory linearly (4 buckets, 32 bytes; 64 buckets, 512 bytes), but the error does not shrink evenly: 31.5% → 0.3% → 1.2% → 1.6% → 0.3%.
• An 8-bucket histogram gave the same p95 as the 16,000-byte full sample using 64 bytes and half a unit of error; a metric’s cost grows with its field count, not its observation count.

Next Step

Two of the three signals are now produced in code: the field-based log traces a request across four services, and the metrics give a fixed-cost summary. What remains is the original question the first lesson left unanswered. The correlation id says which services a request passed through, but it does not say how the durations nest inside one another: did the catalog call start inside the loan request, did it finish before the membership call, where did the waiting happen. The next lesson hand-writes a trace context and carries it across the process boundary. What gets measured is clear: how many traces break where the context is not carried through, how the sampling rate affects the percentage of problem requests caught, and the volume of the trace record.