Metric Collection Models

The previous course closed at the point where the infrastructure was tied to a record, and every change passed through a review and a gate. But once past the gate, no one could see what the system did while it was running. This course’s first question picks up from there and takes one step further: who decides what gets measured?

This course’s unit of measure is not data, it is the question. Whether a collection setup is good is measured not by how many gigabytes it accumulates, but by how many of the questions genuinely asked in operations it answers. This lesson builds that set of questions and runs two collection models — pull-based and push-based — in the same window, applying both to the set.

The measurement is again on the regional measurement network: software that collects readings from water meters, validates the readings, turns them into bills, and opens work orders for field crews. Both the network and the measured numbers are fictional; no real collector is run, what gets counted is a model built with node.

ML1. The twelve questions are fictional and written in advance, as is the resource class, metric, and labels each question requires. ML2. The inventory is fictional: 29 long-lived instances across three environments, 11 restarts within a one-hour window, and 42 nightly validation runs. ML3. The pull-based collector scrapes every 15 seconds and refreshes its target inventory every 30 seconds; the push-based sender flushes every 10 seconds in a long-lived process, and the batch job sends at the end of every step. ML4. A record is 16 bytes, and a series descriptor is 120 bytes. ML5. A resource class is counted as covered only if at least two measurements can be taken from at least 90% of its lifetimes; a single measurement does not show a change. ML6. Restart times and run durations are produced with our own generator, seed 20260901.

The Twelve Questions Asked in Operations

The question set is a product of operations, not of the measurement tool: these are sentences genuinely asked during on-call, in the morning report, and during incident review. Each question is tied to three things — which resource class the data must come from, which metric, and which labels.

// network.mjs — regional measurement network model and the operational question set. Both are fictional.
export const INSTANCES = {                            // environment -> service -> long-lived instances
  production: { "reading-collector": 8, validator: 6, billing: 4, "work-order": 3 },
  staging: { "reading-collector": 2, validator: 2, billing: 1, "work-order": 1 },
  development: { "reading-collector": 1, validator: 1, billing: 0, "work-order": 0 } };
export const LONG_COUNT = Object.values(INSTANCES)
  .reduce((t, o) => t + Object.values(o).reduce((a, b) => a + b, 0), 0);

export const METRICS_LONG = { "request-count": { result: 5 }, "request-duration": { bucket: 12 },
  "queue-depth": { queue: 3 }, "processed-reading": { region: 11, result: 5 },
  "process-start-time": {}, "gc-pause": { bin: 8 }, "file-descriptor-count": {} };
export const METRICS_SHORT = { "job-step-duration": { step: 6 }, "job-processed-records": {} };
export const seriesCount = (o) => Object.values(o)
  .reduce((t, e) => t + Object.values(e).reduce((c, k) => c * k, 1), 0);

// the twelve questions genuinely asked in operations: [id, text, resource class, metric, required labels]
export const QUESTIONS = [
  ["Q1", "which service slowed down", "long", "request-duration", ["service"]],
  ["Q2", "what is the tail bucket of latency", "long", "request-duration", ["service", "bucket"]],
  ["Q3", "which environment saw a rise in errors", "long", "request-count", ["environment", "result"]],
  ["Q4", "which version broke it", "long", "request-count", ["version"]],
  ["Q5", "how many readings did the nightly job process", "short", "job-processed-records", ["job"]],
  ["Q6", "why did the nightly job finish late yesterday", "short", "job-step-duration", ["job", "step"]],
  ["Q7", "which region's readings were not validated", "long", "processed-reading", ["region", "result"]],
  ["Q8", "which instance restarted", "long", "request-count", ["instance"]],
  ["Q9", "which validation run was slow", "short", "job-step-duration", ["job", "instance"]],
  ["Q10", "which error code rose", "long", "request-count", ["result"]],
  ["Q11", "why did the work-order queue back up", "long", "queue-depth", ["service", "queue"]],
  ["Q12", "which reading a bill was generated from", "request", "-", ["transaction-id"]],
];
export const answer = (scope, labels) =>
  QUESTIONS.filter(([, , k, , a]) => scope.includes(k) && a.every((e) => labels.includes(e)));

export const makeRng = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;
export const RECORD_BYTES = 16, SERIES_BYTES = 120;
export const mb = (b) => (b / 1024 ** 2).toFixed(1);

There are three resource classes. The long-lived class is a service instance running for hours. The short-lived class is every run of the nightly batch job; it lives half a minute on average. The request class is the single transaction itself, and no metric can carry it, because the cardinality of a transaction ID is unbounded. Q12 is therefore marked from the start: a metric pipeline cannot answer it no matter how much data it collects.

Pull and Push in the Same Window

The two models run in the same one-hour window. The difference collects in three places: what gets seen (coverage), when it gets seen (discovery delay), and whose reading it is recorded as (resource identity).

// collect.mjs — pull-based and push-based collection run in the same window and are applied to the question set.
import { INSTANCES, LONG_COUNT, METRICS_LONG, METRICS_SHORT, seriesCount, QUESTIONS, answer, makeRng, RECORD_BYTES, SERIES_BYTES, mb }
  from "./network.mjs";

const WINDOW = 3600, SCRAPE = 15, INVENTORY = 30, PUSH = 10;   // seconds
const rand = makeRng(20260901);                                // seed made visible
const lifetimes = [];                                          // 29 instances from window start, 11 restarts
for (let i = 0; i < LONG_COUNT; i++) lifetimes.push({ class: "long", start: 0, end: WINDOW });
for (let i = 0; i < 11; i++) {
  const h = lifetimes[Math.floor(rand() * LONG_COUNT)], t = 300 + Math.floor(rand() * 3000);
  h.end = t; lifetimes.push({ class: "long", start: t + 20, end: WINDOW });
}
const runs = Array.from({ length: 42 }, (_, i) => {             // nightly validation runs
  const start = 60 + i * 80 + Math.floor(rand() * 30);
  return { class: "short", start, end: start + 6 + Math.floor(rand() * 33) };
});
const all = [...lifetimes, ...runs];

const pull = (o) => {                                           // scrapes after the inventory refreshes
  const t = [], visible = Math.ceil(o.start / INVENTORY) * INVENTORY;
  for (let s = Math.ceil(visible / SCRAPE) * SCRAPE; s < o.end; s += SCRAPE) t.push(s);
  return t;
};
const push = (o) => {                                            // process sends itself; no discovery round
  const t = [];
  if (o.class === "short") {                                     // batch job sends at the end of every step
    for (let a = 1; a <= 6; a++) t.push(o.start + Math.round(((o.end - o.start) * a) / 6));
    return t;
  }
  for (let s = o.start + PUSH; s < o.end; s += PUSH) t.push(s);
  return t;
};
const covered = (f, s) => all.filter((o) => o.class === s && f(o).length >= 2).length;
const total = (s) => all.filter((o) => o.class === s).length;
const discovery = (f) => {                                       // only lifetimes that start inside the window
  const g = lifetimes.filter((o) => o.start > 0).map((o) => f(o)[0] - o.start);   // restarted instances
  return [(g.reduce((a, b) => a + b, 0) / g.length).toFixed(1), Math.max(...g)];
};

const LABELS = ["service", "environment", "version", "result", "bucket", "queue", "region", "job", "step"];
const MODELS = [["pull", pull, LONG_COUNT, covered(pull, "short")],  // identity comes from the inventory
  ["push", push, lifetimes.length, 3]];                          // identity is the process's own report, job template is fixed
console.log(`regional measurement network (fictional): ${LONG_COUNT} long-lived instances, ${Object.keys(INSTANCES).length} environments,` +
  ` ${lifetimes.length - LONG_COUNT} restarts, ${runs.length} nightly validation runs`);
console.log(`${seriesCount(METRICS_LONG)} series per instance, ${seriesCount(METRICS_SHORT)} series per run; window ${WINDOW} s\n`);
console.log("model    long-lived  short-lived  discovery avg/max  identity long  identity short");
const summary = [];
for (const [name, f, identityLong, identityShort] of MODELS) {
  const [avg, max] = discovery(f), lc = covered(f, "long"), sc = covered(f, "short");
  const measured = all.reduce((t, o) => t + f(o).length * seriesCount(o.class === "long" ? METRICS_LONG : METRICS_SHORT), 0);
  const seriesTotal = identityLong * seriesCount(METRICS_LONG) + identityShort * seriesCount(METRICS_SHORT);
  summary.push([name, lc, sc, measured, seriesTotal, identityLong === LONG_COUNT]);   // does identity survive a restart
  console.log(`${name.padEnd(7)}${String(`${lc}/${total("long")}`).padStart(12)}` +
    `${String(`${sc}/${total("short")}`).padStart(13)}${String(`${avg} / ${max} s`).padStart(19)}` +
    `${String(`${identityLong}/${lifetimes.length}`).padStart(15)}${String(`${identityShort}/${runs.length}`).padStart(16)}`);
}

console.log("\nagainst the question set (12 questions)");
console.log("model    answered  bytes/hour  per question  left unanswered");
for (const [name, lc, sc, measured, seriesTotal, stable] of summary) {
  const scope = [lc / total("long") >= 0.9 && "long", sc / total("short") >= 0.9 && "short"].filter(Boolean);
  const labels = stable ? [...LABELS, "instance"] : LABELS;      // no instance label if identity is unstable
  const y = answer(scope, labels), bytes = measured * RECORD_BYTES + seriesTotal * SERIES_BYTES;
  console.log(`${name.padEnd(7)}${String(`${y.length}/${QUESTIONS.length}`).padStart(10)}` +
    `${String(`${mb(bytes)} MB`).padStart(12)}${String(`${mb(bytes / y.length)} MB`).padStart(14)}  ` +
    QUESTIONS.filter((s) => !y.includes(s)).map(([k]) => k).join(" "));
}

const unused = Object.keys(METRICS_LONG).filter((a) => !QUESTIONS.some(([, , , o]) => o === a));
const unusedSeries = unused.reduce((t, a) => t + seriesCount({ [a]: METRICS_LONG[a] }), 0);
console.log(`\nmetrics answering no question: ${unused.join(", ")}`);
console.log(`  ${unusedSeries}/${seriesCount(METRICS_LONG)} series per instance (${(100 * unusedSeries / seriesCount(METRICS_LONG)).toFixed(1)}%),` +
  ` ${mb(LONG_COUNT * unusedSeries * (WINDOW / SCRAPE) * RECORD_BYTES)} MB/hour under pull`);
const missedRuns = runs.length - covered(pull, "short"), REPORT = 7 * 3600, START = 2 * 3600;
const wait = runs.reduce((t, o) => t + (REPORT - START - o.end), 0) / runs.length / 3600;
console.log(`runs that produce no series under pull: ${missedRuns}: the failure surfaces only in the morning report,` +
  ` ${wait.toFixed(1)} hours later on average`);

regional measurement network (fictional): 29 long-lived instances, 3 environments, 11 restarts, 42 nightly validation runs
85 series per instance, 7 series per run; window 3600 s

model    long-lived  short-lived  discovery avg/max  identity long  identity short
pull          40/40        13/42        14.6 / 27 s          29/40           13/42
push          40/40        42/42        10.0 / 10 s          40/40            3/42

against the question set (12 questions)
model    answered  bytes/hour  per question  left unanswered
pull         8/12      9.7 MB        1.2 MB  Q5 Q6 Q9 Q12
push         9/12     14.5 MB        1.6 MB  Q8 Q9 Q12

metrics answering no question: process-start-time, gc-pause, file-descriptor-count
  10/85 series per instance (11.8%), 1.1 MB/hour under pull
runs that produce no series under pull: 29: the failure surfaces only in the morning report, 4.5 hours later on average

On the long-lived side the two models are equal: all forty of the forty lifetimes are covered under both models. The split opens on the short-lived side. The pull-based collector can take two or more measurements from only thirteen of the 42 validation runs. The reason is that two delays stack on top of each other: the inventory refreshes every thirty seconds, the scrape happens every fifteen seconds, and a run’s average lifetime is under thirty seconds. A run ends before the round that would have seen it. Twenty-nine runs pass without producing a single series, and their passing is invisible too — a missing series is not an error, only an absence.

The push side has no such problem, because no one has to discover the run: a run sends its own data and sends once more at the end of every step. All forty-two of the forty-two runs are covered.

The identity columns flip the split. In pull, identity comes from the inventory; when an instance restarts, the inventory entry stays the same, so the series stays the same too — forty lifetimes collect under twenty-nine identities, and a restart shows up as a gap in the series. In push, identity is the process’s own report; every new process generates itself a new identity, so forty lifetimes create forty separate identities, and a restarted instance cannot be told apart from a brand-new one never seen before. On the short-lived side the same mechanism breaks in the opposite direction: because the batch template writes a fixed job name, forty-two runs merge under three identities and individual runs blur into each other.

The Questions the Two Models Leave Unanswered

Pull answers eight questions, push answers nine. What is left unanswered is not the same, and that is the most important line in the table for this lesson.

Under pull, Q5, Q6, and Q9 drop out: how many readings the nightly job processed, why it finished late yesterday, and which run was slow. All three depend on the short-lived resource class, and that class is out of coverage. The cost of this is in the last line: when one of the twenty-nine runs that produce no series fails, the only signal that shows it is the morning reconciliation report; the job runs between two and three at night, and the report is not checked until seven — four and a half hours on average. When a validation run at the head of the billing chain drops silently, that four and a half hours multiplies directly into the number of records that need correcting.

Under push, Q8 and Q9 drop out: which instance restarted and which validation run was slow. Both depend on the reliability of a single label — instance. The label technically exists and is written on every record; what is not reliable is the guarantee that the same instance will carry the same value at two different times. This shows that collection can close off a question without losing any data at all: the data is not missing, it is unmeasurable.

Q12 is unanswered under both models and stays that way for the rest of this lesson. Asking which reading a bill was generated from means asking for a single transaction’s path across four services; a metric pipeline cannot carry that path, because carrying it would require turning the transaction ID into a label, and that label’s cardinality would equal the transaction count.

The Metric That Answers No Question

The last block counts which portion of the collected data has no payoff. Three of the seven metrics — process start time, garbage-collection pause, and file descriptor count — appear in none of the twelve questions. That is ten of the eighty-five series per instance, twelve percent. Under the pull model that is 1.1 megabytes an hour, more than eight hundred megabytes over a thirty-day retention, and it closes not a single question in return.

These three metrics are not bad; they are a cost because they have no question of their own. If garbage-collection pause enters a question set tomorrow, the same bytes become meaningful. What justifies collecting something is not the metric itself, but the existence of a question that asks for it.

Cost per question lets the two models be compared on the same scale: 1.2 megabytes for pull, 1.6 for push. Push answers more questions but costs thirty percent more per question, because a ten-second send interval is more frequent than a fifteen-second scrape, and identity multiplication spreads the series count across forty identities. The answer to “which is better” is not in a single number; it is in reading these two columns together.

Summary

• The question set consists of twelve questions; each question is tied to a resource class, a metric, and a set of labels. The unit of measure is not the bytes collected, but the questions answered.
• Pull-based collection covers 13 of the 42 short-lived runs; because the inventory-refresh (30 s) and scrape (15 s) delays stack on top of each other, a run ends before the round that would have seen it.
• Push-based collection covers 42/42 of the short-lived class, but because the process reports its own identity, 40 lifetimes produce 40 separate identities; on the short-lived side, 42 runs merge under 3 identities.
• Pull answers 8/12 questions, push answers 9/12; what is left unanswered does not overlap — Q5, Q6, Q9 under pull, Q8, Q9 under push. Q12 (which reading a bill was generated from) is unanswered under both models.
• 10/85 series (11.8%) of the collected metrics answer no question; that is 1.1 MB an hour under pull. Cost per question is 1.2 MB under pull and 1.6 MB under push.
• The cost of an uncovered run ties to a number: the failure of one of the 29 runs that produce no series surfaces in the morning report an average of 4.5 hours later.

Next Step

This lesson treated the collected data as a box: measurement count, series count, bytes. But the series itself is a structure, and inside that structure sits a decision — which labels exist. Each of the eight or nine questions answered depended on the presence of a specific label; when the instance label dropped, two questions closed. The next lesson measures this link in the opposite direction: what question does adding a label open, how many series and bytes does it cost, and which label opens no question at all when added.