--- title: 'Metric Collection Models' source: 'https://academia.sh/en/courses/devops-observability/metric-collection-models' course: 'Observability and Operations' language: en updated: '2026-09-13T05:08:22+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds a measurement set from operational questions and counts two collection models against it' - 'Shows with numbers how short-lived jobs slip through in pull collection and how resource identity gets confused in push collection' - 'Measures which portion of the collected data answers no question at all and derives the cost per question' --- # Metric Collection Models Pull-based and push-based collection run in the same window and are applied to a set of twelve questions genuinely asked in operations: which model answers how many questions, why short-lived jobs slip through in pull, why resource identity gets confused in push, and how large the discovery delay and the cost per question are. The previous course closed at the point where the infrastructure was tied to a record, and every change passed through a review and a gate. But once past the gate, no one could see what the system did while it was running. This course's first question picks up from there and takes one step further: **who decides what gets measured?** This course's unit of measure is not data, it is the **question**. Whether a collection setup is good is measured not by how many gigabytes it accumulates, but by how many of the questions genuinely asked in operations it answers. This lesson builds that set of questions and runs two collection models — **pull-based** and **push-based** — in the same window, applying both to the set. The measurement is again on the **regional measurement network**: software that collects readings from water meters, validates the readings, turns them into bills, and opens work orders for field crews. Both the network and the measured numbers are **fictional**; no real collector is run, what gets counted is a **model** built with `node`. **ML1.** The twelve questions are fictional and written in advance, as is the resource class, metric, and labels each question requires. **ML2.** The inventory is fictional: 29 long-lived instances across three environments, 11 restarts within a one-hour window, and 42 nightly validation runs. **ML3.** The pull-based collector scrapes every 15 seconds and refreshes its target inventory every 30 seconds; the push-based sender flushes every 10 seconds in a long-lived process, and the batch job sends at the end of every step. **ML4.** A record is 16 bytes, and a series descriptor is 120 bytes. **ML5.** A resource class is counted as covered only if at least two measurements can be taken from at least 90% of its lifetimes; a single measurement does not show a change. **ML6.** Restart times and run durations are produced with our own generator, seed `20260901`. ## The Twelve Questions Asked in Operations The question set is a product of operations, not of the measurement tool: these are sentences genuinely asked during on-call, in the morning report, and during incident review. Each question is tied to three things — which **resource class** the data must come from, which **metric**, and which **labels**. ```js // network.mjs — regional measurement network model and the operational question set. Both are fictional. export const INSTANCES = { // environment -> service -> long-lived instances production: { "reading-collector": 8, validator: 6, billing: 4, "work-order": 3 }, staging: { "reading-collector": 2, validator: 2, billing: 1, "work-order": 1 }, development: { "reading-collector": 1, validator: 1, billing: 0, "work-order": 0 } }; export const LONG_COUNT = Object.values(INSTANCES) .reduce((t, o) => t + Object.values(o).reduce((a, b) => a + b, 0), 0); export const METRICS_LONG = { "request-count": { result: 5 }, "request-duration": { bucket: 12 }, "queue-depth": { queue: 3 }, "processed-reading": { region: 11, result: 5 }, "process-start-time": {}, "gc-pause": { bin: 8 }, "file-descriptor-count": {} }; export const METRICS_SHORT = { "job-step-duration": { step: 6 }, "job-processed-records": {} }; export const seriesCount = (o) => Object.values(o) .reduce((t, e) => t + Object.values(e).reduce((c, k) => c * k, 1), 0); // the twelve questions genuinely asked in operations: [id, text, resource class, metric, required labels] export const QUESTIONS = [ ["Q1", "which service slowed down", "long", "request-duration", ["service"]], ["Q2", "what is the tail bucket of latency", "long", "request-duration", ["service", "bucket"]], ["Q3", "which environment saw a rise in errors", "long", "request-count", ["environment", "result"]], ["Q4", "which version broke it", "long", "request-count", ["version"]], ["Q5", "how many readings did the nightly job process", "short", "job-processed-records", ["job"]], ["Q6", "why did the nightly job finish late yesterday", "short", "job-step-duration", ["job", "step"]], ["Q7", "which region's readings were not validated", "long", "processed-reading", ["region", "result"]], ["Q8", "which instance restarted", "long", "request-count", ["instance"]], ["Q9", "which validation run was slow", "short", "job-step-duration", ["job", "instance"]], ["Q10", "which error code rose", "long", "request-count", ["result"]], ["Q11", "why did the work-order queue back up", "long", "queue-depth", ["service", "queue"]], ["Q12", "which reading a bill was generated from", "request", "-", ["transaction-id"]], ]; export const answer = (scope, labels) => QUESTIONS.filter(([, , k, , a]) => scope.includes(k) && a.every((e) => labels.includes(e))); export const makeRng = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648; export const RECORD_BYTES = 16, SERIES_BYTES = 120; export const mb = (b) => (b / 1024 ** 2).toFixed(1); ``` There are three resource classes. The **long-lived** class is a service instance running for hours. The **short-lived** class is every run of the nightly batch job; it lives half a minute on average. The **request** class is the single transaction itself, and no metric can carry it, because the cardinality of a transaction ID is unbounded. Q12 is therefore marked from the start: a metric pipeline cannot answer it no matter how much data it collects. ## Pull and Push in the Same Window The two models run in the same one-hour window. The difference collects in three places: **what gets seen** (coverage), **when it gets seen** (discovery delay), and **whose reading it is recorded as** (resource identity). ```js // collect.mjs — pull-based and push-based collection run in the same window and are applied to the question set. import { INSTANCES, LONG_COUNT, METRICS_LONG, METRICS_SHORT, seriesCount, QUESTIONS, answer, makeRng, RECORD_BYTES, SERIES_BYTES, mb } from "./network.mjs"; const WINDOW = 3600, SCRAPE = 15, INVENTORY = 30, PUSH = 10; // seconds const rand = makeRng(20260901); // seed made visible const lifetimes = []; // 29 instances from window start, 11 restarts for (let i = 0; i < LONG_COUNT; i++) lifetimes.push({ class: "long", start: 0, end: WINDOW }); for (let i = 0; i < 11; i++) { const h = lifetimes[Math.floor(rand() * LONG_COUNT)], t = 300 + Math.floor(rand() * 3000); h.end = t; lifetimes.push({ class: "long", start: t + 20, end: WINDOW }); } const runs = Array.from({ length: 42 }, (_, i) => { // nightly validation runs const start = 60 + i * 80 + Math.floor(rand() * 30); return { class: "short", start, end: start + 6 + Math.floor(rand() * 33) }; }); const all = [...lifetimes, ...runs]; const pull = (o) => { // scrapes after the inventory refreshes const t = [], visible = Math.ceil(o.start / INVENTORY) * INVENTORY; for (let s = Math.ceil(visible / SCRAPE) * SCRAPE; s < o.end; s += SCRAPE) t.push(s); return t; }; const push = (o) => { // process sends itself; no discovery round const t = []; if (o.class === "short") { // batch job sends at the end of every step for (let a = 1; a <= 6; a++) t.push(o.start + Math.round(((o.end - o.start) * a) / 6)); return t; } for (let s = o.start + PUSH; s < o.end; s += PUSH) t.push(s); return t; }; const covered = (f, s) => all.filter((o) => o.class === s && f(o).length >= 2).length; const total = (s) => all.filter((o) => o.class === s).length; const discovery = (f) => { // only lifetimes that start inside the window const g = lifetimes.filter((o) => o.start > 0).map((o) => f(o)[0] - o.start); // restarted instances return [(g.reduce((a, b) => a + b, 0) / g.length).toFixed(1), Math.max(...g)]; }; const LABELS = ["service", "environment", "version", "result", "bucket", "queue", "region", "job", "step"]; const MODELS = [["pull", pull, LONG_COUNT, covered(pull, "short")], // identity comes from the inventory ["push", push, lifetimes.length, 3]]; // identity is the process's own report, job template is fixed console.log(`regional measurement network (fictional): ${LONG_COUNT} long-lived instances, ${Object.keys(INSTANCES).length} environments,` + ` ${lifetimes.length - LONG_COUNT} restarts, ${runs.length} nightly validation runs`); console.log(`${seriesCount(METRICS_LONG)} series per instance, ${seriesCount(METRICS_SHORT)} series per run; window ${WINDOW} s\n`); console.log("model long-lived short-lived discovery avg/max identity long identity short"); const summary = []; for (const [name, f, identityLong, identityShort] of MODELS) { const [avg, max] = discovery(f), lc = covered(f, "long"), sc = covered(f, "short"); const measured = all.reduce((t, o) => t + f(o).length * seriesCount(o.class === "long" ? METRICS_LONG : METRICS_SHORT), 0); const seriesTotal = identityLong * seriesCount(METRICS_LONG) + identityShort * seriesCount(METRICS_SHORT); summary.push([name, lc, sc, measured, seriesTotal, identityLong === LONG_COUNT]); // does identity survive a restart console.log(`${name.padEnd(7)}${String(`${lc}/${total("long")}`).padStart(12)}` + `${String(`${sc}/${total("short")}`).padStart(13)}${String(`${avg} / ${max} s`).padStart(19)}` + `${String(`${identityLong}/${lifetimes.length}`).padStart(15)}${String(`${identityShort}/${runs.length}`).padStart(16)}`); } console.log("\nagainst the question set (12 questions)"); console.log("model answered bytes/hour per question left unanswered"); for (const [name, lc, sc, measured, seriesTotal, stable] of summary) { const scope = [lc / total("long") >= 0.9 && "long", sc / total("short") >= 0.9 && "short"].filter(Boolean); const labels = stable ? [...LABELS, "instance"] : LABELS; // no instance label if identity is unstable const y = answer(scope, labels), bytes = measured * RECORD_BYTES + seriesTotal * SERIES_BYTES; console.log(`${name.padEnd(7)}${String(`${y.length}/${QUESTIONS.length}`).padStart(10)}` + `${String(`${mb(bytes)} MB`).padStart(12)}${String(`${mb(bytes / y.length)} MB`).padStart(14)} ` + QUESTIONS.filter((s) => !y.includes(s)).map(([k]) => k).join(" ")); } const unused = Object.keys(METRICS_LONG).filter((a) => !QUESTIONS.some(([, , , o]) => o === a)); const unusedSeries = unused.reduce((t, a) => t + seriesCount({ [a]: METRICS_LONG[a] }), 0); console.log(`\nmetrics answering no question: ${unused.join(", ")}`); console.log(` ${unusedSeries}/${seriesCount(METRICS_LONG)} series per instance (${(100 * unusedSeries / seriesCount(METRICS_LONG)).toFixed(1)}%),` + ` ${mb(LONG_COUNT * unusedSeries * (WINDOW / SCRAPE) * RECORD_BYTES)} MB/hour under pull`); const missedRuns = runs.length - covered(pull, "short"), REPORT = 7 * 3600, START = 2 * 3600; const wait = runs.reduce((t, o) => t + (REPORT - START - o.end), 0) / runs.length / 3600; console.log(`runs that produce no series under pull: ${missedRuns}: the failure surfaces only in the morning report,` + ` ${wait.toFixed(1)} hours later on average`); ``` ``` regional measurement network (fictional): 29 long-lived instances, 3 environments, 11 restarts, 42 nightly validation runs 85 series per instance, 7 series per run; window 3600 s model long-lived short-lived discovery avg/max identity long identity short pull 40/40 13/42 14.6 / 27 s 29/40 13/42 push 40/40 42/42 10.0 / 10 s 40/40 3/42 against the question set (12 questions) model answered bytes/hour per question left unanswered pull 8/12 9.7 MB 1.2 MB Q5 Q6 Q9 Q12 push 9/12 14.5 MB 1.6 MB Q8 Q9 Q12 metrics answering no question: process-start-time, gc-pause, file-descriptor-count 10/85 series per instance (11.8%), 1.1 MB/hour under pull runs that produce no series under pull: 29: the failure surfaces only in the morning report, 4.5 hours later on average ``` On the long-lived side the two models are equal: all forty of the forty lifetimes are covered under both models. The split opens on the short-lived side. The pull-based collector can take two or more measurements from only **thirteen** of the 42 validation runs. The reason is that two delays stack on top of each other: the inventory refreshes every thirty seconds, the scrape happens every fifteen seconds, and a run's average lifetime is under thirty seconds. A run ends before the round that would have seen it. Twenty-nine runs pass without producing a single series, and **their passing is invisible too** — a missing series is not an error, only an absence. The push side has no such problem, because no one has to discover the run: a run sends its own data and sends once more at the end of every step. All forty-two of the forty-two runs are covered. The identity columns flip the split. In pull, identity comes from **the inventory**; when an instance restarts, the inventory entry stays the same, so the series stays the same too — forty lifetimes collect under twenty-nine identities, and a restart shows up as a gap in the series. In push, identity is **the process's own report**; every new process generates itself a new identity, so forty lifetimes create forty separate identities, and a restarted instance cannot be told apart from a brand-new one never seen before. On the short-lived side the same mechanism breaks in the opposite direction: because the batch template writes a fixed job name, forty-two runs merge under **three identities** and individual runs blur into each other. ## The Questions the Two Models Leave Unanswered Pull answers eight questions, push answers nine. What is left unanswered is not the same, and that is the most important line in the table for this lesson. Under pull, Q5, Q6, and Q9 drop out: how many readings the nightly job processed, why it finished late yesterday, and which run was slow. All three depend on the short-lived resource class, and that class is out of coverage. The cost of this is in the last line: when one of the twenty-nine runs that produce no series fails, the only signal that shows it is the morning reconciliation report; the job runs between two and three at night, and the report is not checked until seven — **four and a half hours on average**. When a validation run at the head of the billing chain drops silently, that four and a half hours multiplies directly into the number of records that need correcting. Under push, Q8 and Q9 drop out: which instance restarted and which validation run was slow. Both depend on the reliability of a single label — `instance`. The label technically exists and is written on every record; what is not reliable is the guarantee that the same instance will carry the same value at two different times. This shows that collection can close off a question **without losing any data at all**: the data is not missing, it is unmeasurable. Q12 is unanswered under both models and stays that way for the rest of this lesson. Asking which reading a bill was generated from means asking for a single transaction's path across four services; a metric pipeline cannot carry that path, because carrying it would require turning the transaction ID into a label, and that label's cardinality would equal the transaction count. ## The Metric That Answers No Question The last block counts which portion of the collected data has no payoff. Three of the seven metrics — process start time, garbage-collection pause, and file descriptor count — appear in none of the twelve questions. That is **ten** of the eighty-five series per instance, twelve percent. Under the pull model that is 1.1 megabytes an hour, more than eight hundred megabytes over a thirty-day retention, and it closes not a single question in return. These three metrics are not bad; they are a cost because they have no question of their own. If garbage-collection pause enters a question set tomorrow, the same bytes become meaningful. What justifies collecting something is not the metric itself, but the existence of a question that asks for it. Cost per question lets the two models be compared on the same scale: 1.2 megabytes for pull, 1.6 for push. Push answers more questions but costs thirty percent more per question, because a ten-second send interval is more frequent than a fifteen-second scrape, and identity multiplication spreads the series count across forty identities. The answer to "which is better" is not in a single number; it is in reading these two columns together. ## Summary - The question set consists of twelve questions; each question is tied to a resource class, a metric, and a set of labels. The unit of measure is not the bytes collected, but the questions answered. - Pull-based collection covers 13 of the 42 short-lived runs; because the inventory-refresh (30 s) and scrape (15 s) delays stack on top of each other, a run ends before the round that would have seen it. - Push-based collection covers 42/42 of the short-lived class, but because the process reports its own identity, 40 lifetimes produce 40 separate identities; on the short-lived side, 42 runs merge under 3 identities. - Pull answers 8/12 questions, push answers 9/12; what is left unanswered does not overlap — Q5, Q6, Q9 under pull, Q8, Q9 under push. Q12 (which reading a bill was generated from) is unanswered under both models. - 10/85 series (11.8%) of the collected metrics answer no question; that is 1.1 MB an hour under pull. Cost per question is 1.2 MB under pull and 1.6 MB under push. - The cost of an uncovered run ties to a number: the failure of one of the 29 runs that produce no series surfaces in the morning report an average of 4.5 hours later. ## Next Step This lesson treated the collected data as a box: measurement count, series count, bytes. But the series itself is a structure, and inside that structure sits a decision — which labels exist. Each of the eight or nine questions answered depended on the presence of a specific label; when the `instance` label dropped, two questions closed. The next lesson measures this link in the opposite direction: what question does **adding** a label open, how many series and bytes does it cost, and which label opens no question at all when added.