--- title: 'Cluster Observability' source: 'https://academia.sh/en/courses/orchestration/cluster-observability' course: 'Container Orchestration' language: en updated: '2026-09-13T05:08:27+00:00' license: 'CC BY-SA 4.0' objectives: - "Ties series count to a single formula: resource count times the product of each metric's label cardinalities" - 'Shows with numbers why adding one label grows series multiplicatively, not additively' - "Measures sampling and aggregation's two faces together: bytes won and the question that can no longer be answered" --- # Cluster Observability The cost of collecting logs, metrics, and traces at cluster scale is counted: how many resources, how many series per resource, how many records and how many bytes once multiplied by a retention period. Label cardinality's multiplicative growth in series count is shown; the bytes won and the detail lost are measured together once sampling and aggregation are applied. Every number in the previous lesson came out of a single pipeline's eighty-round trace, and what kept that trace was measurement: three items per round, two hundred forty reads in total. A cluster holds not one but dozens of resources, and every one of their controllers does the same reading. Measurement itself becomes a load. This lesson does not argue over what a metric is — counter, gauge, and histogram were established in M16/K07 and appear here only by name. The only thing measured is **the cost of collection at cluster scale**: how many resources, how many series per resource, how many records and how many bytes once multiplied by a retention period. The delegated decision follows from this too: **which question stays answerable later** gets decided at the moment of collection, and that decision cannot be undone, because detail that is not collected sits nowhere. The measurement is again over the **regional measurement network** and is **fictional**; no real cluster is run, and what is counted is a **model** built with `node`. - **OO49.** The cluster inventory is fictional: 34 pods and 9 nodes belonging to five applications. - **OO50.** Metric types were established in M16/K07; they are referred to here only by name. - **OO51.** A record is 16 bytes (timestamp 8, value 8), a series' identity is 120 bytes; the sampling interval is 15 seconds, retention is 30 days. - **OO52.** Series count is the sum, over resource counts, of the product of each metric's label cardinalities. - **OO53.** The latency signal is produced with our own generator, seed `20260801`; the twelve spikes are **hand-written**. - **OO54.** Twelve operating questions and the label each one requires are hand-written. ## Series Arithmetic at Cluster Scale ```js // cluster.mjs — cluster inventory, metric definitions, and series / record / byte arithmetic. export const RESOURCE = { pod: 34, node: 9 }; // five applications, 34 pods, 9 nodes export const SAMPLE_INTERVAL_SEC = 15, RETENTION_DAYS = 30; export const RECORD_BYTES = 16, SERIES_BYTES = 120; // timestamp 8 + value 8; series identity 120 export const METRIC = { // name: [type, { label: cardinality }] pod: [ ["request-count", "counter", { path: 12, method: 3, result: 4 }], ["request-duration", "histogram", { path: 12, bucket: 12 }], ["queue-length", "gauge", {}], ["processed-readings", "counter", { pipeline: 4 }], ["memory-usage", "gauge", {}], ], node: [ ["node-cpu", "gauge", { core: 8 }], ["node-memory", "gauge", {}], ["disk-io", "counter", { device: 3, direction: 2 }], ], }; export const seriesCount = (metric, resource = RESOURCE) => Object.entries(metric).reduce((t, [type, list]) => t + resource[type] * list.reduce((s, [, , e]) => s + Object.values(e).reduce((c, k) => c * k, 1), 0), 0); export const dropLabel = (metric, name) => Object.fromEntries(Object.entries(metric).map(([t, l]) => [t, l.map(([a, type, e]) => [a, type, Object.fromEntries(Object.entries(e).filter(([k]) => k !== name))])])); export const records = (series, interval = SAMPLE_INTERVAL_SEC, days = RETENTION_DAYS) => series * Math.floor(86400 / interval) * days; export const bytes = (series, interval = SAMPLE_INTERVAL_SEC, days = RETENTION_DAYS) => records(series, interval, days) * RECORD_BYTES + series * SERIES_BYTES; export const gb = (b) => (b / 1024 ** 3).toFixed(1); // adding a label multiplies the affected metric's series count export const addLabel = (metric, target, name, cardinality) => Object.fromEntries(Object.entries(metric).map(([t, l]) => [t, l.map(([a, type, e]) => target.includes(a) ? [a, type, { ...e, [name]: cardinality }] : [a, type, e])])); ``` ```js // run15.mjs — cost of collection at cluster scale, and the multiplicative growth from label cardinality. import { RESOURCE, METRIC, SAMPLE_INTERVAL_SEC, RETENTION_DAYS, seriesCount, records, bytes, gb, addLabel } from "./cluster.mjs"; const series = seriesCount(METRIC); const product = (l) => l.reduce((s, [, , e]) => s + Object.values(e).reduce((c, k) => c * k, 1), 0); console.log(`resources ${RESOURCE.pod + RESOURCE.node} (pods ${RESOURCE.pod}, nodes ${RESOURCE.node})` + ` series per pod ${product(METRIC.pod)} series per node ${product(METRIC.node)} total series ${series}`); console.log(`sampling ${SAMPLE_INTERVAL_SEC} sec, retention ${RETENTION_DAYS} days` + ` -> records ${records(series).toLocaleString("en-US")} bytes ${gb(bytes(series))} GB`); console.log("\nas labels get added (only to the named metrics)"); console.log("label added total series factor records GB"); const show = (name, oo) => { const s = seriesCount(oo); console.log(`${name.padEnd(43)} ${String(s).padStart(9)} ${(s / series).toFixed(1).padStart(6)}` + ` ${records(s).toLocaleString("en-US").padStart(13)} ${gb(bytes(s)).padStart(7)}`); return oo; }; let o = show("baseline", METRIC); o = show("+ region (7) to request metrics", addLabel(o, ["request-count", "request-duration"], "region", 7)); o = show("+ counter-type (23) to processed-readings", addLabel(o, ["processed-readings"], "counter-type", 23)); o = show("+ version (4) to every pod metric", addLabel(o, METRIC.pod.map(([a]) => a), "version", 4)); show("+ request-id (5000) to duration", addLabel(o, ["request-duration"], "request-id", 5000)); const days = RETENTION_DAYS * 86400; // thirty-day cost of three signals on the same cluster const SIGNAL = [["metric (15 sec sampling)", records(series) * 16], ["log (12 lines/sec per pod, 180 bytes)", RESOURCE.pod * 12 * 180 * days], ["trace (220 req/sec, 6 spans, 260 bytes)", 220 * 6 * 260 * days], ["trace (with one percent sampling)", (220 * 6 * 260 * days) / 100]]; console.log("\nthirty-day collection cost for three signals"); for (const [name, b] of SIGNAL) console.log(` ${name.padEnd(44)} ${gb(b).padStart(9)} GB`); ``` ``` resources 43 (pods 34, nodes 9) series per pod 294 series per node 15 total series 10131 sampling 15 sec, retention 30 days -> records 1,750,636,800 bytes 26.1 GB as labels get added (only to the named metrics) label added total series factor records GB baseline 10131 1.0 1,750,636,800 26.1 + region (7) to request metrics 68883 6.8 11,902,982,400 177.4 + counter-type (23) to processed-readings 71875 7.1 12,420,000,000 185.1 + version (4) to every pod metric 287095 28.3 49,610,016,000 739.3 + request-id (5000) to duration 685590007 67672.5 118,469,953,209,600 1765416.5 thirty-day collection cost for three signals metric (15 sec sampling) 26.1 GB log (12 lines/sec per pod, 180 bytes) 177.3 GB trace (220 req/sec, 6 spans, 260 bytes) 828.5 GB trace (with one percent sampling) 8.3 GB ``` Forty-three resources, two hundred ninety-four series per pod, fifteen series per node — **ten thousand one hundred thirty-one series** in total. Under thirty-day retention, that is one billion seven hundred fifty million records and twenty-six gigabytes. This is the baseline state with no label added, and it grows linearly as long as the resource count stays at thirty-four: double the pod count, and series doubles too. Adding a label is not linear. When the **region** label's seven values are added to two metrics, series climbs from ten thousand to sixty-nine thousand — the resource count has not changed, the added information is seven distinct values, but the result is a 6.8x multiplier. The reason sits in one line: a metric's series count is the **product** of its labels' cardinalities. The `request-count` metric already carried twelve paths, three methods, and four results; adding seven regions takes it from a hundred forty-four series to a thousand and eight. The last two rows show the limit. When the version label gets added to every pod metric, the total climbs to a 28.3x multiplier, seven hundred thirty-nine gigabytes. When request id — five thousand distinct values — gets added to a single metric, series climbs to six hundred eighty-five million, and storage reaches petabyte territory. This row is not a recommendation, it is a **warning sign**: use a field with unbounded cardinality as a label, and a metric store turns into a record store. Comparing the three signals also shows where the cost sits. On the same cluster, metrics are twenty-six gigabytes, logs a hundred seventy-seven, traces eight hundred twenty-eight. Traces are the most expensive, because they carry each individual request; with one percent sampling they drop to eight gigabytes, and at that point ninety-nine of every hundred requests lose their trail. ## Sampling and Aggregation ```js // sample.mjs — sampling and aggregation: lost detail counted together with bytes won. import { RESOURCE, METRIC, seriesCount, dropLabel, bytes, gb } from "./cluster.mjs"; const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648; const rand = generator(20260801); // seed visible const SPIKE = [180, 420, 655, 900, 1210, 1480, 1755, 2010, 2290, 2640, 2980, 3310]; const latency = Array.from({ length: 3600 }, (_, i) => 118 + Math.round(rand() * 8) + (SPIKE.some((a) => i >= a && i < a + 20) ? 780 : 0)); console.log(`one hour of latency signal: 3600 seconds, ${SPIKE.length} spikes (each 20 sec, +780 ms)`); console.log("interval samples spikes caught seen peak series records/day total GB"); const series = seriesCount(METRIC); for (const k of [1, 15, 60, 300]) { const idx = [...latency.keys()].filter((i) => i % k === 0); const caught = SPIKE.filter((a) => idx.some((i) => i >= a && i < a + 20 && latency[i] > 500)).length; const peak = Math.max(...idx.map((i) => latency[i])); console.log(`${String(k).padStart(4)} sec ${String(idx.length).padStart(7)}` + `${String(`${caught}/${SPIKE.length}`).padStart(14)}${String(`${peak} ms`).padStart(15)}` + `${String(Math.floor(86400 / k)).padStart(24)}${gb(bytes(series, k)).padStart(12)}`); } const QUESTION = [["which path got slow", ["path"]], ["which method is erroring", ["method"]], ["which result code increased", ["result"]], ["latency's upper percentile", ["bucket"]], ["which pipeline fell behind", ["pipeline"]], ["which pod is consuming memory", ["resource"]], ["which core filled up", ["core"]], ["total request count", []], ["which application is slow", []], ["which pod restarted", ["resource"]], ["which result on which path", ["path", "result"]], ["disk direction: read or write", ["direction"]]]; console.log("\naggregation series GB questions answered detail lost"); const level = [["none", [], RESOURCE], ["bucket dropped", ["bucket"], RESOURCE], ["+ path dropped", ["bucket", "path"], RESOURCE], ["+ pods aggregated into apps", ["bucket", "path", "resource"], { pod: 5, node: 9 }]]; for (const [name, dropped, resource] of level) { const o = dropped.filter((d) => d !== "resource").reduce((x, d) => dropLabel(x, d), METRIC); const s = seriesCount(o, resource); const answered = QUESTION.filter((q) => q[1].every((e) => !dropped.includes(e))).length; console.log(`${name.padEnd(30)} ${String(s).padStart(5)} ${gb(bytes(s)).padStart(5)}` + `${String(`${answered}/${QUESTION.length}`).padStart(15)} ${dropped.length ? dropped.join(", ") : "-"}`); } ``` ``` one hour of latency signal: 3600 seconds, 12 spikes (each 20 sec, +780 ms) interval samples spikes caught seen peak series records/day total GB 1 sec 3600 12/12 906 ms 86400 391.3 15 sec 240 12/12 905 ms 5760 26.1 60 sec 60 5/12 903 ms 1440 6.5 300 sec 12 1/12 903 ms 288 1.3 aggregation series GB questions answered detail lost none 10131 26.1 12/12 - bucket dropped 5643 14.5 11/12 bucket + path dropped 781 2.0 9/12 bucket, path + pods aggregated into apps 230 0.6 7/12 bucket, path, resource ``` The sampling table lines up two columns side by side. A fifteen-second interval takes up one-fifteenth the bytes of second-by-second collection — twenty-six gigabytes instead of three hundred ninety-one — and **catches all twelve of the twelve spikes**, because a spike lasts twenty seconds and the fifteen-second grid lands at least once in every twenty-second window. Up to this point, sampling looks free. At sixty seconds, bytes drop to a quarter, six and a half gigabytes, and the spikes caught fall from twelve to **five**. At three hundred seconds, bytes are one point three gigabytes, and only one of the twelve spikes shows up at all. Where the loss comes from is clear: once the sampling interval exceeds the event's duration, whether the event becomes visible is left to chance. The peak seen sits around nine hundred milliseconds across all three intervals — meaning **a caught spike is measured correctly**, and what gets lost is not magnitude but how many spikes never show up at all. The metric's average stays correct, and stays misleading. The aggregation table builds the same duality on the label side. When the histogram bucket is dropped, series falls from ten thousand to five thousand six hundred forty-three, and eleven of twelve questions stay answerable — the one lost question is latency's upper percentile. When the path label is also dropped, series falls to seven hundred eighty-one, bytes to two gigabytes, and two more questions close. When pods get aggregated into applications, series falls to two hundred thirty, bytes to six hundred megabytes; which pod is consuming memory, or which one restarted, can no longer be asked. The three steps' payoff is not equal. The first step buys back 11.6 gigabytes and costs one question; the second buys back 12.5 gigabytes and costs two; the third buys back only 1.4 gigabytes and again costs two. Gain per question falls from 11.6 to 6.3 to 0.7 gigabytes. What is cheap is dropping **the product's large factor** — the twelve-bucket histogram and the twelve-path distinction. Once those two factors are gone, the remaining series are already few, and every further aggregation closes a question for close to free bytes. ## Delegation's Balance Sheet | Measure | Value | |---|---| | resources | 43 (34 pods, 9 nodes) | | series per pod / per node | 294 / 15 | | total series, records, bytes (15 sec, 30 days) | 10,131 / 1.75 billion / 26.1 GB | | one label (7 values) added to two metrics | series 6.8x, bytes 177.4 GB | | unbounded-cardinality label (5000 values) | series 67,672x | | metric / log / trace (30 days) | 26.1 / 177.3 / 828.5 GB | | sampling 15 → 60 sec | bytes 26.1 → 6.5 GB, spikes caught 12/12 → 5/12 | | aggregation, three steps | bytes 26.1 → 0.6 GB, questions answered 12/12 → 7/12 | The delegated decision is this: **which question stays answerable later gets decided before the question is asked.** The information items this needs are countable too — which metric, which labels, which interval, how much retention. Information's cost is the product of four numbers, and changing a single item in that product multiplies the result many times over. What the system does not know also does something, and this time it is silent. When a sixty-second sample interval is set up, no warning appears; because seven spikes never show up, **their not showing up also does not show up.** When the pod label is dropped, a query produces no error, it returns an application-level number instead. Both of these losses get noticed months after the decision was delegated, and only once that question is genuinely asked; at that point there is no way to reach back and collect the data that was never gathered. ## Summary - 43 resources, 294 series per pod and 15 per node; 10,131 series in total, 1.75 billion records and 26.1 GB over 30 days. - A metric's series count is the product of its label cardinalities: adding a 7-value label to two metrics grows the total 6.8x, adding a 4-value label to every pod metric grows it 28.3x. - Turning a 5000-value field into a label multiplies series by 67,672x; unbounded cardinality turns a metric store into a record store. - On the same cluster, the 30-day cost is 26.1 GB for metrics, 177.3 for logs, 828.5 for traces; one-percent sampling cuts traces to 8.3 GB and erases the trail of 99 of every 100 requests. - Moving sampling from 15 to 60 seconds drops bytes from 26.1 to 6.5 GB and spikes caught from 12/12 to 5/12; three aggregation steps drop bytes to 0.6 GB while dropping questions answered from 12/12 to 7/12. ## Next Step Every measurement up to this point was over **a single cluster**: one control plane, one resource inventory, one series store. The regional measurement network's production runs across two regions, and each region has its own cluster — the same chart, the same resource type, the same controller, separate control planes. The next lesson counts what this multiplication costs: how many clusters the same desired state reaches by how many separate routes, how drift between clusters gets measured, and, when a decision has to be made not for a single cluster but for a set of clusters, which information item comes from whom.