Cluster Observability

Every number in the previous lesson came out of a single pipeline’s eighty-round trace, and what kept that trace was measurement: three items per round, two hundred forty reads in total. A cluster holds not one but dozens of resources, and every one of their controllers does the same reading. Measurement itself becomes a load.

This lesson does not argue over what a metric is — counter, gauge, and histogram were established in M16/K07 and appear here only by name. The only thing measured is the cost of collection at cluster scale: how many resources, how many series per resource, how many records and how many bytes once multiplied by a retention period. The delegated decision follows from this too: which question stays answerable later gets decided at the moment of collection, and that decision cannot be undone, because detail that is not collected sits nowhere.

The measurement is again over the regional measurement network and is fictional; no real cluster is run, and what is counted is a model built with node.

• OO49. The cluster inventory is fictional: 34 pods and 9 nodes belonging to five applications.
• OO50. Metric types were established in M16/K07; they are referred to here only by name.
• OO51. A record is 16 bytes (timestamp 8, value 8), a series’ identity is 120 bytes; the sampling interval is 15 seconds, retention is 30 days.
• OO52. Series count is the sum, over resource counts, of the product of each metric’s label cardinalities.
• OO53. The latency signal is produced with our own generator, seed 20260801; the twelve spikes are hand-written.
• OO54. Twelve operating questions and the label each one requires are hand-written.

Series Arithmetic at Cluster Scale

// cluster.mjs — cluster inventory, metric definitions, and series / record / byte arithmetic.
export const RESOURCE = { pod: 34, node: 9 };        // five applications, 34 pods, 9 nodes
export const SAMPLE_INTERVAL_SEC = 15, RETENTION_DAYS = 30;
export const RECORD_BYTES = 16, SERIES_BYTES = 120;  // timestamp 8 + value 8; series identity 120

export const METRIC = {                              // name: [type, { label: cardinality }]
  pod: [
    ["request-count", "counter", { path: 12, method: 3, result: 4 }],
    ["request-duration", "histogram", { path: 12, bucket: 12 }],
    ["queue-length", "gauge", {}],
    ["processed-readings", "counter", { pipeline: 4 }],
    ["memory-usage", "gauge", {}],
  ],
  node: [
    ["node-cpu", "gauge", { core: 8 }],
    ["node-memory", "gauge", {}],
    ["disk-io", "counter", { device: 3, direction: 2 }],
  ],
};

export const seriesCount = (metric, resource = RESOURCE) => Object.entries(metric).reduce((t, [type, list]) =>
  t + resource[type] * list.reduce((s, [, , e]) =>
    s + Object.values(e).reduce((c, k) => c * k, 1), 0), 0);
export const dropLabel = (metric, name) => Object.fromEntries(Object.entries(metric).map(([t, l]) =>
  [t, l.map(([a, type, e]) => [a, type, Object.fromEntries(Object.entries(e).filter(([k]) => k !== name))])]));

export const records = (series, interval = SAMPLE_INTERVAL_SEC, days = RETENTION_DAYS) =>
  series * Math.floor(86400 / interval) * days;
export const bytes = (series, interval = SAMPLE_INTERVAL_SEC, days = RETENTION_DAYS) =>
  records(series, interval, days) * RECORD_BYTES + series * SERIES_BYTES;
export const gb = (b) => (b / 1024 ** 3).toFixed(1);

// adding a label multiplies the affected metric's series count
export const addLabel = (metric, target, name, cardinality) =>
  Object.fromEntries(Object.entries(metric).map(([t, l]) => [t, l.map(([a, type, e]) =>
    target.includes(a) ? [a, type, { ...e, [name]: cardinality }] : [a, type, e])]));

// run15.mjs — cost of collection at cluster scale, and the multiplicative growth from label cardinality.
import { RESOURCE, METRIC, SAMPLE_INTERVAL_SEC, RETENTION_DAYS, seriesCount, records, bytes, gb, addLabel } from "./cluster.mjs";

const series = seriesCount(METRIC);
const product = (l) => l.reduce((s, [, , e]) => s + Object.values(e).reduce((c, k) => c * k, 1), 0);
console.log(`resources ${RESOURCE.pod + RESOURCE.node} (pods ${RESOURCE.pod}, nodes ${RESOURCE.node})` +
  `   series per pod ${product(METRIC.pod)}   series per node ${product(METRIC.node)}   total series ${series}`);
console.log(`sampling ${SAMPLE_INTERVAL_SEC} sec, retention ${RETENTION_DAYS} days` +
  ` -> records ${records(series).toLocaleString("en-US")}   bytes ${gb(bytes(series))} GB`);

console.log("\nas labels get added (only to the named metrics)");
console.log("label added                                 total series  factor   records        GB");
const show = (name, oo) => {
  const s = seriesCount(oo);
  console.log(`${name.padEnd(43)} ${String(s).padStart(9)}   ${(s / series).toFixed(1).padStart(6)}` +
    `   ${records(s).toLocaleString("en-US").padStart(13)}  ${gb(bytes(s)).padStart(7)}`);
  return oo;
};
let o = show("baseline", METRIC);
o = show("+ region (7) to request metrics", addLabel(o, ["request-count", "request-duration"], "region", 7));
o = show("+ counter-type (23) to processed-readings", addLabel(o, ["processed-readings"], "counter-type", 23));
o = show("+ version (4) to every pod metric", addLabel(o, METRIC.pod.map(([a]) => a), "version", 4));
show("+ request-id (5000) to duration", addLabel(o, ["request-duration"], "request-id", 5000));

const days = RETENTION_DAYS * 86400;                 // thirty-day cost of three signals on the same cluster
const SIGNAL = [["metric  (15 sec sampling)", records(series) * 16],
  ["log     (12 lines/sec per pod, 180 bytes)", RESOURCE.pod * 12 * 180 * days],
  ["trace   (220 req/sec, 6 spans, 260 bytes)", 220 * 6 * 260 * days],
  ["trace   (with one percent sampling)", (220 * 6 * 260 * days) / 100]];
console.log("\nthirty-day collection cost for three signals");
for (const [name, b] of SIGNAL) console.log(`  ${name.padEnd(44)} ${gb(b).padStart(9)} GB`);

resources 43 (pods 34, nodes 9)   series per pod 294   series per node 15   total series 10131
sampling 15 sec, retention 30 days -> records 1,750,636,800   bytes 26.1 GB

as labels get added (only to the named metrics)
label added                                 total series  factor   records        GB
baseline                                        10131      1.0   1,750,636,800     26.1
+ region (7) to request metrics                 68883      6.8   11,902,982,400    177.4
+ counter-type (23) to processed-readings       71875      7.1   12,420,000,000    185.1
+ version (4) to every pod metric              287095     28.3   49,610,016,000    739.3
+ request-id (5000) to duration             685590007   67672.5   118,469,953,209,600  1765416.5

thirty-day collection cost for three signals
  metric  (15 sec sampling)                         26.1 GB
  log     (12 lines/sec per pod, 180 bytes)        177.3 GB
  trace   (220 req/sec, 6 spans, 260 bytes)        828.5 GB
  trace   (with one percent sampling)                8.3 GB

Forty-three resources, two hundred ninety-four series per pod, fifteen series per node — ten thousand one hundred thirty-one series in total. Under thirty-day retention, that is one billion seven hundred fifty million records and twenty-six gigabytes. This is the baseline state with no label added, and it grows linearly as long as the resource count stays at thirty-four: double the pod count, and series doubles too.

Adding a label is not linear. When the region label’s seven values are added to two metrics, series climbs from ten thousand to sixty-nine thousand — the resource count has not changed, the added information is seven distinct values, but the result is a 6.8x multiplier. The reason sits in one line: a metric’s series count is the product of its labels’ cardinalities. The request-count metric already carried twelve paths, three methods, and four results; adding seven regions takes it from a hundred forty-four series to a thousand and eight.

The last two rows show the limit. When the version label gets added to every pod metric, the total climbs to a 28.3x multiplier, seven hundred thirty-nine gigabytes. When request id — five thousand distinct values — gets added to a single metric, series climbs to six hundred eighty-five million, and storage reaches petabyte territory. This row is not a recommendation, it is a warning sign: use a field with unbounded cardinality as a label, and a metric store turns into a record store.

Comparing the three signals also shows where the cost sits. On the same cluster, metrics are twenty-six gigabytes, logs a hundred seventy-seven, traces eight hundred twenty-eight. Traces are the most expensive, because they carry each individual request; with one percent sampling they drop to eight gigabytes, and at that point ninety-nine of every hundred requests lose their trail.

Sampling and Aggregation

// sample.mjs — sampling and aggregation: lost detail counted together with bytes won.
import { RESOURCE, METRIC, seriesCount, dropLabel, bytes, gb } from "./cluster.mjs";

const generator = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648;
const rand = generator(20260801);                    // seed visible
const SPIKE = [180, 420, 655, 900, 1210, 1480, 1755, 2010, 2290, 2640, 2980, 3310];
const latency = Array.from({ length: 3600 }, (_, i) => 118 + Math.round(rand() * 8) +
  (SPIKE.some((a) => i >= a && i < a + 20) ? 780 : 0));

console.log(`one hour of latency signal: 3600 seconds, ${SPIKE.length} spikes (each 20 sec, +780 ms)`);
console.log("interval  samples  spikes caught  seen peak   series records/day   total GB");
const series = seriesCount(METRIC);
for (const k of [1, 15, 60, 300]) {
  const idx = [...latency.keys()].filter((i) => i % k === 0);
  const caught = SPIKE.filter((a) => idx.some((i) => i >= a && i < a + 20 && latency[i] > 500)).length;
  const peak = Math.max(...idx.map((i) => latency[i]));
  console.log(`${String(k).padStart(4)} sec ${String(idx.length).padStart(7)}` +
    `${String(`${caught}/${SPIKE.length}`).padStart(14)}${String(`${peak} ms`).padStart(15)}` +
    `${String(Math.floor(86400 / k)).padStart(24)}${gb(bytes(series, k)).padStart(12)}`);
}

const QUESTION = [["which path got slow", ["path"]], ["which method is erroring", ["method"]],
  ["which result code increased", ["result"]], ["latency's upper percentile", ["bucket"]],
  ["which pipeline fell behind", ["pipeline"]], ["which pod is consuming memory", ["resource"]],
  ["which core filled up", ["core"]], ["total request count", []],
  ["which application is slow", []], ["which pod restarted", ["resource"]],
  ["which result on which path", ["path", "result"]], ["disk direction: read or write", ["direction"]]];

console.log("\naggregation                   series   GB   questions answered   detail lost");
const level = [["none", [], RESOURCE], ["bucket dropped", ["bucket"], RESOURCE],
  ["+ path dropped", ["bucket", "path"], RESOURCE],
  ["+ pods aggregated into apps", ["bucket", "path", "resource"], { pod: 5, node: 9 }]];
for (const [name, dropped, resource] of level) {
  const o = dropped.filter((d) => d !== "resource").reduce((x, d) => dropLabel(x, d), METRIC);
  const s = seriesCount(o, resource);
  const answered = QUESTION.filter((q) => q[1].every((e) => !dropped.includes(e))).length;
  console.log(`${name.padEnd(30)} ${String(s).padStart(5)} ${gb(bytes(s)).padStart(5)}` +
    `${String(`${answered}/${QUESTION.length}`).padStart(15)}   ${dropped.length ? dropped.join(", ") : "-"}`);
}

one hour of latency signal: 3600 seconds, 12 spikes (each 20 sec, +780 ms)
interval  samples  spikes caught  seen peak   series records/day   total GB
   1 sec    3600         12/12         906 ms                   86400       391.3
  15 sec     240         12/12         905 ms                    5760        26.1
  60 sec      60          5/12         903 ms                    1440         6.5
 300 sec      12          1/12         903 ms                     288         1.3

aggregation                   series   GB   questions answered   detail lost
none                           10131  26.1          12/12   -
bucket dropped                  5643  14.5          11/12   bucket
+ path dropped                   781   2.0           9/12   bucket, path
+ pods aggregated into apps      230   0.6           7/12   bucket, path, resource

The sampling table lines up two columns side by side. A fifteen-second interval takes up one-fifteenth the bytes of second-by-second collection — twenty-six gigabytes instead of three hundred ninety-one — and catches all twelve of the twelve spikes, because a spike lasts twenty seconds and the fifteen-second grid lands at least once in every twenty-second window. Up to this point, sampling looks free.

At sixty seconds, bytes drop to a quarter, six and a half gigabytes, and the spikes caught fall from twelve to five. At three hundred seconds, bytes are one point three gigabytes, and only one of the twelve spikes shows up at all. Where the loss comes from is clear: once the sampling interval exceeds the event’s duration, whether the event becomes visible is left to chance. The peak seen sits around nine hundred milliseconds across all three intervals — meaning a caught spike is measured correctly, and what gets lost is not magnitude but how many spikes never show up at all. The metric’s average stays correct, and stays misleading.

The aggregation table builds the same duality on the label side. When the histogram bucket is dropped, series falls from ten thousand to five thousand six hundred forty-three, and eleven of twelve questions stay answerable — the one lost question is latency’s upper percentile. When the path label is also dropped, series falls to seven hundred eighty-one, bytes to two gigabytes, and two more questions close. When pods get aggregated into applications, series falls to two hundred thirty, bytes to six hundred megabytes; which pod is consuming memory, or which one restarted, can no longer be asked.

The three steps’ payoff is not equal. The first step buys back 11.6 gigabytes and costs one question; the second buys back 12.5 gigabytes and costs two; the third buys back only 1.4 gigabytes and again costs two. Gain per question falls from 11.6 to 6.3 to 0.7 gigabytes. What is cheap is dropping the product’s large factor — the twelve-bucket histogram and the twelve-path distinction. Once those two factors are gone, the remaining series are already few, and every further aggregation closes a question for close to free bytes.

Delegation’s Balance Sheet

The delegated decision is this: which question stays answerable later gets decided before the question is asked. The information items this needs are countable too — which metric, which labels, which interval, how much retention. Information’s cost is the product of four numbers, and changing a single item in that product multiplies the result many times over.

What the system does not know also does something, and this time it is silent. When a sixty-second sample interval is set up, no warning appears; because seven spikes never show up, their not showing up also does not show up. When the pod label is dropped, a query produces no error, it returns an application-level number instead. Both of these losses get noticed months after the decision was delegated, and only once that question is genuinely asked; at that point there is no way to reach back and collect the data that was never gathered.

Summary

• 43 resources, 294 series per pod and 15 per node; 10,131 series in total, 1.75 billion records and 26.1 GB over 30 days.
• A metric’s series count is the product of its label cardinalities: adding a 7-value label to two metrics grows the total 6.8x, adding a 4-value label to every pod metric grows it 28.3x.
• Turning a 5000-value field into a label multiplies series by 67,672x; unbounded cardinality turns a metric store into a record store.
• On the same cluster, the 30-day cost is 26.1 GB for metrics, 177.3 for logs, 828.5 for traces; one-percent sampling cuts traces to 8.3 GB and erases the trail of 99 of every 100 requests.
• Moving sampling from 15 to 60 seconds drops bytes from 26.1 to 6.5 GB and spikes caught from 12/12 to 5/12; three aggregation steps drop bytes to 0.6 GB while dropping questions answered from 12/12 to 7/12.

Next Step

Every measurement up to this point was over a single cluster: one control plane, one resource inventory, one series store. The regional measurement network’s production runs across two regions, and each region has its own cluster — the same chart, the same resource type, the same controller, separate control planes. The next lesson counts what this multiplication costs: how many clusters the same desired state reaches by how many separate routes, how drift between clusters gets measured, and, when a decision has to be made not for a single cluster but for a set of clusters, which information item comes from whom.