Time Series Data

The previous lesson measured collection like a box: how many measurements, how many series, how many bytes. But each of the eight or nine questions answered depended on the presence of a specific label, and when the instance label became unreliable, two questions closed at once. This lesson measures that link in the opposite direction: what question does adding a label open, and how many bytes does that question cost to open?

Why a label grows the series count multiplicatively rather than additively was measured in the Cluster Observability lesson of the Container Orchestration course: a base of ten thousand series rose to 6.8x with a single seven-valued label, and to 28.3x when a four-valued label was added to every metric. That measurement is not repeated here; the multiplication rule is this lesson’s input. What is measured here is something else — which question opens in exchange for the multiplier.

The series set and the numbers are both fictional; what gets measured is a model built with node.

ML7. The label set, the cardinalities, and which label applies to which metric are fictional and written in advance. ML8. The instance label resolves within a service; at most eight instances per service are assumed (the 29 instances from the previous lesson spread across four services). ML9. Every question has a backward-looking horizon: how many days of data must be looked at for it to be answerable is written in advance. ML10. Sampling is 15 seconds, a record is 16 bytes, and a series descriptor is 120 bytes (the same as the previous lesson). ML11. For a question to count as opened, all the labels it requires must be present in the set; no single label is sufficient on its own. ML12. The metric set derives from the previous lesson’s metrics; the three metrics that answer no question were counted there and are not carried into this set.

The Decision Inside the Series

A time series consists of a name and a label set, and in storage every distinct combination of that pair is a separate series. The label set is therefore not a storage setting, it is a question decision: which breakdown you can perform later is fixed at the moment of collection.

// series.mjs — the time series set, label cardinalities, and the label/horizon requirement of questions.
export const METRICS = { "request-count": "long", "request-duration": "long", "queue-depth": "long",
  "processed-reading": "long", "job-step-duration": "short", "job-processed-records": "short" };
export const LABELS = {                              // name: [cardinality, metrics it applies to]
  service: [4, ["request-count", "request-duration", "queue-depth", "processed-reading"]],
  job: [3, ["job-step-duration", "job-processed-records"]],
  environment: [3, Object.keys(METRICS)], version: [4, Object.keys(METRICS)], instance: [8, Object.keys(METRICS)],
  result: [5, ["request-count", "processed-reading"]], bucket: [12, ["request-duration"]],
  queue: [3, ["queue-depth"]], region: [11, ["processed-reading"]], step: [6, ["job-step-duration"]],
  "node-name": [9, Object.keys(METRICS)],
};
// question: [id, required labels, required horizon (days)]
export const QUESTIONS = [["Q1", ["service"], 1], ["Q2", ["service", "bucket"], 1],
  ["Q3", ["environment", "result"], 1], ["Q4", ["version"], 30], ["Q5", ["job"], 7],
  ["Q6", ["job", "step"], 7], ["Q7", ["region", "result"], 365], ["Q8", ["instance"], 7],
  ["Q9", ["job", "instance"], 7], ["Q10", ["result"], 1], ["Q11", ["service", "queue"], 1],
  ["Q12", ["transaction-id"], 1]];

export const seriesCount = (open) => Object.keys(METRICS).reduce((t, m) => t +
  open.reduce((c, e) => c * (LABELS[e][1].includes(m) ? LABELS[e][0] : 1), 1), 0);
export const bytes = (series, days, interval = 15) => series * Math.floor(86400 / interval) * days * 16 + series * 120;
export const gb = (b) => (b / 1024 ** 3).toFixed(2);

Labels are not spread evenly across metrics. region applies only to the processed-reading metric; environment, version, instance, and node-name apply to every metric. This split determines the cost: an eleven-valued label added to a single metric can be cheaper than a four-valued label added to every metric.

The Label Ladder

The ladder starts from an empty set — only service and job names — and adds labels one at a time. At every rung, four things are written together: series, bytes, questions answered, and the marginal byte cost of the question opened at that rung.

// labels.mjs — the question a label opens when added, the question closed by removing it, and the retention window.
import { METRICS, LABELS, QUESTIONS, seriesCount, bytes, gb } from "./series.mjs";

const opened = (a) => QUESTIONS.filter(([, e]) => e.every((x) => a.includes(x))).map(([k]) => k);
const LADDER = ["environment", "result", "bucket", "queue", "region", "step", "version", "instance", "node-name"];
let open = ["service", "job"], prevQuestions = opened(open), prevBytes = bytes(seriesCount(open), 30);
console.log(`${Object.keys(METRICS).length} metrics, 15 s sampling, 30-day retention; base labels ${open.join(" + ")}`);
console.log("added label      cardinality   series        GB     answered   opened question    marginal GB/question");
const print = (name, card) => {
  const s = seriesCount(open), b = bytes(s, 30), y = opened(open);
  const added = y.filter((q) => !prevQuestions.includes(q));
  const marginal = added.length ? gb((b - prevBytes) / added.length) : "-";
  console.log(`${name.padEnd(15)}${String(card).padStart(13)}${s.toLocaleString("en-US").padStart(9)}` +
    `${gb(b).padStart(10)}${String(`${y.length}/12`).padStart(13)}   ${(added.join(" ") || "-").padEnd(17)}${marginal.padStart(22)}`);
  prevQuestions = y; prevBytes = b;
};
print("base", "-");
for (const e of LADDER) { open = [...open, e]; print(`+ ${e}`, LABELS[e][0]); }

const FULL = open, fullBytes = bytes(seriesCount(FULL), 30);
console.log(`\nwhen a label is removed (full set ${FULL.length} labels,` +
  ` ${seriesCount(FULL).toLocaleString("en-US")} series, ${gb(fullBytes)} GB)`);
console.log("removed label         gained GB   closed question     GB/closed question");
for (const e of FULL) {
  const remaining = FULL.filter((x) => x !== e), gain = fullBytes - bytes(seriesCount(remaining), 30);
  const closed = opened(FULL).filter((q) => !opened(remaining).includes(q));
  console.log(`${e.padEnd(18)}${gb(gain).padStart(13)}   ${(closed.join(" ") || "-").padEnd(17)}` +
    `${(closed.length ? gb(gain / closed.length) : "-").padStart(21)}`);
}

const PRUNED = FULL.filter((e) => e !== "node-name");
console.log(`\nretention window (pruned set ${PRUNED.length} labels,` +
  ` ${seriesCount(PRUNED).toLocaleString("en-US")} series)`);
console.log("window          GB     answered in horizon  GB/question   question the window opens");
let prevHorizon = [];
for (const day of [7, 30, 90, 395]) {
  const y = opened(PRUNED).filter((q) => QUESTIONS.find(([k]) => k === q)[2] <= day);
  const b = bytes(seriesCount(PRUNED), day);
  console.log(`${String(`${day} d`).padEnd(9)}${gb(b).padStart(9)}${String(`${y.length}/12`).padStart(24)}` +
    `${gb(b / y.length).padStart(13)}   ${y.filter((q) => !prevHorizon.includes(q)).join(" ") || "-"}`);
  prevHorizon = y;
}

6 metrics, 15 s sampling, 30-day retention; base labels service + job
added label      cardinality   series        GB     answered   opened question    marginal GB/question
base                       -       22      0.06         2/12   -                                     -
+ environment              3       66      0.17         2/12   -                                     -
+ result                   5      162      0.42         4/12   Q3 Q10                             0.12
+ bucket                  12      294      0.76         5/12   Q2                                 0.34
+ queue                    3      318      0.82         6/12   Q11                                0.06
+ region                  11      918      2.36         7/12   Q7                                 1.55
+ step                     6      963      2.48         8/12   Q6                                 0.12
+ version                  4    3,852      9.92         9/12   Q4                                 7.44
+ instance                 8   30,816     79.35        11/12   Q8 Q9                             34.72
+ node-name                9  277,344    714.17        11/12   -                                     -

when a label is removed (full set 11 labels, 277,344 series, 714.17 GB)
removed label         gained GB   closed question     GB/closed question
service                  500.59   Q1 Q2 Q11                       166.86
job                       31.15   Q5 Q6 Q9                         10.38
environment              476.11   Q3                              476.11
result                   427.17   Q3 Q7 Q10                       142.39
bucket                    97.89   Q2                               97.89
queue                     17.80   Q11                              17.80
region                   444.97   Q7                              444.97
step                      33.37   Q6                               33.37
version                  535.63   Q4                              535.63
instance                 624.90   Q8 Q9                           312.45
node-name                634.82   -                                    -

retention window (pruned set 10 labels, 30,816 series)
window          GB     answered in horizon  GB/question   question the window opens
7 d          18.52                    9/12         2.06   Q1 Q2 Q3 Q5 Q6 Q8 Q9 Q10 Q11
30 d         79.35                   10/12         7.94   Q4
90 d        238.05                   10/12        23.80   -
395 d      1044.76                   11/12        94.98   Q7

The first half of the ladder is cheap. The result label opens two questions for 0.12 gigabytes, the queue label opens one question for 0.06 gigabytes, the step label opens one question for 0.12 gigabytes. These are labels that apply to a single metric or carry low cardinality; the product grows by a small multiplier.

The second half is expensive. The version label is four-valued, but because it applies to every metric it quadruples the series and opens a single question — which version broke it — for 7.44 gigabytes. The instance label opens two questions, at 34.72 gigabytes per question. Between two different labels that each open two questions in the same set there is a three-hundred-fold difference, and the source of that difference is not the importance of the question but how many metrics the label applies to.

The environment row must be read separately: it opens no question at the rung where it is added. But Q3 requires both the environment and result labels; without environment, the result label alone could not have opened Q3. A label “opening no question” and a label “not being sufficient on its own” are different things, and the ladder cannot distinguish between them.

Removing a Label, and Pruning

The second table is what makes the distinction: every label is removed from the full set and the questions that close are counted. When environment is removed, Q3 closes — so it does have a share, and that share costs 476 gigabytes. When node-name is removed, no question closes. The bytes gained are 634.82 gigabytes: eighty-nine percent of the full set.

This label’s name needs to be written down because it looks harmless. It is nine-valued, short, easy to read, and the collector adds it on its own — the node name is already known. The problem is that none of the twelve questions ask for the node name. Whichever node the pod runs on, what operations asks for is the service, the environment, the version, and the instance. Until a label is tied to a question, it is not information, it is a multiplier.

The pruning order also comes from the table. First the label with no payoff is dropped; this single step brings the series from 277,344 down to 30,816. After that, the marginal cost of every remaining label drops to one-ninth — the version label costs 59 gigabytes instead of 535, the instance label costs 69 instead of 624. A label with a payoff often looks expensive not because of itself, but because of the payoff-less label sitting next to it.

The Product Is an Upper Bound

All the numbers on the ladder came out of a product, and a product assumes that every label combination will actually occur. The inventory does not confirm that: the four services do not have equal instance counts, two services never run in the development environment, and only two services emit the processed-reading metric. The actual series set can be counted from the inventory.

// actual.mjs — the product is an upper bound: the actual series count and the correction of the measure.
import { METRICS, LABELS, QUESTIONS, seriesCount, bytes, gb } from "./series.mjs";

const INSTANCES = { "reading-collector": [8, 2, 1], validator: [6, 2, 1],   // production, staging, development
  billing: [4, 1, 0], "work-order": [3, 1, 0] };
const EMITTERS = { "processed-reading": ["reading-collector", "validator"] };  // others on all four services
const RUNNERS = 3 * 2;                                     // three nightly jobs, runner in two environments
const PRUNED = ["service", "job", "environment", "version", "instance", "result", "bucket", "queue", "region", "step"];
const BASE = ["service", "job", "environment", "instance"];   // the triple actually counted from inventory

const actual = Object.keys(METRICS).reduce((t, m) => {
  const base = METRICS[m] === "short" ? RUNNERS
    : (EMITTERS[m] || Object.keys(INSTANCES)).reduce((s, v) => s + INSTANCES[v].reduce((a, b) => a + b, 0), 0);
  return t + base * PRUNED.filter((e) => !BASE.includes(e) && LABELS[e][1].includes(m))
    .reduce((c, e) => c * LABELS[e][0], 1);
}, 0);

const theoretical = seriesCount(PRUNED), answered = QUESTIONS.filter(([, e]) => e.every((x) => PRUNED.includes(x))).length;
console.log("metric                   actual  theoretical     overcount");
for (const m of Object.keys(METRICS)) {
  const base = METRICS[m] === "short" ? RUNNERS
    : (EMITTERS[m] || Object.keys(INSTANCES)).reduce((s, v) => s + INSTANCES[v].reduce((a, b) => a + b, 0), 0);
  const multiplier = PRUNED.filter((e) => !BASE.includes(e) && LABELS[e][1].includes(m))
    .reduce((c, e) => c * LABELS[e][0], 1);
  const k = PRUNED.reduce((c, e) => c * (LABELS[e][1].includes(m) ? LABELS[e][0] : 1), 1);
  console.log(`${m.padEnd(22)}${(base * multiplier).toLocaleString("en-US").padStart(9)}${k.toLocaleString("en-US").padStart(13)}` +
    `${String(`${(k / (base * multiplier)).toFixed(1)}x`).padStart(14)}`);
}
console.log(`${"total".padEnd(22)}${actual.toLocaleString("en-US").padStart(9)}` +
  `${theoretical.toLocaleString("en-US").padStart(13)}${String(`${(theoretical / actual).toFixed(1)}x`).padStart(14)}`);
console.log(`\n30-day retention   theoretical ${gb(bytes(theoretical, 30))} GB` +
  `   actual ${gb(bytes(actual, 30))} GB`);
console.log(`for ${answered}/12 questions, per question   theoretical ${gb(bytes(theoretical, 30) / answered)} GB` +
  `   actual ${gb(bytes(actual, 30) / answered)} GB`);

metric                   actual  theoretical     overcount
request-count               580        1,920          3.3x
request-duration          1,392        4,608          3.3x
queue-depth                 348        1,152          3.3x
processed-reading         4,400       21,120          4.8x
job-step-duration           144        1,728         12.0x
job-processed-records        24          288         12.0x
total                     6,888       30,816          4.5x

30-day retention   theoretical 79.35 GB   actual 17.74 GB
for 11/12 questions, per question   theoretical 7.21 GB   actual 1.61 GB

The product overcounts by four and a half times overall, and by twelve times on the nightly-job metrics. The reason differs per metric: of the theoretical ninety-six combinations of the service-environment-instance triple, only twenty-nine exist; only two of the four services emit processed-reading; and the nightly job runs a single runner in two environments rather than eight instances across three environments.

This correction changes the decision. A retention discussion that looks at the theoretical number prices thirty days at 79 gigabytes, 7.21 gigabytes per question; the actual set counted from the inventory brings the same thirty days down to 17.74 gigabytes, 1.61 gigabytes per question. The difference is not a saving, it is a measurement error — a cost inflated four and a half times leads not to skipping a label that should never have been added, but to skipping a label that is actually needed.

The Retention Window

The label set determines which breakdown a question can perform; the retention window determines how far back it can look. The third table measures the same pruned set across four windows; the numbers are over the theoretical series set, and in the actual set all of them divide by four and a half.

The seven-day window holds eighteen and a half gigabytes and answers nine questions — 2.06 gigabytes per question. Thirty days opens Q4: a version comparison must cover the time both versions stayed in production, and deployment frequency requires thirty days for that. Its cost is 7.94 gigabytes per question.

The ninety-day window opens no new question. Three times the bytes, zero questions — the cost per question rises to 23.80 gigabytes. This line is the exact price of a decision that stretches retention on the reasoning that it should be kept a little longer.

Three hundred ninety-five days opens a single question: which region’s readings did not pass validation, compared against the same period the year before. Its cost is 965 gigabytes of extra retention. The cost of leaving it unanswered can also be counted: when the seasonal comparison cannot be made, which region is chronically problematic cannot be told apart, and the field crew has to close work orders for all eleven regions instead of just one. Which number is larger depends on the regional measurement network’s own cost structure; what carries the decision is that the two numbers stand in the same table.

Q12 appears in none of these tables. Neither adding a label nor extending retention opens it; the question of which reading a bill was generated from cannot be solved with a label, because the breakdown it requires is per transaction.

Summary

• A label’s cost is not its cardinality, it is the product of its cardinality and the number of metrics it applies to: the 11-valued region opens a question for 1.55 GB, while the 4-valued version opens a question for 7.44 GB.
• As the ladder climbs from 22 series to 277,344 series, questions answered climb from 2/12 to 11/12; marginal bytes per question range from 0.06 GB to 34.72 GB, a three-hundred-fold spread.
• node-name opens no question and closes no question; removing it gains 634.82 GB, 89% of the full set. Pruning is the first step; after it, the cost of every other label drops to one-ninth.
• A label that opens no question at the rung where it is added is different from a label with no share in any question: environment opens no question on its own, but removing it closes Q3.
• The product is an upper bound: the actual set counted from the inventory is 6,888 series, a quarter of the theoretical 30,816; cost per question drops from 7.21 GB to 1.61 GB.
• The retention window sets the question’s horizon: 7 days answers 9/12 questions at 2.06 GB/question, 30 days answers 10/12 at 7.94 GB/question, 395 days answers 11/12 at 94.98 GB/question. The 90-day window opens no new question in exchange for three times the bytes.

Next Step

Up to this point the series set was treated like a warehouse: how many series exist, how many bytes they hold, which breakdown they store. But no question is answered by the series itself; the answer comes out of an aggregation — average, percentile, total, rate. The next lesson answers the same question with different aggregations and shows that the answer changes: which aggregation hides what, which detail pre-aggregation erases irrecoverably, and what it buys back in query cost in return.