Retention and Cost

In the five measurements so far, retention was always held fixed. The metric pipeline was priced over thirty days, the log pipeline over one hour, and every lesson measured at a single level of detail. A real pipeline does not work that way: as data ages, its resolution is lowered, its labels are downsampled, its lines are discarded, and it eventually collapses into a summary.

This lesson scans through those tiers. The measure does not change — at every tier, the number of answerable questions is counted together with the bytes, and cost per question is derived tier by tier. In the end, two numbers stand side by side: the marginal cost of adding a tier, and the name of the question that tier closes.

The numbers come from the regional measurement network’s pipeline and are fictional; what gets measured is a model built with node, and its inputs come from this topic’s previous lessons.

ML31. The age range, resolution, dropped labels, and kept log line types of the three tiers are fictional and written in advance. ML32. The metric series count is taken from the second lesson’s actual set: 6,888 series. At the summary tier, the instance label is rolled up to the service level, and the bucket label is dropped entirely. ML33. Log volume derives from the fourth lesson’s stream: a base of 232 lines/second, 540,000 lines from the nightly job; an error line is 512 bytes, the truncation limit. ML34. Every question’s horizon comes from the second lesson, its resolution requirement from the third, and its required log line type from the fourth and fifth. ML35. A tier is counted as answering a question if its age range covers the question’s horizon, its resolution is sufficient, it has not dropped the required label, and it keeps the required line type. ML36. Q12 is answered at no tier; the data it requires is not present in any tier’s content.

Three Tiers and Their Distinguishing Measures

// tiers.mjs — retention tiers, the questions answered at each tier, and the cost per question.
export const METRICS = {              // metric: [instance base, service base, other-label multiplier, bucket]
  "request-count": [29, 4, 4 * 5, 1], "request-duration": [29, 4, 4, 12], "queue-depth": [29, 4, 4 * 3, 1],
  "processed-reading": [20, 2, 4 * 11 * 5, 1], "job-step-duration": [6, 3, 4 * 6, 1], "job-processed-records": [6, 3, 4, 1],
};
export const seriesCount = (summary) => Object.values(METRICS).reduce((t, [a, b, c, d]) =>
  t + (summary ? b : a) * c * (summary ? 1 : d), 0);       // summary rolls instance up to service, drops bucket entirely

export const LOG_LINES = { request: 232 * 0.86 * 86400, error: 232 * 0.02 * 86400,
  "job-step": 232 * 0.12 * 86400 + 540000 };         // lines/day; nightly job adds 540,000 lines
export const LINE_BYTES = { request: 180, error: 512, "job-step": 240 };

// tier: [name, start day, end day, resolution (s), is summary, log line types it keeps]
export const TIERS = [["raw", 0, 7, 15, false, ["request", "error", "job-step"]],
  ["downsampled", 7, 90, 300, false, ["error", "job-step"]],
  ["summary", 90, 395, 3600, true, []]];

// question: [id, required horizon (days), required resolution (s), required labels, required line type]
export const QUESTIONS = [["Q1", 1, 300, ["service"], null], ["Q2", 1, 15, ["bucket"], null],
  ["Q3", 1, 3600, ["environment", "result"], null], ["Q4", 30, 3600, ["version"], null],
  ["Q5", 7, null, [], "job-step"], ["Q6", 7, null, [], "job-step"],
  ["Q7", 365, 3600, ["region", "result"], null], ["Q8", 7, 300, ["instance"], null],
  ["Q9", 7, 300, ["instance"], "job-step"], ["Q10", 1, null, [], "error"],
  ["Q11", 1, 300, ["queue"], null], ["Q12", 1, null, [], "tracing"]];

export const DROPPED = ["instance", "bucket"];             // labels dropped at the summary tier
export const fits = (tier, question, checkAge = true) => {      // does the tier answer the question
  const [, start, end, resolution, summary, types] = tier, [, horizon, reqResolution, reqLabels, reqType] = question;
  if (checkAge && !(start <= horizon && horizon <= end)) return false;
  if (reqResolution !== null && resolution > reqResolution) return false;
  if (summary && reqLabels.some((e) => DROPPED.includes(e))) return false;
  return reqType === null || types.includes(reqType);
};
export const gb = (b) => (b / 1024 ** 3).toFixed(2);

Four measures define a tier, and all four can close off a question: age range (does it cover the question’s horizon), resolution (the query-window measurement from the third lesson), dropped label (the label ladder from the second lesson), and kept line type (the layer loss from the fourth lesson). Designing a tier means deciding on all four of these at once.

Bytes and Questions, Tier by Tier

// cost.mjs — bytes tier by tier, questions answered, and cost per question.
import { METRICS, seriesCount, LOG_LINES, LINE_BYTES, TIERS, QUESTIONS, DROPPED, fits, gb } from "./tiers.mjs";

const metricBytes = (b) => {
  const [, start, end, resolution, summary] = b;
  return seriesCount(summary) * Math.floor(86400 / resolution) * (end - start) * 16 + seriesCount(summary) * 120;
};
const logBytes = (b) => (b[2] - b[1]) * b[5].reduce((t, type) => t + LOG_LINES[type] * LINE_BYTES[type], 0);

console.log(`metric series: raw/downsampled ${seriesCount(false).toLocaleString("en-US")},` +
  ` summary ${seriesCount(true).toLocaleString("en-US")} (dropped labels: ${DROPPED.join(", ")})`);
console.log("\ntier          days       resolution  metric GB   log GB   total GB   within age     by detail  GB/question");
const rows = [];
for (const b of TIERS) {
  const m = metricBytes(b), l = logBytes(b), t = m + l;
  const withinAge = QUESTIONS.filter((s) => fits(b, s)), byDetail = QUESTIONS.filter((s) => fits(b, s, false));
  rows.push([b, t, withinAge, byDetail]);
  console.log(`${b[0].padEnd(14)}${`${b[1]}-${b[2]}`.padEnd(9)}${`${b[3]} s`.padStart(12)}` +
    `${gb(m).padStart(11)}${gb(l).padStart(9)}${gb(t).padStart(11)}${`${withinAge.length}/12`.padStart(13)}` +
    `${`${byDetail.length}/12`.padStart(14)}${gb(t / (withinAge.length || 1)).padStart(13)}`);
}

console.log("\ncumulative (tiers stacked on top of each other)");
console.log("tiers                          total GB     answered  GB/question  marginal GB/new question   newly opened");
let total = 0, prev = [];
for (let i = 0; i < rows.length; i++) {
  const tierBytes = rows[i][1], name = TIERS.slice(0, i + 1).map((b) => b[0]).join(" + ");
  total += tierBytes;
  const answered = [...new Set(rows.slice(0, i + 1).flatMap(([, , withinAge]) => withinAge))];
  const opened = answered.filter((s) => !prev.includes(s));
  console.log(`${name.padEnd(29)}${gb(total).padStart(10)}${`${answered.length}/12`.padStart(13)}` +
    `${gb(total / answered.length).padStart(13)}${(opened.length ? gb(tierBytes / opened.length) : "-").padStart(26)}   ` +
    (opened.map(([k]) => k).join(" ") || "-"));
  prev = answered;
}

const cheapest = rows[rows.length - 1];
const lost = QUESTIONS.filter((s) => !cheapest[3].includes(s) && s[0] !== "Q12");
console.log(`\ncheapest tier: ${cheapest[0][0]}, ${gb(cheapest[1])} GB, ${cheapest[3].length}/12 questions by detail` +
  `, ${gb(cheapest[1] / cheapest[3].length)} GB per question`);
console.log("questions it loses and why");
for (const s of lost) {
  const reasons = [cheapest[0][3] > (s[2] ?? Infinity) && `resolution ${cheapest[0][3]} s`,
    s[3].some((e) => DROPPED.includes(e)) && `${s[3].filter((e) => DROPPED.includes(e)).join("/")} label dropped`,
    s[4] && !cheapest[0][5].includes(s[4]) && `${s[4]} lines not kept`].filter(Boolean);
  console.log(`  ${s[0].padEnd(4)} ${reasons.join("; ")}`);
}

metric series: raw/downsampled 6,888, summary 668 (dropped labels: instance, bucket)

tier          days       resolution  metric GB   log GB   total GB   within age     by detail  GB/question
raw           0-7              15 s       4.14    26.18      30.31         9/12         11/12         3.37
downsampled   7-90            300 s       2.45    70.51      72.96         5/12         10/12        14.59
summary       90-395         3600 s       0.07     0.00       0.07         1/12          3/12         0.07

cumulative (tiers stacked on top of each other)
tiers                          total GB     answered  GB/question  marginal GB/new question   newly opened
raw                               30.31         9/12         3.37                      3.37   Q1 Q2 Q3 Q5 Q6 Q8 Q9 Q10 Q11
raw + downsampled                103.28        10/12        10.33                     72.96   Q4
raw + downsampled + summary      103.35        11/12         9.40                      0.07   Q7

cheapest tier: summary, 0.07 GB, 3/12 questions by detail, 0.02 GB per question
questions it loses and why
  Q1   resolution 3600 s
  Q2   resolution 3600 s; bucket label dropped
  Q5   job-step lines not kept
  Q6   job-step lines not kept
  Q8   resolution 3600 s; instance label dropped
  Q9   resolution 3600 s; instance label dropped; job-step lines not kept
  Q10  error lines not kept
  Q11  resolution 3600 s

The two columns measure separate things, and seeing that distinction is the key to this table. The within age column counts questions whose horizon falls inside that tier’s age window; the by detail column counts how many questions that tier’s detail could answer if age were disregarded entirely. The raw tier carries eleven questions by detail but answers only nine, because the horizons of the version comparison and the seasonal region comparison fall outside seven days. Keeping the detail is not enough; the detail has to sit alongside old enough data.

The byte split is also different from what might be expected. Of the raw tier’s 30.31 gigabytes, 26.18 is log, and only 4.14 is metric. At the downsampled tier, even though request lines are discarded, the log still holds 70.51 gigabytes — because error and job-step lines also accumulate over an eighty-three-day window. The summary tier is close to free on both sides: three hundred five days of retention comes down to 0.07 gigabytes.

The Reversal of Marginal Cost

The cumulative table gives the number that carries the decision. The raw tier opens nine questions for 30.31 gigabytes — 3.37 per question. When the downsampled tier is added, the total rises to 103.28 gigabytes, and questions answered rise from nine to ten: seventy-three gigabytes, a single question. The marginal cost is 72.96 gigabytes per question, twenty-one times the raw tier.

When the summary tier is added, the total becomes 103.35 gigabytes, and questions answered reach eleven. The marginal cost is 0.07 gigabytes per question — a thousand times cheaper than the downsampled tier. That ratio is this lesson’s sharpest number: both tiers open exactly one question, and one is a thousand times more expensive than the other.

The reason for the difference is not the tier’s length, it is what it keeps. The downsampled tier keeps the log line and the fully labeled series for eighty-three days; the single question it opens asks only for the version label at an hourly resolution. So almost all of the data kept for a thirty-day version comparison goes unused by that question. The same question could have been answered by the summary tier’s own detail.

This shows the real decision in tier design: lowering the downsampled tier’s resolution and line selection to the minimum the questions require drops the cost of a single question from seventy-three gigabytes to the single-gigabyte range. Shortening the tier’s duration, on the other hand, closes Q4 entirely and gains nothing.

The Questions the Cheapest Tier Loses

The summary tier answers three questions by detail for 0.07 gigabytes — 0.02 gigabytes per question, the cheapest number on the whole pipeline. A decision that looked only at this number would reduce everything to the summary. The losses stand in the list by name, and each one’s reason is a separate design item.

Because resolution rises to 3600 seconds, Q1 (which service slowed down), Q11 (why the work-order queue backed up), and partly Q8 drop — it was measured in the third lesson that a ninety-second degradation in a one-hour window shows up in neither the average nor the percentile. Q2 drops because the bucket label is dropped: without a distribution bucket, the tail cannot be computed. Q8 and Q9 drop because the instance label is rolled up to the service — which instance restarted and which run was slow can no longer be asked. Q5, Q6, and Q10 drop because log lines are not kept at all.

All eight of the eight questions close silently. At the summary tier, these queries do not error; they return an empty result or a single aggregate value. This has the same shape as the truncation finding from the fifth lesson: the signal of the loss disappears along with the loss itself.

Tier Layouts Side by Side

Now that the individual cost of each tier is known, the layout itself can be compared: in which layout does the same twelve-question set get answered for how many gigabytes?

// policy.mjs — tier layouts are compared: how many gigabytes does the same question set cost under each.
import { seriesCount, LOG_LINES, LINE_BYTES, TIERS, QUESTIONS, fits, gb } from "./tiers.mjs";

const resize = (name, start, end) => {                    // move a tier to a different age range
  const b = TIERS.find((x) => x[0] === name);
  return [name, start, end, b[3], b[4], b[5]];
};
const bytes = (b) => seriesCount(b[4]) * Math.floor(86400 / b[3]) * (b[2] - b[1]) * 16 + seriesCount(b[4]) * 120 +
  (b[2] - b[1]) * b[5].reduce((t, type) => t + LOG_LINES[type] * LINE_BYTES[type], 0);

const POLICIES = [
  ["raw only, 7 days", [resize("raw", 0, 7)]],
  ["raw only, 30 days", [resize("raw", 0, 30)]],
  ["raw only, 395 days", [resize("raw", 0, 395)]],
  ["three tiers 7 / 90 / 395", [resize("raw", 0, 7), resize("downsampled", 7, 90), resize("summary", 90, 395)]],
  ["raw 7 + summary 395 (no middle)", [resize("raw", 0, 7), resize("summary", 7, 395)]],
  ["raw 30 + summary 395", [resize("raw", 0, 30), resize("summary", 30, 395)]],
];
console.log("policy                            total GB     answered  GB/question   unanswered");
const results = POLICIES.map(([name, bs]) => {
  const t = bs.reduce((s, b) => s + bytes(b), 0);
  const y = QUESTIONS.filter((s) => bs.some((b) => fits(b, s)));
  console.log(`${name.padEnd(33)}${gb(t).padStart(9)}${`${y.length}/12`.padStart(13)}${gb(t / y.length).padStart(13)}   ` +
    QUESTIONS.filter((s) => !y.includes(s)).map(([k]) => k).join(" "));
  return [name, t, y.length];
});
const [, threeTierBytes, threeTierAnswered] = results[3], [, noMidBytes, noMidAnswered] = results[4];
console.log(`\nthe middle tier's contribution: ${gb(threeTierBytes - noMidBytes)} GB more, ${threeTierAnswered - noMidAnswered} new question`);
console.log(`the cost of keeping everything raw: ${gb(results[2][1] - noMidBytes)} GB more,` +
  ` ${results[2][2] - noMidAnswered} new question`);

policy                            total GB     answered  GB/question   unanswered
raw only, 7 days                     30.31         9/12         3.37   Q4 Q7 Q12
raw only, 30 days                   129.92        10/12        12.99   Q7 Q12
raw only, 395 days                 1710.56        11/12       155.51   Q12
three tiers 7 / 90 / 395            103.35        11/12         9.40   Q12
raw 7 + summary 395 (no middle)      30.41        11/12         2.76   Q12
raw 30 + summary 395                130.00        11/12        11.82   Q12

the middle tier's contribution: 72.94 GB more, 0 new question
the cost of keeping everything raw: 1680.16 GB more, 0 new question

Four layouts answer the same eleven questions, and there is a fifty-six-fold price difference between them. The cheapest is the layout that never sets up the middle tier: seven days of raw, then straight to summary. Total 30.41 gigabytes, 2.76 per question.

The middle layer of the three-tier layout holds 72.94 gigabytes extra and opens zero new questions. The reason was seen in the fourth section: the downsampled tier’s only contribution was Q4, and because Q4 only needs the version label at hourly resolution, it is also answered by the summary tier’s own detail. The middle tier was built only on the intuition that “seven days is too little, ninety days would be better”; counted against the question set, it has no payoff.

The “keep everything raw” layout answers the same eleven questions for 1710.56 gigabytes — 1680 gigabytes extra, zero new questions. Its cost per question is 155.51 gigabytes, fifty-six times the cheapest layout. These two lines pin this course’s rule to a number: data that answers no question, no matter how carefully it is collected, is a cost.

Summary

• Four items define a tier, and all four can close off a question: age range, resolution, dropped label, and kept line type.
• The raw tier carries 11/12 questions by detail but answers 9/12 because of its age range; keeping the detail is not enough, it has to sit alongside old enough data.
• Bytes accumulate on the log side: of the raw tier’s 30.31 GB, 26.18 is log, 4.14 is metric; at the downsampled tier, even though request lines are discarded, the log holds 70.51 GB.
• Marginal cost reverses: the downsampled tier opens one question (Q4) for 72.96 GB, the summary tier opens one question (Q7) for 0.07 GB — a thousandfold difference, both exactly one question.
• The cheapest tier (summary) answers 3/12 questions at 0.02 GB/question and silently loses eight questions by name: Q1, Q11 from resolution; Q2 from the bucket label; Q8, Q9 from the instance label; Q5, Q6, Q10 from log lines. The queries do not error, they return an empty or a single aggregate value.
• In the layout comparison, four different layouts answer the same 11/12: the cheapest is the one that never sets up the middle tier (30.41 GB, 2.76 GB/question). The middle tier holds 72.94 GB extra and opens zero questions; “keep everything raw” holds 1680 GB extra and opens zero questions.

Next Step

This topic took data end to end: how it is collected, which labels it is stored with, which aggregation it is queried through, which layer of the log pipeline loses what, how text is turned into a searchable field, and how much it costs at which tier as it ages. Eleven of the twelve questions became answerable somewhere, and the price of every answer is known.

One question remains, and it appeared at no tier, in no label set, on no log line: which reading a bill was generated from. Asking this means asking for a single transaction’s path from the reading collector to the validator, from there to billing and the work order. A metric cannot carry it, because a transaction ID’s cardinality is unbounded; a log line cannot carry it, because every service generates its own identity and the lines cannot be linked to each other. Not a single byte of the hundred gigabytes collected shows that path. The next topic starts from this gap: what does making a request’s path across services visible require, and which questions does it open.