--- title: 'Retention and Cost' source: 'https://academia.sh/en/courses/devops-observability/retention-and-cost' course: 'Observability and Operations' language: en updated: '2026-09-13T05:08:22+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines retention tiers by age range, resolution, dropped label, and kept line type' - 'Counts the number of answerable questions together with the bytes at every tier and derives cost per question' - 'Compares the marginal cost per question of adding a tier and names the questions the cheapest tier loses' --- # Retention and Cost Three retention tiers are scanned, and at every tier the number of answerable questions is counted together with the bytes: cost per question is derived tier by tier, one tier's marginal cost differs by a thousandfold, and which questions the cheapest tier loses, and why, is written down by name. In the five measurements so far, retention was always held fixed. The metric pipeline was priced over thirty days, the log pipeline over one hour, and every lesson measured at a single level of detail. A real pipeline does not work that way: as data ages, its resolution is lowered, its labels are downsampled, its lines are discarded, and it eventually collapses into a summary. This lesson scans through those tiers. The measure does not change — at every tier, the **number of answerable questions** is counted together with the **bytes**, and cost per question is derived tier by tier. In the end, two numbers stand side by side: the marginal cost of adding a tier, and the name of the question that tier closes. The numbers come from the **regional measurement network's** pipeline and are **fictional**; what gets measured is a **model** built with `node`, and its inputs come from this topic's previous lessons. **ML31.** The age range, resolution, dropped labels, and kept log line types of the three tiers are fictional and written in advance. **ML32.** The metric series count is taken from the second lesson's actual set: 6,888 series. At the summary tier, the `instance` label is rolled up to the service level, and the `bucket` label is dropped entirely. **ML33.** Log volume derives from the fourth lesson's stream: a base of 232 lines/second, 540,000 lines from the nightly job; an error line is 512 bytes, the truncation limit. **ML34.** Every question's horizon comes from the second lesson, its resolution requirement from the third, and its required log line type from the fourth and fifth. **ML35.** A tier is counted as answering a question if its age range covers the question's horizon, its resolution is sufficient, it has not dropped the required label, and it keeps the required line type. **ML36.** Q12 is answered at no tier; the data it requires is not present in any tier's content. ## Three Tiers and Their Distinguishing Measures ```js // tiers.mjs — retention tiers, the questions answered at each tier, and the cost per question. export const METRICS = { // metric: [instance base, service base, other-label multiplier, bucket] "request-count": [29, 4, 4 * 5, 1], "request-duration": [29, 4, 4, 12], "queue-depth": [29, 4, 4 * 3, 1], "processed-reading": [20, 2, 4 * 11 * 5, 1], "job-step-duration": [6, 3, 4 * 6, 1], "job-processed-records": [6, 3, 4, 1], }; export const seriesCount = (summary) => Object.values(METRICS).reduce((t, [a, b, c, d]) => t + (summary ? b : a) * c * (summary ? 1 : d), 0); // summary rolls instance up to service, drops bucket entirely export const LOG_LINES = { request: 232 * 0.86 * 86400, error: 232 * 0.02 * 86400, "job-step": 232 * 0.12 * 86400 + 540000 }; // lines/day; nightly job adds 540,000 lines export const LINE_BYTES = { request: 180, error: 512, "job-step": 240 }; // tier: [name, start day, end day, resolution (s), is summary, log line types it keeps] export const TIERS = [["raw", 0, 7, 15, false, ["request", "error", "job-step"]], ["downsampled", 7, 90, 300, false, ["error", "job-step"]], ["summary", 90, 395, 3600, true, []]]; // question: [id, required horizon (days), required resolution (s), required labels, required line type] export const QUESTIONS = [["Q1", 1, 300, ["service"], null], ["Q2", 1, 15, ["bucket"], null], ["Q3", 1, 3600, ["environment", "result"], null], ["Q4", 30, 3600, ["version"], null], ["Q5", 7, null, [], "job-step"], ["Q6", 7, null, [], "job-step"], ["Q7", 365, 3600, ["region", "result"], null], ["Q8", 7, 300, ["instance"], null], ["Q9", 7, 300, ["instance"], "job-step"], ["Q10", 1, null, [], "error"], ["Q11", 1, 300, ["queue"], null], ["Q12", 1, null, [], "tracing"]]; export const DROPPED = ["instance", "bucket"]; // labels dropped at the summary tier export const fits = (tier, question, checkAge = true) => { // does the tier answer the question const [, start, end, resolution, summary, types] = tier, [, horizon, reqResolution, reqLabels, reqType] = question; if (checkAge && !(start <= horizon && horizon <= end)) return false; if (reqResolution !== null && resolution > reqResolution) return false; if (summary && reqLabels.some((e) => DROPPED.includes(e))) return false; return reqType === null || types.includes(reqType); }; export const gb = (b) => (b / 1024 ** 3).toFixed(2); ``` Four measures define a tier, and all four can close off a question: **age range** (does it cover the question's horizon), **resolution** (the query-window measurement from the third lesson), **dropped label** (the label ladder from the second lesson), and **kept line type** (the layer loss from the fourth lesson). Designing a tier means deciding on all four of these at once. ## Bytes and Questions, Tier by Tier ```js // cost.mjs — bytes tier by tier, questions answered, and cost per question. import { METRICS, seriesCount, LOG_LINES, LINE_BYTES, TIERS, QUESTIONS, DROPPED, fits, gb } from "./tiers.mjs"; const metricBytes = (b) => { const [, start, end, resolution, summary] = b; return seriesCount(summary) * Math.floor(86400 / resolution) * (end - start) * 16 + seriesCount(summary) * 120; }; const logBytes = (b) => (b[2] - b[1]) * b[5].reduce((t, type) => t + LOG_LINES[type] * LINE_BYTES[type], 0); console.log(`metric series: raw/downsampled ${seriesCount(false).toLocaleString("en-US")},` + ` summary ${seriesCount(true).toLocaleString("en-US")} (dropped labels: ${DROPPED.join(", ")})`); console.log("\ntier days resolution metric GB log GB total GB within age by detail GB/question"); const rows = []; for (const b of TIERS) { const m = metricBytes(b), l = logBytes(b), t = m + l; const withinAge = QUESTIONS.filter((s) => fits(b, s)), byDetail = QUESTIONS.filter((s) => fits(b, s, false)); rows.push([b, t, withinAge, byDetail]); console.log(`${b[0].padEnd(14)}${`${b[1]}-${b[2]}`.padEnd(9)}${`${b[3]} s`.padStart(12)}` + `${gb(m).padStart(11)}${gb(l).padStart(9)}${gb(t).padStart(11)}${`${withinAge.length}/12`.padStart(13)}` + `${`${byDetail.length}/12`.padStart(14)}${gb(t / (withinAge.length || 1)).padStart(13)}`); } console.log("\ncumulative (tiers stacked on top of each other)"); console.log("tiers total GB answered GB/question marginal GB/new question newly opened"); let total = 0, prev = []; for (let i = 0; i < rows.length; i++) { const tierBytes = rows[i][1], name = TIERS.slice(0, i + 1).map((b) => b[0]).join(" + "); total += tierBytes; const answered = [...new Set(rows.slice(0, i + 1).flatMap(([, , withinAge]) => withinAge))]; const opened = answered.filter((s) => !prev.includes(s)); console.log(`${name.padEnd(29)}${gb(total).padStart(10)}${`${answered.length}/12`.padStart(13)}` + `${gb(total / answered.length).padStart(13)}${(opened.length ? gb(tierBytes / opened.length) : "-").padStart(26)} ` + (opened.map(([k]) => k).join(" ") || "-")); prev = answered; } const cheapest = rows[rows.length - 1]; const lost = QUESTIONS.filter((s) => !cheapest[3].includes(s) && s[0] !== "Q12"); console.log(`\ncheapest tier: ${cheapest[0][0]}, ${gb(cheapest[1])} GB, ${cheapest[3].length}/12 questions by detail` + `, ${gb(cheapest[1] / cheapest[3].length)} GB per question`); console.log("questions it loses and why"); for (const s of lost) { const reasons = [cheapest[0][3] > (s[2] ?? Infinity) && `resolution ${cheapest[0][3]} s`, s[3].some((e) => DROPPED.includes(e)) && `${s[3].filter((e) => DROPPED.includes(e)).join("/")} label dropped`, s[4] && !cheapest[0][5].includes(s[4]) && `${s[4]} lines not kept`].filter(Boolean); console.log(` ${s[0].padEnd(4)} ${reasons.join("; ")}`); } ``` ``` metric series: raw/downsampled 6,888, summary 668 (dropped labels: instance, bucket) tier days resolution metric GB log GB total GB within age by detail GB/question raw 0-7 15 s 4.14 26.18 30.31 9/12 11/12 3.37 downsampled 7-90 300 s 2.45 70.51 72.96 5/12 10/12 14.59 summary 90-395 3600 s 0.07 0.00 0.07 1/12 3/12 0.07 cumulative (tiers stacked on top of each other) tiers total GB answered GB/question marginal GB/new question newly opened raw 30.31 9/12 3.37 3.37 Q1 Q2 Q3 Q5 Q6 Q8 Q9 Q10 Q11 raw + downsampled 103.28 10/12 10.33 72.96 Q4 raw + downsampled + summary 103.35 11/12 9.40 0.07 Q7 cheapest tier: summary, 0.07 GB, 3/12 questions by detail, 0.02 GB per question questions it loses and why Q1 resolution 3600 s Q2 resolution 3600 s; bucket label dropped Q5 job-step lines not kept Q6 job-step lines not kept Q8 resolution 3600 s; instance label dropped Q9 resolution 3600 s; instance label dropped; job-step lines not kept Q10 error lines not kept Q11 resolution 3600 s ``` The two columns measure separate things, and seeing that distinction is the key to this table. The **within age** column counts questions whose horizon falls inside that tier's age window; the **by detail** column counts how many questions that tier's detail could answer if age were disregarded entirely. The raw tier carries eleven questions by detail but answers only nine, because the horizons of the version comparison and the seasonal region comparison fall outside seven days. Keeping the detail is not enough; the detail has to sit alongside old enough data. The byte split is also different from what might be expected. Of the raw tier's 30.31 gigabytes, 26.18 is log, and only 4.14 is metric. At the downsampled tier, even though request lines are discarded, the log still holds 70.51 gigabytes — because error and job-step lines also accumulate over an eighty-three-day window. The summary tier is close to free on both sides: three hundred five days of retention comes down to 0.07 gigabytes. ## The Reversal of Marginal Cost The cumulative table gives the number that carries the decision. The raw tier opens nine questions for 30.31 gigabytes — 3.37 per question. When the downsampled tier is added, the total rises to 103.28 gigabytes, and questions answered rise from nine to **ten**: seventy-three gigabytes, a single question. The marginal cost is 72.96 gigabytes per question, twenty-one times the raw tier. When the summary tier is added, the total becomes 103.35 gigabytes, and questions answered reach eleven. The marginal cost is **0.07 gigabytes per question** — a thousand times cheaper than the downsampled tier. That ratio is this lesson's sharpest number: both tiers open exactly one question, and one is a thousand times more expensive than the other. The reason for the difference is not the tier's length, it is **what it keeps**. The downsampled tier keeps the log line and the fully labeled series for eighty-three days; the single question it opens asks only for the `version` label at an hourly resolution. So almost all of the data kept for a thirty-day version comparison goes unused by that question. The same question could have been answered by the summary tier's own detail. This shows the real decision in tier design: lowering the downsampled tier's resolution and line selection to the minimum the questions require drops the cost of a single question from seventy-three gigabytes to the single-gigabyte range. Shortening the tier's duration, on the other hand, closes Q4 entirely and gains nothing. ## The Questions the Cheapest Tier Loses The summary tier answers three questions by detail for 0.07 gigabytes — 0.02 gigabytes per question, the cheapest number on the whole pipeline. A decision that looked only at this number would reduce everything to the summary. The losses stand in the list by name, and each one's reason is a separate design item. Because resolution rises to 3600 seconds, **Q1** (which service slowed down), **Q11** (why the work-order queue backed up), and partly **Q8** drop — it was measured in the third lesson that a ninety-second degradation in a one-hour window shows up in neither the average nor the percentile. **Q2** drops because the `bucket` label is dropped: without a distribution bucket, the tail cannot be computed. **Q8** and **Q9** drop because the `instance` label is rolled up to the service — which instance restarted and which run was slow can no longer be asked. **Q5**, **Q6**, and **Q10** drop because log lines are not kept at all. All eight of the eight questions close silently. At the summary tier, these queries do not error; they return an empty result or a single aggregate value. This has the same shape as the truncation finding from the fifth lesson: the signal of the loss disappears along with the loss itself. ## Tier Layouts Side by Side Now that the individual cost of each tier is known, the layout itself can be compared: in which layout does the same twelve-question set get answered for how many gigabytes? ```js // policy.mjs — tier layouts are compared: how many gigabytes does the same question set cost under each. import { seriesCount, LOG_LINES, LINE_BYTES, TIERS, QUESTIONS, fits, gb } from "./tiers.mjs"; const resize = (name, start, end) => { // move a tier to a different age range const b = TIERS.find((x) => x[0] === name); return [name, start, end, b[3], b[4], b[5]]; }; const bytes = (b) => seriesCount(b[4]) * Math.floor(86400 / b[3]) * (b[2] - b[1]) * 16 + seriesCount(b[4]) * 120 + (b[2] - b[1]) * b[5].reduce((t, type) => t + LOG_LINES[type] * LINE_BYTES[type], 0); const POLICIES = [ ["raw only, 7 days", [resize("raw", 0, 7)]], ["raw only, 30 days", [resize("raw", 0, 30)]], ["raw only, 395 days", [resize("raw", 0, 395)]], ["three tiers 7 / 90 / 395", [resize("raw", 0, 7), resize("downsampled", 7, 90), resize("summary", 90, 395)]], ["raw 7 + summary 395 (no middle)", [resize("raw", 0, 7), resize("summary", 7, 395)]], ["raw 30 + summary 395", [resize("raw", 0, 30), resize("summary", 30, 395)]], ]; console.log("policy total GB answered GB/question unanswered"); const results = POLICIES.map(([name, bs]) => { const t = bs.reduce((s, b) => s + bytes(b), 0); const y = QUESTIONS.filter((s) => bs.some((b) => fits(b, s))); console.log(`${name.padEnd(33)}${gb(t).padStart(9)}${`${y.length}/12`.padStart(13)}${gb(t / y.length).padStart(13)} ` + QUESTIONS.filter((s) => !y.includes(s)).map(([k]) => k).join(" ")); return [name, t, y.length]; }); const [, threeTierBytes, threeTierAnswered] = results[3], [, noMidBytes, noMidAnswered] = results[4]; console.log(`\nthe middle tier's contribution: ${gb(threeTierBytes - noMidBytes)} GB more, ${threeTierAnswered - noMidAnswered} new question`); console.log(`the cost of keeping everything raw: ${gb(results[2][1] - noMidBytes)} GB more,` + ` ${results[2][2] - noMidAnswered} new question`); ``` ``` policy total GB answered GB/question unanswered raw only, 7 days 30.31 9/12 3.37 Q4 Q7 Q12 raw only, 30 days 129.92 10/12 12.99 Q7 Q12 raw only, 395 days 1710.56 11/12 155.51 Q12 three tiers 7 / 90 / 395 103.35 11/12 9.40 Q12 raw 7 + summary 395 (no middle) 30.41 11/12 2.76 Q12 raw 30 + summary 395 130.00 11/12 11.82 Q12 the middle tier's contribution: 72.94 GB more, 0 new question the cost of keeping everything raw: 1680.16 GB more, 0 new question ``` Four layouts answer the same eleven questions, and there is a fifty-six-fold price difference between them. The cheapest is the layout that **never sets up** the middle tier: seven days of raw, then straight to summary. Total 30.41 gigabytes, 2.76 per question. The middle layer of the three-tier layout holds 72.94 gigabytes extra and opens **zero new questions**. The reason was seen in the fourth section: the downsampled tier's only contribution was Q4, and because Q4 only needs the `version` label at hourly resolution, it is also answered by the summary tier's own detail. The middle tier was built only on the intuition that "seven days is too little, ninety days would be better"; counted against the question set, it has no payoff. The "keep everything raw" layout answers the same eleven questions for 1710.56 gigabytes — 1680 gigabytes extra, zero new questions. Its cost per question is 155.51 gigabytes, fifty-six times the cheapest layout. These two lines pin this course's rule to a number: data that answers no question, no matter how carefully it is collected, is a cost. ## Summary - Four items define a tier, and all four can close off a question: age range, resolution, dropped label, and kept line type. - The raw tier carries 11/12 questions by detail but answers 9/12 because of its age range; keeping the detail is not enough, it has to sit alongside old enough data. - Bytes accumulate on the log side: of the raw tier's 30.31 GB, 26.18 is log, 4.14 is metric; at the downsampled tier, even though request lines are discarded, the log holds 70.51 GB. - Marginal cost reverses: the downsampled tier opens one question (Q4) for 72.96 GB, the summary tier opens one question (Q7) for 0.07 GB — a thousandfold difference, both exactly one question. - The cheapest tier (summary) answers 3/12 questions at 0.02 GB/question and silently loses eight questions by name: Q1, Q11 from resolution; Q2 from the `bucket` label; Q8, Q9 from the `instance` label; Q5, Q6, Q10 from log lines. The queries do not error, they return an empty or a single aggregate value. - In the layout comparison, four different layouts answer the same 11/12: the cheapest is the one that never sets up the middle tier (30.41 GB, 2.76 GB/question). The middle tier holds 72.94 GB extra and opens zero questions; "keep everything raw" holds 1680 GB extra and opens zero questions. ## Next Step This topic took data end to end: how it is collected, which labels it is stored with, which aggregation it is queried through, which layer of the log pipeline loses what, how text is turned into a searchable field, and how much it costs at which tier as it ages. Eleven of the twelve questions became answerable somewhere, and the price of every answer is known. One question remains, and it appeared at no tier, in no label set, on no log line: which reading a bill was generated from. Asking this means asking for a single transaction's **path** from the reading collector to the validator, from there to billing and the work order. A metric cannot carry it, because a transaction ID's cardinality is unbounded; a log line cannot carry it, because every service generates its own identity and the lines cannot be linked to each other. Not a single byte of the hundred gigabytes collected shows that path. The next topic starts from this gap: what does making a request's path across services visible require, and which questions does it open.