Versioning and Lifecycle Rules

The previous lesson’s class distribution rested on an assumption: every object’s annual read count is known and it is placed by hand into the right class. Across millions of objects this is neither known nor done by hand. Object storage offers a lifecycle rule for this: objects under a prefix change class or get deleted once they reach a given age. The rule looks at age, not access, and this distinction carries a measurable cost.

The same lesson also introduces versioning. In this model versioning is a bucket-level switch, and once turned on there is no going back; with real providers the level of that switch and whether it can be turned off vary, but the old versions that pile up stay everywhere. Both are measured together: versioning grows bytes, the lifecycle rule shrinks them.

• ST13 — The 36-month object lifetime is a model: monthly production, average size, and the age at which reads halve are written into the model. There is no randomness, every number comes from a cohort.
• ST14 — A transition is a request, billed per object; the minimum retention period starts when the class is entered.
• ST15 — The rule only looks at age; it does not see whether an object was actually read.

// lifecycle/model.mjs — a 36-month object lifetime is a MODEL: no randomness, every number comes from a cohort.
// Fees are a fictional unit; classes are called only by access frequency.
export const MONTHS = 36;
export const CLASS = {                 // storage unit/GiB/month, request and transition unit/request, retrieval unit/GiB
  frequent: { storage: 0.0230, request: 0.00040 / 1000, retrieval: 0.000, minMonths: 0, transition: 0.0050 / 1000 },
  infrequent: { storage: 0.0125, request: 0.00100 / 1000, retrieval: 0.010, minMonths: 1, transition: 0.0100 / 1000 },
  archive: { storage: 0.0010, request: 0.00500 / 1000, retrieval: 0.030, minMonths: 6, transition: 0.0500 / 1000 },
};
// births: objects created each month; base: monthly reads at age zero; halfLife: the month reads halve
export const FAMILIES = [
  { name: "meter-readings", births: 120_000, avg: 2400, base: 3.0, halfLife: 1.5 },
  { name: "invoice-documents", births: 7_200, avg: 380 * 1024, base: 2.2, halfLife: 4 },
  { name: "field-photos", births: 3_500, avg: Math.round(3.6 * 1024 * 1024), base: 0.9, halfLife: 2 },
  { name: "work-order-records", births: 26_000, avg: 11 * 1024, base: 4.0, halfLife: 3 },
];
export const gib = (bytes) => bytes / 1024 ** 3;
export const reads = (f, age) => f.base * Math.pow(0.5, age / f.halfLife);
// Rule: change class at an age (month) threshold; remove the object at the delete threshold.
export const RULE = {
  "no rule": { transitions: [], delete: null },
  moderate: { transitions: [[3, "infrequent"], [12, "archive"]], delete: 30 },
  aggressive: { transitions: [[1, "infrequent"], [3, "archive"]], delete: 7 },
  "early-archive": { transitions: [[1, "archive"]], delete: null },
};
export const classAt = (rule, age) => {    // the class occupied at this age and the age it was entered
  let name = "frequent", entered = 0;
  for (const [threshold, c] of rule.transitions) if (age >= threshold) { name = c; entered = threshold; }
  return { c: CLASS[name], name, entered };
};

The Rule Looks at Age, Not Access

// lifecycle/rule.mjs — lifecycle rules are run over cohorts across 36 months
import { MONTHS, CLASS, FAMILIES, RULE, gib, reads, classAt } from "./model.mjs";

const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
const s = (x, n) => String(x).padStart(n);
const empty = () => ({ storage: 0, request: 0, retrieval: 0, transitionFee: 0, transitionCount: 0, debt: 0, deleted: 0, remainingBytes: 0 });
const run = (rule, family) => {                       // one family's 36-month lifetime
  const r = empty();
  for (let birth = 0; birth < MONTHS; birth++)        // the cohort born each month is aged month by month
    for (let month = birth; month < MONTHS; month++) {
      const age = month - birth;
      if (rule.delete !== null && age >= rule.delete) {   // minimum retention debt falls due at deletion
        if (age === rule.delete) {
          const at = classAt(rule, age - 1);
          r.debt += family.births * Math.max(0, at.c.minMonths - (age - at.entered)) * at.c.storage * gib(family.avg);
          r.deleted += family.births;
        }
        continue;
      }
      const at = classAt(rule, age), readCount = reads(family, age);
      if (rule.transitions.some(([t]) => t === age)) { r.transitionCount += family.births; r.transitionFee += family.births * at.c.transition; }
      r.storage += family.births * gib(family.avg) * at.c.storage;
      r.request += family.births * readCount * at.c.request;
      r.retrieval += family.births * readCount * gib(family.avg) * at.c.retrieval;
      if (month === MONTHS - 1) r.remainingBytes += family.births * family.avg;
    }
  r.total = r.storage + r.request + r.retrieval + r.transitionFee + r.debt;
  return r;
};

const T = {};                                          // T[rule][family] and T[rule].all
for (const [name, rule] of Object.entries(RULE)) {
  T[name] = { all: empty() };
  for (const f of FAMILIES) {
    const r = T[name][f.name] = run(rule, f);
    for (const a of ["storage", "request", "retrieval", "transitionFee", "transitionCount", "debt", "deleted", "remainingBytes", "total"])
      T[name].all[a] = (T[name].all[a] ?? 0) + r[a];
  }
}
console.log(`${FAMILIES.reduce((a, f) => a + f.births, 0).toLocaleString("en-US")} objects/month, ${MONTHS} months; ` +
  `reads halve with age, no randomness. Fees are a fictional unit.`);
console.log("\nrule           |    GiB | transitions | storage | retrieval | transition fee  | total");
console.log("---------------|--------|-------------|---------|-----------|-----------------|------");
for (const [name, r] of Object.entries(T).map(([a, x]) => [a, x.all]))
  console.log(`${name.padEnd(14)} | ${g(gib(r.remainingBytes), 6, 1)} | ${s(r.transitionCount.toLocaleString("en-US"), 11)} | ` +
    `${g(r.storage, 7)} | ${g(r.retrieval, 9)} | ${g(r.transitionFee, 15)} | ${g(r.total, 6)}`);

console.log("\n36-month total, family by family (fictional unit):");
console.log("family             |" + Object.keys(RULE).map((a) => a.padStart(14)).join(" |") + " | cheapest rule");
console.log("-------------------|" + Object.keys(RULE).map(() => "-".repeat(15)).join("|") + "|--------------");
for (const f of FAMILIES) {
  const v = Object.keys(RULE).map((a) => T[a][f.name].total);
  console.log(`${f.name.padEnd(18)} |` + v.map((x) => g(x, 14)).join(" |") +
    ` | ${Object.keys(RULE)[v.indexOf(Math.min(...v))]}`);
}

const [none, moderate, aggressive, early] = ["no rule", "moderate", "aggressive", "early-archive"].map((a) => T[a]);
console.log(`\nwith no rule, ${gib(none.all.remainingBytes).toFixed(1)} GiB accumulates by month 36, no object is ever deleted. ` +
  `The moderate rule deletes ${moderate.all.deleted.toLocaleString("en-US")} objects, cutting storage by ` +
  `${(100 * (1 - moderate.all.storage / none.all.storage)).toFixed(1)} points, but multiplies the transition-fee ` +
  `total by ${(moderate.all.total / none.all.total).toFixed(2)}x.`);
const mixed = FAMILIES.reduce((a, f) => a + Math.min(...Object.keys(RULE).map((k) => T[k][f.name].total)), 0);
console.log(`the mixed rule chosen by family costs ${mixed.toFixed(2)} units: below the best single rule's ` +
  `(${Math.min(...Object.values(T).map((x) => x.all.total)).toFixed(2)}). A rule is written by object type.`);
console.log(`the cost of an early transition is in the retrieval column: moderate ${moderate.all.retrieval.toFixed(2)}, early-archive ` +
  `${early.all.retrieval.toFixed(2)} units (${(early.all.retrieval / moderate.all.retrieval).toFixed(1)}x); early-archive saves ` +
  `${(moderate.all.storage - early.all.storage).toFixed(2)} in storage and loses ` +
  `${(early.all.retrieval + early.all.request - moderate.all.retrieval - moderate.all.request).toFixed(2)} units in retrieval and requests.`);
console.log(`the aggressive rule deletes at month 7, and archive carries a minimum ${CLASS.archive.minMonths}-month debt: ` +
  `${aggressive.all.debt.toFixed(2)} units are paid for objects that no longer exist.`);

156,700 objects/month, 36 months; reads halve with age, no randomness. Fees are a fictional unit.

rule           |    GiB | transitions | storage | retrieval | transition fee  | total
---------------|--------|-------------|---------|-----------|-----------------|------
no rule        |  556.4 |           0 |  236.74 |      0.00 |            0.00 | 257.94
moderate       |  463.6 |   8,931,900 |   92.06 |     13.05 |          239.75 | 379.90
aggressive     |  108.2 |  10,655,600 |   28.07 |     30.28 |          313.40 | 460.64
early-archive  |  556.4 |   5,484,500 |   22.53 |     58.48 |          274.22 | 540.24

36-month total, family by family (fictional unit):
family             |       no rule |      moderate |    aggressive | early-archive | cheapest rule
-------------------|---------------|---------------|---------------|---------------|--------------
meter-readings     |         17.46 |        203.62 |        292.04 |        318.98 | no rule
invoice-documents  |         41.19 |         37.32 |         38.33 |         56.13 | moderate
field-photos       |        188.63 |         83.22 |         44.92 |         51.49 | aggressive
work-order-records |         10.66 |         55.74 |         85.34 |        113.64 | no rule

with no rule, 556.4 GiB accumulates by month 36, no object is ever deleted. The moderate rule deletes 940,200 objects, cutting storage by 61.1 points, but multiplies the transition-fee total by 1.47x.
the mixed rule chosen by family costs 110.36 units: below the best single rule's (257.94). A rule is written by object type.
the cost of an early transition is in the retrieval column: moderate 13.05, early-archive 58.48 units (4.5x); early-archive saves 69.52 in storage and loses 195.39 units in retrieval and requests.
the aggressive rule deletes at month 7, and archive carries a minimum 6-month debt: 0.90 units are paid for objects that no longer exist.

With no rule, 556.4 GiB accumulates over thirty-six months and no object is ever deleted; storage is nearly the whole total. The moderate rule cuts this item by 61.1%, deletes 940,200 objects, pulls bytes down to 463.6 GiB — and raises the total. The reason is in the transition-fee column: 8,931,900 transition requests cost 239.75 units, more than what was gained in storage.

The family table shows where the rule pays off. For field photos the aggressive rule brings the total from 188.63 down to 44.92 units: the object is large, so the transition request carries many bytes. For meter readings the same rule raises it from 17.46 to 292.04 units; for a 2.3 KiB object, the transition request costs more than storing that object for years. This is why the rule is written at the prefix level: the mixed rule, at 110.36 units, is under half of the best single rule.

Early transition opens a third item. The early-archive rule gains 69.52 units in storage and loses 195.39 units in retrieval and requests: when an object still being read in its first month is sent to archive, every read incurs a retrieval fee. The objects the aggressive rule deletes at month seven, meanwhile, never fill archive’s six-month period; 0.90 units are paid for objects that no longer exist.

Versioning: Where Deletion Doesn’t Delete

• ST16 — The update rate is a model: overwrites grow rarer as an object ages, and every overwrite leaves an old version behind.
• ST17 — Versioning is bucket-level; while it is on, deletion places a delete marker and turns the current version into an old version.
• ST18 — Old versions are cleaned up only by a separate lifecycle rule.

// lifecycle/versioning.mjs — what deleting and disabling do while versioning is on is counted (MODEL)
import { MONTHS, CLASS, FAMILIES, gib } from "./model.mjs";

// Update MODEL: overwrites grow rarer as an object ages. Every overwrite leaves an old version behind.
const UPDATE = { "work-order-records": { base: 0.85, halfLife: 2 }, "invoice-documents": { base: 0.30, halfLife: 3 } };
const DELETE_MONTH = 30, VERSION_LIFE = 3, DISABLE_MONTH = 24;    // old-version rule: clean up after 3 months
const selected = FAMILIES.filter((f) => UPDATE[f.name]);
const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
const s = (x, n) => String(x).padStart(n);

// Every month: current-object bytes and produced old-version bytes (by the month produced) are summed.
const run = ({ versioning, versionRule, disable }) => {
  const r = { current: 0, currentBytes: 0, oldVersions: 0, oldVersionBytes: 0, storage: 0, deleted: 0, freed: 0 };
  for (const f of selected) {
    const u = UPDATE[f.name], produced = [];             // produced[m] = old versions born that month
    for (let m = 0; m < MONTHS; m++) produced.push(0);
    for (let m = 0; m < MONTHS; m++) {
      let current = 0;
      for (let birth = 0; birth <= m; birth++) {
        const age = m - birth;
        if (age >= DELETE_MONTH) {                        // delete: bytes are freed if versioning is off
          if (age === DELETE_MONTH) {
            r.deleted += f.births;
            if (versioning && !(disable && m >= DISABLE_MONTH)) produced[m] += f.births;
            else r.freed += f.births * f.avg;
          }
          continue;
        }
        current += f.births;
        const write = f.births * u.base * Math.pow(0.5, age / u.halfLife);
        if (versioning && !(disable && m >= DISABLE_MONTH)) produced[m] += write;
      }
      let old = 0;                                        // old versions still standing that month
      for (let k = 0; k <= m; k++) if (!versionRule || m - k < VERSION_LIFE) old += produced[k];
      r.storage += gib(f.avg) * (current + old) * CLASS.frequent.storage;
      if (m === MONTHS - 1) {
        r.current += current; r.currentBytes += current * f.avg;
        r.oldVersions += old; r.oldVersionBytes += old * f.avg;
      }
    }
  }
  return r;
};

const D = {
  "versioning off": run({ versioning: false }),
  "versioning on": run({ versioning: true }),
  "on + version rule": run({ versioning: true, versionRule: true }),
  "disabled at month 24": run({ versioning: true, disable: true }),
};
console.log(`${selected.map((f) => f.name).join(" + ")}; ${MONTHS} months, delete at month ${DELETE_MONTH}, ` +
  `old-version rule ${VERSION_LIFE} months. Update rate is in the model.`);
console.log("\nstate                 | current objects | old versions | old version GiB  | 36-month storage");
console.log("----------------------|-----------------|--------------|------------------|-----------------");
for (const [name, r] of Object.entries(D))
  console.log(`${name.padEnd(21)} | ${s(Math.round(r.current).toLocaleString("en-US"), 15)} | ` +
    `${s(Math.round(r.oldVersions).toLocaleString("en-US"), 12)} | ${g(gib(r.oldVersionBytes), 16, 1)} | ${g(r.storage, 16)}`);

const [off, on, rule, disabled] = Object.values(D);
console.log(`\ndeletion experiment: ${Math.round(off.deleted).toLocaleString("en-US")} objects deleted. Bytes freed — ` +
  `versioning off ${gib(off.freed).toFixed(1)} GiB, on ${gib(on.freed).toFixed(1)} GiB.`);
console.log(`  old-version to current-data ratio ${(on.oldVersionBytes / on.currentBytes).toFixed(2)}; the version rule ` +
  `recovers ${gib(on.oldVersionBytes - rule.oldVersionBytes).toFixed(1)} GiB.`);
console.log(`disable experiment: disabled at month ${DISABLE_MONTH}. No new versions are produced, the accumulated ` +
  `${gib(disabled.oldVersionBytes).toFixed(1)} GiB still stands and is billed for ${MONTHS - DISABLE_MONTH} more months;`);
console.log(`  deletions after disabling free ${gib(disabled.freed).toFixed(1)} GiB; the only thing that can be done ` +
  `for the earlier ones is to write a separate version rule.`);

invoice-documents + work-order-records; 36 months, delete at month 30, old-version rule 3 months. Update rate is in the model.

state                 | current objects | old versions | old version GiB  | 36-month storage
----------------------|-----------------|--------------|------------------|-----------------
versioning off        |         996,000 |            0 |              0.0 |            42.75
versioning on         |         996,000 |    3,070,001 |            165.9 |           102.12
on + version rule     |         996,000 |      357,335 |             22.4 |            53.68
disabled at month 24  |         996,000 |    1,839,941 |             93.6 |            92.51

deletion experiment: 199,200 objects deleted. Bytes freed — versioning off 17.3 GiB, on 0.0 GiB.
  old-version to current-data ratio 1.92; the version rule recovers 143.5 GiB.
disable experiment: disabled at month 24. No new versions are produced, the accumulated 93.6 GiB still stands and is billed for 12 more months;
  deletions after disabling free 17.3 GiB; the only thing that can be done for the earlier ones is to write a separate version rule.

The current object count is the same across all four rows: 996,000. What the application sees does not change; what changes is the bill. Once versioning is turned on, 3,070,001 old versions accumulate, taking up 165.9 GiB — 1.92 times the current data, bytes produced by overwriting itself.

The deletion experiment settles the concept in place. 199,200 objects are deleted: with versioning off, 17.3 GiB is freed; with it on, 0.0 GiB. The deletion succeeds, the object no longer appears in listings — and the bytes stay put, because the deleted object has turned into an old version. Once a separate rule is written for old versions, 143.5 GiB comes back, and storage drops from 102.12 to 53.68 units; until that rule is written, turning on versioning silently makes deletion meaningless.

The disable experiment shows the unrecoverability. Versioning is disabled in month twenty-four: no new version is produced, subsequent deletions free bytes again, but the accumulated 93.6 GiB stands and is billed for twelve more months. Versioning is this course’s second unrecoverable item: turning it on is one switch, turning it off is not the same switch. The provider only runs the rule; the thresholds, the deletion, and the old-version lifetime all stay with you.

Summary

• The rule looks at age, not access: the early-archive rule gains 69.52 units in storage and loses 195.39 units in retrieval and requests, a 4.5x increase in the retrieval item.
• The transition fee is per object: the moderate rule cuts storage by 61.1% but multiplies the total by 1.47x. A rule is written at the prefix level, not the bucket level.
• The mixed rule costs 110.36 units, the best single rule 257.94: the aggressive rule pays off for field photos, the same rule costs seventeen times as much for meter readings.
• Early deletion incurs minimum retention debt: 0.90 units are paid for objects deleted at month seven, and the objects no longer exist.
• With versioning on, deleting 199,200 objects frees 0.0 GiB, and 165.9 GiB of old versions accumulate. Disabling it does not clean up the accumulated versions, it only stops producing new ones.

Next Step

These three lessons share a single assumption: data is written whole and read whole. One part of the fictional measurement network does not work this way — the collector process makes hundreds of small writes a second to a database file, and every write changes a few kilobytes in the middle of the file. The 190x write amplification measured in the first lesson makes this workload alone impossible on object storage. The next lesson moves to the type that presents data as byte blocks. Three things will be measured — the input/output budget that a volume’s type and size give, the point where workload profiles saturate that budget, and the design trap opened by size and performance being tied together.