Build Artifact Management

The previous lesson measured eight gates and never counted what the 713 runs produced: the build artifact the packaging gate outputs. At the end of every run that finishes green, there is a package, and that package is what goes to production. This package has two questions, and both of them are gate questions.

The first is identity: what is the package named after, and can that name be traced back to the source that produced the package? The second is retention: how long does the package stay, and when a failure occurs in production, is the version to roll back to still there? The two are linked: a deleted output can only be rebuilt if its identity resolves back to the source. The build artifact repository’s growth in bytes was measured in the Fundamental Concepts topic, and the resolution of the movable tag in the Images topic; those measurements are not repeated here, and repository size is counted as the number of retained outputs.

Output History

The model is the history of outputs the regional measurement network’s pipeline produces over twenty workdays. Both the network and the history are fictional. Durations are model minutes.

PD19 — 400 build artifacts are produced in twenty workdays, one every 24 minutes. 70 percent of runs are from the main branch, the rest from feature branches; 85 percent of feature branches are deleted once the work is done. 6 percent of runs are a rerun of the same commit. Every fifteenth run carries a version tag.

PD20 — six identity schemes are compared. What information a scheme carries is written with a single field: does the identity text contain the commit hash.

PD21 — resolving from an identity back to the source requires three conditions together: the identity must be unique, it must carry the commit or a pipeline log must still exist (0.92 probability), and the commit must be accessible in the repository. A deleted branch’s commit is not accessible.

PD22 — 40 rollback events occur in twenty workdays. How many production outputs back the target is drawn with these weights: 1 step back 50, 2 steps 18, 3 steps 10, 5 steps 7, 10 steps 6, 25 steps 5, 60 steps 4.

PD23 — rolling back from a retained output takes 6 model minutes; if the output has been deleted, rebuilding from source takes 55 minutes and is possible only if the identity resolves to the source. If it does not resolve, the request cannot be met.

PD24 — repository size is measured by the number of retained outputs, not in bytes.

// history.mjs — fictional history of build artifacts the pipeline produces over twenty workdays
const generator = (seed) => () => {
  seed = (seed * 1664525 + 1013904223) >>> 0;
  return seed / 4294967296;
};

export const OUTPUTS = 400, INTERVAL = 24;    // PD19: one output every 24 minutes, a 9600-minute window
export const LOG = 0.92;                      // PD21: probability the pipeline log still exists

const hex = (r, n) => Array.from({ length: n }, () => '0123456789abcdef'[Math.floor(r() * 16)]).join('');

export const generateHistory = (seed) => {
  const r = generator(seed);
  const branches = new Map();
  const list = [];
  let commit = hex(r, 12), version = [1, 4, 0];
  for (let i = 0; i < OUTPUTS; i++) {
    const rerun = r() < 0.06;              // PD19: a rerun of the same commit
    if (!rerun) commit = hex(r, 12);
    const mainBranch = r() < 0.70;
    const branch = mainBranch ? 'main' : `feature-${Math.floor(r() * 40)}`;
    if (!branches.has(branch)) branches.set(branch, { deleted: mainBranch ? false : r() < 0.85 });
    if (i % 15 === 0 && i > 0) version = [version[0], version[1], version[2] + 1];
    list.push({ number: i + 1, commit, branch, mainBranch, t: i * INTERVAL,
      version: version.join('.'), tagged: i % 15 === 0,
      accessible: !branches.get(branch).deleted,   // a deleted branch's commit does not stay in the repository
      logged: r() < LOG });
  }
  return list;
};

// PD20: identity schemes. commit = does the identity carry the commit, make = the identity text.
export const schemes = [
  { name: 'movable tag', commit: false, make: () => 'latest' },
  { name: 'run number', commit: false, make: (c) => `${c.number}` },
  { name: 'semantic version', commit: false, make: (c) => c.version },
  { name: 'branch + run number', commit: false, make: (c) => `${c.branch}-${c.number}` },
  { name: 'semantic version + commit', commit: true, make: (c) => `${c.version}+${c.commit.slice(0, 7)}` },
  { name: 'commit + run number', commit: true, make: (c) => `${c.commit.slice(0, 7)}-${c.number}` },
];

// PD21: resolving back to the source requires the identity to be unique, to carry the commit or
// have a pipeline log, and for the commit to be accessible in the repository.
export const resolve = (scheme, history) => {
  const counter = new Map();
  for (const c of history) counter.set(scheme.make(c), (counter.get(scheme.make(c)) ?? 0) + 1);
  let unique = 0, source = 0, sourceMain = 0, main = 0;
  const resolved = new Set();
  for (const c of history) {
    const single = counter.get(scheme.make(c)) === 1;
    if (single) unique++;
    const resolves = single && (scheme.commit || c.logged) && c.accessible;
    if (resolves) { source++; resolved.add(c.number); }
    if (c.mainBranch) { main++; if (resolves) sourceMain++; }
  }
  return { unique, source, sourceMain, main, resolved,
    colliding: history.length - unique, length: scheme.make(history[history.length - 1]).length };
};

// identity.mjs — uniqueness per scheme and the rate at which identity resolves back to source
import { generateHistory, schemes, resolve, OUTPUTS } from './history.mjs';

const history = generateHistory(30719);
const pad = (x, w) => String(x).padStart(w);
const pct = (a, b) => `${(100 * a / b).toFixed(1)}`;
const stranded = history.filter((c) => !c.accessible).length;

console.log(`outputs ${OUTPUTS}, main branch ${history.filter((c) => c.mainBranch).length}, `
  + `stranded in a deleted branch ${stranded}, reruns ${OUTPUTS - new Set(history.map((c) => c.commit)).size}`);
console.log(`\n${'scheme'.padEnd(28)}${pad('length', 9)}${pad('colliding', 11)}${pad('resolves to source', 20)}${pad('on main', 11)}`);
for (const scheme of schemes) {
  const c = resolve(scheme, history);
  console.log(`${scheme.name.padEnd(28)}${pad(c.length, 9)}${pad(c.colliding, 11)}`
    + `${pad(`${c.source} (${pct(c.source, OUTPUTS)}%)`, 20)}${pad(`${pct(c.sourceMain, c.main)}%`, 11)}`);
}

outputs 400, main branch 274, stranded in a deleted branch 103, reruns 28

scheme                         length  colliding  resolves to source    on main
movable tag                         6        400            0 (0.0%)       0.0%
run number                          3          0         269 (67.3%)      90.9%
semantic version                    6        400            0 (0.0%)       0.0%
branch + run number                 8          0         269 (67.3%)      90.9%
semantic version + commit          14         50         261 (65.3%)      87.6%
commit + run number                11          0         297 (74.3%)     100.0%

What Identity Carries

The first two rows separate out the uniqueness condition of identity. The movable tag is rewritten on every run: 400 of the 400 outputs share the same identity, and none of them can resolve back to source. Semantic version is in the same position: a number that increments every fifteenth run cannot tell the fourteen outputs in between apart.

The run number is unique, and resolution back to source rises to 67.3 percent. The remaining gap has two sources: since the identity does not carry the commit, the pipeline log it relies on stands only with 0.92 probability, and the commit of the 103 outputs in deleted branches is not accessible in the repository. The second constraint is eliminated in the main-branch row — for the run number, resolution on the main branch is 90.9 percent, meaning the remaining 9.1 points to reach 100 percent come entirely from a lost pipeline log.

The last scheme, which carries the commit hash, closes this gap: 100 percent on the main branch. When the identity carries the source within itself, no outside record is needed. The overall rate still sitting at 74.3 percent is a matter of deleted feature branches, and that is not an identity problem. The fifth scheme is a middle ground: when semantic version is combined with the commit hash, 50 outputs collide, because rerunning the same commit produces the same identity. The identity shows the source, but it does not show which run produced it; adding the run number solves exactly that.

The Retention Rule Is a Gate

The retention rule decides which output stays available to roll back to, which makes it a gate, and it is measured with the same three numbers. The scan below runs seven rules under two identity schemes.

// retention.mjs — rollback window and repository size when the retention rule is scanned
import { generateHistory, schemes, resolve, OUTPUTS, INTERVAL } from './history.mjs';

const DAY_MIN = 480, DIRECT_MIN = 6, REBUILD_MIN = 55;   // PD23: rollback and rebuild
const generator = (seed) => () => {
  seed = (seed * 1664525 + 1013904223) >>> 0;
  return seed / 4294967296;
};

// PD22: 40 rollback events in twenty workdays; how many production outputs back the target is.
const DEPTH = [[1, 50], [2, 18], [3, 10], [5, 7], [10, 6], [25, 5], [60, 4]];
const generateEvents = (seed, releases) => {
  const r = generator(seed), total = DEPTH.reduce((a, d) => a + d[1], 0);
  return Array.from({ length: 40 }, () => {
    const T = 1200 + r() * (OUTPUTS * INTERVAL - 1200);
    let x = r() * total, depth = 1;
    for (const [d, w] of DEPTH) { x -= w; if (x <= 0) { depth = d; break; } }
    const priorReleases = releases.filter((c) => c.t < T);
    return { T, depth, target: priorReleases[priorReleases.length - depth] ?? null };
  });
};

const rules = [
  { name: 'last 10 runs', keep: (c, n) => c.number > n - 10 },
  { name: 'last 25 runs', keep: (c, n) => c.number > n - 25 },
  { name: 'last 50 runs', keep: (c, n) => c.number > n - 50 },
  { name: 'last 100 runs', keep: (c, n) => c.number > n - 100 },
  { name: 'last 7 days', keep: (c, n, T) => T - c.t <= 7 * DAY_MIN },
  { name: 'tagged + last 25', keep: (c, n) => c.tagged || c.number > n - 25 },
  { name: 'all', keep: () => true },
];

const history = generateHistory(30719);
const releases = history.filter((c) => c.mainBranch);
const events = generateEvents(60417, releases);
const now = OUTPUTS * INTERVAL;

const measure = (rule, resolved) => {
  let direct = 0, rebuilt = 0, unmet = 0, min = 0;
  for (const e of events) {
    if (e.target === null) { unmet++; continue; }
    const n = history.filter((c) => c.t < e.T).length;
    if (rule.keep(e.target, n, e.T)) { direct++; min += DIRECT_MIN; }
    else if (resolved.has(e.target.number)) { rebuilt++; min += REBUILD_MIN; }
    else unmet++;
  }
  const retained = history.filter((c) => rule.keep(c, OUTPUTS, now)).length;
  const remainingReleases = releases.filter((c) => rule.keep(c, OUTPUTS, now));
  // unbroken depth: how many are retained without a gap, counting back from the latest release
  let unbroken = 0;
  for (let i = releases.length - 1; i >= 0 && rule.keep(releases[i], OUTPUTS, now); i--) unbroken++;
  // wasted: retained output that is not the target of any rollback event
  const targets = new Set(events.filter((e) => e.target).map((e) => e.target.number));
  const wasted = history.filter((c) => rule.keep(c, OUTPUTS, now) && !targets.has(c.number)).length;
  return { direct, rebuilt, unmet, min, retained, unbroken, wasted,
    window: remainingReleases.length ? (now - remainingReleases[0].t) / DAY_MIN : 0 };
};

const pad = (x, w) => String(x).padStart(w);
console.log(`releases ${releases.length}, rollback events ${events.length}, `
  + `deepest target ${Math.max(...events.map((e) => e.depth))} versions back`);
console.log(`\n${'retention rule'.padEnd(20)}${pad('retained', 10)}${pad('window days', 13)}${pad('unbroken depth', 21)}${pad('wasted retained', 17)}`);
for (const r of rules) {
  const m = measure(r, new Set());
  console.log(`${r.name.padEnd(20)}${pad(m.retained, 10)}${pad(m.window.toFixed(2), 13)}${pad(m.unbroken, 21)}${pad(m.wasted, 17)}`);
}
for (const name of ['movable tag', 'commit + run number']) {
  const scheme = schemes.find((x) => x.name === name);
  const { resolved } = resolve(scheme, history);
  console.log(`\nidentity scheme: ${name} (resolves to source for ${resolved.size} outputs)`);
  console.log(`${'retention rule'.padEnd(20)}${pad('direct', 10)}${pad('rebuild', 16)}${pad('unmet', 14)}${pad('recovery min', 14)}`);
  for (const r of rules) {
    const m = measure(r, resolved);
    console.log(`${r.name.padEnd(20)}${pad(m.direct, 10)}${pad(m.rebuilt, 16)}${pad(m.unmet, 14)}${pad(m.min, 14)}`);
  }
}

releases 274, rollback events 40, deepest target 60 versions back

retention rule        retained  window days       unbroken depth  wasted retained
last 10 runs                10         0.50                    8               10
last 25 runs                25         1.25                   17               23
last 50 runs                50         2.50                   31               47
last 100 runs              100         5.00                   67               89
last 7 days                140         7.00                   98              125
tagged + last 25            50        20.00                   17               44
all                        400        20.00                  274              365

identity scheme: movable tag (resolves to source for 0 outputs)
retention rule          direct         rebuild         unmet  recovery min
last 10 runs                35               0             5           210
last 25 runs                37               0             3           222
last 50 runs                38               0             2           228
last 100 runs               40               0             0           240
last 7 days                 40               0             0           240
tagged + last 25            37               0             3           222
all                         40               0             0           240

identity scheme: commit + run number (resolves to source for 297 outputs)
retention rule          direct         rebuild         unmet  recovery min
last 10 runs                35               5             0           485
last 25 runs                37               3             0           387
last 50 runs                38               2             0           338
last 100 runs               40               0             0           240
last 7 days                 40               0             0           240
tagged + last 25            37               3             0           387
all                         40               0             0           240

The Advertised Window versus the Unbroken Depth

The most instructive row of the second table is tagged + last 25. This rule retains 50 outputs and advertises a 20-day window: the oldest retained production output sits at the start of the window. The last 50 runs rule, which retains the same 50 outputs, has a window of only 2.5 days. The eight-times-wider window looks better. Rollback events say the opposite: the tagged rule meets 37 requests directly, while last 50 runs meets 38.

The unbroken depth column explains the difference. The tagged rule’s window has holes in it; only 17 production outputs are retained without a gap counting backward, against 31 for the last 50 runs rule. A rollback request does not say “go back to twenty days ago,” it says “go back three versions”; if the version in between has been deleted, the window’s length is useless. A retention rule is defended by unbroken depth, not by window length.

The wasted retained column is the gate’s false hold. The all rule retains 400 outputs, 365 of which are never the target of a rollback event, and it does not meet a single request more than the last 100 runs rule: four times the repository for zero marginal gain. The difference between last 100 runs and last 50 runs, meanwhile, is 2 requests.

Where Identity and Retention Intersect

The last two tables combine the two gates. Under the movable-tag scheme, the last 10 runs rule cannot meet five requests: the output has been deleted, and since the identity cannot resolve to the source, it cannot be rebuilt either. When the same retention rule runs with an identity that carries the commit hash, unmet requests drop to zero; the five requests are met with a 55-minute rebuild each, and recovery time rises from 210 minutes to 485 minutes.

The trade-off is clear. A strict retention rule brings the repository down from 400 to 10 and costs 245 extra minutes of recovery time — but only if the identity scheme allows it. If the identity does not resolve, the same strictness loses five requests entirely. The identity scheme sets the upper bound on how strict the retention rule can be.

The retention gate’s four numbers can now be written. For the last 50 runs rule: what it holds is 38 rollback requests, what it alone holds — when chosen over last 25 — is 1 request, what it misses is 2 requests, what it falsely holds is 47 outputs, and what it adds is a 50-output repository and 338 minutes of recovery time.

Summary

• Under the movable-tag and semantic-version schemes, all 400 outputs collide and the rate of resolving back to source is zero; the run number is unique and gives 67.3 percent, and the scheme carrying the commit hash gives 74.3 percent.
• The main-branch row isolates the constraint: the run number stays at 90.9 percent (a lost pipeline log), while the identity carrying the commit rises to 100 percent. The gap in the overall rate belongs to deleted feature branches, and no identity scheme closes it.
• The advertised window is misleading: the tagged + last 25 rule shows a 20-day window, but its unbroken depth is 17, and with the same repository size it meets one fewer request than the rule that gives a 2.5-day window.
• Quadrupling the repository can have zero marginal value: the all rule retains 400 outputs, 365 of which are never used, and it meets the same 40 requests as last 100 runs.
• The identity scheme sets retention’s upper bound: the last 10 runs rule produces zero unmet requests and 485 minutes with a resolvable identity, and five unmet requests with one that does not resolve.

Next Step

The four lessons so far counted what the pipeline holds and always wrote duration as a cost item: the minutes a gate adds, the minutes waiting costs, the minutes a rebuild takes. Duration itself was never the target. Yet the first lesson’s table said something — as a run gets cheaper, the best merge interval shifts toward more frequent merging, meaning shortening pipeline time does not just make the wait easier, it also changes the gates’ economics. The next lesson looks directly at this duration: what work can be reused between stages, where the limit of running in parallel sits, and past what point adding resources stops shortening the pipeline.