Pipeline as Code

The previous two lessons shortened pipeline duration and decided where it runs. Both touched the same object, but the object itself was never questioned: eight stages, the dependencies between them, which stage is a gate, and how many slots are used are written down somewhere. Raising the slot count from two to four, and removing a gate, both mean touching that written thing.

Where it is written down is a choice between two options. The pipeline can be configured by hand through an interface: fields are filled in, saved, and take effect immediately. Or it can sit as a file in the repository: the change arrives as a commit, is reviewed, and is versioned. This lesson compares the two forms on four numbers — traceability, the fraction of changes that pass without review, the rollback step, and the duplication of the definition across five services.

PD37. The schema, the validator, and the file form are defined in this lesson; the field names pipeline, slots, stage, command, deps, gate are not the fields of a real tool. PD38. Twenty pipeline changes stand in for a quarter of the fictional measurement network; six are faulty. PD39. The interface form’s picker structurally blocks three kinds of error — an unknown field cannot be written, a nonexistent stage cannot be linked to, a number field cannot take text; against that, the interface has no review step. PD40. In the repository form, changes 3, 11, and 17 passed without review on grounds of urgency. PD41. Review wait is 42 minutes, schema validation is 4 seconds; the rollback steps are counted, their minutes are fictional. PD42. Line counts are counted from the actually produced text.

The Schema Itself

The first thing gained when a pipeline definition sits as a file is this: the definition is now data, and data can be read. The schema declares fields and their types; the validator reads the schema and checks the definition.

// schema.mjs — pipeline definition schema, validator, and writer (field names defined in this lesson)
export const SCHEMA = { pipeline: "text", slots: "number", stages: "array" };
const STAGE = { name: "text", command: "text", deps: "array", gate: "yn" };
const types = { text: (d) => typeof d === "string", number: (d) => Number.isInteger(d),
  array: (d) => Array.isArray(d), yn: (d) => d === "yes" || d === "no" };
const checkFields = (obj, schema, prefix, errors) => {
  for (const key of Object.keys(obj)) if (!schema[key]) errors.push(`unknown field: ${prefix}${key}`);
  for (const [key, type] of Object.entries(schema))
    if (!(key in obj)) errors.push(`missing field: ${prefix}${key}`);
    else if (!types[type](obj[key])) errors.push(`type error: ${prefix}${key}, expected ${type}`);
};

export const validate = (def) => {
  const errors = [];
  checkFields(def, SCHEMA, "", errors);
  if (!Array.isArray(def.stages)) return errors;
  const names = def.stages.map((s) => s.name);
  for (const s of def.stages) {
    checkFields(s, STAGE, `${s.name}.`, errors);
    for (const dep of s.deps || []) if (!names.includes(dep)) errors.push(`unknown dep: ${s.name} -> ${dep}`);
  }
  const inCycle = (name, seen) => (seen.includes(name) ? true
    : (def.stages.find((s) => s.name === name)?.deps || []).some((dep) => inCycle(dep, [...seen, name])));
  for (const s of def.stages) if (inCycle(s.name, [])) { errors.push(`cycle: ${s.name}`); break; }
  return errors;
};

export const BASE = { pipeline: "reading-collector", slots: 3, stages: [
  ["dependency-resolution", "resolve", [], "no"], ["build", "build", ["dependency-resolution"], "no"],
  ["static-analysis", "analyze-static", ["dependency-resolution"], "yes"], ["unit-test", "test-unit", ["build"], "yes"],
  ["integration-test", "test-integration", ["build"], "yes"], ["packaging", "package", ["build"], "no"],
  ["contract-test", "test-contract", ["packaging"], "yes"], ["image-build", "build-image", ["packaging"], "no"],
].map(([name, command, deps, gate]) => ({ name, command, deps, gate })) };

// The definition's file form: every stage is three or four lines.
export const write = (def) => [`pipeline ${def.pipeline}`, `slots ${def.slots}`, ...def.stages.flatMap((s) => [`stage ${s.name}`,
  `  command ${s.command}`, ...(s.deps.length ? [`  deps ${s.deps.join(" ")}`] : []), `  gate ${s.gate}`])];

// Five services of the fictional measurement network: base pipeline + service-specific diffs.
export const SERVICES = [
  { name: "reading-collector", diff: [] }, { name: "verifier", diff: [["slots", 2]] },
  { name: "billing", diff: [["remove", "contract-test"]] },
  { name: "work-order", diff: [["add", { name: "latency-test", command: "test-latency", deps: ["packaging"], gate: "yes" }]] },
  { name: "nightly-batch", diff: [["slots", 4], ["remove", "static-analysis"]] },
];
export const apply = (s) => {
  const def = { ...BASE, pipeline: s.name, stages: BASE.stages.map((a) => ({ ...a })) };
  for (const [kind, data] of s.diff) {
    if (kind === "slots") def.slots = data;
    if (kind === "remove") def.stages = def.stages.filter((a) => a.name !== data);
    if (kind === "add") def.stages.push(data);
  }
  return def;
};

The validator checks four things: unknown and missing fields, type mismatches, a deps reference to a stage that does not exist, and a cycle in the dependencies. All four are decided from the definition’s text; there is no need to run the pipeline.

Two Forms

Six faulty changes are actually constructed, actually run through the validator, and for each one, what the two forms hold is written down separately.

// forms.mjs — the same pipeline in two forms: traceability, review, gate, rollback
import { validate, BASE } from "./schema.mjs";

const clone = () => ({ ...BASE, stages: BASE.stages.map((a) => ({ ...a })) });
const N = 20, PASSED = [3, 11, 17]; // three pipeline changes that pass without review
const TRACE = ["who", "when", "why", "what changed"];

// Six faulty pipeline changes. no: which change number it arrives on. iface: an error the interface
// form cannot write by construction (a picker prevents it). human: an error only a human can see.
const ERRORS = [
  ["E1 unknown field", 2, 1, 0, (def) => { def.stage = def.stages; delete def.stages; }],
  ["E2 cycle", 5, 0, 1, (def) => { def.stages[1].deps = ["packaging"]; }],
  ["E3 unknown dep", 8, 1, 0, (def) => { def.stages[3].deps = ["build-old"]; }],
  ["E4 type error (slots)", 11, 1, 0, (def) => { def.slots = "three"; }],
  ["E5 gate removed", 14, 0, 1, (def) => { def.stages.splice(2, 1); }],
  ["E6 gate disabled", 17, 0, 1, (def) => { def.stages[4].gate = "no"; }],
];

const tally = { iface: 0, repo: 0, schema: 0, alone: 0 };
console.log("error                    no  what the schema validator says              iface  repo");
for (const [name, no, iface, human, corrupt] of ERRORS) {
  const def = clone(); corrupt(def);
  const findings = validate(def), schemaCaught = findings.length > 0;
  const repoCaught = schemaCaught || (human && !PASSED.includes(no)); // a human does not look at a passed change
  tally.schema += schemaCaught; tally.iface += iface; tally.repo += repoCaught; tally.alone += repoCaught && !schemaCaught;
  console.log(name.padEnd(24) + String(no).padStart(2) + "  " + (findings[0] || "— (schema passed)").slice(0, 42).padEnd(44) +
    (iface ? "holds" : "escapes").padEnd(9) + (repoCaught ? "holds" : "escapes"));
}

const FORMS = {
  interface: { trace: TRACE.slice(0, 2), unreviewed: N, min: 35,
    rollback: ["find the change", "recall the old value", "write it back by hand", "save", "trigger manually", "verify"] },
  repo: { trace: TRACE, unreviewed: PASSED.length, min: 4, rollback: ["revert to the previous version", "let the pipeline run itself"] },
};
console.log(`\ntraceability (${N} changes x ${TRACE.length} fields):`);
for (const [name, f] of Object.entries(FORMS))
  console.log(`  ${name.padEnd(11)}${f.trace.length * N}/${TRACE.length * N} fields — ${f.trace.join(", ")}`);
console.log(`passes without review: interface ${N}/${N} (100%), repo ${PASSED.length}/${N} (${(PASSED.length * 100) / N}%)`);
console.log(`gate account (6 errors): interface ${tally.iface}/6, repo ${tally.repo}/6 (schema ${tally.schema}, review alone ${tally.alone})`);
console.log("\nrollback when the pipeline breaks:");
for (const [name, f] of Object.entries(FORMS))
  console.log(`  ${name.padEnd(11)}${f.rollback.length} steps, ${f.min} min — ${f.rollback.join(" / ")}`);
console.log(`\nthe time the gate adds: schema 4 s x ${N} = ${4 * N} s; review 42 min x ${N - PASSED.length} = ${42 * (N - PASSED.length)} min`);

error                    no  what the schema validator says              iface  repo
E1 unknown field         2  unknown field: stage                        holds    holds
E2 cycle                 5  cycle: build                                escapes  holds
E3 unknown dep           8  unknown dep: unit-test -> build-old         holds    holds
E4 type error (slots)   11  type error: slots, expected number          holds    holds
E5 gate removed         14  — (schema passed)                           escapes  holds
E6 gate disabled        17  — (schema passed)                           escapes  escapes

traceability (20 changes x 4 fields):
  interface  40/80 fields — who, when
  repo       80/80 fields — who, when, why, what changed
passes without review: interface 20/20 (100%), repo 3/20 (15%)
gate account (6 errors): interface 3/6, repo 5/6 (schema 4, review alone 1)

rollback when the pipeline breaks:
  interface  6 steps, 35 min — find the change / recall the old value / write it back by hand / save / trigger manually / verify
  repo       2 steps, 4 min — revert to the previous version / let the pipeline run itself

the time the gate adds: schema 4 s x 20 = 80 s; review 42 min x 17 = 714 min

The Gate’s Account

The table starts from an unexpected place: the interface form holds three errors by construction. An unknown field cannot be written because the field list is fixed; a nonexistent stage cannot be linked to because the choice comes from a picker; a number field cannot take text. In the repository form, these three errors can be written, and are caught only if the schema validator runs. Without a validator, the repository form is worse than the interface on these three errors.

Once the validator runs, the table turns. The schema holds four errors; the fourth is the cycle, an error the interface cannot see: when the build stage is linked to packaging, the graph closes on itself, but every field is individually valid. An interface checks fields; it does not check the graph.

Two errors remain, and both touch the gate itself: E5 deletes the static analysis stage, E6 turns the integration test’s gate field to no. Both are flawless definitions according to the schema. Only a human can ask “why does this change remove a gate?” The repository form holds E5; E6 escapes, because change 17 is one of the three that pass without review.

The total is interface 3/6, repository 5/6. The gate’s marginal value is read separately: the schema holds four errors and adds a total of 80 seconds over 20 changes. Review holds only one error alone, and holds up 17 changes for 714 minutes. Twelve hours for one defect — by this course’s measure, this is the hardest gate to defend. Its defense sits in one place: the error it holds is a change that removes a gate — exactly the kind of thing that silently erases what every measurement in this course rests on. Against that, a much cheaper way to hold the same error is visible — adding a “gate count cannot decrease” rule to the schema. This lesson does not measure that rule; but if it did, review’s marginal value would drop to zero, and the 714-minute gate could be removed.

Traceability splits on two fields. The interface records 40 of 80 fields — who and when. The two missing are “why” and “what changed,” exactly the two questions to ask when the pipeline breaks at night. The rollback table is the second bill for the same gap: in the interface, rollback’s first step is “find the change,” and its answer is written down nowhere — six steps, 35 minutes. In the repository, reverting to the previous version is two steps and 4 minutes, because “what changed” is already on record.

Five Services

The fictional measurement network’s five services use almost the same pipeline. The definition can be copied to every service, or the shared part can be separated out and only the diffs declared. The two forms’ lines are actually produced and counted.

// duplication.mjs — five services: full copy vs. shared-part form
import { SERVICES, apply, write, BASE } from "./schema.mjs";

const full = SERVICES.map((s) => ({ name: s.name, lines: write(apply(s)) }));
const totalFull = full.reduce((t, s) => t + s.lines.length, 0);

// Shared part: everything except the pipeline name. Service file: inherit + name + diffs only.
const shared = write(BASE).filter((l) => !l.startsWith("pipeline "));
const split = SERVICES.map((s) => ({
  name: s.name,
  lines: ["inherit shared", `pipeline ${s.name}`, ...s.diff.flatMap(([kind, d]) => kind === "slots" ? [`slots ${d}`]
    : kind === "remove" ? [`stage-remove ${d}`]
      : [`stage ${d.name}`, `  command ${d.command}`, `  deps ${d.deps.join(" ")}`, `  gate ${d.gate}`])],
}));
const totalSplit = shared.length + split.reduce((t, s) => t + s.lines.length, 0);

// Lines identical across all five files (counting repeats).
const count = (a) => a.reduce((m, l) => m.set(l, (m.get(l) || 0) + 1), new Map());
const counts = full.map((s) => count(s.lines));
let inAll = 0;
for (const [l, c] of counts[0]) inAll += Math.min(c, ...counts.slice(1).map((m) => m.get(l) || 0));

console.log("service              full copy  with shared part");
full.forEach((s, i) => console.log(s.name.padEnd(20) + String(s.lines.length).padStart(6) +
  String(split[i].lines.length).padStart(12)));
console.log("shared file".padEnd(20) + "     —" + String(shared.length).padStart(12));
console.log("total".padEnd(20) + String(totalFull).padStart(6) + String(totalSplit).padStart(12) +
  `   (${(((totalFull - totalSplit) * 100) / totalFull).toFixed(1)}% fewer lines)`);
console.log(`\nlines identical across all five files: ${inAll} — in the full copy, ` +
  `${inAll * 4} of the ${inAll * 5} lines are repeats`);

console.log("\nreadability cost");
console.log(`  reading one service's pipeline: full copy 1 file, ${full[1].lines.length} lines; ` +
  `shared part 2 files, ${shared.length + split[1].lines.length} lines`);
console.log(`  services affected when one line in the shared file changes: ${SERVICES.length}/${SERVICES.length}`);
console.log(`  adding a new gate to five services: full copy ${5 * 4} lines / 5 separate changes; ` +
  `shared part 4 lines / 1 change`);

service              full copy  with shared part
reading-collector       33           2
verifier                33           3
billing                 29           3
work-order              37           6
nightly-batch           29           4
shared file              —          32
total                  161          50   (68.9% fewer lines)

lines identical across all five files: 24 — in the full copy, 96 of the 120 lines are repeats

readability cost
  reading one service's pipeline: full copy 1 file, 33 lines; shared part 2 files, 35 lines
  services affected when one line in the shared file changes: 5/5
  adding a new gate to five services: full copy 20 lines / 5 separate changes; shared part 4 lines / 1 change

The gain is large: 161 lines drop to 50, a 68.9 percent reduction. Its source is plain — 24 lines are identical across all five files, and in the full copy, these lines are written 120 times, meaning 96 lines are repeats.

The cost is not visible in the line count. Reading one service’s pipeline means a single 33-line file in the full copy, and 35 lines across two files once the shared part is separated. The difference is two lines, but the line slots 2 in that service’s file says nothing on its own; what it overrides is only visible in the other file, and the effective definition sits nowhere in one whole piece.

The real cost is blast radius. A single line changed in the shared file affects all five of the five services at once — which is also the gain: adding a new gate to five services is 20 lines and five separate changes in the full copy, 4 lines and a single change with the shared part. Duplication is scattered, but it lets one service be changed on its own; sharing makes every change a joint decision of all five services. If the five pipelines are meant to genuinely stay identical, sharing is correct; if they are expected to diverge, the 96 lines of repetition are a paid price for independence.

Summary

• Once the definition is a file, it becomes data: the schema declares fields and types, and the validator catches unknown fields, type errors, a reference to a nonexistent stage, and a dependency cycle without running the pipeline.
• The interface form holds three errors by construction (the field list is fixed, the choice comes from a picker), but it does not check the graph and misses the cycle. Total: interface 3/6, repository 5/6.
• The gate’s marginal value: the schema holds four errors and adds 80 seconds over 20 changes; review holds only one error alone and holds up 17 changes for 714 minutes. The error it holds is a change that removes a gate; if a “gate count cannot decrease” rule is added to the schema, this gate becomes indefensible.
• Traceability is 40/80 fields against 80/80. The two fields missing in the interface are “why” and “what changed”; rollback’s first step is exactly that too — 6 steps and 35 minutes against 2 steps and 4 minutes.
• For five services, the full copy is 161 lines, the shared-part form is 50 (68.9 percent fewer); the 96 repeats of the 24 lines identical across all five files are removed. The cost is readability and blast radius: reading one service grows to 2 files and 35 lines, and a single line in the shared file affects all five services at once.

Next Step

The pipeline is now defined, validated, versioned, and can be rolled back. One question still remains: when does this pipeline run? The previous lesson gave a run count between 10 and 80 a day and calculated the threshold from that number, but where the number came from was never said. What starts a run is a trigger, and a trigger is a gate too: it decides whether the run happens at all. There are three types — bound to an event, bound to a schedule, and pulled by hand — and the three produce very different numbers on the same calendar: how many runs, how many of them with nothing changed, how much delay between a change and its run, how many runs overlap each other. The next lesson runs all three on the same two-week fictional calendar, then adds a path-based filter and counts, together, the drop in unnecessary runs and the real change the filter misses.