Pull-Based Deployment

The previous lesson counted what keeping the desired state in the repository buys: unrecorded changes fell from 63 to 5, drifted items at period’s end from 45 to 1. One item stayed silent in that measurement — who carries the repository’s writing over to the cluster. In every arrangement so far, the carrier has been the pipeline: the pipeline connects to the cluster’s control plane, identifies itself with credentials, and writes objects. This is called push.

There is a second way: a reconciler agent running inside the cluster reads the repository, compares it against its own environment’s actual state, and closes the gap itself. This is called pull. The mechanics of the reconciliation loop were measured in the cluster architecture topic — the period, the lagging snapshot, the oscillation — and are not repeated here. The only thing this lesson measures is what the source being a repository changes across these two directions of transport.

The measurement is again over the regional measurement network and is fictional; what runs is a model built with node.

RD7. The model carries 4 services × 3 environments = 12 namespaces, 6 object kinds, 72 object instances, and 4 operations. RD8. In push, a single subject — the pipeline runner — is authorized across all 12 namespaces. In pull, each environment has one agent, authorized only in its own environment’s 4 namespaces. RD9. 60 manual changes occur over thirty days; the generator is self-written, seed 31337. Three classes: the field the deployment writes on every run, the field the deployment never writes, and the object the agent does not own. RD10. A portion of the manual changes are genuine interventions that close a real outage; a portion are short-lived — the operator reverts them themselves within minutes. RD11. The deployment schedule is fixed in writing: the four services deploy every 720, 480, 2880, and 4320 minutes respectively. RD12. On the twelfth day there is a 24-hour pipeline outage; the deployment does not run during that window, the agent does.

The Authority Surface and the Direction of the Connection

// authority.mjs — counts the authority surface in push and pull arrangements. No
// randomness; the count runs over a model's object tree, not a real cluster.
const SERVICE = ["collector", "verifier", "billing", "workorder"];
const ENV = ["development", "staging", "production"];
const OBJECT = ["deployment", "service", "config", "secret-claim", "scheduled-job", "network-rule"];
const WRITE = ["write", "delete", "patch"], OP = ["read", ...WRITE];

const namespace = ENV.flatMap((e) => SERVICE.map((s) => `${e}/${s}`));
const instance = namespace.flatMap((n) => OBJECT.map((o) => `${n}/${o}`));   // object instances

const arrangement = {
  push: {                                        // pipeline writes to the cluster
    subject: "pipeline runner", scope: namespace,
    exposed: ENV.length,                          // each environment's control plane is externally reachable
    pipelineHeld: ENV.length,                     // cluster identities the pipeline holds
    direction: "pipeline -> cluster (outside-in)",
  },
  pull: {                                        // agent reads the repository
    subject: "one agent per environment", scope: namespace.filter((n) => n.startsWith("production/")),
    exposed: 0, pipelineHeld: 0,
    direction: "cluster -> repository (inside-out)",
  },
};

const count = (k) => ({ authEntries: k.length * OBJECT.length * OP.length,
  writeEntries: k.length * OBJECT.length * WRITE.length,
  writable: instance.filter((o) => k.some((n) => o.startsWith(n + "/"))).length });

console.log(`model: ${SERVICE.length} services x ${ENV.length} environments = ${namespace.length} namespaces, ` +
  `${OBJECT.length} object kinds, ${instance.length} object instances, ${OP.length} operations`);
console.log("\n" + "arrangement".padEnd(13) + "subject".padEnd(28) + "namespace".padStart(11) +
  "auth".padStart(6) + "write".padStart(7) + "writable if compromised".padStart(26));
for (const [ad, d] of Object.entries(arrangement)) {
  const s = count(d.scope);
  console.log(ad.padEnd(13) + d.subject.padEnd(28) + String(d.scope.length).padStart(11) +
    String(s.authEntries).padStart(6) + String(s.writeEntries).padStart(7) +
    String(s.writable).padStart(26));
}
console.log("\n" + "arrangement".padEnd(13) + "exposed control planes".padStart(25) +
  "pipeline-held cluster identities".padStart(35) + "  connection direction");
for (const [ad, d] of Object.entries(arrangement))
  console.log(ad.padEnd(13) + String(d.exposed).padStart(25) + String(d.pipelineHeld).padStart(35) +
    "  " + d.direction);

model: 4 services x 3 environments = 12 namespaces, 6 object kinds, 72 object instances, 4 operations

arrangement  subject                       namespace  auth  write   writable if compromised
push         pipeline runner                      12   288    216                        72
pull         one agent per environment             4    96     72                        24

arrangement     exposed control planes   pipeline-held cluster identities  connection direction
push                                 3                                  3  pipeline -> cluster (outside-in)
pull                                 0                                  0  cluster -> repository (inside-out)

The table’s first reading is misleading. In the pull arrangement, 96 authority entries appear per agent, versus 288 in push; but since there is one agent per environment, the total authority entries in pull are also 3 × 96 = 288. The authority surface does not shrink, it splits. What changes is the last column: a compromised pipeline runner can write to all 72 object instances, a compromised agent only to its own environment’s 24. The production environment, among the three, sits behind a separate subject, and a problem in the development environment cannot reach it.

The second table shows a sharper difference. In push, three control planes must be externally reachable and the pipeline holds three cluster identities; in pull, both are zero, because the cluster establishes the connection. The operational counterpart of this is concrete: in push, wherever the pipeline runs has to be treated as if it were inside the security boundary — the pipeline runner is a subject that can write to three production environments. In pull, the pipeline’s only authority is to write to the repository; the clusters do not even recognize it.

This is not a gate count, but it determines the ground the gates stand on: in push, the repository-defined desired state principle is a contract — if it wants to, the pipeline can bypass the gates and write directly. In pull, the principle is a constraint: there is no authority to write from outside the cluster.

Who Closes the Drift, and When

// drift.mjs — a 30-day manual-change flow is counted in two arrangements: push (pipeline
// writes to cluster) and pull (agent reads repository). Model runs; generator self-written, seed visible.
const generator = (t) => { let s = t >>> 0;                     // same linear congruential generator
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const DURATION = 30 * 1440, OUTAGE = [17820, 19260];            // day 12, a 24-hour pipeline outage
const SCHEDULE = { collector: 720, verifier: 480, billing: 2880, workorder: 4320 };
const SERVICE = Object.keys(SCHEDULE);

function events(seed = 31337, count = 60) {
  const r = generator(seed), l = [];
  for (let i = 0; i < count; i++) {
    const p = r();
    l.push({ t: Math.floor(r() * DURATION), service: SERVICE[Math.floor(r() * 4)],
      class: p < 0.45 ? "written" : p < 0.85 ? "not-written" : "out-of-scope",
      genuine: r() < 0.2,                                       // a fix that closes a real outage
      lifespan: r() < 0.25 ? 5 + Math.floor(r() * 25) : Infinity });// reverted by the operator themselves
  }
  return l;
}

const nextDeploy = (service, t) => {                            // the outage window is skipped
  const p = SCHEDULE[service];
  for (let k = Math.floor(t / p) + 1; k * p <= DURATION; k++)
    if (!(k * p >= OUTAGE[0] && k * p < OUTAGE[1])) return k * p;
  return null;
};

function run(l, U) {                                            // U=null means push arrangement
  const s = { closed: 0, marginal: 0, delay: 0, unseen: 0, missed: 0,
    wrong: 0, remaining: 0, windowOpen: 0, windowClosed: 0 };
  for (const o of l) {
    const closesAt = U === null ? (o.class === "written" ? nextDeploy(o.service, o.t) : null)
      : (o.class === "out-of-scope" ? null : (Math.floor(o.t / U) + 1) * U);
    if (o.lifespan !== Infinity && (closesAt === null || o.t + o.lifespan < closesAt)) { s.unseen++; continue; }
    const end = (closesAt === null || closesAt > DURATION) ? DURATION : closesAt;
    if (o.t < OUTAGE[1] && end > OUTAGE[0]) s.windowOpen++;        // open drift throughout the outage
    if (closesAt !== null && closesAt >= OUTAGE[0] && closesAt < OUTAGE[1]) s.windowClosed++;
    if (closesAt === null || closesAt > DURATION) { s.remaining++; if (o.class === "out-of-scope") s.missed++; continue; }
    s.closed++; s.delay += closesAt - o.t;
    if (o.class === "not-written") s.marginal++;                // the class push can never close
    if (o.genuine) s.wrong++;                                   // a genuine fix reverted/overwritten
  }
  s.avgDelay = s.closed ? s.delay / s.closed : 0;
  s.propagation = U === null ? 3 : U / 2 + 1;                   // commit -> apply lag
  return s;
}

const W = [13, 8, 10, 11, 8, 8, 7, 11, 13];
const row = (h) => h.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0])).join("");

const l = events();
console.log(`seed 31337 | ${l.length} manual changes / 30 days | ` +
  `${l.filter((o) => o.genuine).length} are genuine fixes closing a real outage`);
console.log(`class: written ${l.filter((o) => o.class === "written").length}, ` +
  `not-written ${l.filter((o) => o.class === "not-written").length}, ` +
  `out of scope ${l.filter((o) => o.class === "out-of-scope").length}`);
console.log("\n" + row(["arrangement", "closed", "marginal", "avg delay", "unseen", "missed",
  "wrong", "remaining", "propagation"]));
console.log(row(["push", ...(({ closed, marginal, avgDelay, unseen, missed, wrong, remaining,
  propagation }) => [closed, marginal, avgDelay.toFixed(0), unseen, missed, wrong, remaining,
  propagation])(run(l, null))]));
for (const U of [1, 5, 15, 60, 240]) {
  const x = run(l, U);
  console.log(row([`pull U=${U}`, x.closed, x.marginal, x.avgDelay.toFixed(0), x.unseen,
    x.missed, x.wrong, x.remaining, x.propagation.toFixed(1)]));
}
const missedDeploy = SERVICE.reduce((a, sv) => a + Object.keys([...Array(DURATION / SCHEDULE[sv] | 0)])
  .filter((k) => (+k + 1) * SCHEDULE[sv] >= OUTAGE[0] && (+k + 1) * SCHEDULE[sv] < OUTAGE[1]).length, 0);
const i = run(l, null), c = run(l, 15);
console.log(`\n24-hour pipeline outage: ${missedDeploy} scheduled deploys missed, ` +
  `agent ran ${(OUTAGE[1] - OUTAGE[0]) / 15} rounds`);
console.log(`open drift throughout the window — push ${i.windowOpen}, pull(U=15) ${c.windowOpen}; ` +
  `closed within the window — push ${i.windowClosed}, pull ${c.windowClosed}`);

seed 31337 | 60 manual changes / 30 days | 10 are genuine fixes closing a real outage
class: written 32, not-written 17, out of scope 11

arrangement    closed  marginal  avg delay  unseen  missed  wrong  remaining  propagation
push               26         0       1206      16       6      4         18            3
pull U=1           49        17          1       5       6      9          6          1.5
pull U=5           49        17          3       5       6      9          6          3.5
pull U=15          48        17          8       6       6      9          6          8.5
pull U=60          42        14         30      12       6      8          6         31.0
pull U=240         39        12        135      15       6      7          6        121.0

24-hour pipeline outage: 5 scheduled deploys missed, agent ran 96 rounds
open drift throughout the window — push 11, pull(U=15) 7; closed within the window — push 0, pull 3

The push row does not look like a gate, because it is not one. 26 of the sixty manual changes closed, but closing is not a matter of noticing — it is an overwrite: the drift is erased because the next deployment rewrites that field. The average duration is 1206 minutes — twenty hours — and the entire duration depends on the deployment schedule. No one noticed the drift, no one kept a record; 18 items remained drifted at period’s end.

The reconciliation round’s gate ledger stands against this. In a fifteen-minute period, 48 drifts closed, with an average delay of 8 minutes. The marginal column gives the work this gate does only by itself: 17 drifts in fields the deployment never writes at all. These are items that, under the push arrangement, would never close over thirty days — a manually added network rule, a manually enlarged replica count, a manually disabled scheduled job. The gate’s value is not in the 48, it is in the 17.

What it misses is just as clear: 6 drifts in object kinds the agent does not own were missed at every period, and shrinking the period did not change this. Scope is not a setting, it is a definition; the agent cannot see what it does not manage. The second class of misses is the unseen column: changes made between two rounds and reverted by the operator themselves. 5 of them were never seen at U=1, 15 at U=240. An agent with a long period is not merely slow; it keeps no record at all of short-lived interventions.

What the Reconciliation Round Wrongly Holds

The wrong column is this gate’s price. Over thirty days, 10 manual changes are genuine interventions closing a real outage. In the pull arrangement, 9 of these were reverted within an average of 9 minutes — the agent looks at the repository’s writing, does not know why the intervention was made, and brings the outage back. In push, the same number is 4, and the reversion happens silently, thirty-four hours later on average.

Both are wrongful holding, but their forms differ, and this is where the confusion starts. In pull, the wrongful hold is fast and visible: the agent leaves a record, the person who made the intervention sees it reverted within nine minutes and learns the correct behavior — commit the fix to the repository. In push, the wrongful hold is slow and silent: the intervention runs for thirty-four hours, then a deployment overwrites it, and no one connects the two. The gate’s count is twice as high in pull; the harm the gate does is greater in push, because it does not teach.

The correct tuning for this gate is not lengthening the period. In the U=240 row, wrongful reversion falls from 9 to 7, but marginal value also drops from 17 to 12 and unseen drift rises to 15. The correct tuning for the gate is scope: if the fields where urgent intervention happens are moved into the set the agent does not own, both wrongful reversion and learning disappear. The measurement gives this trade-off’s price — the drifts in those fields are now added to the 6 items no one closes at all.

When the Pipeline Goes Down, and the Price of Pull

The 24-hour pipeline outage separates the two arrangements’ resilience. 5 scheduled deploys were missed in that window; in the push arrangement, none of the 11 drifts left open throughout the window closed, because the closing mechanism was the deployment itself. The agent ran 96 rounds in the same window and closed 3 of the 7 open drifts. The distinction is this: in push, the pipeline both carries the new change and closes drift, so the pipeline’s outage stops both at once. In pull, the pipeline only carries the new change; preserving the last known state is independent of the pipeline. New releases stop in both arrangements — this is not an advantage, it is a separation of concerns.

Pull’s bill is in the last column. In push, a commit turning into an application takes 3 minutes; in pull it is half the period plus application, i.e. 8.5 minutes at U=15 and 121 minutes at U=240. Keeping the reconciliation round’s marginal value high requires shortening the period, but a short period improves propagation while placing a constant read load on the repository and the control plane. The second item is the agent itself: one agent per environment, three separate upgrade windows, three separate sets of authority entries, and a separate watch to notice when the agent itself goes down. The repository-defined desired state principle does not come bundled with the agent; the agent is only one of the ways to implement the principle, and it demands its own upkeep.

Summary

• Total authority entries are 288 in both arrangements; in pull, authority splits across three subjects, and the object instances a compromised subject can write to fall from 72 to 24. Externally exposed control planes drop from 3 to 0.
• In push, drift goes unnoticed, it gets overwritten: 26 of 60 manual changes were erased after an average of 1206 minutes, and 18 were still standing at period’s end.
• The reconciliation round’s marginal value is 17 drifts — fields the deployment never writes. The gate’s value is not in the 48 it closes, it is in the 17 push could never close.
• The gate misses two classes: 6 objects the agent does not own (independent of period), and changes made and reverted between two rounds (5 at U=1, 15 at U=240).
• Pull wrongly reverted a genuine emergency intervention 9 times in an average of 9 minutes; push silently overwrote it 4 times after thirty-four hours; pull’s wrongful hold is more frequent, push’s is more harmful because it does not teach.
• In the 24-hour pipeline outage, push closed 0 of 11 open drifts, the agent closed 3 of 7; the price is propagation delay rising from 3 minutes to 8.5 minutes, and the upkeep of one agent per environment.

Next Step

Two lessons measured the desired state staying in the repository and being carried from there to the cluster. One item remains, and its entry into the repository is problematic from the start: the secrets services need. The database connection, the credentials going out to the measurement network’s remote endpoints, and the signing keys cannot sit in the repository as plain text — but if they do not sit in the repository, the desired state stays incomplete and the agent cannot apply it. The next lesson measures the sealed secret: where the key sits, how many subjects can decrypt it, how many records get re-encrypted and how many files get touched when the key is rotated, and how long a stale encrypted value remains readable.