Scheduler–Agent–Supervisor

Every failure in the previous lesson announced itself clearly: a step returned an error, and the compensation chain either worked or did not. Most failures in a real workflow do not speak up like that. The orchestrator crashes after sending a command, a step stalls without replying, a compensation call is tried once and dropped. Such a job appears to be “running” forever, and because no one asks, no one notices.

Scheduler–agent–supervisor closes this gap with three roles. The scheduler writes the workflow’s state to a durable record and starts the next step; where the job stands lives not in memory but in a record that can still be read after the process crashes. The agent runs the step on the remote service and writes the result to the record. The supervisor scans the record at regular intervals; it finds steps that were started but have not recorded a result within the allotted time, re-drives them, and starts compensation once the retry threshold is exceeded. The time allotted to each started step is called the lease.

Why the Three Roles Are Separated

The rationale for the separation fits in one sentence: no role can notice its own failure. When the agent crashes, there is no party left to say “I crashed”; when the scheduler crashes, no one is left to wait for the result of the steps it started. Noticing is handed to a third party that does no work itself and looks only at the durable record.

The model separates two kinds of failure. Stall: the agent has run the job but failed to record the result — the call went unanswered, or the process stopped before recording. Persistent failure: the step never passes on any attempt, because the contract record is missing or the tariff is undefined. The two look identical — the record has not landed — and the supervisor can only tell them apart by retrying.

// supervisor/run.mjs — scheduler, agent, and supervisor; a round-based in-process model. A round
// is an abstract step. Stall and persistent failure are triggered by parameters, not measured durations.
export const STEP = ["tariff", "discount", "invoice", "notification"];
export const JOBS = Array.from({ length: 12 }, (_, i) => `S${i + 1}`);
export const STALLED = { S2: "discount", S5: "tariff", S9: "invoice" };  // agent did the work, record did not land
export const PERSISTENT = { S7: "discount" };                          // step never passes on any attempt

export function run({ supervisor = true, scan = 2, lease = 2, retryLimit = 3, round = 80 }) {
  const jobs = JOBS.map((job) => ({ job, step: 0, state: "waiting", started: 0, attempt: 0 }));
  const s = { completed: 0, steps: 0, redriven: 0, compensatedSteps: 0, wasted: 0,
    recordsRead: 0, recovery: [], finishedRound: 0 };

  for (let t = 1; t <= round; t++) {
    for (const k of jobs) {                                // agent: the step that started the previous round
      if (k.state !== "running" || t !== k.started + 1) continue;
      const st = STEP[k.step];
      if (PERSISTENT[k.job] === st) continue;              // the record never lands
      if (STALLED[k.job] === st && k.attempt === 0) continue; // the work was done, the record did not land
      s.steps += 1; k.step += 1; k.attempt = 0; k.state = "waiting";
    }
    if (supervisor && t % scan === 0) {                    // supervisor: finds steps whose time has expired
      s.recordsRead += jobs.length;
      for (const k of jobs) {
        if (k.state !== "running" || t - k.started < lease) continue;
        s.redriven += 1;
        s.recovery.push(t - (k.started + 1));              // rounds elapsed since the expected record
        if (STALLED[k.job] === STEP[k.step]) s.wasted += 1;    // the step will run a second time
        k.attempt += 1; k.state = "waiting";
      }
    }
    for (const k of jobs) {                                // scheduler: starts the next step
      if (k.state !== "waiting") continue;
      if (k.step >= STEP.length) { k.state = "done"; s.completed += 1; s.finishedRound = t; continue; }
      if (k.attempt >= retryLimit) {                        // threshold exceeded: completed steps are compensated
        s.compensatedSteps += k.step; k.state = "compensated"; s.finishedRound = t; continue;
      }
      k.state = "running"; k.started = t;
    }
    if (jobs.every((k) => k.state === "done" || k.state === "compensated")) break;  // all jobs stopped
  }
  const avg = s.recovery.length ? s.recovery.reduce((a, b) => a + b, 0) / s.recovery.length : 0;
  return { ...s, halfFinished: jobs.filter((k) => k.state === "running").length,
    compensatedJobs: jobs.filter((k) => k.state === "compensated").length,
    avgRecovery: avg, worstRecovery: s.recovery.length ? Math.max(...s.recovery) : 0 };
}

// supervisor/measure.mjs — a run without and with the supervisor, then a scan of the scan interval
import { run, JOBS, STEP, STALLED, PERSISTENT } from "./run.mjs";

const D = [["no supervisor", run({ supervisor: false })], ["supervisor on", run({ supervisor: true })]];
const MEASURES = [["completed", "completed jobs"], ["steps", "steps run"],
  ["redriven", "steps re-driven"], ["wasted", "wasted steps"],
  ["compensatedJobs", "compensated jobs"], ["compensatedSteps", "compensated steps"],
  ["halfFinished", "half-finished jobs"], ["finishedRound", "last job round"]];

console.log(`${JOBS.length} jobs, ${STEP.length} steps; ${Object.keys(STALLED).length} step(s) stalled ` +
  `(${Object.entries(STALLED).map(([i, a]) => `${i}@${a}`).join(", ")}), ` +
  `${Object.keys(PERSISTENT).length} step(s) with a persistent failure (${Object.entries(PERSISTENT).map(([i, a]) => `${i}@${a}`).join(", ")})`);
console.log();
console.log(`${"measure".padEnd(24)}${D.map(([a]) => a.padStart(14)).join("")}`);
for (const [k, label] of MEASURES)
  console.log(`${label.padEnd(24)}${D.map(([, r]) => String(r[k]).padStart(14)).join("")}`);

console.log(`\n${"scan".padStart(7)}${"jobs recovered".padStart(15)}${"avg recovery".padStart(14)}` +
  `${"worst".padStart(9)}${"last round".padStart(11)}${"records read".padStart(14)}`);
for (const scan of [1, 2, 4, 8]) {
  const r = run({ supervisor: true, scan });
  console.log(`${String(scan).padStart(7)}${String(r.completed).padStart(15)}` +
    `${r.avgRecovery.toFixed(2).padStart(14)}${String(r.worstRecovery).padStart(9)}` +
    `${String(r.finishedRound).padStart(11)}${String(r.recordsRead).padStart(14)}`);
}

12 jobs, 4 steps; 3 step(s) stalled (S2@discount, S5@tariff, S9@invoice), 1 step(s) with a persistent failure (S7@discount)

measure                  no supervisor supervisor on
completed jobs                       8            11
steps run                           36            45
steps re-driven                      0             6
wasted steps                         0             3
compensated jobs                     0             1
compensated steps                    0             1
half-finished jobs                   4             0
last job round                       5             8

   scan jobs recovered  avg recovery    worst last round  records read
      1             11          1.00        1          8            96
      2             11          1.33        2          8            48
      4             11          2.33        4         12            36
      8             11          5.67        7         24            36

Reading the Numbers

The no-supervisor column shows what a silent failure means. Eight of twelve jobs completed, 36 steps ran, and the last job finished on round 5. On the surface everything looks fine: the error counter is zero, compensation is zero, no one saw an exception. The bottom row carries the real result: 4 jobs are half-finished and sit in the “running” state forever. The scheduler started the step, the agent did not reply, and there is no one to ask the question.

In the supervisor-on column, 3 of the 4 jobs are recovered, one is compensated, and half-finished jobs drop to 0. Completed jobs rise from 8 to 11 — not 12, because job S7 has a persistent failure, and the supervisor counted it as past the threshold after driving it three times; the one completed step of that job was compensated. The supervisor does not turn a failed job into a successful one; it turns an uncertain state into a determined one. A job either finishes or is compensated; the count of jobs left in the “running” state is zero.

Two rows show the cost. Steps run rises from 36 to 45, and 3 of those are wasted steps: the agent had already done the work, it had only failed to record it. The supervisor cannot know this; the only way to tell a stall apart from a persistent failure is to retry. This is why the scheduler–agent–supervisor arrangement has a mandatory precondition: steps must be idempotent. Running the same step a second time must not produce an additional effect; if a step does not carry this property naturally, it is protected with an idempotency key. Idempotency and the idempotency key were established in the Web API Design and The Data Access Layer and Business Logic courses; here they precondition the supervisor’s work. A non-idempotent invoice step produces three extra invoice lines across three wasted runs.

The table below shows what the scan interval buys. Recovered jobs are 11 across all four intervals: the supervisor finds them sooner or later. What changes is how late it finds them. Average recovery delay rises from 1.00 to 5.67 rounds, the worst case from 1 to 7; the batch’s last job shifts from round 8 to round 24. In exchange, the records the supervisor reads drop from 96 to 36. When the interval doubles, the delay roughly doubles and the reads are halved; frequent scanning speeds up recovery, infrequent scanning reduces the read load on the durable record. This trade-off is deterministic, read directly from the model, not a duration measurement dependent on the environment.

Back to the Estimate

The model’s 12 jobs are not a volume. The volume comes from K01: 4,000 workflows a day and four steps per job, that is, 16,000 steps a day. The previous lesson’s KK2 assumption (the end-of-day workflow’s persistent failure ratio of 0.02) carries forward. This lesson adds one more assumption.

KK3 — a step’s stall ratio is 0.005. The rationale is that agent process restarts, calls cut off before recording, and steps that time out and lose their reply produce a steady baseline. It is not added to K01’s table; its sensitivity is calculated at 0.01.

// supervisor/cost.mjs — applies the model's ratios to K01's end-of-day job volume
import { run, STEP } from "./run.mjs";

const INVOICE_LINES = 4000;                 // K01: daily invoice lines (computed value) = daily workflows
const STEPS_PER_DAY = INVOICE_LINES * STEP.length;
const KK2 = 0.02;                           // previous lesson: end-of-day workflow's persistent failure ratio
const RETRY_LIMIT = 3;                      // the supervisor's threshold (model parameter)
const r = run({ supervisor: true, scan: 2 });
const wastedRatio = r.wasted / r.redriven;   // the share of re-drives that go to waste

console.log(`daily workflows ${INVOICE_LINES}, steps/day ${STEPS_PER_DAY}, ` +
  `model's wasted re-drive ratio ${wastedRatio.toFixed(3)}`);
console.log();
console.log(`${"KK3".padStart(6)}${"stalled steps/day".padStart(19)}${"re-drives/day".padStart(19)}` +
  `${"wasted steps/day".padStart(20)}${"half-finished, no supervisor".padStart(31)}${"job share".padStart(10)}`);
for (const KK3 of [0.005, 0.01]) {          // KK3: a step's stall ratio
  const stalled = STEPS_PER_DAY * KK3;
  const persistent = INVOICE_LINES * KK2;
  const halfFinished = stalled + persistent;
  console.log(`${KK3.toFixed(3).padStart(6)}${stalled.toFixed(0).padStart(19)}` +
    `${(stalled + persistent * RETRY_LIMIT).toFixed(0).padStart(19)}${stalled.toFixed(0).padStart(20)}` +
    `${halfFinished.toFixed(0).padStart(31)}${`${(100 * halfFinished / INVOICE_LINES).toFixed(2)}%`.padStart(10)}`);
}

console.log(`\nthe cost of the scan interval (model measurement, deterministic):`);
for (const scan of [1, 8]) {
  const k = run({ supervisor: true, scan });
  console.log(`  scan ${scan}: avg recovery ${k.avgRecovery.toFixed(2)} rounds, ` +
    `worst ${k.worstRecovery}, last round ${k.finishedRound}, records read ${k.recordsRead}`);
}

daily workflows 4000, steps/day 16000, model's wasted re-drive ratio 0.500

   KK3  stalled steps/day      re-drives/day    wasted steps/day   half-finished, no supervisor job share
 0.005                 80                320                  80                            160     4.00%
 0.010                160                400                 160                            240     6.00%

the cost of the scan interval (model measurement, deterministic):
  scan 1: avg recovery 1.00 rounds, worst 1, last round 8, records read 96
  scan 8: avg recovery 5.67 rounds, worst 7, last round 24, records read 36

Once the number is read, the decision makes itself. A stall ratio of five in a thousand means 80 stalled steps a day; together with persistent failures, an unsupervised arrangement leaves 160 workflows half-finished a day, which is 4 percent of the daily 4,000 invoice lines. If the ratio rises to 0.01, that becomes 240 and 6 percent. None of these jobs increment an error counter; their first symptom is that month-end reconciliation does not balance.

The same table carries the supervisor’s cost: 320 re-drives a day, 80 of them wasted steps. The wasted-run ratio in the model is 0.500, because half of the re-drives come from retrying a persistently failed job. The eighty are absorbed by idempotency; if they are not, 80 extra invoice lines appear a day — that is, the supervisor produces new errors equal to exactly half of the error it set out to fix. The scheduler–agent–supervisor arrangement is therefore not a pattern on its own; together with idempotent steps, it is a single decision.

Summary

• The three roles are separated because no role can notice its own failure: the scheduler writes the job to a durable record and starts the step, the agent runs the step, the supervisor only looks at the record.
• In the unsupervised run, 4 of 12 jobs stayed in the “running” state without producing a single error; the error counter was zero and completed jobs stopped at 8.
• The supervisor turns an uncertain state into a determined one: 3 jobs were recovered, 1 job was compensated, half-finished jobs dropped from 4 to 0, and completed jobs rose from 8 to 11.
• The cost sits in steps run (36 → 45), 3 of which are wasted runs; this is why idempotent steps are a precondition of the arrangement.
• The scan interval trades recovery delay against read load: as the interval rises from 1 to 8, average recovery climbs from 1.00 to 5.67 rounds and the last job shifts from round 8 to round 24, while records read drop from 96 to 36.
• Back to K01: at 16,000 steps/day and KK3 = 0.005, the unsupervised arrangement leaves 160 half-finished jobs a day (4 percent of invoice lines); the supervisor zeroes this out and in exchange brings 320 re-drives and 80 wasted runs a day.

Next Step

This lesson’s supervisor was treated as a single party. In reality, the supervisor, and the scheduler that starts the end-of-day job, both run on more than one node — because a supervisor running on a single node becomes, when it goes down, exactly the kind of workflow no one notices. But the scheduler’s job is singular: end-of-day billing must be started exactly once. If two nodes start it at the same time, two invoice lines are produced for the same seller-day, and that can only be fixed by hand, not by compensation. The next lesson builds the arrangement that gives this singular responsibility to a single node: one of the nodes taking over responsibility for the duration of a lease, how many rounds a handover takes when that node goes down, and counting the rounds in which a misconfigured lease runs two nodes at once.