--- title: Scheduler–Agent–Supervisor source: 'https://academia.sh/en/courses/application-layer/scheduler-agent-supervisor' course: 'The Application Layer and Service Interaction' language: en updated: '2026-08-23T14:25:22+00:00' license: 'CC BY-SA 4.0' --- # Scheduler–Agent–Supervisor Making resilient a workflow whose half-finished job no one asks about: a scheduler that durably records the job, an agent that runs the step remotely, and a supervisor that finds an expired step and either re-drives or compensates it; measuring half-finished jobs, re-driven steps, and wasted steps in unsupervised and supervised runs, and the trade-off between the scan interval and recovery delay. Every failure in the previous lesson announced itself clearly: a step returned an error, and the compensation chain either worked or did not. Most failures in a real workflow do not speak up like that. The orchestrator crashes after sending a command, a step stalls without replying, a compensation call is tried once and dropped. Such a job appears to be "running" forever, and because no one asks, no one notices. **Scheduler–agent–supervisor** closes this gap with three roles. The **scheduler** writes the workflow's state to a durable record and starts the next step; where the job stands lives not in memory but in a record that can still be read after the process crashes. The **agent** runs the step on the remote service and writes the result to the record. The **supervisor** scans the record at regular intervals; it finds steps that were started but have not recorded a result within the allotted time, re-drives them, and starts compensation once the retry threshold is exceeded. The time allotted to each started step is called the **lease**. ## Why the Three Roles Are Separated The rationale for the separation fits in one sentence: **no role can notice its own failure.** When the agent crashes, there is no party left to say "I crashed"; when the scheduler crashes, no one is left to wait for the result of the steps it started. Noticing is handed to a third party that does no work itself and looks only at the durable record. The model separates two kinds of failure. **Stall**: the agent has run the job but failed to record the result — the call went unanswered, or the process stopped before recording. **Persistent failure**: the step never passes on any attempt, because the contract record is missing or the tariff is undefined. The two look identical — the record has not landed — and the supervisor can only tell them apart by retrying. ```js // supervisor/run.mjs — scheduler, agent, and supervisor; a round-based in-process model. A round // is an abstract step. Stall and persistent failure are triggered by parameters, not measured durations. export const STEP = ["tariff", "discount", "invoice", "notification"]; export const JOBS = Array.from({ length: 12 }, (_, i) => `S${i + 1}`); export const STALLED = { S2: "discount", S5: "tariff", S9: "invoice" }; // agent did the work, record did not land export const PERSISTENT = { S7: "discount" }; // step never passes on any attempt export function run({ supervisor = true, scan = 2, lease = 2, retryLimit = 3, round = 80 }) { const jobs = JOBS.map((job) => ({ job, step: 0, state: "waiting", started: 0, attempt: 0 })); const s = { completed: 0, steps: 0, redriven: 0, compensatedSteps: 0, wasted: 0, recordsRead: 0, recovery: [], finishedRound: 0 }; for (let t = 1; t <= round; t++) { for (const k of jobs) { // agent: the step that started the previous round if (k.state !== "running" || t !== k.started + 1) continue; const st = STEP[k.step]; if (PERSISTENT[k.job] === st) continue; // the record never lands if (STALLED[k.job] === st && k.attempt === 0) continue; // the work was done, the record did not land s.steps += 1; k.step += 1; k.attempt = 0; k.state = "waiting"; } if (supervisor && t % scan === 0) { // supervisor: finds steps whose time has expired s.recordsRead += jobs.length; for (const k of jobs) { if (k.state !== "running" || t - k.started < lease) continue; s.redriven += 1; s.recovery.push(t - (k.started + 1)); // rounds elapsed since the expected record if (STALLED[k.job] === STEP[k.step]) s.wasted += 1; // the step will run a second time k.attempt += 1; k.state = "waiting"; } } for (const k of jobs) { // scheduler: starts the next step if (k.state !== "waiting") continue; if (k.step >= STEP.length) { k.state = "done"; s.completed += 1; s.finishedRound = t; continue; } if (k.attempt >= retryLimit) { // threshold exceeded: completed steps are compensated s.compensatedSteps += k.step; k.state = "compensated"; s.finishedRound = t; continue; } k.state = "running"; k.started = t; } if (jobs.every((k) => k.state === "done" || k.state === "compensated")) break; // all jobs stopped } const avg = s.recovery.length ? s.recovery.reduce((a, b) => a + b, 0) / s.recovery.length : 0; return { ...s, halfFinished: jobs.filter((k) => k.state === "running").length, compensatedJobs: jobs.filter((k) => k.state === "compensated").length, avgRecovery: avg, worstRecovery: s.recovery.length ? Math.max(...s.recovery) : 0 }; } ``` ```js // supervisor/measure.mjs — a run without and with the supervisor, then a scan of the scan interval import { run, JOBS, STEP, STALLED, PERSISTENT } from "./run.mjs"; const D = [["no supervisor", run({ supervisor: false })], ["supervisor on", run({ supervisor: true })]]; const MEASURES = [["completed", "completed jobs"], ["steps", "steps run"], ["redriven", "steps re-driven"], ["wasted", "wasted steps"], ["compensatedJobs", "compensated jobs"], ["compensatedSteps", "compensated steps"], ["halfFinished", "half-finished jobs"], ["finishedRound", "last job round"]]; console.log(`${JOBS.length} jobs, ${STEP.length} steps; ${Object.keys(STALLED).length} step(s) stalled ` + `(${Object.entries(STALLED).map(([i, a]) => `${i}@${a}`).join(", ")}), ` + `${Object.keys(PERSISTENT).length} step(s) with a persistent failure (${Object.entries(PERSISTENT).map(([i, a]) => `${i}@${a}`).join(", ")})`); console.log(); console.log(`${"measure".padEnd(24)}${D.map(([a]) => a.padStart(14)).join("")}`); for (const [k, label] of MEASURES) console.log(`${label.padEnd(24)}${D.map(([, r]) => String(r[k]).padStart(14)).join("")}`); console.log(`\n${"scan".padStart(7)}${"jobs recovered".padStart(15)}${"avg recovery".padStart(14)}` + `${"worst".padStart(9)}${"last round".padStart(11)}${"records read".padStart(14)}`); for (const scan of [1, 2, 4, 8]) { const r = run({ supervisor: true, scan }); console.log(`${String(scan).padStart(7)}${String(r.completed).padStart(15)}` + `${r.avgRecovery.toFixed(2).padStart(14)}${String(r.worstRecovery).padStart(9)}` + `${String(r.finishedRound).padStart(11)}${String(r.recordsRead).padStart(14)}`); } ``` ``` 12 jobs, 4 steps; 3 step(s) stalled (S2@discount, S5@tariff, S9@invoice), 1 step(s) with a persistent failure (S7@discount) measure no supervisor supervisor on completed jobs 8 11 steps run 36 45 steps re-driven 0 6 wasted steps 0 3 compensated jobs 0 1 compensated steps 0 1 half-finished jobs 4 0 last job round 5 8 scan jobs recovered avg recovery worst last round records read 1 11 1.00 1 8 96 2 11 1.33 2 8 48 4 11 2.33 4 12 36 8 11 5.67 7 24 36 ``` ## Reading the Numbers The no-supervisor column shows what a silent failure means. Eight of twelve jobs completed, 36 steps ran, and the last job finished on round 5. On the surface everything looks fine: the error counter is zero, compensation is zero, no one saw an exception. The bottom row carries the real result: **4 jobs are half-finished** and sit in the "running" state forever. The scheduler started the step, the agent did not reply, and there is no one to ask the question. In the supervisor-on column, 3 of the 4 jobs are recovered, one is compensated, and half-finished jobs drop to 0. Completed jobs rise from 8 to 11 — not 12, because job `S7` has a persistent failure, and the supervisor counted it as past the threshold after driving it three times; the one completed step of that job was compensated. The supervisor does not turn a failed job into a successful one; **it turns an uncertain state into a determined one.** A job either finishes or is compensated; the count of jobs left in the "running" state is zero. Two rows show the cost. Steps run rises from 36 to 45, and 3 of those are **wasted** steps: the agent had already done the work, it had only failed to record it. The supervisor cannot know this; the only way to tell a stall apart from a persistent failure is to retry. This is why the scheduler–agent–supervisor arrangement has a mandatory precondition: **steps must be idempotent.** Running the same step a second time must not produce an additional effect; if a step does not carry this property naturally, it is protected with an idempotency key. Idempotency and the idempotency key were established in the Web API Design and The Data Access Layer and Business Logic courses; here they precondition the supervisor's work. A non-idempotent `invoice` step produces three extra invoice lines across three wasted runs. The table below shows what the scan interval buys. Recovered jobs are 11 across all four intervals: the supervisor finds them sooner or later. What changes is how late it finds them. Average recovery delay rises from 1.00 to 5.67 rounds, the worst case from 1 to 7; the batch's last job shifts from round 8 to round 24. In exchange, the records the supervisor reads drop from 96 to 36. When the interval doubles, the delay roughly doubles and the reads are halved; frequent scanning speeds up recovery, infrequent scanning reduces the read load on the durable record. This trade-off is deterministic, read directly from the model, not a duration measurement dependent on the environment. ## Back to the Estimate The model's 12 jobs are not a volume. The volume comes from K01: 4,000 workflows a day and four steps per job, that is, 16,000 steps a day. The previous lesson's KK2 assumption (the end-of-day workflow's persistent failure ratio of 0.02) carries forward. This lesson adds one more assumption. **KK3 — a step's stall ratio is 0.005.** The rationale is that agent process restarts, calls cut off before recording, and steps that time out and lose their reply produce a steady baseline. It is not added to K01's table; its sensitivity is calculated at 0.01. ```js // supervisor/cost.mjs — applies the model's ratios to K01's end-of-day job volume import { run, STEP } from "./run.mjs"; const INVOICE_LINES = 4000; // K01: daily invoice lines (computed value) = daily workflows const STEPS_PER_DAY = INVOICE_LINES * STEP.length; const KK2 = 0.02; // previous lesson: end-of-day workflow's persistent failure ratio const RETRY_LIMIT = 3; // the supervisor's threshold (model parameter) const r = run({ supervisor: true, scan: 2 }); const wastedRatio = r.wasted / r.redriven; // the share of re-drives that go to waste console.log(`daily workflows ${INVOICE_LINES}, steps/day ${STEPS_PER_DAY}, ` + `model's wasted re-drive ratio ${wastedRatio.toFixed(3)}`); console.log(); console.log(`${"KK3".padStart(6)}${"stalled steps/day".padStart(19)}${"re-drives/day".padStart(19)}` + `${"wasted steps/day".padStart(20)}${"half-finished, no supervisor".padStart(31)}${"job share".padStart(10)}`); for (const KK3 of [0.005, 0.01]) { // KK3: a step's stall ratio const stalled = STEPS_PER_DAY * KK3; const persistent = INVOICE_LINES * KK2; const halfFinished = stalled + persistent; console.log(`${KK3.toFixed(3).padStart(6)}${stalled.toFixed(0).padStart(19)}` + `${(stalled + persistent * RETRY_LIMIT).toFixed(0).padStart(19)}${stalled.toFixed(0).padStart(20)}` + `${halfFinished.toFixed(0).padStart(31)}${`${(100 * halfFinished / INVOICE_LINES).toFixed(2)}%`.padStart(10)}`); } console.log(`\nthe cost of the scan interval (model measurement, deterministic):`); for (const scan of [1, 8]) { const k = run({ supervisor: true, scan }); console.log(` scan ${scan}: avg recovery ${k.avgRecovery.toFixed(2)} rounds, ` + `worst ${k.worstRecovery}, last round ${k.finishedRound}, records read ${k.recordsRead}`); } ``` ``` daily workflows 4000, steps/day 16000, model's wasted re-drive ratio 0.500 KK3 stalled steps/day re-drives/day wasted steps/day half-finished, no supervisor job share 0.005 80 320 80 160 4.00% 0.010 160 400 160 240 6.00% the cost of the scan interval (model measurement, deterministic): scan 1: avg recovery 1.00 rounds, worst 1, last round 8, records read 96 scan 8: avg recovery 5.67 rounds, worst 7, last round 24, records read 36 ``` Once the number is read, the decision makes itself. A stall ratio of five in a thousand means 80 stalled steps a day; together with persistent failures, an unsupervised arrangement leaves **160 workflows half-finished** a day, which is 4 percent of the daily 4,000 invoice lines. If the ratio rises to 0.01, that becomes 240 and 6 percent. None of these jobs increment an error counter; their first symptom is that month-end reconciliation does not balance. The same table carries the supervisor's cost: 320 re-drives a day, 80 of them wasted steps. The wasted-run ratio in the model is 0.500, because half of the re-drives come from retrying a persistently failed job. The eighty are absorbed by idempotency; if they are not, 80 extra invoice lines appear a day — that is, the supervisor produces new errors equal to exactly half of the error it set out to fix. The scheduler–agent–supervisor arrangement is therefore not a pattern on its own; together with idempotent steps, it is a single decision. ## Summary - The three roles are separated because no role can notice its own failure: the scheduler writes the job to a durable record and starts the step, the agent runs the step, the supervisor only looks at the record. - In the unsupervised run, 4 of 12 jobs stayed in the "running" state without producing a single error; the error counter was zero and completed jobs stopped at 8. - The supervisor turns an uncertain state into a determined one: 3 jobs were recovered, 1 job was compensated, half-finished jobs dropped from 4 to 0, and completed jobs rose from 8 to 11. - The cost sits in steps run (36 → 45), 3 of which are wasted runs; this is why idempotent steps are a precondition of the arrangement. - The scan interval trades recovery delay against read load: as the interval rises from 1 to 8, average recovery climbs from 1.00 to 5.67 rounds and the last job shifts from round 8 to round 24, while records read drop from 96 to 36. - Back to K01: at 16,000 steps/day and KK3 = 0.005, the unsupervised arrangement leaves 160 half-finished jobs a day (4 percent of invoice lines); the supervisor zeroes this out and in exchange brings 320 re-drives and 80 wasted runs a day. ## Next Step This lesson's supervisor was treated as a single party. In reality, the supervisor, and the scheduler that starts the end-of-day job, both run on more than one node — because a supervisor running on a single node becomes, when it goes down, exactly the kind of workflow no one notices. But the scheduler's job is singular: end-of-day billing must be started exactly once. If two nodes start it at the same time, two invoice lines are produced for the same seller-day, and that can only be fixed by hand, not by compensation. The next lesson builds the arrangement that gives this singular responsibility to a single node: one of the nodes taking over responsibility for the duration of a lease, how many rounds a handover takes when that node goes down, and counting the rounds in which a misconfigured lease runs two nodes at once.