Workflow Orchestration

A queue safely links two functions: one writes, the other reads, and if something fails the message comes back. Invoice generation in the measurement network, though, is not two steps but five — a reading is validated, consumption is computed, a tariff is applied, an invoice is written, a notification is sent. When the fourth step fails permanently, nothing in any queue records what should happen to the first three; a queue only knows its own message.

The dispatched workflow that runs on compensating transactions — the saga pattern — was measured earlier. That pattern is not repeated here, only named. This lesson’s question is: when the whole flow is handed to a workflow orchestrator, which decision is handed over and what number comes back in exchange?

Who Holds the Flow

Writing all five steps inside a single function is the shortest path: one function, five calls, error handling with if blocks in between. The flow’s state lives in that function’s local variables. This causes no trouble until the flow crashes — when it does, nothing is left behind. Which step a function died at on its third attempt, which step left its side effect, is written down nowhere.

An orchestrator takes this state out of the function. The start, end, input, output, and attempt count of every step is written to the orchestrator’s record; the function only performs a single step and returns its result. That is the handed-over decision: the responsibility of knowing where the flow is. The first constraint that comes back follows from the same place — every piece of data passed between steps goes through that record, so it is subject to a size limit, and large payloads cannot be carried between steps; they can only be written to storage with their key passed along instead.

The second handed-over decision is the retry itself. In a flow written inside a function, how many times to retry, how long to wait, and which errors count as retryable are in the code; in an orchestrator these sit as a policy next to the step definition. The gain is that the policy is separated from the code and visible in one place. The cost is that the flow definition falls into the same place as the rule in the previous lesson: because a definition is a configuration item, it does not go through code review, and when a step’s retry count is raised from three to five, no function changes — the difference only shows up in billing and in latency.

The measurement again runs on the fictional regional measurement network.

• ED13. The flow has five steps and step durations are in fictional units: 2, 3, 2, 4, 1.
• ED14. Each step has its own error rate; the retry policy is three attempts with increasing wait between attempts (2 and 4 units).
• ED15. In one in seventeen of 200 flows, writing the invoice conflicts permanently; these flows exhaust all three attempts and end in failure.
• ED16. Compensation is defined only for steps that have a side effect: apply-tariff 1 unit, write-invoice 2 units. The first two steps have no compensation.
• ED17. Transition count is the sum of step attempts and compensation steps.
• ED18. The model builds only the inside of a single flow; concurrency and ordering across flows are not measured.

The State Machine

// flow.mjs — workflow orchestrator model: steps, retry policy, compensation path
// Model: not a real orchestrator, but a countable setup of a step-based state machine.
const SEED = 20260603;                        // generator seed: run is reproducible
let state = SEED;
const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648;

const STEPS = [                               // duration and compensation: fictional unit
  { name: "validate-reading", duration: 2, errorRate: 0.06, compensation: 0 },
  { name: "compute-consumption", duration: 3, errorRate: 0.04, compensation: 0 },
  { name: "apply-tariff", duration: 2, errorRate: 0.10, compensation: 1 },
  { name: "write-invoice", duration: 4, errorRate: 0.08, compensation: 2 },
  { name: "send-notification", duration: 1, errorRate: 0.15, compensation: 0 },
];
const MAX_TRIES = 3, WAIT = [2, 4];           // 3 attempts; increasing wait between attempts
const N = 200;
const PERMANENT = (n) => n % 17 === 5;        // in these flows, writing the invoice conflicts every attempt

const s = { attempts: STEPS.map(() => 0), errors: STEPS.map(() => 0), succeeded: 0, compensated: 0,
            compensationSteps: 0, work: 0, wall: 0, transitions: 0, longest: 0 };

for (let n = 0; n < N; n++) {
  const done = [], start = s.wall;
  let cut = false;
  for (let a = 0; a < STEPS.length && !cut; a++) {
    let attempt = 0, ok = false;
    while (attempt < MAX_TRIES) {
      attempt++; s.attempts[a]++; s.transitions++; s.work += STEPS[a].duration; s.wall += STEPS[a].duration;
      const permanent = PERMANENT(n) && STEPS[a].name === "write-invoice";
      if (!permanent && random() >= STEPS[a].errorRate) { ok = true; break; }
      s.errors[a]++;
      if (attempt < MAX_TRIES) s.wall += WAIT[attempt - 1];   // wait: no work, but time passes
    }
    if (ok) { done.push(a); continue; }
    for (const t of done.reverse()) {                          // compensation: unwind in reverse order
      if (STEPS[t].compensation === 0) continue;
      s.transitions++; s.work += STEPS[t].compensation; s.wall += STEPS[t].compensation; s.compensationSteps++;
    }
    s.compensated++; cut = true;
  }
  if (!cut) s.succeeded++;
  s.longest = Math.max(s.longest, s.wall - start);        // longest wall time of a single flow
}

console.log("step".padEnd(22) + "attempts".padStart(10) + "errors".padStart(8) +
  "attempts/flow".padStart(15));
STEPS.forEach((a, i) => console.log(a.name.padEnd(22) + String(s.attempts[i]).padStart(10) +
  String(s.errors[i]).padStart(8) + (s.attempts[i] / N).toFixed(2).padStart(15)));
console.log(`\nflows: ${N}, succeeded: ${s.succeeded}, went to compensation: ${s.compensated}, ` +
  `compensation steps: ${s.compensationSteps}`);
console.log(`work time: ${s.work} units, wall time: ${s.wall} units, transitions: ${s.transitions}`);
console.log(`longest single flow: ${s.longest} units, seed: ${SEED}`);

node flow.mjs

step                    attempts  errors  attempts/flow
validate-reading             213      13           1.06
compute-consumption          211      11           1.05
apply-tariff                 229      29           1.15
write-invoice                234      46           1.17
send-notification            220      32           1.10

flows: 200, succeeded: 188, went to compensation: 12, compensation steps: 12
work time: 2685 units, wall time: 2967 units, transitions: 1119
longest single flow: 31 units, seed: 20260603

The step table shows where retries pile up. If every one of the five steps had passed on its first attempt, 200 flows would have made 1,000 attempts; the measurement shows 1,107 attempts. The difference is the second and third attempts of the steps that error. The heaviest step is writing the invoice: 234 attempts, 46 errors. 36 of those 46 come from the twelve flows with a permanent conflict; the remaining 10 are transient errors closed by retrying.

The number in the last row looks backward at first glance: even though sending the notification has the highest error rate, its attempts-per-flow is 1.10, lower than the tariff step’s. The reason is the denominator — only 188 flows reached this step, because 12 flows were cut at the previous one. Divided by the same 188 flows that reached it, the number comes out to 1.17. Step metrics should be divided not by the total flow count but by the number of flows that reached that step; otherwise every error rate toward the end of a flow looks smaller than it is.

The compensation path produced two numbers: 12 flows went to compensation and 12 compensation steps ran in total, a ratio of one per flow, because when writing the invoice fails, the only side effect that needs undoing is the tariff application; the first two steps have no defined compensation. This shows the real question of compensation design: the number of steps to unwind depends on which step the failure happened at. A failure later in the flow means a longer unwind, and a flow that fails on its last step can cost more to compensate than the flow itself.

The retry policy itself is inside these numbers too. If the maximum attempt count had been five instead of three, the 12 permanently conflicting flows would have spent 60 attempts on writing the invoice instead of 36: 24 extra attempts, 96 extra units of work, and — since the waits increase — a longer delay per flow before compensation even starts. On a permanent error, retrying rescues nothing; it only delays accepting the failure. That is the cost of loosening the policy; the cost of tightening it runs the other way — two attempts instead of three turns part of the tariff step’s 29 transient errors into permanent failures and raises the number of flows that go to compensation.

Billing the Long-Running Flow

Total work time is 2,685 units, wall time is 2,967 units. The 282-unit gap between them is the wait between retries — time in which no work happens, but the flow stays open. In this measurement, wait is about 10 percent of the total, because the wait values in the policy were kept small. In flows waiting on an approval or on a response from an external service, the ratio reverses and wait grows to multiples of work time. The calculation below scans that difference with a multiplier; each row runs the same 200 flows, only the weight of the wait changes. Who ends up paying for it depends on where the flow is built.

// flow-cost.mjs — same 200 flows: is the flow built inside the function, or in the orchestrator
// Numbers read from the first run. The billing unit is fictional; the ratio is preserved, there is no currency.
const N = 200, WORK = 2685, WALL = 2967, TRANSITIONS = 1119;
const WAIT = WALL - WORK;                     // wait between steps: time with no work done
const DURATION_RATE = 1;                      // fictional unit: per unit of running time
const TRANSITION_RATE = 0.25;                 // fictional unit: per state transition in the orchestrator

console.log("wait multiplier".padEnd(17) + "in function".padStart(14) +
  "in orchestrator".padStart(17) + "locked (function)".padStart(20) +
  "locked (orch.)".padStart(19));
for (const k of [1, 5, 20, 100]) {
  const wait = WAIT * k;
  const inFunction = (WORK + wait) * DURATION_RATE;          // the function stays up throughout the wait too
  const inOrchestrator = WORK * DURATION_RATE + TRANSITIONS * TRANSITION_RATE;  // wait is free, transitions are billed
  console.log(String(k).padEnd(17) + inFunction.toFixed(0).padStart(14) + inOrchestrator.toFixed(0).padStart(17) +
    String(WORK + wait).padStart(20) + String(WORK).padStart(19));
}
console.log(`flows: ${N}, work: ${WORK} units, wait (multiplier 1): ${WAIT} units, transitions: ${TRANSITIONS}`);

node flow-cost.mjs

wait multiplier     in function  in orchestrator   locked (function)     locked (orch.)
1                          2967             2965                2967               2685
5                          4095             2965                4095               2685
20                         8325             2965                8325               2685
100                       30885             2965               30885               2685
flows: 200, work: 2685 units, wait (multiplier 1): 282 units, transitions: 1119

The first row is an honest result: while wait is short, the cost of the two designs is nearly the same — 2,967 against 2,965. The 1,119 × 0.25 = 280 units the orchestrator charges per transition almost exactly matches the 282 units the function pays for waiting. Per-step billing is not free: as step count grows, transition cost grows with it, and in flows with many steps that finish fast the orchestrator can stay expensive.

A side effect of this billing shape is that it prices the flow’s granularity. If each of the five steps were split into three — its own validation, its own work, its own write — work time would not change but transition count would triple from 1,119 and transition cost would rise from 280 units to 840. Where step boundaries are drawn directly determines the cost of a flow running in the orchestrator. The reverse holds too: merging two steps into one lowers transition cost, but the merged step’s retry now reruns both at once and the failure point gets coarser.

The gap widens as wait lengthens. At a multiplier of 100 — a flow waiting on an approval, on a response from an external service, or one that sits until the next day — the in-function design climbs to 30,885 units while the orchestrator stays at 2,965: 10.4 times. The break-even point can be read straight from the measurement; the moment wait exceeds the total transition cost, the orchestrator gets cheaper.

The last two columns show a harder constraint. Locked time is the slot held from the concurrency pool. A flow built inside a function occupies a slot through the wait as well; at a multiplier of 100, 30,885 units of slot-time get locked. In the orchestrator this number stays fixed at 2,685, because no function is up during the wait. In an environment with a concurrency limit, this difference hits capacity before it hits cost: waiting flows block the processing of newly arriving readings.

Visibility, Restartability, Limit

Beyond cost and capacity there are three more differences, and all three read straight from the measurement’s numbers.

Visibility. Because the orchestrator records every transition, all 1,119 of them form a queryable record: which flow, at which step, on which attempt, waited for how long. When the same flow is built inside a function, this information lives only in log lines, and stitching it back together per flow becomes a separate job. Here the handed-over decision comes back as a gain.

Restartability. The 12 flows that went to compensation can be restarted from the failed step in the orchestrator; the result of the first three steps sits in the record and does not rerun. In the in-function design, restarting means starting over — three more steps for 12 flows, 12 × 7 = 84 extra units of work. Once the permanent conflict’s cause is fixed, this difference is counted directly.

Limit. The longest single flow took 31 units. This number is the critical constraint of the in-function design: a flow’s wall time cannot exceed the function-execution environment’s time limit. 31 units stays well under a reasonable limit, but wall time grows with the wait multiplier; a flow waiting on an approval or held over to the next day fits no function’s time limit. The orchestrator has no such limit, because nothing is running during the wait.

In exchange, the orchestrator brings its own limits, and these are measured not in time but in count: the maximum number of steps a flow can take, the record size it can hold, the lifetime of retained history. Because the measured flow has five steps and averages 5.6 transitions per flow, it approaches none of these limits; but if step count depends on the size of the data — for example a batch flow that opens one step per meter — the limit makes the design directly impossible. In that case the only path is turning the flow into an outer loop with a fixed step count, starting a new flow on every iteration; the cost is that flow count, and transition cost with it, grows together with the data.

The way around the constraint is to split the flow in two: short, tight steps inside a function, transitions that involve waiting in the orchestrator. The cost is that the flow is now defined in two places at once — visibility gets split again, and which step runs where becomes a separate piece of information.

Summary

• The responsibility of holding the flow’s state is handed to the orchestrator; in exchange, every piece of data passed between steps goes through the orchestrator’s record and is subject to a size limit.
• 200 flows produced 1,107 step attempts; the heaviest step is writing the invoice with 234 attempts and 46 errors, 36 of which come from twelve permanently conflicting flows.
• Step metrics must be divided by the number of flows that reached that step, not by the total flow count: the last step’s attempt count comes out to 1.10 divided by 200, and 1.17 divided by the 188 flows that reached it.
• 12 flows went to compensation and ran 12 compensation steps; the number of steps to unwind depends on which step of the flow the failure happened at.
• The two designs cost the same while wait is short (2,967 against 2,965); once wait grows a hundredfold, the in-function flow costs 30,885 while the orchestrator stays at 2,965, and locked concurrency is 30,885 slot-units against 2,685.
• Per-transition billing prices the flow’s granularity: splitting the steps into three raises transition cost from 280 units to 840, while work time stays unchanged.

Next Step

This entire flow started with an event: a reading arrived, validation was triggered, the chain ran. Some of the measurement network’s jobs have no such event. The nightly batch invoicing, the daily reconciliation, and the weekly meter health scan depend on no reading’s arrival; the only thing that starts them is the clock. Something holds the clock too, and it is a service. The next lesson measures that trigger: the kind of guarantee the service gives, what firing twice in the same minute does to invoicing, and why a missed run is not automatically compensated for.