Compensating Transactions

The previous lesson stopped duplication: the idempotency key kept the same work from running a second time. The drill’s second open problem still stands. When the end-of-day billing workflow fails at its third step, the first two steps’ effects stay final; those are not duplicated effects but half-finished ones, and the idempotency key says nothing about them.

A compensating transaction is a step that balances a completed step’s effect with an inverse operation — the same concept as The Data Access Layer and Business Logic course’s compensating step, which was built there inside a saga arrangement, where the possibility of the compensation itself failing was already measured; the mechanics are not repeated here. Do not let the similar name mislead: the Caching, Queues and Asynchronous Processing course’s catch-up policy is a separate decision about how a lagging consumer catches up. This lesson has three questions: where an uncompensatable step can be placed in the ordering, how many jobs stay stuck when a compensation call itself fails, and how long the intermediate state stays visible in the rollback window.

The Ordering Is Not Free

End-of-day billing has four steps: closing the period, writing the invoice line, opening the collection entry, and sending the seller notification. Two of them cannot be rolled back. Closing the period cannot be rolled back, because reopening a closed period drops the carrier events that arrived in between out of scope and makes the same day billable twice. The seller notification cannot be rolled back, because a notification already sent can only be corrected by a second notification.

The rule is well known: an uncompensatable step goes at the end of the ordering. The problem is that the steps carry dependencies. The invoice line cannot be written before the period closes, the collection entry cannot open without the invoice line, and the notification cannot be sent before the invoice line. How many orderings both satisfy the dependencies and honor the rule is a countable question.

// compensating/flow.mjs — the in-process model of the end-of-day billing workflow.
// There is no service, network, or store: steps are plain objects, rollback is a loop.
// A compensating transaction is the same thing as M16/K04's compensating step; its mechanics were built there.

export const STEP = [
  { name: "close-period", compensate: null, requires: [] },                          // cannot be rolled back
  { name: "invoice-line", compensate: "cancel-line", requires: ["close-period"] },
  { name: "collection-entry", compensate: "revert-entry", requires: ["invoice-line"] },
  { name: "seller-notification", compensate: null, requires: ["invoice-line"] },     // cannot be rolled back
];

export function* permutations(steps) {                 // every ordering
  if (steps.length === 0) { yield []; return; }
  for (let i = 0; i < steps.length; i += 1)
    for (const k of permutations(steps.filter((_, j) => j !== i)))
      yield [steps[i], ...k];
}

export const valid = (d) => d.every((a, i) =>
  a.requires.every((o) => d.slice(0, i).some((x) => x.name === o)));

// One run: the failure occurs at step k; completed steps are compensated in reverse order.
// compensationFailure: the compensation call's failure probability; attempts: compensation attempt count.
export function run(ordering, k, { compensationFailure, attempts, rnd }) {
  const completed = ordering.slice(0, k);
  const r = { uncompensated: 0, compensationCalls: 0, stuck: 0, round: 0 };
  for (const a of [...completed].reverse()) {
    r.round += 1;
    if (a.compensate === null) { r.uncompensated += 1; continue; }
    let succeeded = false;
    for (let d = 0; d < attempts && !succeeded; d += 1) {
      r.compensationCalls += 1;
      if (d > 0) r.round += 1;
      succeeded = rnd() >= compensationFailure;
    }
    if (!succeeded) r.stuck += 1;
  }
  return r;
}

export function generator(seed) {              // 32-bit linear congruential generator, seed exposed
  let s = seed >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; };
}

DD5 — the compensation call’s failure probability is 0.05. Rationale: compensation is itself a remote call and runs right after a failure; the target service may still be broken at that moment, so it carries a higher error margin than an ordinary call. Its sensitivity is given by sweeping the attempt count. The number is an assumption and is not added to K01’s table.

// compensating/measure.mjs — the uncompensated step and rollback rounds across valid orderings
import { STEP, permutations, valid, run, generator } from "./flow.mjs";

const ALL = [...permutations(STEP)];
const VALID = ALL.filter(valid);
console.log(`${ALL.length} orderings, ${VALID.length} of them satisfy the dependencies`);
console.log(`cannot be rolled back: ${STEP.filter((a) => a.compensate === null).map((a) => a.name).join(", ")}`);

const noFailure = () => 1;               // failure-free compensation run: rnd() = 1 >= failure rate
console.log(`\n${"ordering".padEnd(66)}${"uncompensated".padStart(15)}${"rounds".padStart(8)}`);
const summary = [];
for (const d of VALID) {
  let uncompensated = 0, rounds = 0;
  for (let k = 1; k <= STEP.length; k += 1) {           // failure at step k
    const r = run(d, k - 1, { compensationFailure: 0, attempts: 1, rnd: noFailure });
    uncompensated += r.uncompensated; rounds += r.round;
  }
  summary.push([d.map((a) => a.name).join(" > "), uncompensated, rounds]);
  console.log(`${summary.at(-1)[0].padEnd(66)}${String(uncompensated).padStart(15)}${String(rounds).padStart(8)}`);
}

const chosen = VALID[summary.findIndex((o) => o[1] === Math.min(...summary.map((x) => x[1])))];
console.log(`\nchosen ordering: ${chosen.map((a) => a.name).join(" > ")}`);
console.log(`${"step that fails".padEnd(28)}${"completed".padStart(12)}${"compensated".padStart(14)}` +
  `${"uncompensated".padStart(16)}${"rollback rounds".padStart(18)}`);
for (let k = 1; k <= STEP.length; k += 1) {
  const r = run(chosen, k - 1, { compensationFailure: 0, attempts: 1, rnd: noFailure });
  console.log(`${`${k}. (${chosen[k - 1].name})`.padEnd(28)}${String(k - 1).padStart(12)}` +
    `${String(r.compensationCalls).padStart(14)}${String(r.uncompensated).padStart(16)}${String(r.round).padStart(18)}`);
}

// DD5: the compensation call's failure probability is 0.05. Attempt count is swept.
const N = 100_000, FAILURE_RATE = 0.05;
console.log(`\n${N} runs, DD5 compensation failure rate ${FAILURE_RATE}; the failing step is chosen with an even distribution`);
console.log(`${"attempts".padStart(9)}${"stuck jobs".padStart(13)}${"ratio".padStart(10)}` +
  `${"calls/job".padStart(12)}${"avg rollback rounds".padStart(22)}`);
for (const attempts of [1, 2, 3]) {
  const rnd = generator(20260730);
  let stuck = 0, calls = 0, rounds = 0;
  for (let i = 0; i < N; i += 1) {
    const k = i % STEP.length;                          // failing step, even distribution
    const r = run(chosen, k, { compensationFailure: FAILURE_RATE, attempts, rnd });
    stuck += r.stuck > 0 ? 1 : 0; calls += r.compensationCalls; rounds += r.round;
  }
  console.log(`${String(attempts).padStart(9)}${String(stuck).padStart(13)}` +
    `${(stuck / N).toFixed(5).padStart(10)}${(calls / N).toFixed(3).padStart(12)}` +
    `${(rounds / N).toFixed(3).padStart(22)}`);
}

24 orderings, 2 of them satisfy the dependencies
cannot be rolled back: close-period, seller-notification

ordering                                                            uncompensated  rounds
close-period > invoice-line > collection-entry > seller-notification              3       6
close-period > invoice-line > seller-notification > collection-entry              4       6

chosen ordering: close-period > invoice-line > collection-entry > seller-notification
step that fails                completed   compensated   uncompensated   rollback rounds
1. (close-period)                      0             0               0                 0
2. (invoice-line)                      1             0               1                 1
3. (collection-entry)                  2             1               1                 2
4. (seller-notification)               3             2               1                 3

100000 runs, DD5 compensation failure rate 0.05; the failing step is chosen with an even distribution
 attempts   stuck jobs     ratio   calls/job   avg rollback rounds
        1         3724   0.03724       0.750                 1.500
        2          214   0.00214       0.788                 1.538
        3            8   0.00008       0.790                 1.540

These numbers are in the measurement class: they come from a process-internal model and a generator seeded at 20260730, and they reproduce under the same seed.

The Rule Only Applies as Far as the Dependencies Allow

Twenty-two of the twenty-four orderings violate a dependency; two orderings remain, and both start with close-period. This is the rule’s limit: one of the two uncompensatable steps cannot be moved to the end, because the other three steps depend on it. The notification can be moved to the end; closing the period cannot.

The table below shows the cost step by step. If the failure occurs at the first step, nothing has completed; the uncompensated step count is 0. If it occurs at the second, third, or fourth step, the period is closed and stays closed: the uncompensated step count is 1 in all three cases. Changing the ordering does not lower this number, it only moves the position of the second uncompensatable step — when the notification is moved to third place, the total uncompensated step count climbs from 3 to 4.

The design rule that follows from this has nothing to do with ordering. If an uncompensatable step cannot be moved to the end, making it compensatable is the only way out: if closing the period is defined as writing a flag rather than a deletion, it has a compensation. The ordering rule is a placement tool; it does not substitute for step design.

Stuck Work and the Intermediate State Window

The block below counts compensation’s own failure. At one attempt, 3.724 percent of jobs stay stuck: the compensation call failed, the step’s effect stands, and the job is neither completed nor rolled back. A second attempt brings the ratio down to 0.00214, a third to 0.00008. In exchange, compensation calls per job climb from 0.750 to 0.790, and rollback rounds from 1.500 to 1.540 — a second attempt cuts stuck work seventeenfold for 5 percent more calls.

The intermediate state window is the invoice line appearing “issued” while the rollback is still underway. This window’s length is measured in rounds; giving a round a duration requires a conversion factor.

DD6 — one rollback round is 0.25 seconds. Rationale: a round is a remote call plus its confirmation. Its sensitivity is given at four times that value. DD7 — stuck work is closed within 4 hours, the same as K01’s V10 batch window; that is the billing job’s window.

// compensating/cost.mjs — converting the measured ratios into daily numbers at K01 scale
const INVOICE_LINES = 4000;                  // K01 computed value: daily invoice lines
const KK2 = 0.02;                            // M19/K03 choreography lesson's assumption: workflow failure rate
const ROUND_SEC = 0.25, STUCK_HOURS = 4;     // DD6 round duration, DD7 time to close stuck work
const failed = INVOICE_LINES * KK2;

console.log(`daily workflows ${INVOICE_LINES}, KK2 = ${KK2} -> failed ${failed}/day`);
console.log(`\n${"ordering".padEnd(22)}${"uncompensated steps/day".padStart(26)}${"a year".padStart(9)}`);
for (const [name, ratio] of [["notification last", 3 / 4], ["notification third", 4 / 4]])
  console.log(`${name.padEnd(22)}${(failed * ratio).toFixed(1).padStart(26)}` +
    `${(failed * ratio * 365).toFixed(0).padStart(9)}`);

// measure.mjs measurements: stuck work ratio and average rollback rounds, by compensation attempts
const MEASURED = [[1, 0.03724, 1.500], [2, 0.00214, 1.538], [3, 0.00008, 1.540]];
console.log(`\n${"attempts".padStart(9)}${"stuck work/day".padStart(16)}${"a year".padStart(8)}` +
  `${"rollback s".padStart(12)}${"intermediate s/day".padStart(20)}${"h/day from stuck".padStart(19)}`);
for (const [attempts, ratio, rounds] of MEASURED) {
  const stuck = failed * ratio, rollback = rounds * ROUND_SEC;
  console.log(`${String(attempts).padStart(9)}${stuck.toFixed(3).padStart(16)}` +
    `${(stuck * 365).toFixed(0).padStart(8)}${rollback.toFixed(3).padStart(12)}` +
    `${(failed * rollback).toFixed(1).padStart(20)}${(stuck * STUCK_HOURS).toFixed(3).padStart(19)}`);
}
const [, ratio1, rounds1] = MEASURED[0];
const factor = (t) => (failed * ratio1 * STUCK_HOURS * 3600) / (failed * rounds1 * t);
console.log(`\nat one attempt, from stuck work / from rollback = ${factor(ROUND_SEC).toFixed(0)}x`);
console.log(`DD6 sensitivity: at a round duration of ${ROUND_SEC * 4} s instead of ${ROUND_SEC} s, intermediate state is ` +
  `${(failed * rounds1 * ROUND_SEC * 4).toFixed(1)} s/day, ratio ${factor(ROUND_SEC * 4).toFixed(0)}x`);
console.log(`\nfailure-free day's cost: 2 of the 4 steps carry a compensation function, ` +
  `${(INVOICE_LINES * 2).toLocaleString("en-US")} written compensation paths a day; compensation calls run ` +
  `${(failed * 0.75).toFixed(0)}/day, ${(0.75 * KK2).toFixed(4)} per workflow`);

daily workflows 4000, KK2 = 0.02 -> failed 80/day

ordering                 uncompensated steps/day   a year
notification last                           60.0    21900
notification third                          80.0    29200

 attempts  stuck work/day  a year  rollback s  intermediate s/day   h/day from stuck
        1           2.979    1087       0.375                30.0             11.917
        2           0.171      62       0.385                30.8              0.685
        3           0.006       2       0.385                30.8              0.026

at one attempt, from stuck work / from rollback = 1430x
DD6 sensitivity: at a round duration of 1 s instead of 0.25 s, intermediate state is 120.0 s/day, ratio 358x

failure-free day's cost: 2 of the 4 steps carry a compensation function, 8,000 written compensation paths a day; compensation calls run 60/day, 0.0150 per workflow

The Numbers for Two Days

The failure-free day’s cost sits in the code path. Two of the four steps must carry a compensation function, and those functions sit in every workflow definition: 8000 written compensation paths a day for 4000 workflows. How often they run is very small — 0.0150 per workflow, 60 calls a day. The cost is not a runtime load, it is untested code: alongside a path that runs 21,900 times a year, the codebase carries a path that runs with a probability of 1.5 percent. Closing the period being uncompensatable also produces a constraint on the failure-free day: no correction can be made on a closed period.

The failure day’s gain is two numbers. At KK2 = 0.02, 80 workflows fail a day. With the ordering rule applied, the uncompensated step count is 60 a day, 21,900 a year; if the notification is moved to third place, it is 80 and 29,200. Ordering alone makes a difference of 7300 uncompensated steps a year. Adding one compensation attempt brings stuck work down from 2.979 to 0.171 a day, from 1087 to 62 a year.

The intermediate state window has two sources, and their sizes are not comparable. Rollback itself produces a total of 30.0 seconds of intermediate state a day. Stuck work produces 11.917 hours a day — 1430 times as much. Even at four times DD6, the ratio stays at 358x, so the result is not sensitive to the round-duration assumption. The intermediate state window is set by compensation’s own failure, not by rollback’s speed; a team that wants to shorten the window should tune the compensation attempt count, not the round duration.

Summary

• Only two of the twenty-four orderings satisfy the dependencies, and both start with the uncompensatable close-period step: the move-to-the-end rule cannot be applied to this step.
• When the failure occurs at the second, third, or fourth step, the uncompensated step count is 1 in every case; what saves a step that cannot be moved to the end is not ordering but designing the step to be compensatable.
• The notification’s position makes a difference of 7300 uncompensated steps a year: 60/day and 21,900/year at the end, 80/day and 29,200/year at third place.
• Compensation carries its own failure: at DD5 = 0.05, one attempt leaves 3.724 percent of jobs stuck; a second attempt brings the ratio down to 0.00214 and raises calls per job from 0.750 to 0.790.
• The overwhelming share of the intermediate state window comes from stuck work, not rollback: 30.0 seconds a day against 11.917 hours, 1430 times; even at four times DD6, still 358 times.
• The failure-free day’s cost is 8000 written compensation paths, and only 60 of them run a day; what the codebase carries is not runtime load but rarely run code.

Next Step

Compensation kept the workflow’s steps correct. The data the same workflow carries was never in question. When the carrier reports a delivery, it attaches proof of delivery alongside it: the recipient’s signature or a photo of the delivery point. This addition turns the state event’s record into something far larger than K01’s 220 bytes, and that something passes through the queue, gets copied to every consumer, and is carried again on every redelivery. The next lesson takes on pulling the large payload out of the message: the message’s size, the bytes crossing the queue, how long the payload lives in a separate store, and how many times the reference left behind after the payload is deleted dangles.