Distributed Transaction Alternatives

The previous topic drew the boundary, established communication across the two sides of it, and showed how the address gets resolved. None of those three tasks touched data. The catalog service keeps the book, the membership service keeps the loan count, each in its own store; yet the “issue loan” request is still written on the assumption that both writes happen together or neither happens at all. That assumption rested on a single transaction boundary, and the boundary disappeared the moment the two sides split.

This particular loss was already established and measured in the Distributed Transaction Problem lesson of M16/K04: when writes go to two separate stores and the second call fails, the first stays committed. That measurement is not repeated here. What gets measured here is the implementation-side cost of the classic fix that lesson mentioned in a single sentence — two-phase commit: how long the lock stays held, how many requests get blocked during that window, and what remains behind when the coordinator crashes.

Mechanism

The two services are genuinely two separate processes; each opens its own node:sqlite file, and the only connection between them is a call over the local network.

DC1. The store holds its write lock at the file level, not the row level. So the “blocked request” figure counted below is an upper bound; in a store that locks rows, the blocked set would stay limited to the rows the prepared transaction touches. The shape of the measurement — a resource held while a decision is awaited — is the same in both cases; only the size differs.

DC2. The membership service spends a fixed 400 ms before voting. This models a slow participant; it is not a delay measured from a real network. So the size of the lock window is a chosen quantity — what gets measured is what happens inside a window of that size.

# setup.sh — closes the previous run, rebuilds each service's own store from scratch
pkill -f participant.mjs 2>/dev/null; sleep 0.3
rm -f catalog.db* membership.db* service.log
sqlite3 catalog.db >/dev/null <<'SQL'
PRAGMA journal_mode=WAL;
CREATE TABLE book (book_id INTEGER PRIMARY KEY, title TEXT NOT NULL, status TEXT NOT NULL);
INSERT INTO book VALUES (1,'Blindness','on_shelf'),(2,'The Disconnected','on_shelf'),(3,'The Book of Sand','on_shelf');
SQL
sqlite3 membership.db >/dev/null <<'SQL'
PRAGMA journal_mode=WAL;
CREATE TABLE member (member_id INTEGER PRIMARY KEY, name TEXT NOT NULL, open_loans INTEGER NOT NULL, loan_limit INTEGER NOT NULL);
INSERT INTO member VALUES (4,'Ethan',2,3),(5,'Sara',3,3),(6,'Blake',0,9);
SQL

The participant side is the same module for both services; the business rule and the store are supplied from outside. The distinguishing point is the /prepare endpoint: the write happens, the transaction is not committed, and the connection stays held until the decision arrives.

// participant.mjs — one service's participant side; each service is a separate process, a separate store
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const [name, file, port, ownSql, rivalSql] = process.argv.slice(2);
const slowMs = Number(process.env.SLOW_MS ?? 0);
const pool = Array.from({ length: 2 }, () => {
  const db = new DatabaseSync(file);
  db.exec("PRAGMA busy_timeout = 50");
  return { db, busy: false };
});
const pending = new Map();
const wait = (ms) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
const available = () => pool.find((b) => !b.busy);

createServer((request, response) => {
  const u = new URL(request.url, "http://y");
  const send = (o) => response.end(JSON.stringify(o) + "\n");
  const tx = u.searchParams.get("tx");
  const key = Number(u.searchParams.get("key"));

  if (u.pathname === "/prepare" || u.pathname === "/single") {
    const b = available();
    if (!b) return send({ vote: "no", reason: "no available connection" });
    b.busy = true;
    if (slowMs) wait(slowMs);
    b.db.exec("BEGIN IMMEDIATE");
    if (b.db.prepare(ownSql).run(key).changes !== 1) {
      b.db.exec("ROLLBACK"); b.busy = false; return send({ vote: "no", reason: "condition not met" });
    }
    if (u.pathname === "/single") { b.db.exec("COMMIT"); b.busy = false; return send({ vote: "yes" }); }
    pending.set(tx, b);                    // the lock stays held after this response too
    return send({ vote: "yes" });
  }
  if (u.pathname === "/commit" || u.pathname === "/abort") {
    const b = pending.get(tx);
    if (!b) return send({ result: "unknown transaction" });
    b.db.exec(u.pathname === "/commit" ? "COMMIT" : "ROLLBACK");
    b.busy = false; pending.delete(tx);
    return send({ result: u.pathname.slice(1) });
  }
  if (u.pathname === "/rival") {           // an independent local write that is not part of the transaction
    const b = available();
    if (!b) return send({ result: "blocked", reason: "no available connection" });
    b.busy = true;
    try { b.db.exec("BEGIN IMMEDIATE"); b.db.prepare(rivalSql).run(key); b.db.exec("COMMIT");
          return send({ result: "written" }); }
    catch (h) { return send({ result: "blocked", reason: h.code ?? h.message }); }
    finally { b.busy = false; }
  }
  if (u.pathname === "/pending")
    return send({ service: name, pending: [...pending.keys()], held: pool.filter((b) => b.busy).length });
  send({ error: "unknown endpoint" });
}).listen(Number(port), () => console.log(`${name} up: ${port}`));

# start.sh — two services, two separate processes, two separate stores; membership set up as the slow participant
node participant.mjs catalog catalog.db 8801 \
  "UPDATE book SET status='on_loan' WHERE book_id=? AND status='on_shelf'" \
  "UPDATE book SET status = CASE status WHEN 'on_shelf' THEN 'on_loan' ELSE 'on_shelf' END WHERE book_id=?" \
  >>service.log 2>&1 &
SLOW_MS=400 node participant.mjs membership membership.db 8802 \
  "UPDATE member SET open_loans=open_loans+1 WHERE member_id=? AND open_loans<loan_limit" \
  "UPDATE member SET open_loans=open_loans WHERE member_id=?" \
  >>service.log 2>&1 &
sleep 1

Same Work, Two Implementations

The coordinator is a single file and runs the same workflow in two forms. --sequential mode has each service commit its own transaction immediately — the arrangement whose inconsistency was measured in M16/K04. The default mode takes two rounds: votes are collected from every participant first, then the decision is distributed.

// coordinator.mjs — same work, two implementations: --two-phase and --sequential
const mode = process.argv.includes("--sequential") ? "sequential" : "two-phase";
const crash = process.argv.includes("--crash");       // coordinator drops before the decision is distributed
const [bookId, memberId] = process.argv.slice(2).filter((a) => !a.startsWith("--")).map(Number);
const tx = `tx-${bookId}-${memberId}`;
const participants = [
  { name: "catalog",    endpoint: "http://localhost:8801", key: bookId },
  { name: "membership", endpoint: "http://localhost:8802", key: memberId },
];
let messages = 0;
const call = async (p, path) => {
  messages += 1;
  const c = await fetch(`${p.endpoint}/${path}?tx=${tx}&key=${p.key}`);
  return c.json();
};
const ms = (a, b) => (b - a).toFixed(1);

if (mode === "sequential") {                          // one round: each service commits its own work
  const t0 = performance.now();
  for (const p of participants) {
    const c = await call(p, "single");
    console.log(`  ${p.name}: ${c.vote}${c.reason ? " (" + c.reason + ")" : ""}`);
  }
  console.log(`sequential | round=1 messages=${messages} | coordinator lock window=0 ms | duration=${ms(t0, performance.now())} ms`);
} else {                                              // two rounds: vote first, then decision
  const votes = [];
  const t0 = performance.now();
  for (const p of participants) {
    const c = await call(p, "prepare");
    votes.push([p, c, performance.now()]);
    console.log(`  ${p.name}: vote=${c.vote}${c.reason ? " (" + c.reason + ")" : ""}`);
  }
  const decision = votes.every(([, c]) => c.vote === "yes") ? "commit" : "abort";
  if (crash) {
    console.log(`coordinator dropped before distributing the decision (decision=${decision} reached no participant)`);
    process.exit(1);
  }
  for (const [p, c] of votes) if (c.vote === "yes") await call(p, decision);
  const t1 = performance.now();
  const [, , tVote] = votes[0];
  console.log(`two-phase | round=2 messages=${messages} | decision=${decision}`);
  console.log(`  catalog lock window=${ms(tVote, t1)} ms | total=${ms(t0, t1)} ms (in this run)`);
}

Being able to count the blocked resource needs the system given ordinary, unrelated load.

// rival.mjs — requests that are not part of the transaction; tries to write at regular intervals through the window
const [endpoint, key, count, interval] = process.argv.slice(2);
let written = 0, blocked = 0, totalWait = 0;
for (let n = 0; n < Number(count); n++) {
  const t = performance.now();
  const c = await (await fetch(`${endpoint}/rival?key=${key}`)).json();
  totalWait += performance.now() - t;
  c.result === "written" ? (written += 1) : (blocked += 1);
  await new Promise((r) => setTimeout(r, Number(interval)));
}
console.log(`rival: requests=${count} written=${written} blocked=${blocked}` +
            ` | average wait=${(totalWait / Number(count)).toFixed(1)} ms (in this run)`);

Resources Blocked in the Lock Window

The same load, the same work, two implementations. The load goes to the catalog service; it has nothing to do with the loan request, it only wants to write to the same store.

sh setup.sh; sh start.sh
node rival.mjs http://localhost:8801 2 40 10 &
node coordinator.mjs 1 4
wait

  catalog: vote=yes
  membership: vote=yes
two-phase | round=2 messages=4 | decision=commit
  catalog lock window=459.5 ms | total=489.6 ms (in this run)
rival: requests=40 written=34 blocked=6 | average wait=12.4 ms (in this run)

sh setup.sh; sh start.sh
node rival.mjs http://localhost:8801 2 40 10 &
node coordinator.mjs 1 4 --sequential
wait

  catalog: yes
  membership: yes
sequential | round=1 messages=2 | coordinator lock window=0 ms | duration=443.0 ms
rival: requests=40 written=40 blocked=0 | average wait=2.6 ms (in this run)

Two quantities that do not depend on the run are decisive here: round count, 2 versus 1, and message count, 4 versus 2. As the number of participants grows, the message count grows as 2n, and the window becomes tied to the speed of the slowest participant.

The run-dependent figures are consistent too: the catalog lock was held for 459.5 ms, and 6 of the 40 rival requests in that window were blocked, and the average wait rose from 2.6 ms to 12.4 ms. The catalog had finished its own work in 30.1 ms; the reason it held the lock for 459.5 ms was not its own work, it was waiting for another service to vote.

When the Coordinator Crashes

This is where the real cost of two-phase commit lies. Once a participant has said “yes,” it cannot decide on its own: if it commits, the other side may have aborted; if it aborts, the other side may have committed. Only the coordinator knows the decision.

sh setup.sh; sh start.sh
node coordinator.mjs 1 4 --crash
echo "-- participants' view --"
curl -s "http://localhost:8801/pending"; curl -s "http://localhost:8802/pending"
echo "-- reading from outside, and a rival write --"
sqlite3 catalog.db "SELECT book_id, status FROM book WHERE book_id=1;"
node rival.mjs http://localhost:8801 2 10 10
echo "-- if the decision is supplied from outside --"
TX=$(curl -s "http://localhost:8801/pending" | sed 's/.*\["//;s/"\].*//')
curl -s "http://localhost:8801/commit?tx=$TX"
curl -s "http://localhost:8802/commit?tx=$TX"
node rival.mjs http://localhost:8801 2 10 10

  catalog: vote=yes
  membership: vote=yes
coordinator dropped before distributing the decision (decision=commit reached no participant)
-- participants' view --
{"service":"catalog","pending":["tx-1-4"],"held":1}
{"service":"membership","pending":["tx-1-4"],"held":1}
-- reading from outside, and a rival write --
1|on_shelf
rival: requests=10 written=0 blocked=10 | average wait=67.4 ms (in this run)
-- if the decision is supplied from outside --
{"result":"commit"}
{"result":"commit"}
rival: requests=10 written=10 blocked=0 | average wait=3.8 ms (in this run)

Both services show 1 pending transaction each, and 1 connection is held out of each two-connection pool. From the outside, the book’s status still shows as on_shelf — the write is invisible, but the lock is real: all 10 of the 10 rival requests were blocked, and the average wait rose to 67.4 ms. This figure does not drop until the coordinator comes back up. The lock duration is no longer a delay; it is the length of an outage.

The last two calls supply the decision from outside. In real systems this step is the job of a person or a recovery procedure, and it must know the correct decision; whether a pending transaction should be committed or aborted cannot be derived from the information the participants hold.

Comparison

Measure	Sequential local commit	Two-phase commit
Round count	1	2
Message count (2 participants)	2	4
Lock window	0 ms	459.5 ms (in this run)
Rival requests blocked in the window	0 / 40	6 / 40 (in this run)
Pending transactions if the coordinator crashes	0	2 (1 per service)
Rival requests blocked if the coordinator crashes	0 / 10	10 / 10

What it made cheaper: two-phase commit eliminated the partial outcome. In --sequential mode, when the catalog commits and membership rejects, a loaned book with no owner is left behind; in two-phase mode this cannot happen, because no participant commits without seeing the other’s vote. No compensating logic needs to be written in the application code.

What it made more expensive: the round count rose to two, the message count to 2n; a resource stayed held for as long as the system’s slowest participant took, and unrelated requests were rejected during that time.

Which new failure mode was born: indeterminate state. A participant that has cast its vote but not received the decision holds the lock indefinitely; the only way to free the resource is a decision supplied from outside, and whoever supplies it cannot find the correct answer from local information. This is a mode that sequential commit does not have.

The alternatives choose where this cost gets moved. A compensating arrangement never holds a lock; it makes the inconsistency visible for a while and constrains the ordering. An outbox arrangement, on the other hand, confines atomicity inside a single store and hands the publishing off to a separate job. Both give up “all or nothing” and choose “eventually correct” instead.

Summary

• Two-phase commit was built across two service processes; between the vote round and the decision round, the catalog service held its lock for 459.5 ms, even though its own work took 30.1 ms.
• The window’s length is set by the slowest participant; the run-independent cost is 2 rounds instead of 1, and 2n messages instead of n.
• In that window, 6 of 40 unrelated requests were blocked (in this run), and the average wait rose almost fivefold; in the single-round implementation the blocked request count was zero.
• When the coordinator dropped before distributing the decision, each participant was left with 1 pending transaction and 1 held connection; every rival request was blocked, and this lasted until a decision was supplied from outside.
• Whether a pending transaction should be committed or aborted cannot be derived from the participants’ local information; this is a new failure mode that the single-round implementation does not have.

Next Step

The way to zero out the lock window is to commit each step individually instead of preparing them, and to roll back whatever completed when a step fails. M16/K04 built a compensating step for this arrangement, and M19/K05 measured the cost of a compensating transaction. What remains is a constraint neither of them touched: steps with unequal compensability have to be placed in an order, and that order is not arbitrary. The next lesson runs the same loan flow through two different step orders and counts the unrecoverable side effect at each one, then implements the same flow twice, as choreography and as an executor, and measures the difference on the code side — the services touched and the number of steps each one knows about.