Payment Flow

In the previous case, the reservation made a promise, and the step that keeps it was left out of scope: the payment itself. That step has two sides; each side keeps its own record, neither can see the other’s ledger, and message exchange is the only link between them.

Idempotence and the idempotency key were built in the Resilience and Reliability course’s Idempotent Operations lesson and are not repeated here; a resent request never opens a second transaction. This lesson asks something else: how far the two ledgers diverge when messages are lost or late, how that gap is counted, and when it closes.

Constraints and Scope

Functional requirements: initiating a payment, processing the status notification from the other party, querying the status, initiating a refund, and writing every status transition to the audit trail.

Out of scope: the authentication flow, fraud scoring, closing the accounting ledger, and the refund rules themselves.

The non-functional requirements are written with a threshold and its source: a transaction’s status on both sides matches within 30 minutes at the latest (source: how long a gap may close on its own before manual review), the unmatched-record rate per reconciliation cycle does not exceed 10 per mille (source: how many records one cycle can review by hand), and every status transition appears in the audit trail and the record never changes afterward (source: a transaction’s history must be reconstructible).

Assumptions and Scale

Code	Assumption	Value	Rationale
OD1	daily payment transactions	1,200,000	the sum of completed and rejected attempts
OD2	peak multiplier	4	the ratio of evening-hour volume to the daily average
OD3	notification delay	0.92 / 0.06 / 0.02	shares of notifications arriving within five seconds, within five minutes, and later
OD4	notification loss	0.03	the other party’s notification never arriving at all
OD5	poll interval	60 seconds	how often a pending transaction is queried
OD6	reconciliation cycle	900 seconds	how often the two ledgers are compared
OD7	audit trail	5 transitions × 260 bytes	number of status transitions per transaction and record length
OD8	retention period	2,555 days	disputes can be reopened as far back as seven years
OD9	settlement on the other side	0.90 / 0.08 / 0.02	shares of transactions settling in 3, 3–60, and 60–900 seconds; the other party’s own approval steps produce a queue
OD10	synchronous-verification budget	20 seconds	the upper bound on how long a request can be kept open while waiting for a reply

// payment/measure.mjs — rough sizing derived from the OD assumption table
const OD = { dailyTransactions: 1_200_000, peakMultiplier: 4, notificationLoss: 0.03, poll: 60,
  cycle: 900, auditRecordBytes: 260, transitionCount: 5, retentionDays: 2555 };   // OD1..OD8
const DAY = 86_400;

const peak = (OD.dailyTransactions / DAY) * OD.peakMultiplier;
const auditDaily = OD.dailyTransactions * OD.transitionCount * OD.auditRecordBytes;
console.log(`peak transactions/s              ${peak.toFixed(2)}`);
console.log(`records compared per cycle       ${Math.round(peak * OD.cycle)}`);
console.log(`notifications lost per day       ${OD.dailyTransactions * OD.notificationLoss}`);
console.log(`records/s the poll will find     ${(peak * OD.notificationLoss).toFixed(2)}`);
console.log(`audit trail, daily GB            ${(auditDaily / 1e9).toFixed(2)}`);
console.log(`audit trail, ${OD.retentionDays} days, TB       ${((auditDaily * OD.retentionDays) / 1e12).toFixed(2)}`);
console.log(`audit rows / transaction ratio   ${OD.transitionCount} (one row per transition)`);

console.log(`\n${"reconciliation cycle".padStart(21)}${"records at peak batch".padStart(23)}` +
  `${"cycles per day".padStart(17)}`);
for (const c of [300, OD.cycle, 3600])
  console.log(`${`${c / 60} min`.padStart(21)}${Math.round(peak * c).toString().padStart(23)}` +
    `${(DAY / c).toFixed(0).padStart(17)}`);
console.log(`\ndaily records compared is independent of the cycle: ${OD.dailyTransactions} ` +
  `(each record once); the cycle only changes the batch size and the time to find the difference`);

peak transactions/s              55.56
records compared per cycle       50000
notifications lost per day       36000
records/s the poll will find     1.67
audit trail, daily GB            1.56
audit trail, 2555 days, TB       3.99
audit rows / transaction ratio   5 (one row per transition)

 reconciliation cycle  records at peak batch   cycles per day
                5 min                  16667              288
               15 min                  50000               96
               60 min                 200000               24

daily records compared is independent of the cycle: 1200000 (each record once); the cycle only changes the batch size and the time to find the difference

These numbers are of the computed class; OD9 and OD10 feed only the eliminated alternative’s model. Three of them shape the design. First, the peak rate is 55.56 transactions per second — none of this case’s decisions turn on volume. Second, 36,000 notifications never arrive each day, and finding them costs 1.67 queries per second, a small load, but without it those records diverge. Third, the audit trail is the real volume item: 1.56 GB daily, 3.99 TB over seven years, five rows per transaction.

Measuring the Gap

The measurement is an in-process model: there is no real counterparty, network, or store; delay and loss are parameters, and time is the model’s own clock. The model generates one cycle’s transactions and, for each, computes when it settles on the other side and when the local side learns of it, then compares the two states at cycle end. The gap splits into three sources: lost notifications, late notifications, and transactions not yet settled on the other side — the last is not a gap, both sides say the same thing.

// payment/reconciliation.mjs — an in-process model of the gap between the two sides' records.
// There is no real counterparty, network, or store: delay and loss are parameters, and time is
// the model's own clock (seconds).
const PEAK = 55.56, POLL = 60, LOSS = 0.03;      // computed; OD5, OD4
const SETTLE_TIME = 3;                            // time for the transaction to finish on the other side (s)

function rng(seed) {                              // 32-bit linear congruential generator
  let s = seed >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; };
}

// OD3: notification delay in three tiers — 92% fast, 6% delayed, 2% late
const delay = (r) => (r < 0.92 ? 5 * (r / 0.92)
  : r < 0.98 ? 5 + 295 * ((r - 0.92) / 0.06)
  : 300 + 1500 * ((r - 0.98) / 0.02));

export function run({ cycle, polling, seed }) {
  const rnd = rng(seed);
  const n = Math.round(PEAK * cycle);
  const s = { transactions: n, lateArrived: 0, lost: 0, unsettled: 0 };
  for (let i = 0; i < n; i += 1) {
    const start = (i / PEAK), finish = start + SETTLE_TIME;
    const isLost = rnd() < LOSS, d = delay(rnd());
    const notifiedAt = isLost ? Infinity : finish + d;
    const pollAt = polling ? Math.ceil(finish / POLL) * POLL : Infinity;
    const knownAt = Math.min(notifiedAt, pollAt);
    if (finish >= cycle) { s.unsettled += 1; continue; }  // not finished on the other side either
    if (knownAt < cycle) continue;                        // both sides know
    if (isLost) s.lost += 1; else s.lateArrived += 1;
  }
  s.unmatched = s.lateArrived + s.lost;
  s.perMille = (1000 * s.unmatched) / s.transactions;
  return s;
}

const printRow = (label, r) =>
  console.log(`${label.padEnd(26)}${String(r.transactions).padStart(9)}${String(r.unmatched).padStart(12)}` +
    `${String(r.lateArrived).padStart(14)}${String(r.lost).padStart(10)}` +
    `${String(r.unsettled).padStart(15)}${r.perMille.toFixed(2).padStart(11)}`);

console.log(`model: peak ${PEAK} transactions/s, notification loss ${LOSS}, poll ${POLL} s, seed 20260730`);
console.log(`\n${"regime (900 s cycle)".padEnd(26)}${"txns".padStart(9)}${"unmatched".padStart(12)}` +
  `${"late arrived".padStart(14)}${"lost".padStart(10)}${"unsettled".padStart(15)}${"per mille".padStart(11)}`);
printRow("notification only", run({ cycle: 900, polling: false, seed: 20260730 }));
printRow("notification + poll", run({ cycle: 900, polling: true, seed: 20260730 }));

console.log(`\n${"reconciliation cycle".padStart(21)}${"txns".padStart(9)}${"unmatched".padStart(12)}` +
  `${"per mille".padStart(11)}${"10 per mille limit".padStart(21)}${"closure min".padStart(13)}${"30 min limit".padStart(14)}`);
for (const c of [300, 900, 3600]) {
  const r = run({ cycle: c, polling: true, seed: 20260730 });
  console.log(`${`${c / 60} min`.padStart(21)}${String(r.transactions).padStart(9)}${String(r.unmatched).padStart(12)}` +
    `${r.perMille.toFixed(2).padStart(11)}${(r.perMille <= 10 ? "passes" : "fails").padStart(21)}` +
    `${String((2 * c) / 60).padStart(13)}${((2 * c) / 60 <= 30 ? "passes" : "fails").padStart(14)}`);
}

const a = run({ cycle: 900, polling: false, seed: 20260730 });
const b = run({ cycle: 900, polling: true, seed: 20260730 });
const c60 = run({ cycle: 3600, polling: true, seed: 20260730 });
console.log(`\nthe gap that polling closes: ${a.unmatched} -> ${b.unmatched} records ` +
  `(per mille ${a.perMille.toFixed(2)} -> ${b.perMille.toFixed(2)})`);
console.log(`${b.lateArrived}/${b.unmatched} of the remaining gap is late-arriving confirmations`);
console.log(`cycle 15 min -> 60 min: unmatched count ${b.unmatched} -> ${c60.unmatched} (nearly the same), ` +
  `per mille ${b.perMille.toFixed(2)} -> ${c60.perMille.toFixed(2)}`);

model: peak 55.56 transactions/s, notification loss 0.03, poll 60 s, seed 20260730

regime (900 s cycle)           txns   unmatched  late arrived      lost      unsettled  per mille
notification only             50004        2987          1437      1550            166      59.74
notification + poll           50004         488           369       119            166       9.76

 reconciliation cycle     txns   unmatched  per mille   10 per mille limit  closure min  30 min limit
                5 min    16668         488      29.28                fails           10        passes
               15 min    50004         488       9.76               passes           30        passes
               60 min   200016         482       2.41               passes          120         fails

the gap that polling closes: 2987 -> 488 records (per mille 59.74 -> 9.76)
369/488 of the remaining gap is late-arriving confirmations
cycle 15 min -> 60 min: unmatched count 488 -> 482 (nearly the same), per mille 9.76 -> 2.41

These numbers are of the measured class; they reproduce with seed 20260730.

The first table separates out polling’s contribution. In the notification-only regime, 2,987 records per cycle do not match, 59.74 per mille; since the threshold is 10 per mille, this regime fails on its own. Adding polling drops the gap to 488, 9.76 per mille: lost notifications fall from 1,550 to 119, since the query finds them within 60 seconds at the latest.

Of the remaining 488 records, 369 are late-arriving confirmations, and polling cannot close this gap: the other side has finished the work but has not yet sent the notification, so the poll also gets a “pending” reply. This is not a defect but a genuine interval between the two sides’ clocks; reconciliation’s job is to count it and wait for closure.

The second table selects the cycle interval; the most instructive row is the count itself: unmatched records are 488 at 5 minutes, 488 at 15, and 482 at 60 — practically unchanged. The gap is a boundary event, determined not by the cycle’s length but by the number of transactions at its edge; lengthening the cycle does not resolve it, only dilutes the rate from 29.28 per mille to 2.41 per mille. Closure time runs two cycle lengths: 10, 30, and 120 minutes. The two thresholds press from opposite directions, and only 15 minutes clears both.

Design

Status notification and polling. The other party’s notification is the structure from the Application Layer and Service Interaction course’s Asynchronous Request–Reply lesson; polling, meanwhile, is a pattern eliminated in this topic’s first case, there for producing 140.63x the requests as a primary delivery path. Here it only queries what is pending, so its peak load is 1.67 queries per second — the same pattern, a different role, a different parameter.

Audit trail. The append-only log from the Scaling the Data Layer course’s Event-Sourced Design lesson is the audit trail here; its parameters are the retention period and the transition count: 5 rows per transaction, 260 bytes, 3.99 TB over seven years. The trail is never overwritten, only appended to; a transaction’s current status is a projection derived from it.

What happens once a gap is found. An unmatched record triggers the step from the Resilience and Reliability course’s Compensating Transactions lesson: if the local side says “succeeded” while the other party says “declined,” the local effect is rolled back. Compensation is triggered only by reconciliation; no request path initiates it on its own.

Pattern deliberately not used. Payment status is not cached (Scaling the Data Layer, Cache Architecture): a stale status would be shown to the user as final — here, staleness is not a window, it is a defect.

Eliminated Alternative: Synchronous Verification

A design that removes the reconciliation cycle altogether can be argued for: every payment is queried synchronously, the request waits for the reply, and the two ledgers that could diverge never arise. This regime is sensitive not to load but to settlement time on the other side: as that time grows, transactions fall into the model’s third source above, the bucket that does not count as a gap; in the synchronous regime they turn directly into unanswered requests. The alternative’s model therefore gives settlement its own distribution (OD9) and the request a wait budget (OD10); a synchronous reply counts as lossless, so the measurement favors the alternative.

// payment/synchronous-verification.mjs — an in-process model of the eliminated alternative.
// There is no real counterparty: the settlement time is a parameter, and time is the model's
// own clock (seconds).
const PEAK = 55.56, CYCLE = 900, N = Math.round(PEAK * CYCLE);   // computed; OD6

function rng(seed) {                             // 32-bit linear congruential generator
  let s = seed >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; };
}
const settle = (r) =>                            // OD9: 90% 3 s, 8% 3-60 s, 2% 60-900 s
  (r < 0.9 ? 3 : r < 0.98 ? 3 + 57 * ((r - 0.9) / 0.08) : 60 + 840 * ((r - 0.98) / 0.02));

function run(budget) {                           // every payment keeps the request open until a reply arrives
  const rnd = rng(20260730);
  let unresolved = 0, held = 0;
  for (let i = 0; i < N; i += 1) {
    const s = settle(rnd());
    held += Math.min(s, budget);
    if (s > budget) unresolved += 1;             // closed unanswered, and no cycle to close it either
  }
  return { unresolved, perMille: (1000 * unresolved) / N, avg: held / N };
}

console.log(`model: peak ${PEAK} transactions/s, cycle ${CYCLE} s, ${N} transactions, seed 20260730`);
for (const b of [20, 60, 540]) {
  const r = run(b);
  console.log(`budget ${b} s: unresolved ${r.unresolved} (per mille ${r.perMille.toFixed(2)}, limit ` +
    `${r.perMille <= 10 ? "passes" : "fails"}), average hold ${r.avg.toFixed(2)} s, ` +
    `open requests at peak ${(PEAK * r.avg).toFixed(1)}`);
}

model: peak 55.56 transactions/s, cycle 900 s, 50004 transactions, seed 20260730
budget 20 s: unresolved 3789 (per mille 75.77, limit fails), average hold 4.48 s, open requests at peak 249.1
budget 60 s: unresolved 995 (per mille 19.90, limit fails), average hold 6.38 s, open requests at peak 354.4
budget 540 s: unresolved 421 (per mille 8.42, limit passes), average hold 13.06 s, open requests at peak 725.7

These numbers are of the measured class; they reproduce with seed 20260730. The number that eliminates the alternative is 75.77 per mille: at a 20-second budget, 3,789 transactions per cycle close unanswered — 7.6x the threshold — with no cycle to close them; closure time is undefined, not 30 minutes. Widening the budget is not cheap: at 60 seconds the rate still sits at 19.90 per mille, clearing the threshold only at 540 seconds — nine minutes of client wait — with 725.7 requests open at peak.

What it wins is written down too: a transaction answered within budget has its two ledgers match at the moment of reply, average closure 4.48 seconds, with reconciliation and compensation work never arising. The verdict would flip if the other party settled 99 percent of transactions within budget: the unresolved rate would drop to 10 per mille, the alternative would pass the threshold, and the reconciliation cycle would become unnecessary.

Failure Behavior and What Is Sacrificed

If the other party’s notification path stops entirely, polling becomes the only channel and every transaction is learned of within 60 seconds at most; at peak that is 55.56 queries per second, 33x normal, and the throttling from the Resilience and Reliability course’s Throttling and Load Shedding lesson cuts that rate. If a reconciliation cycle is missed, the gap is not lost — it shows up in the next cycle with twice the records; reconciliation withstands interruption because it reconstructs its state from the trail.

What is sacrificed fits in one sentence: this design sacrifices instantaneous consistency — the two ledgers never say the same thing at the same moment, only a guarantee that they will within 30 minutes.

Summary

• Volume decides nothing here: a peak of 55.56 transactions/s is carried by any structure; the real volume item is the audit trail — 1.56 GB per day, 3.99 TB over seven years.
• In the notification-only regime, 2,987 records per cycle do not match (59.74 per mille); polling brings that to 488 (9.76 per mille) and drops lost notifications from 1,550 to 119.
• 369 of the remaining 488 are late-arriving confirmations; polling cannot close this, since the other party has not reported the outcome yet either.
• Unmatched records are nearly independent of cycle length (488, 488, 482); lengthening the cycle does not resolve the gap, only dilutes the rate from 29.28 to 2.41 per mille.
• The two thresholds squeeze the cycle to a single value: the 10-per-mille limit eliminates 5 minutes, the 30-minute closure limit eliminates 60 minutes, and 15 minutes remains.
• The cycle-free synchronous-verification alternative is eliminated at 75.77 per mille unresolved records; it clears the threshold only with a 540-second wait budget and 725.7 open requests at peak.

Next Step

Five cases belonged to the same family: writes dominated, and a wrong record was unacceptable. Decisions came from delivery guarantees, channel order, window shape, lock type, and the reconciliation cycle. All five share a feature easy to miss: the data was small. A message was 220 bytes, a notification record 300 bytes, a counter 48 bytes, a stock row 64 bytes, a trail record 260 bytes; no design decision came from the byte count itself, all came from the meaning per record. Where a record is meaningless on its own, where its value only emerges in bulk, and where a single object does not fit in one server’s memory, this ordering reverses — the subject of the next topic.