Notification System

In the previous case, delivery moved along a single path, and the only response to failure was retrying that same path. In this case there is more than one path. The same notification can go out by app push, SMS, or email; the channels’ failure rates differ from one another, and one channel’s failure is grounds for trying another.

This changes the resend question. Where an attempt gets spent is now the question of “the same place, or somewhere else,” and the answer moves two numbers at once: the total delivery rate and the number of outbound calls.

Constraints and Scope

The functional requirements: accepting the notification request, choosing a channel by the recipient’s preference, formatting per channel, recording delivery status, and preventing the same event from going to the recipient a second time.

What is not designed: writing the notification text and the templating language, translation, the permissions screen, choosing recipient sets, and the internal workings of the channel providers themselves.

The non-functional requirements are written with a threshold and its source: the total delivery rate does not fall below 0.99 (source: the notification being the sole messenger for a transaction), the average delivery delay for an urgent notification does not exceed 5 seconds (source: the recipient expecting the news at the moment they perform the action), and the number of deliveries reaching the recipient for one event is one (source: receiving the same news over two channels counting as an error for the recipient).

Assumptions and Scale

Code	Assumption	Value	Rationale
BI1	daily active recipients	5,000,000	account able to receive notifications
BI2	notifications per recipient per day	6	transaction news and reminders
BI3	peak factor	5	notifications piling into a few hours of the day
BI4	channel share	0.55 / 0.25 / 0.20	preference split across push, SMS, email
BI5	transient failure per channel	0.05 / 0.02 / 0.01	momentary unreachability, a resend fixes it
BI6	permanent failure per channel	0.06 / 0.02 / 0.01	invalid address, revoked permission, a resend does not fix it
BI7	notification record	300 bytes	event ID, recipient, channel, status, timestamp
BI8	attempt budget	3	most outbound calls made per notification
BI9	wait between attempts	30 seconds	time left for a transient failure to pass

The BI5–BI6 distinction is this case’s decisive assumption. A transient failure clears when retried on the same channel; a permanent failure does not clear, and every attempt spent on the same channel is wasted.

// notification/scale.mjs — the back-of-envelope calculation drawn from the BI assumption table
export const BI = {
  dailyRecipients: 5_000_000,        // BI1
  notificationsPerRecipient: 6,      // BI2
  peakFactor: 5,                     // BI3
  share: { push: 0.55, sms: 0.25, email: 0.20 },          // BI4
  transient: { push: 0.05, sms: 0.02, email: 0.01 },      // BI5
  permanent: { push: 0.06, sms: 0.02, email: 0.01 },      // BI6
  recordBytes: 300,                  // BI7
  budget: 3,                         // BI8
  wait: 30,                          // BI9
};
const DAY = 86_400;

const daily = BI.dailyRecipients * BI.notificationsPerRecipient;
const peak = (daily / DAY) * BI.peakFactor;
console.log(`daily notifications    ${daily}`);
console.log(`peak notifications/s    ${peak.toFixed(2)}`);
console.log(`daily data growth GB    ${((daily * BI.recordBytes) / 1e9).toFixed(2)}`);

console.log(`\n${"channel".padEnd(11)}${"share".padStart(6)}${"peak calls/s".padStart(14)}${"transient".padStart(11)}` +
  `${"permanent".padStart(11)}${"remaining after 3 attempts".padStart(28)}`);
let blendedRemaining = 0;
for (const ch of Object.keys(BI.share)) {
  const remaining = BI.permanent[ch] + BI.transient[ch] ** BI.budget;
  blendedRemaining += BI.share[ch] * remaining;
  console.log(`${ch.padEnd(11)}${BI.share[ch].toFixed(2).padStart(6)}${(peak * BI.share[ch]).toFixed(2).padStart(14)}` +
    `${BI.transient[ch].toFixed(2).padStart(11)}${BI.permanent[ch].toFixed(2).padStart(11)}${remaining.toFixed(6).padStart(28)}`);
}
console.log(`\nsingle channel, ${BI.budget} attempts -> blended remaining error ${blendedRemaining.toFixed(6)}, ` +
  `delivery rate ${(1 - blendedRemaining).toFixed(4)}`);
console.log(`threshold 0.9900 -> single channel ${1 - blendedRemaining >= 0.99 ? "passes" : "fails"}; ` +
  `gap ${((0.99 - (1 - blendedRemaining)) * 100).toFixed(2)} points`);
console.log(`unreached notifications: peak ${(peak * blendedRemaining).toFixed(2)}/s, ` +
  `daily ${Math.round(daily * blendedRemaining)}`);
console.log(`cascading worst-case delivery = (${BI.budget} - 1) x ${BI.wait} s = ${(BI.budget - 1) * BI.wait} s`);

daily notifications    30000000
peak notifications/s    1736.11
daily data growth GB    9.00

channel     share  peak calls/s  transient  permanent  remaining after 3 attempts
push         0.55        954.86       0.05       0.06                    0.060125
sms          0.25        434.03       0.02       0.02                    0.020008
email        0.20        347.22       0.01       0.01                    0.010001

single channel, 3 attempts -> blended remaining error 0.040071, delivery rate 0.9599
threshold 0.9900 -> single channel fails; gap 3.01 points
unreached notifications: peak 69.57/s, daily 1202129
cascading worst-case delivery = (3 - 1) x 30 s = 60 s

These numbers belong to the calculation class. Three of them shape the design. First, peak load does not split evenly across channels: the push channel takes 954.86 calls per second while email takes 347.22, so the most crowded channel is also the channel with the highest permanent failure rate. Second, three attempts on the same channel all but erase the transient failure (0.05 to the third power for push, that is, 0.000125) but do not touch the permanent one; the remaining error comes almost entirely from BI6. Third, the single-channel design’s delivery rate comes out to 0.9599 and stays 3.01 points below the 0.99 threshold: 1,202,129 notifications a day never arrive at all. The threshold forces a channel-order decision.

Design

Intake and queue. The notification request is accepted asynchronously; the Application Layer and Service Interaction course’s Message Queues and Competing Consumers lessons give the structure here. The parameter is priority classes: two classes from the same course’s Priority Queue lesson are used, urgent notification and reminder, because only one of the thresholds carries a delay.

Bulkhead per channel. The resource pool from the Resilience and Reliability course’s Bulkhead Pattern lesson is split per channel, and pool shares are given by the peak call rate: 954.86, 434.03, and 347.22 calls/s. The reason for the split is that one channel slowing down must not make the other channels’ calls wait.

Circuit breaker per channel. The threshold in the same course’s Circuit Breaker lesson is here a choice of what to count: the circuit breaker counts only transient failures. A permanent failure is specific to the recipient and carries no information about the channel’s health; if permanent failures counted toward it, the push channel’s circuit would trip constantly because of a 6% floor.

Placement of the attempt budget. The attempt budget in the Retries and Storm Risk lesson is 3, and this case’s decision is where the budget gets spent: on the same channel, or on the next one in line. The measurement section turns this decision into a number.

Uniqueness. The scope of the uniqueness key from the Resilience and Reliability course’s Idempotent Operations lesson is the (event ID, recipient) pair here; the channel is not part of the scope, because the same event going out over two channels is exactly what is meant to be prevented.

Deliberately unused pattern. The sequential convoy (Application Layer and Service Interaction, Queues and Workflows) has no place in this design: there is no ordering constraint between two notifications, each is the messenger for an independent event, and imposing a lane would cut parallelism for no reason.

Channel Order Is Measured

The measurement is an in-process model: there is no real channel, external service, or network; each attempt is a probability, and a round is an abstract step, one round being 30 seconds by BI9. The model takes a one-minute peak window: 104,167 notifications. Three arrangements spend the same 3-call budget in different places — single channel spends all three on the preferred channel, parallel spends all three on three channels at once, cascading spends one on the preferred channel and the rest on the others in sequence.

// notification/distribution.mjs — in-process model of channel order. There is no real channel,
// external service, or network: each attempt is a probability, a round is an abstract step (BI9: 30 s wait per round).
const CHANNEL = ["push", "sms", "email"];
const SHARE = { push: 0.55, sms: 0.25, email: 0.20 };            // BI4
const TRANSIENT = { push: 0.05, sms: 0.02, email: 0.01 };        // BI5
const PERMANENT = { push: 0.06, sms: 0.02, email: 0.01 };        // BI6
const BUDGET = 3, WAIT = 30;                                     // BI8, BI9
const NOTIFICATIONS = 104_167;               // calculation: peak 1736.11 notifications/s x 60 s window

function generator(seed) {                   // 32-bit linear congruential generator
  let s = seed >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; };
}

export function run({ arrangement, seed, multiplier = 1, down = null }) {
  const rnd = generator(seed);
  const s = { delivered: 0, outboundCalls: 0, duplicated: 0, roundTotal: 0 };
  for (let i = 0; i < NOTIFICATIONS; i += 1) {
    let u = rnd(), preferred = CHANNEL[CHANNEL.length - 1];
    for (const k of CHANNEL) { if (u < SHARE[k]) { preferred = k; break; } u -= SHARE[k]; }
    const isPermanent = {};
    for (const k of CHANNEL) isPermanent[k] = k === down || rnd() < Math.min(1, PERMANENT[k] * multiplier);
    const sequence = arrangement === "single channel" ? [preferred, preferred, preferred]
      : [preferred, ...CHANNEL.filter((k) => k !== preferred)];

    let reached = 0, firstRound = null;
    for (let t = 0; t < BUDGET; t += 1) {
      const k = sequence[t];
      s.outboundCalls += 1;
      const succeeded = isPermanent[k] === false && rnd() >= Math.min(1, TRANSIENT[k] * multiplier);
      if (succeeded) {
        reached += 1;
        if (firstRound === null) firstRound = arrangement === "parallel" ? 0 : t;
        if (arrangement !== "parallel") break;
      }
    }
    if (reached > 0) { s.delivered += 1; s.roundTotal += firstRound; s.duplicated += reached - 1; }
  }
  s.rate = s.delivered / NOTIFICATIONS;
  return s;
}

const ARRANGEMENT = ["single channel", "parallel", "cascading"];
console.log(`model: ${NOTIFICATIONS} notifications, ${CHANNEL.length} channels, attempt budget ${BUDGET}, seed 20260730`);
console.log(`\n${"arrangement".padEnd(16)}${"delivery rate".padStart(15)}${"outbound calls".padStart(16)}` +
  `${"calls/notification".padStart(20)}${"duplicated".padStart(12)}${"avg delivery s".padStart(16)}`);
const R = {};
for (const a of ARRANGEMENT) {
  const r = (R[a] = run({ arrangement: a, seed: 20260730 }));
  console.log(`${a.padEnd(16)}${r.rate.toFixed(4).padStart(15)}${String(r.outboundCalls).padStart(16)}` +
    `${(r.outboundCalls / NOTIFICATIONS).toFixed(3).padStart(20)}${String(r.duplicated).padStart(12)}` +
    `${((r.roundTotal / r.delivered) * WAIT).toFixed(2).padStart(16)}`);
}

console.log(`\n${"failure multiplier".padEnd(20)}${ARRANGEMENT.map((a) => `${a} rate`.padStart(21)).join("")}`);
for (const c of [1, 2, 3]) {
  const row = ARRANGEMENT.map((a) => run({ arrangement: a, seed: 20260730, multiplier: c }).rate.toFixed(4).padStart(21));
  console.log(`${`x${c}`.padEnd(20)}${row.join("")}`);
}

console.log(`\n${"push channel down entirely".padEnd(28)}${ARRANGEMENT.map((a) => a.padStart(17)).join("")}`);
const downed = ARRANGEMENT.map((a) => run({ arrangement: a, seed: 20260730, down: "push" }));
console.log(`${"delivery rate".padEnd(28)}${downed.map((r) => r.rate.toFixed(4).padStart(17)).join("")}`);
console.log(`${"outbound calls".padEnd(28)}${downed.map((r) => String(r.outboundCalls).padStart(17)).join("")}`);

const single = R["single channel"], par = R["parallel"], cas = R["cascading"];
console.log(`\ncascading - single channel delivery rate difference = ${((cas.rate - single.rate) * 100).toFixed(2)} points`);
console.log(`cost of parallel: outbound calls x${(par.outboundCalls / cas.outboundCalls).toFixed(2)}, ` +
  `${par.duplicated} duplicated deliveries, ${(par.duplicated / NOTIFICATIONS).toFixed(2)} extra per notification`);
console.log(`gain of parallel: avg delivery ${((par.roundTotal / par.delivered) * WAIT).toFixed(2)} s, ` +
  `cascading ${((cas.roundTotal / cas.delivered) * WAIT).toFixed(2)} s`);

model: 104167 notifications, 3 channels, attempt budget 3, seed 20260730

arrangement       delivery rate  outbound calls  calls/notification  duplicated  avg delivery s
single channel           0.9590          116305               1.117           0            1.08
parallel                 0.9999          312501               3.000      190861            0.00
cascading                0.9999          112357               1.079           0            2.35

failure multiplier    single channel rate        parallel rate       cascading rate
x1                                 0.9590               0.9999               0.9999
x2                                 0.9199               0.9993               0.9994
x3                                 0.8790               0.9979               0.9979

push channel down entirely     single channel         parallel        cascading
delivery rate                          0.4443           0.9993           0.9991
outbound calls                         220661           312501           166806

cascading - single channel delivery rate difference = 4.09 points
cost of parallel: outbound calls x2.78, 190861 duplicated deliveries, 1.83 extra per notification
gain of parallel: avg delivery 0.00 s, cascading 2.35 s

These numbers belong to the measurement class; they reproduce with seed 20260730.

The first table prices out the placement of the budget. Single channel gives a 0.9590 delivery rate, the same order of magnitude as the 0.9599 found by calculation; cascading climbs to 0.9999 with the same three-call budget, a difference of 4.09 points. What stands out is the second column: the cascading arrangement does this with fewer outbound calls, 1.079 calls per notification, below single channel’s 1.117 calls. A notification that hits a permanent failure wastes all three calls in the single-channel arrangement; in the cascading arrangement, the second call finishes the job. Where a resend goes matters more than how many there are.

The second table gives the sensitivity of the failure rates. When every channel’s failure rate triples, single channel drops to 0.8790, cascading stays at 0.9979. Channel diversity’s gain grows as the failure rate grows: the difference climbs from 4.09 points to 11.89 points.

Eliminated Alternative: Parallel Fan-Out

The parallel arrangement gives the same result as cascading in delivery rate (0.9999) and beats it on delay: average delivery is 0.00 seconds, versus 2.35 seconds for cascading. Even so, it is eliminated by two numbers.

First, the outbound call count is 2.78 times cascading’s, and it is fixed at 3.000 calls per notification; it does not fall even when channel failure falls, because the parallel arrangement sends all three without waiting for a result. Second, and decisive, it produces 190,861 duplicated deliveries: 1.83 extra deliveries per notification. This directly violates the non-functional requirement — the recipient gets the same news over an average of nearly three channels at once.

What the alternative wins is written too: the parallel arrangement zeroes out the 60-second worst-case delivery window. If the delay threshold were 1 second instead of 5 and the “single delivery” requirement were lifted, the choice would reverse; a middle path where two channels are tried at once would also be defensible at this threshold.

Failure Behavior and What Is Sacrificed

The failure scenario is one channel stopping entirely: the push channel fails for every recipient. The third table gives the result. The single-channel arrangement drops to 0.4443 — more than half the notifications never arrive — and its outbound calls rise to 220,661, that is, 1.90 times normal, because attempts that go to waste use up the whole budget. The cascading arrangement stays at 0.9991, and its outbound calls rise to 166,806. This is where the circuit breaker per channel earns its keep: once the push channel’s circuit trips, the sequence starts directly from the second channel, and the 54,449 wasted calls the failure would have added (166,806 minus 112,357) never happen.

What is sacrificed is in one sentence: the cascading order buys its delivery rate with delay — a recipient whose first channel hits a permanent failure gets the notification 30 seconds late, and 60 seconds late if the second one fails too.

Summary

• Channel failure cannot be designed for without splitting it in two: transient failure is erased in three attempts (0.05 to the third power), permanent failure never shrinks on the same channel, and it makes up nearly all of the remaining error.
• The single-channel arrangement gives a delivery rate of 0.9599 calculated, 0.9590 measured; that is 3.01 points below the 0.99 threshold, and 1,202,129 notifications a day do not arrive.
• Cascading order gives 0.9999 with the same three-call budget, and does it with fewer outbound calls: 1.079 calls per notification, against single channel’s 1.117 calls.
• When failure rates triple, single channel drops to 0.8790, cascading stays at 0.9979; diversity’s gain climbs from 4.09 points to 11.89 points.
• Parallel fan-out gives the same delivery rate but produces 2.78 times the outbound calls and 1.83 extra deliveries per notification; it is eliminated for violating the single-delivery requirement.
• When the push channel fails entirely, single channel drops to 0.4443, cascading stays at 0.9991; the cost is the notification arriving 30 or 60 seconds late.

Next Step

In both of these cases the system’s job was to satisfy a request, and failure was a bad outcome. The next case designs the reverse: a component whose job is to reject the request. The rejection decision is made by looking at a shared counter, and the counter is distributed; two questions follow from this. In what form the counter’s window is defined determines how far above the limit it is possible to climb, and how much drift concurrently reading and writing nodes produce in the counter. Both are countable.