--- title: 'Notification System' source: 'https://academia.sh/en/courses/case-studies/notification-system' course: 'Case Studies' language: en updated: '2026-08-23T14:25:22+00:00' license: 'CC BY-SA 4.0' --- # Notification System The case where the same message can go out over more than one path: separating channel-specific transient and permanent failure rates, three attempts on a single channel leaving the delivery rate 3.01 points below the threshold, measuring single-channel against parallel and cascading channel order in delivery rate, outbound calls, duplicate deliveries, and delay, and parallel fan-out eliminated by its call amplification. In the previous case, delivery moved along a single path, and the only response to failure was retrying that same path. In this case there is more than one path. The same notification can go out by app push, SMS, or email; the channels' failure rates differ from one another, and one channel's failure is grounds for trying another. This changes the resend question. Where an attempt gets spent is now the question of "the same place, or somewhere else," and the answer moves two numbers at once: the total delivery rate and the number of outbound calls. ## Constraints and Scope The functional requirements: accepting the notification request, choosing a channel by the recipient's preference, formatting per channel, recording delivery status, and preventing the same event from going to the recipient a second time. What is not designed: writing the notification text and the templating language, translation, the permissions screen, choosing recipient sets, and the internal workings of the channel providers themselves. The non-functional requirements are written with a threshold and its source: the total delivery rate does not fall below 0.99 (source: the notification being the sole messenger for a transaction), the average delivery delay for an urgent notification does not exceed 5 seconds (source: the recipient expecting the news at the moment they perform the action), and the number of deliveries reaching the recipient for one event is one (source: receiving the same news over two channels counting as an error for the recipient). ## Assumptions and Scale | Code | Assumption | Value | Rationale | |---|---|---|---| | BI1 | daily active recipients | 5,000,000 | account able to receive notifications | | BI2 | notifications per recipient per day | 6 | transaction news and reminders | | BI3 | peak factor | 5 | notifications piling into a few hours of the day | | BI4 | channel share | 0.55 / 0.25 / 0.20 | preference split across push, SMS, email | | BI5 | transient failure per channel | 0.05 / 0.02 / 0.01 | momentary unreachability, a resend fixes it | | BI6 | permanent failure per channel | 0.06 / 0.02 / 0.01 | invalid address, revoked permission, a resend does not fix it | | BI7 | notification record | 300 bytes | event ID, recipient, channel, status, timestamp | | BI8 | attempt budget | 3 | most outbound calls made per notification | | BI9 | wait between attempts | 30 seconds | time left for a transient failure to pass | The BI5–BI6 distinction is this case's decisive assumption. A transient failure clears when retried on the same channel; a permanent failure does not clear, and every attempt spent on the same channel is wasted. ```js // notification/scale.mjs — the back-of-envelope calculation drawn from the BI assumption table export const BI = { dailyRecipients: 5_000_000, // BI1 notificationsPerRecipient: 6, // BI2 peakFactor: 5, // BI3 share: { push: 0.55, sms: 0.25, email: 0.20 }, // BI4 transient: { push: 0.05, sms: 0.02, email: 0.01 }, // BI5 permanent: { push: 0.06, sms: 0.02, email: 0.01 }, // BI6 recordBytes: 300, // BI7 budget: 3, // BI8 wait: 30, // BI9 }; const DAY = 86_400; const daily = BI.dailyRecipients * BI.notificationsPerRecipient; const peak = (daily / DAY) * BI.peakFactor; console.log(`daily notifications ${daily}`); console.log(`peak notifications/s ${peak.toFixed(2)}`); console.log(`daily data growth GB ${((daily * BI.recordBytes) / 1e9).toFixed(2)}`); console.log(`\n${"channel".padEnd(11)}${"share".padStart(6)}${"peak calls/s".padStart(14)}${"transient".padStart(11)}` + `${"permanent".padStart(11)}${"remaining after 3 attempts".padStart(28)}`); let blendedRemaining = 0; for (const ch of Object.keys(BI.share)) { const remaining = BI.permanent[ch] + BI.transient[ch] ** BI.budget; blendedRemaining += BI.share[ch] * remaining; console.log(`${ch.padEnd(11)}${BI.share[ch].toFixed(2).padStart(6)}${(peak * BI.share[ch]).toFixed(2).padStart(14)}` + `${BI.transient[ch].toFixed(2).padStart(11)}${BI.permanent[ch].toFixed(2).padStart(11)}${remaining.toFixed(6).padStart(28)}`); } console.log(`\nsingle channel, ${BI.budget} attempts -> blended remaining error ${blendedRemaining.toFixed(6)}, ` + `delivery rate ${(1 - blendedRemaining).toFixed(4)}`); console.log(`threshold 0.9900 -> single channel ${1 - blendedRemaining >= 0.99 ? "passes" : "fails"}; ` + `gap ${((0.99 - (1 - blendedRemaining)) * 100).toFixed(2)} points`); console.log(`unreached notifications: peak ${(peak * blendedRemaining).toFixed(2)}/s, ` + `daily ${Math.round(daily * blendedRemaining)}`); console.log(`cascading worst-case delivery = (${BI.budget} - 1) x ${BI.wait} s = ${(BI.budget - 1) * BI.wait} s`); ``` ``` daily notifications 30000000 peak notifications/s 1736.11 daily data growth GB 9.00 channel share peak calls/s transient permanent remaining after 3 attempts push 0.55 954.86 0.05 0.06 0.060125 sms 0.25 434.03 0.02 0.02 0.020008 email 0.20 347.22 0.01 0.01 0.010001 single channel, 3 attempts -> blended remaining error 0.040071, delivery rate 0.9599 threshold 0.9900 -> single channel fails; gap 3.01 points unreached notifications: peak 69.57/s, daily 1202129 cascading worst-case delivery = (3 - 1) x 30 s = 60 s ``` These numbers belong to the **calculation** class. Three of them shape the design. First, peak load does not split evenly across channels: the push channel takes 954.86 calls per second while email takes 347.22, so the most crowded channel is also the channel with the highest permanent failure rate. Second, three attempts on the same channel all but erase the transient failure (0.05 to the third power for push, that is, 0.000125) but do not touch the permanent one; the remaining error comes almost entirely from BI6. Third, the single-channel design's delivery rate comes out to 0.9599 and stays 3.01 points below the 0.99 threshold: 1,202,129 notifications a day never arrive at all. The threshold forces a channel-order decision. ## Design **Intake and queue.** The notification request is accepted asynchronously; the Application Layer and Service Interaction course's Message Queues and Competing Consumers lessons give the structure here. The parameter is priority classes: two classes from the same course's Priority Queue lesson are used, urgent notification and reminder, because only one of the thresholds carries a delay. **Bulkhead per channel.** The resource pool from the Resilience and Reliability course's Bulkhead Pattern lesson is split per channel, and pool shares are given by the peak call rate: 954.86, 434.03, and 347.22 calls/s. The reason for the split is that one channel slowing down must not make the other channels' calls wait. **Circuit breaker per channel.** The threshold in the same course's Circuit Breaker lesson is here a choice of what to count: the circuit breaker counts only transient failures. A permanent failure is specific to the recipient and carries no information about the channel's health; if permanent failures counted toward it, the push channel's circuit would trip constantly because of a 6% floor. **Placement of the attempt budget.** The attempt budget in the Retries and Storm Risk lesson is 3, and this case's decision is where the budget gets spent: on the same channel, or on the next one in line. The measurement section turns this decision into a number. **Uniqueness.** The scope of the uniqueness key from the Resilience and Reliability course's Idempotent Operations lesson is the `(event ID, recipient)` pair here; the channel is not part of the scope, because the same event going out over two channels is exactly what is meant to be prevented. **Deliberately unused pattern.** The sequential convoy (Application Layer and Service Interaction, Queues and Workflows) has no place in this design: there is no ordering constraint between two notifications, each is the messenger for an independent event, and imposing a lane would cut parallelism for no reason. ## Channel Order Is Measured The measurement is an in-process model: there is no real channel, external service, or network; each attempt is a probability, and a round is an abstract step, one round being 30 seconds by BI9. The model takes a one-minute peak window: 104,167 notifications. Three arrangements spend the same 3-call budget in different places — **single channel** spends all three on the preferred channel, **parallel** spends all three on three channels at once, **cascading** spends one on the preferred channel and the rest on the others in sequence. ```js // notification/distribution.mjs — in-process model of channel order. There is no real channel, // external service, or network: each attempt is a probability, a round is an abstract step (BI9: 30 s wait per round). const CHANNEL = ["push", "sms", "email"]; const SHARE = { push: 0.55, sms: 0.25, email: 0.20 }; // BI4 const TRANSIENT = { push: 0.05, sms: 0.02, email: 0.01 }; // BI5 const PERMANENT = { push: 0.06, sms: 0.02, email: 0.01 }; // BI6 const BUDGET = 3, WAIT = 30; // BI8, BI9 const NOTIFICATIONS = 104_167; // calculation: peak 1736.11 notifications/s x 60 s window function generator(seed) { // 32-bit linear congruential generator let s = seed >>> 0; return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; }; } export function run({ arrangement, seed, multiplier = 1, down = null }) { const rnd = generator(seed); const s = { delivered: 0, outboundCalls: 0, duplicated: 0, roundTotal: 0 }; for (let i = 0; i < NOTIFICATIONS; i += 1) { let u = rnd(), preferred = CHANNEL[CHANNEL.length - 1]; for (const k of CHANNEL) { if (u < SHARE[k]) { preferred = k; break; } u -= SHARE[k]; } const isPermanent = {}; for (const k of CHANNEL) isPermanent[k] = k === down || rnd() < Math.min(1, PERMANENT[k] * multiplier); const sequence = arrangement === "single channel" ? [preferred, preferred, preferred] : [preferred, ...CHANNEL.filter((k) => k !== preferred)]; let reached = 0, firstRound = null; for (let t = 0; t < BUDGET; t += 1) { const k = sequence[t]; s.outboundCalls += 1; const succeeded = isPermanent[k] === false && rnd() >= Math.min(1, TRANSIENT[k] * multiplier); if (succeeded) { reached += 1; if (firstRound === null) firstRound = arrangement === "parallel" ? 0 : t; if (arrangement !== "parallel") break; } } if (reached > 0) { s.delivered += 1; s.roundTotal += firstRound; s.duplicated += reached - 1; } } s.rate = s.delivered / NOTIFICATIONS; return s; } const ARRANGEMENT = ["single channel", "parallel", "cascading"]; console.log(`model: ${NOTIFICATIONS} notifications, ${CHANNEL.length} channels, attempt budget ${BUDGET}, seed 20260730`); console.log(`\n${"arrangement".padEnd(16)}${"delivery rate".padStart(15)}${"outbound calls".padStart(16)}` + `${"calls/notification".padStart(20)}${"duplicated".padStart(12)}${"avg delivery s".padStart(16)}`); const R = {}; for (const a of ARRANGEMENT) { const r = (R[a] = run({ arrangement: a, seed: 20260730 })); console.log(`${a.padEnd(16)}${r.rate.toFixed(4).padStart(15)}${String(r.outboundCalls).padStart(16)}` + `${(r.outboundCalls / NOTIFICATIONS).toFixed(3).padStart(20)}${String(r.duplicated).padStart(12)}` + `${((r.roundTotal / r.delivered) * WAIT).toFixed(2).padStart(16)}`); } console.log(`\n${"failure multiplier".padEnd(20)}${ARRANGEMENT.map((a) => `${a} rate`.padStart(21)).join("")}`); for (const c of [1, 2, 3]) { const row = ARRANGEMENT.map((a) => run({ arrangement: a, seed: 20260730, multiplier: c }).rate.toFixed(4).padStart(21)); console.log(`${`x${c}`.padEnd(20)}${row.join("")}`); } console.log(`\n${"push channel down entirely".padEnd(28)}${ARRANGEMENT.map((a) => a.padStart(17)).join("")}`); const downed = ARRANGEMENT.map((a) => run({ arrangement: a, seed: 20260730, down: "push" })); console.log(`${"delivery rate".padEnd(28)}${downed.map((r) => r.rate.toFixed(4).padStart(17)).join("")}`); console.log(`${"outbound calls".padEnd(28)}${downed.map((r) => String(r.outboundCalls).padStart(17)).join("")}`); const single = R["single channel"], par = R["parallel"], cas = R["cascading"]; console.log(`\ncascading - single channel delivery rate difference = ${((cas.rate - single.rate) * 100).toFixed(2)} points`); console.log(`cost of parallel: outbound calls x${(par.outboundCalls / cas.outboundCalls).toFixed(2)}, ` + `${par.duplicated} duplicated deliveries, ${(par.duplicated / NOTIFICATIONS).toFixed(2)} extra per notification`); console.log(`gain of parallel: avg delivery ${((par.roundTotal / par.delivered) * WAIT).toFixed(2)} s, ` + `cascading ${((cas.roundTotal / cas.delivered) * WAIT).toFixed(2)} s`); ``` ``` model: 104167 notifications, 3 channels, attempt budget 3, seed 20260730 arrangement delivery rate outbound calls calls/notification duplicated avg delivery s single channel 0.9590 116305 1.117 0 1.08 parallel 0.9999 312501 3.000 190861 0.00 cascading 0.9999 112357 1.079 0 2.35 failure multiplier single channel rate parallel rate cascading rate x1 0.9590 0.9999 0.9999 x2 0.9199 0.9993 0.9994 x3 0.8790 0.9979 0.9979 push channel down entirely single channel parallel cascading delivery rate 0.4443 0.9993 0.9991 outbound calls 220661 312501 166806 cascading - single channel delivery rate difference = 4.09 points cost of parallel: outbound calls x2.78, 190861 duplicated deliveries, 1.83 extra per notification gain of parallel: avg delivery 0.00 s, cascading 2.35 s ``` These numbers belong to the **measurement** class; they reproduce with seed 20260730. The first table prices out the placement of the budget. Single channel gives a 0.9590 delivery rate, the same order of magnitude as the 0.9599 found by calculation; cascading climbs to 0.9999 with the same three-call budget, a difference of 4.09 points. What stands out is the second column: the cascading arrangement does this with **fewer** outbound calls, 1.079 calls per notification, below single channel's 1.117 calls. A notification that hits a permanent failure wastes all three calls in the single-channel arrangement; in the cascading arrangement, the second call finishes the job. Where a resend goes matters more than how many there are. The second table gives the sensitivity of the failure rates. When every channel's failure rate triples, single channel drops to 0.8790, cascading stays at 0.9979. Channel diversity's gain grows as the failure rate grows: the difference climbs from 4.09 points to 11.89 points. ## Eliminated Alternative: Parallel Fan-Out The parallel arrangement gives the same result as cascading in delivery rate (0.9999) and beats it on delay: average delivery is 0.00 seconds, versus 2.35 seconds for cascading. Even so, it is eliminated by two numbers. First, the outbound call count is 2.78 times cascading's, and it is fixed at 3.000 calls per notification; it does not fall even when channel failure falls, because the parallel arrangement sends all three without waiting for a result. Second, and decisive, it produces 190,861 duplicated deliveries: 1.83 extra deliveries per notification. This directly violates the non-functional requirement — the recipient gets the same news over an average of nearly three channels at once. What the alternative wins is written too: the parallel arrangement zeroes out the 60-second worst-case delivery window. If the delay threshold were 1 second instead of 5 and the "single delivery" requirement were lifted, the choice would reverse; a middle path where two channels are tried at once would also be defensible at this threshold. ## Failure Behavior and What Is Sacrificed The failure scenario is one channel stopping entirely: the push channel fails for every recipient. The third table gives the result. The single-channel arrangement drops to 0.4443 — more than half the notifications never arrive — and its outbound calls rise to 220,661, that is, 1.90 times normal, because attempts that go to waste use up the whole budget. The cascading arrangement stays at 0.9991, and its outbound calls rise to 166,806. This is where the circuit breaker per channel earns its keep: once the push channel's circuit trips, the sequence starts directly from the second channel, and the 54,449 wasted calls the failure would have added (166,806 minus 112,357) never happen. What is sacrificed is in one sentence: the cascading order buys its delivery rate with delay — a recipient whose first channel hits a permanent failure gets the notification 30 seconds late, and 60 seconds late if the second one fails too. ## Summary - Channel failure cannot be designed for without splitting it in two: transient failure is erased in three attempts (0.05 to the third power), permanent failure never shrinks on the same channel, and it makes up nearly all of the remaining error. - The single-channel arrangement gives a delivery rate of 0.9599 calculated, 0.9590 measured; that is 3.01 points below the 0.99 threshold, and 1,202,129 notifications a day do not arrive. - Cascading order gives 0.9999 with the same three-call budget, and does it with fewer outbound calls: 1.079 calls per notification, against single channel's 1.117 calls. - When failure rates triple, single channel drops to 0.8790, cascading stays at 0.9979; diversity's gain climbs from 4.09 points to 11.89 points. - Parallel fan-out gives the same delivery rate but produces 2.78 times the outbound calls and 1.83 extra deliveries per notification; it is eliminated for violating the single-delivery requirement. - When the push channel fails entirely, single channel drops to 0.4443, cascading stays at 0.9991; the cost is the notification arriving 30 or 60 seconds late. ## Next Step In both of these cases the system's job was to satisfy a request, and failure was a bad outcome. The next case designs the reverse: a component whose job is to **reject** the request. The rejection decision is made by looking at a shared counter, and the counter is distributed; two questions follow from this. In what form the counter's window is defined determines how far above the limit it is possible to climb, and how much drift concurrently reading and writing nodes produce in the counter. Both are countable.