Chat System

The Read-Heavy Systems topic produced four cases, and in all four reading outdid writing by a wide margin. That balance makes one thing cheap: a response being a few seconds stale was an affordable cost, because the same record was read thousands of times and its freshness depended on a single write. The staleness window was a design tool there.

This topic takes the reverse balance. In chat, a record is written once and read a small number of times; the record itself is a message a user sent. Its being lost, appearing a second time, or appearing in the wrong order is not acceptable. The same patterns are still on hand, but their parameters sit elsewhere.

Constraints and Scope

The functional requirements are five items: one-to-one message sending, group messages with at most 50 members, presence (online or last seen), delivery and read receipts, and holding messages for an offline recipient.

What is not designed is stated explicitly: voice and video calling, file attachments, in-message search, end-to-end encryption, and history export are outside this case.

The non-functional requirements are written with a threshold and the threshold’s source: lost messages are zero (source: the message being assumed to exist once “sent” is shown to the sender), the same message being shown a second time is zero (source: the recipient’s seen list being equal to the sent list), an ordering violation within a chat is zero (source: messages being replies to one another), and the median delivery delay to a connected recipient does not exceed 200 milliseconds (source: the concurrency the sender expects from the peer’s typing indicator).

Assumptions and Scale

Code	Assumption	Value	Rationale
SO1	daily active users	5,000,000	account that opens the app at least once a day
SO2	messages sent per user per day	40	short exchanges spread across a few chats
SO3	message body	220 bytes	text, sender, chat ID, timestamp
SO4	peak factor	4	ratio of evening hours to the daily average
SO5	concurrently connected user share	0.25	connection held while the app is open
SO6	recipients per message	2.4	average of the one-to-one and group mix
SO7	offline recipient share	0.35	share of recipients not connected at delivery time
SO8	retention period	365 days	history opens a year back
SO9	connections per node	50,000	open connections a single node carries

// chat/scale.mjs — the back-of-envelope calculation drawn from the SO assumption table
export const SO = {
  dailyUsers: 5_000_000,        // SO1
  messagesPerUser: 40,          // SO2
  messageBytes: 220,            // SO3
  peakFactor: 4,                // SO4
  connectedShare: 0.25,         // SO5
  avgRecipients: 2.4,           // SO6
  offlineShare: 0.35,           // SO7
  retentionDays: 365,           // SO8
  connectionsPerNode: 50_000,   // SO9
};
const DAY = 86_400;

export function calculate(v) {
  const dailyMessages = v.dailyUsers * v.messagesPerUser;
  const avgSend = dailyMessages / DAY;
  const peakSend = avgSend * v.peakFactor;
  const peakDelivery = peakSend * v.avgRecipients;
  const connected = v.dailyUsers * v.connectedShare;
  return {
    "daily messages": dailyMessages,
    "average sends/s": avgSend,
    "peak sends/s": peakSend,
    "peak deliveries/s": peakDelivery,
    "peak mailbox writes/s": peakDelivery * v.offlineShare,
    "concurrent connected users": connected,
    "connection nodes": connected / v.connectionsPerNode,
    "peak egress Mbit/s": (peakDelivery * v.messageBytes * 8) / 1e6,
    "daily data growth GB": (dailyMessages * v.messageBytes) / 1e9,
    "stored data TB": (dailyMessages * v.messageBytes * v.retentionDays) / 1e12,
  };
}

const r = calculate(SO);
const format = (x) => (Number.isInteger(x) ? String(x) : x.toFixed(2));
for (const [name, d] of Object.entries(r)) console.log(`${name.padEnd(28)}${format(d).padStart(13)}`);

const RESULT = ["peak deliveries/s", "connection nodes", "daily data growth GB"];
console.log(`\n${"assumption doubled".padEnd(25)}${RESULT.map((s) => s.split(" ")[0].padStart(11)).join("")}`);
for (const name of ["messagesPerUser", "connectedShare", "avgRecipients"]) {
  const y = calculate({ ...SO, [name]: SO[name] * 2 });
  console.log(`${name.padEnd(25)}${RESULT.map((s) => `x${(y[s] / r[s]).toFixed(2)}`.padStart(11)).join("")}`);
}

daily messages                  200000000
average sends/s                   2314.81
peak sends/s                      9259.26
peak deliveries/s                22222.22
peak mailbox writes/s             7777.78
concurrent connected users        1250000
connection nodes                       25
peak egress Mbit/s                  39.11
daily data growth GB                   44
stored data TB                      16.06

assumption doubled              peak connection      daily
messagesPerUser                x2.00      x1.00      x2.00
connectedShare                 x1.00      x2.00      x1.00
avgRecipients                  x2.00      x1.00      x1.00

These numbers belong to the calculation class. Four of them shape the design. First, peak sends are 9259.26 while peak deliveries are 22,222.22: every write produces 2.4 deliveries on average, and the load the system sees is not the write rate but the delivery rate. Second, 1,250,000 concurrent connections mean 25 connection nodes; that is 25 units of state held somewhere. Third, peak mailbox writes are 7777.78 records per second: more than a third of deliveries go to a recipient who is not connected and get written to a persistent mailbox. Fourth, bandwidth is small: 39.11 Mbit/s of egress, 44 GB of daily growth. In this case the bottleneck is not bytes but record count and connection count.

The sensitivity table separates two sources: when the connected share doubles, only the node count doubles; when the recipient count doubles, only the delivery rate doubles.

Design

Connection node. The long-lived connection turns the stickiness covered in the Traffic Layer course’s Session Stickiness lesson from a choice into a requirement: a connection is bound to a node and stays on that node. The parameter is the stickiness key: the device ID, not the user ID, because a second device for the same user is a separate connection. The mapping from device to node is done with the consistent hashing from the same course’s Balancing Algorithms lesson; on a 25-node ring, adding one node moves a connection share of 1/26.

Routing directory. The sender’s node must know which node the recipient is connected to. This is the state store from the Application Layer and Service Interaction course’s Stateless Services lesson; the record it holds is user ID, node name, and last heartbeat timestamp — the same record as presence.

Delivery path. Every delivery passes through a message broker. Order within a chat is kept with the lane from the same course’s Sequential Convoy lesson, and the lane key is the chat ID, not the sender ID: the ordering constraint exists within the chat, not across all of a sender’s chats.

Uniqueness. The client generates the message ID, and a resend carries the same ID. The scope of the uniqueness key from the Resilience and Reliability course’s Idempotent Operations lesson is the (sender, client message ID) pair here; the chat ID is not part of the scope, because there is no case of the same message going to two chats.

Offline mailbox. Messages for a recipient who is not connected are written to a persistent mailbox; the partition key from the Scaling the Data Layer course’s Partitioning Strategies lesson is the recipient ID.

Deliberately unused pattern. Edge caching (Traffic Layer, Entry Points) has no place in this design: every message goes to a single set of recipients exactly once, and no shared hit ever forms.

Delivery Guarantee and Order Are Measured

The measurement is an in-process model: there is no real socket, network, or broker; the channel is a probability and a round is an abstract step. Three more assumptions are added. SO10 — channel loss probability 0.02, justified by short cellular-connection drops; SO11 — acknowledgment loss probability 0.01, justified by the acknowledgment’s return path being shorter; SO12 — resend backoff 3 rounds. The model takes a one-second peak window: 22,222 deliveries, 40 chat lanes.

// chat/delivery.mjs — in-process model of the delivery guarantee and ordering breakdown.
// There is no real network, socket, or queue: the channel is a probability, the round is an abstract step.

export function generator(seed) {                // 32-bit linear congruential generator
  let s = seed >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; };
}

const DELIVERIES = 22222, CHATS = 40;             // calculation: peak deliveries/s x 1-second window
const LOSS = 0.02, ACK_LOSS = 0.01;               // SO10, SO11
const BACKOFF = 3;                                // SO12: resend backoff rounds

export function run({ attempts, keyed, convoy, seed }) {
  const rnd = generator(seed);
  const laneLastArrival = new Array(CHATS).fill(0);
  const laneStream = Array.from({ length: CHATS }, () => []);
  const s = { channelAttempts: 0, lost: 0, duplicated: 0, waitRounds: 0 };

  for (let i = 0; i < DELIVERIES; i += 1) {
    const chat = i % CHATS;
    const seq = Math.floor(i / CHATS);                  // the message's position in the lane and its send round
    let round = convoy ? Math.max(seq, laneLastArrival[chat]) : seq;
    s.waitRounds += round - seq;                        // waiting on the earlier message in the lane
    let reached = 0, arrival = null;
    for (let d = 0; d < attempts; d += 1) {
      s.channelAttempts += 1;
      if (rnd() < LOSS) { round += BACKOFF; continue; }        // delivery dropped on the channel
      reached += 1;
      if (arrival === null) arrival = round;
      if (rnd() < ACK_LOSS) { round += BACKOFF; continue; }    // ack dropped, sender resends
      break;
    }
    if (reached === 0) s.lost += 1;
    else {
      if (keyed === false) s.duplicated += reached - 1;
      laneStream[chat].push({ seq, arrival });
      laneLastArrival[chat] = arrival;
    }
  }

  s.inversions = laneStream.reduce((t, stream) => {
    const arrivals = stream.map((o, i) => ({ ...o, i })).sort((a, b) => a.arrival - b.arrival || a.i - b.i);
    let n = 0;
    for (let a = 0; a < arrivals.length; a += 1)
      for (let b = a + 1; b < arrivals.length; b += 1) if (arrivals[a].seq > arrivals[b].seq) n += 1;
    return t + n;
  }, 0);
  return s;
}

const SCHEMES = [
  ["at-most-once / free", { attempts: 1, keyed: false, convoy: false }],
  ["at-least-once / free", { attempts: 5, keyed: false, convoy: false }],
  ["at-least-once + key", { attempts: 5, keyed: true, convoy: false }],
  ["same + sequential convoy", { attempts: 5, keyed: true, convoy: true }],
];

console.log(`model: ${DELIVERIES} deliveries, ${CHATS} chat lanes, loss ${LOSS}, ack loss ${ACK_LOSS}, seed 20260730`);
console.log(`\n${"scheme".padEnd(26)}${"channel attempts".padStart(17)}${"lost".padStart(10)}` +
  `${"duplicated".padStart(11)}${"inversions".padStart(11)}${"wait rounds".padStart(14)}`);
const R = {};
for (const [name, choice] of SCHEMES) {
  const r = (R[name] = run({ ...choice, seed: 20260730 }));
  console.log(`${name.padEnd(26)}${String(r.channelAttempts).padStart(17)}${String(r.lost).padStart(10)}` +
    `${String(r.duplicated).padStart(11)}${String(r.inversions).padStart(11)}${String(r.waitRounds).padStart(14)}`);
}

const mostOnce = R["at-most-once / free"], atLeastOnce = R["at-least-once / free"];
const convoy = R["same + sequential convoy"];
console.log(`\nexpected loss = ${DELIVERIES} x ${LOSS} = ${(DELIVERIES * LOSS).toFixed(2)}; measured ${mostOnce.lost}`);
console.log(`cost of resending: channel attempts x${(atLeastOnce.channelAttempts / mostOnce.channelAttempts).toFixed(4)}, ` +
  `duplicate display rate ${((100 * atLeastOnce.duplicated) / DELIVERIES).toFixed(2)}%`);
console.log(`cost of the convoy: ${(convoy.waitRounds / DELIVERIES).toFixed(3)} wait rounds per message; ` +
  `inversions ${atLeastOnce.inversions} -> ${convoy.inversions}`);

model: 22222 deliveries, 40 chat lanes, loss 0.02, ack loss 0.01, seed 20260730

scheme                     channel attempts      lost duplicated inversions   wait rounds
at-most-once / free                   22222       408          0          0             0
at-least-once / free                  22870         0        229        827             0
at-least-once + key                   22870         0          0        827             0
same + sequential convoy              22870         0          0          0          1414

expected loss = 22222 x 0.02 = 444.44; measured 408
cost of resending: channel attempts x1.0292, duplicate display rate 1.03%
cost of the convoy: 0.064 wait rounds per message; inversions 827 -> 0

These numbers belong to the measurement class; they reproduce with seed 20260730.

The first row shows what at-most-once delivery gets: in the one-second peak window, 408 messages are lost, and in exchange no duplication and no ordering violation arise. Expected loss was 444.44; the measured 408 is this run’s count. Lost messages do not disrupt the order of the rest, because nothing is resent. At-most-once delivery being clean with respect to order is not a coincidence — it is the loss itself.

The second row hands over the bill for at-least-once delivery. Loss drops to zero; the cost is channel attempts rising to 1.0292 times, 229 messages reaching the recipient a second time (1.03% of deliveries), and 827 inversions. The duplication comes from the acknowledgment being lost: the message has arrived, the sender does not know it.

The third row isolates the uniqueness key’s contribution: the 229 duplicate displays drop to zero, inversions stay at 827. The key ledger resolves duplication, not order; the two are separate problems and call for separate patterns.

The fourth row is the sequential convoy’s contribution: the 827 inversions drop to zero, in exchange for 0.064 wait rounds paid per message, 1414 rounds total in the window. The wait arises from loss — when one delivery drops, the messages behind it wait in the lane. In a round without loss, the convoy’s cost is zero.

Eliminated Alternative: Polling

A design that holds no connection can be built: messages are written to the mailbox, and the client asks at a fixed interval. This design removes the 25 connection nodes, the routing directory, and session stickiness all at once. The reason for its elimination is a ratio.

// chat/alternative.mjs — numeric comparison of the push model against the polling model
const CONNECTED = 1_250_000, PEAK_DELIVERY = 22222.22;   // scale.mjs calculation: connected users, peak deliveries/s
const THRESHOLD_MS = 200;                                // delivery delay threshold (source: the write-response expectation)

console.log(`${"poll interval".padStart(16)}${"poll requests/s".padStart(17)}${"ratio to push".padStart(14)}` +
  `${"non-empty polls".padStart(16)}${"avg delay ms".padStart(17)}`);
for (const interval of [(2 * THRESHOLD_MS) / 1000, 1, 3, 10, 30]) {
  const requests = CONNECTED / interval;
  const nonEmpty = Math.min(1, (PEAK_DELIVERY * interval) / CONNECTED);
  console.log(`${`${interval} s`.padStart(16)}${requests.toFixed(0).padStart(17)}` +
    `${`x${(requests / PEAK_DELIVERY).toFixed(2)}`.padStart(14)}${`${(100 * nonEmpty).toFixed(2)}%`.padStart(16)}` +
    `${((interval * 1000) / 2).toFixed(0).padStart(17)}`);
}

const breakeven = CONNECTED / PEAK_DELIVERY;
console.log(`\nbreakeven interval = connected / peak delivery = ${breakeven.toFixed(2)} s`);
console.log(`average delivery delay at this interval = ${((breakeven * 1000) / 2 / 1000).toFixed(2)} s`);
console.log(`interval needed for a ${THRESHOLD_MS} ms threshold = ${((2 * THRESHOLD_MS) / 1000).toFixed(1)} s, ` +
  `that is, x${(CONNECTED / ((2 * THRESHOLD_MS) / 1000) / PEAK_DELIVERY).toFixed(2)} of push's requests`);

   poll interval  poll requests/s ratio to push non-empty polls     avg delay ms
           0.4 s          3125000       x140.63           0.71%              200
             1 s          1250000        x56.25           1.78%              500
             3 s           416667        x18.75           5.33%             1500
            10 s           125000         x5.63          17.78%             5000
            30 s            41667         x1.88          53.33%            15000

breakeven interval = connected / peak delivery = 56.25 s
average delivery delay at this interval = 28.13 s
interval needed for a 200 ms threshold = 0.4 s, that is, x140.63 of push's requests

The elimination number is the last line: a 200-millisecond average delay threshold demands a 0.4-second interval, which is 3,125,000 requests per second — 140.63 times the delivery rate in the push model. What is more, 99.29% of these requests return empty. Polling is a rate paid for work that is not carried.

What the alternative wins is written too. The breakeven interval is 56.25 seconds: if the delay threshold rises above this, polling produces fewer requests, and the 25 connection nodes and the routing directory drop out of the design. The elimination is a delivery-delay-threshold decision; if the threshold were one minute, the choice would reverse.

Failure Behavior and What Is Sacrificed

When a connection node goes down, 50,000 connections drop at once and all of them try to reconnect. This is the scenario from the Performance Anti-Patterns and Monitoring course’s Retry Storm lesson; its counterpart is the jitter and retry budget from the Resilience and Reliability course’s Retries and Storm Risk lesson. The parameter is a ceiling: the 50,000 connections spread across the remaining 24 nodes mean 2083 extra connections per node, that is, 4.17% utilization.

When the broker goes down, the design follows the path from the Graceful Degradation lesson: the connection stays up, presence and read receipts stop, and message sending falls through to the offline mailbox. What is sacrificed, in one sentence: a single lane per chat accepts giving up chat-level parallelism for the sake of in-chat order, and a 1414-round wait during rounds with loss.

Summary

• When writing dominates, the measure is record count: peak sends are 9259.26 while peak deliveries are 22,222.22, and bandwidth stays small at 39.11 Mbit/s.
• 1,250,000 concurrent connections mean 25 nodes; when the connected share doubles, the delivery rate does not change, only the node count doubles.
• At-most-once delivery lost 408 messages in the one-second peak window and never disrupted order; at-least-once delivery zeroed out loss but produced 229 duplicate displays and 827 inversions.
• The uniqueness key zeroed out duplication and left inversions at 827; the sequential convoy zeroed out inversions and took 0.064 wait rounds per message.
• Polling demands 140.63 times push’s requests at the 200-millisecond threshold, and 99.29% of them return empty; the breakeven interval is 56.25 seconds.
• Edge caching was deliberately not used: every message goes to a single set of recipients and never produces a shared hit.

Next Step

In this case delivery moved along a single path: the recipient was either connected or the message was written to their mailbox, and failure was met with a resend. The next case asks the same question where more than one path exists. The same notification can go out over different channels, the channels’ failure rates differ from one another, and one channel’s failure is grounds for trying another. The number to ask then is this: how much does trying the channels in sequence versus at the same time move the total delivery rate and the resend load.