---
title: 'The Application Layer and Service Interaction'
source: 'https://academia.sh/en/courses/application-layer'
language: en
updated: '2026-08-23T07:01:22+00:00'
license: 'CC BY-SA 4.0'
lessons: 15
outcomes:
    - 'Makes horizontal scaling possible with stateless service design'
    - 'Chooses between synchronous and asynchronous communication'
    - 'Builds workflows through choreography or orchestration'
---

# The Application Layer and Service Interaction

What follows once the request reaches the application: making horizontal scaling possible by moving state out of the service, services finding one another and choosing how they talk, contracts evolving while compatibility is preserved, the effect of timeout and retry budget on total latency in a call chain, spreading load over time with queues, and building workflows that span several services through choreography or orchestration.

## Service Design

The decisions that determine whether a service is scalable and changeable: moving state out of the service, service discovery through dynamic addressing and health information, the selection criteria for remote procedure call, resource-based, and query-based calling, preserving compatibility through schema evolution, and the effect of timeout and retry budget on total latency in a call chain.

### Stateless Services

Source: https://academia.sh/en/courses/application-layer/stateless-services

The effect of moving state out of the service process on horizontal scaling: measuring the same load in two modes with local processes, the session held in a replica's memory splitting across replicas and being lost on restart, reversing the replica count measured in the traffic layer, and computing the cost that moving state outward charges to the state store and the body's byte count.

The Traffic Layer course brought the request to the application's door: its name was resolved, it
passed through the edge cache, it landed on a replica, it passed through the gateway and the gate.
Behind the door stands an area that was never designed. Whether the services are stateless, how
they find each other's addresses, how long they wait and how many times they retry when a call
does not answer, and how their contracts change were all silently assumed in that course. This
course opens those assumptions.

The sequence starts at the bottom: **what the service holds in its own memory.** The previous
course's last measurement already showed why this is the first question — once a session sits in
one replica's memory, the balancer loses the authority to distribute the request, the replica
count needed for the 0.50 utilization target rises from 3 to 4, and added replicas stop lowering
the busiest replica's load. This lesson takes up the source of that bond. A **stateless service**
is a service that keeps no data connecting two requests in its own process memory. The stateless
process and the twelve-factor principle were established in the Server-Side Foundations course;
the definition is not repeated here, it is **measured**.

#### Which State Travels

Not every piece of data sitting in a process has to move. The question that separates them is
this: when this data is lost, or another replica cannot see it, **does the response change**.

Data connecting two requests — a session, a multi-step job's intermediate result, a pagination
cursor, an idempotency key record — must travel; the response changes when it is lost. Data kept
only for speed — a hot record cache, an open connection pool, compiled configuration — can stay;
the response is the same when it is lost, only slower. This distinction also decides what
restarting a replica means: in the first set, a restart is a data loss; in the second, it is a
warm-up period.

The measurement takes up the first set. A seller integration opens a session on the tracking
service and sends tracking queries one after another; the service keeps track of which shipments
are being watched in that session.

#### Setup

The measurement is built with local processes. Two replicas do the same work and run in two modes
with a single parameter: in `memory` mode the session list sits in the replica's own memory, in
`external` mode it sits in a separate state store process. No real cluster, cloud environment, or
session store product is set up.

```js
// state/store.mjs — the state store that keeps the session list outside the process (single process, in-memory)
import { createServer } from "node:http";

const sessions = new Map();
const port = Number(process.argv[2]);

if (Number.isInteger(port) === false) console.log("usage: node state/store.mjs <port>");
else createServer((request, response) => {
  const name = new URL(request.url, "http://local").pathname.split("/").pop();
  if (request.method === "PUT") {
    let m = "";
    request.on("data", (p) => (m += p));
    request.on("end", () => { sessions.set(name, JSON.parse(m)); response.writeHead(204).end(); });
    return;
  }
  response.writeHead(200, { "content-type": "application/json" });
  response.end(JSON.stringify(sessions.get(name) ?? null));
}).listen(port, "127.0.0.1");
```

```js
// state/replica.mjs — tracking service replica; session list lives in memory or in the state store.
// Usage: node state/replica.mjs <port> <name> <memory|external> <store-port>
import { createServer } from "node:http";

const [port, name, mode, store] = process.argv.slice(2);
const local = new Map();

const read = async (s) => (mode === "memory" ? local.get(s) ?? null
  : (await fetch(`http://127.0.0.1:${store}/session/${s}`)).json());
const write = async (s, list) => {
  if (mode === "memory") { local.set(s, list); return; }
  await fetch(`http://127.0.0.1:${store}/session/${s}`, { method: "PUT", body: JSON.stringify(list) });
};

if (Number.isInteger(Number(port)) === false) console.log("usage: node state/replica.mjs <port> <name> <memory|external> <store-port>");
else createServer(async (request, response) => {
  const u = new URL(request.url, "http://local");
  const session = u.searchParams.get("session") ?? "-";
  const json = (code, body) => response.writeHead(code, { "content-type": "application/json" }).end(JSON.stringify({ name, ...body }));
  if (u.pathname === "/session") { await write(session, []); return json(201, { tracked: 0 }); }
  const list = await read(session);
  if (list === null) return json(409, { error: "no_session" });
  if (u.pathname === "/tracking") { list.push(u.searchParams.get("no")); await write(session, list); }
  json(200, { tracked: list.length });
}).listen(Number(port), "127.0.0.1");
```

The client opens the session on the first replica, then sends eight tracking requests to the
replicas in turn. Sequential dispatch makes the distribution deterministic; the dispatch rule was
the Traffic Layer course's subject, and it is taken here in its plainest form.

```js
// state/client.mjs — opens the session on the first replica, sends eight tracking requests to the replicas in turn, then reads the summary
// Usage: node state/client.mjs <label> <port...>
const [label, ...port] = process.argv.slice(2);
const get = async (n, path) => {
  const y = await fetch(`http://127.0.0.1:${n}${path}`);
  return { code: y.status, ...(await y.json()) };
};

if (port.length === 0) console.log("usage: node state/client.mjs <label> <port...>");
else {
  await get(port[0], "/session?session=S1");
  let found = 0;
  for (let i = 0; i < 8; i += 1) {
    const s = await get(port[i % port.length], `/tracking?session=S1&no=TR-${4820 + i}`);
    if (s.code === 200) found += 1;
  }
  const summary = [];
  for (const n of port) {
    const s = await get(n, "/monitor?session=S1");
    summary.push(`${s.name}=${s.code === 200 ? s.tracked : s.error}`);
  }
  console.log(`${label.padEnd(24)} session found ${found}/8   summary ${summary.join("  ")}`);
}
```

The driver script runs the same load in two modes and restarts one replica in each mode. Port
numbers depend on the environment; change them if they are already in use on your machine.

```bash
# measure.sh — same load in two modes: state in the replica's memory and state in the state store
node state/store.mjs 8891 & D=$!
node state/replica.mjs 8881 k1 memory 8891 & A=$!
node state/replica.mjs 8882 k2 memory 8891 & B=$!
sleep 1
node state/client.mjs "in memory" 8881 8882
kill $A; sleep 1
node state/replica.mjs 8881 k1 memory 8891 & A=$!
sleep 1
echo "in memory, k1 restarted -> $(curl -s '127.0.0.1:8881/monitor?session=S1')"
kill $A $B; sleep 1

node state/replica.mjs 8881 k1 external 8891 & A=$!
node state/replica.mjs 8882 k2 external 8891 & B=$!
sleep 1
node state/client.mjs "in state store" 8881 8882
kill $A; sleep 1
node state/replica.mjs 8881 k1 external 8891 & A=$!
sleep 1
echo "in store,   k1 restarted -> $(curl -s '127.0.0.1:8881/monitor?session=S1')"
kill $A $B $D
```

```
in memory                session found 4/8   summary k1=4  k2=no_session
in memory, k1 restarted -> {"name":"k1","error":"no_session"}
in state store           session found 8/8   summary k1=8  k2=8
in store,   k1 restarted -> {"name":"k1","tracked":8}
```

#### Measured Effect

These numbers are in the **measurement** class and come from local processes; they are
deterministic, because requests are sent in sequence.

The first row shows the bond itself: in `memory` mode, only four of the eight requests find the
session. All the ones that find it are requests that land on the replica that opened the session;
the others get `no_session`. The summary row carries the consequence — `k1` tracks four numbers,
`k2` never sees the session. The list was not lost, it was **split**. No replica can return a
split list correctly.

The second row gives the second cost: when `k1` restarts, the four tracked numbers are gone too. A
replica restarting here means a release, a scale change, or recovery after a failure; all three
are routine events, and all three produce data loss.

The third and fourth rows give the same load in `external` mode. All eight requests find the
session, both replicas count the same eight numbers, and the restarted replica reads the list back
whole. The replicas have become each other's equals: which replica answers no longer changes the
answer.

What the measurement shows is not a performance gain — the `external` mode does more work. What it
shows is a **condition**: the load being freely splittable across replicas depends on the data
connecting two requests being outside the process.

#### Back to the Calculation

The cost of meeting the condition was already turned into a number in the Traffic Layer course's
last lesson. That lesson measured the imbalance in the sticky arrangement, found the replica count
needed for the 0.50 utilization target to be 4, and wrote the largest session's share as a serial
fraction. In the stateless arrangement, these numbers are reversed.

```js
// state/capacity.mjs — statelessness's effect on replica count and the cost of moving state outward
const PEAK_EDGE = 513.89;            // K01 Back-of-the-Envelope Estimation: peak edge req/s
const SATURATION = 400, SAFE = 200;  // K02 Role of the Load Balancer: assumption Y1 and Y1 x Y2
const SERIAL_FRACTION = 0.1980;      // K02 Session Stickiness: largest session's share
const STORE_REQUESTS = 138.89;       // K01: requests/s reaching the store
const WRITE_INGRESS = 0.17;          // K01: write ingress Mbit/s
const ARRANGEMENT = [["sticky (K02)", 3, 1.173], ["sticky (K02)", 4, 1.505],
  ["stateless", 3, 1.000], ["stateless", 4, 1.000]];

console.log(`${"arrangement".padEnd(16)}${"replica".padStart(8)}${"imbalance".padStart(12)}` +
  `${"busiest req/s".padStart(18)}${"utilization".padStart(13)}${"replicas needed".padStart(17)}`);
for (const [name, n, d] of ARRANGEMENT) {
  const busiest = (d * PEAK_EDGE) / n;
  console.log(`${name.padEnd(16)}${String(n).padStart(8)}${d.toFixed(3).padStart(12)}` +
    `${busiest.toFixed(2).padStart(18)}${(busiest / SATURATION).toFixed(3).padStart(13)}` +
    `${String(Math.ceil((d * PEAK_EDGE) / SAFE)).padStart(17)}`);
}
console.log(`\nsticky arrangement scaling upper bound = 1/${SERIAL_FRACTION} = ${(1 / SERIAL_FRACTION).toFixed(2)}x;` +
  ` no such upper bound in the stateless arrangement`);

// U1 cost: each tracking request does one read and one write of the session list (behavior in the setup)
const ops = 2 * PEAK_EDGE;
console.log(`\nU1 -> state store ops/s = ${ops.toFixed(2)}, ${(ops / STORE_REQUESTS).toFixed(2)}x ` +
  `K01's ${STORE_REQUESTS} req/s reaching the store`);
console.log(`if U1 were read-only = ${PEAK_EDGE.toFixed(2)} ops/s (${(PEAK_EDGE / STORE_REQUESTS).toFixed(2)}x)`);

// Second path: state travels in the request; the body grows
const list = (k) => JSON.stringify(Array.from({ length: k }, (_, i) => `TR-${4820 + i}`));
console.log(`\n${"numbers in list".padEnd(18)}${"session body bytes".padStart(21)}` +
  `${"added ingress Mbit/s".padStart(22)}${"ratio to K01 write ingress".padStart(27)}`);
for (const k of [8, 40]) {
  const bytes = Buffer.byteLength(list(k));
  const mbit = (PEAK_EDGE * bytes * 8) / 1e6;
  console.log(`${String(k).padEnd(18)}${String(bytes).padStart(21)}${mbit.toFixed(3).padStart(22)}` +
    `${(mbit / WRITE_INGRESS).toFixed(2).padStart(27)}`);
}
```

```
arrangement      replica   imbalance     busiest req/s  utilization  replicas needed
sticky (K02)           3       1.173            200.93        0.502                4
sticky (K02)           4       1.505            193.35        0.483                4
stateless              3       1.000            171.30        0.428                3
stateless              4       1.000            128.47        0.321                3

sticky arrangement scaling upper bound = 1/0.198 = 5.05x; no such upper bound in the stateless arrangement

U1 -> state store ops/s = 1027.78, 7.40x K01's 138.89 req/s reaching the store
if U1 were read-only = 513.89 ops/s (3.70x)

numbers in list      session body bytes  added ingress Mbit/s ratio to K01 write ingress
8                                    81                 0.333                       1.96
40                                  401                 1.649                       9.70
```

The numbers in the top table are in the **computed value** class. Because the imbalance in the
stateless arrangement is 1.000, the 513.89 req/s at the edge splits evenly across three replicas:
171.30 req/s per replica and 0.428 utilization. The required replica count falls from 4 to 3 — the
sticky arrangement's bill was one replica, and it is reversed. The bottom two rows say something
more important: in the stateless arrangement, adding a replica works. The fourth replica lowers
the busiest replica from 171.30 to 128.47 req/s, while in the sticky arrangement the same addition
had only lowered it from 200.93 to 193.35. The scaling upper bound disappears too: with a serial
fraction of 0.1980 the upper bound was 5.05x, and in the stateless arrangement there is no such
bound.

The cost sits in the sections below and is written with **this course's assumption**; it is not
added to K01's table.

**U1 — each tracking request does one read and one write of the session state.** Rationale: this
is the behavior in the setup, the tracking list is read and updated on every query. The
sensitivity is computed below.

With U1, the state store takes 1027.78 operations per second. That is 7.40x K01's 138.89 req/s
reaching the shipment store. The size of the number is a warning: moving state out of the process
does not eliminate it, it **moves it somewhere else**, and that somewhere is now the system's
busiest component. The sensitivity runs one way — if the session were only read, it would be
513.89 ops/s, or 3.70x; since the write is behavior-dependent, this is a number that design can
shrink.

The second path is to never store the state at all: the list travels in the request itself. The
table below gives the bill for that. A list of eight numbers is 81 bytes and means 0.333 Mbit/s of
added ingress at 513.89 req/s — 1.96x K01's write ingress. When the list grows to forty numbers,
401 bytes and 1.649 Mbit/s, or 9.70x. The distinction between the two paths is read from here: if
the state is small and bounded it can travel in the request; if it can grow, it cannot.

#### Summary

- A stateless service is a service that keeps no data connecting two requests in its own process
  memory; the question that separates the two is whether the response changes when the data is
  lost.
- In `memory` mode, four of eight requests found the session and the list split between the two
  replicas (k1=4, k2 sees nothing); when the replica restarted, the list was lost entirely.
- In `external` mode, all eight of eight requests found the session, both replicas counted the
  same eight numbers, and the restarted replica read the list back whole.
- In the stateless arrangement the imbalance is 1.000: the required replicas fall from 4 to 3, the
  load per replica falls from 200.93 to 171.30 req/s, and the fourth replica now works
  (128.47 req/s).
- The scaling upper bound disappears: in the sticky arrangement a serial fraction of 0.1980 held
  the upper bound at 5.05x.
- The cost moves, it does not vanish: with U1 the state store takes 1027.78 ops/s (7.40x K01's
  store load), or if the state travels in the request, 0.333 Mbit/s of added ingress for 81 bytes,
  1.649 Mbit/s for a list of forty.

#### Next Step

The replicas are now each other's equals and the load can be freely split. But one part of the
measurement still stood written by hand: the driver script took the replicas' ports from the
command line, and the client carried the same numbers by hand. The Traffic Layer course's balancer
had taken the replica list the same way. When the replica count changes, when a replica drops, or
when a new replica comes up, who updates that list is not written anywhere. The next lesson takes
up that question: it measures where address information lives, how long it takes for a replica to
enter and leave the list, and how many requests go to the wrong address in that window — and it
shows why this work is a decision separate from the load balancer's health check.

### Service Discovery

Source: https://academia.sh/en/courses/application-layer/service-discovery

Moving replica addresses out of a hand-written list and into a registry: separating service discovery from the load balancer's health check, using a model to measure how much a registration's time to live and the resolution cache extend the window for detecting a failure, graceful shutdown resetting that window, and converting the window into the outage budget measured in the introductory course.

The previous lesson made the replicas each other's equals: state moved out of the process, load
became freely splittable, and the required replica count fell to 3. But one part of the
measurement setup still stood written by hand — the driver script took the replicas' ports from
the command line, and the client carried the same numbers by hand. The Traffic Layer course's load
balancer had taken the replica list the same way. Nowhere is it written who updates that list when
the replica count changes, when a replica drops, or when a new replica comes up.

**Service discovery** is finding, at runtime, the addresses of a service's currently live
replicas. It is easy to confuse with the load balancer, and the two do separate jobs: **the
balancer distributes the request, discovery finds the address.** The balancer's pool is the list
discovery produces; without discovery that list sits in a configuration file. This lesson measures
where the list comes from, how long it takes to update, and how many requests go to a dead address
in that time.

#### Two Placements

**Static list.** Addresses are written into every calling service's configuration. When a replica
drops, no caller notices on its own; when the list changes, every caller's configuration is
updated separately.

**Registry.** Replicas register themselves with a registry when they come up and send a
**heartbeat** at regular intervals. A registration has a **time to live**; when the heartbeat
stops, the registration drops once that period elapses. The caller resolves the address from the
registry at call time.

Both placements carry a delay, and that delay is the number this lesson measures: the
**detection window** between a replica going silent and dropping out of the list.

#### Model

The setup below is a **model**: the registry, heartbeat, and resolution are modules, durations
advance on a logical clock, and a failure and a new replica are each triggered by a parameter. No
real discovery infrastructure, service mesh product, or cloud environment is set up. Two of the
model's numbers are **this course's assumption** and are not added to K01's table.

**U2 — heartbeat interval 1000 ms, registration time to live 3000 ms.** Rationale: the time to
live is chosen as a few multiples of the interval so that a single missed heartbeat does not drop
the registration; three times was chosen here. The sensitivity is computed below.

```js
// discovery/registry.mjs — service REGISTRY model: registration, heartbeat, time to live, and graceful shutdown.
// Time is not real, it is a logical clock (ms); failure and new replica are triggered by parameters.
export const HEARTBEAT = 1000;         // assumption U2: heartbeat interval (ms)
export const TTL = 3000;               // assumption U2: registration's time to live (ms)

export function registry() {
  const records = new Map();
  return {
    heartbeat: (name, address, t) => records.set(address, { name, last: t }),
    deleteRecord: (address) => records.delete(address),
    resolve: (name, t) => [...records.entries()]
      .filter(([, r]) => r.name === name && t - r.last < TTL)
      .map(([address]) => address),
  };
}

// Resolution cache: the caller holds the resolved list for a period of time.
export function resolver(r, name, cacheMs) {
  let list = [], refreshed = -Infinity;
  return (t) => {
    if (t - refreshed >= cacheMs) { list = r.resolve(name, t); refreshed = t; }
    return list;
  };
}
```

The run repeats the same failure across four placements. Replica `k2` goes silent at the fifth
second, replica `k4` comes up at the sixth second; the caller sends a request every ten
milliseconds and picks an address from the list in turn.

```js
// discovery/measure.mjs — same failure in four placements: static list, registry, resolution cache, graceful shutdown
import { registry, resolver, HEARTBEAT, TTL } from "./registry.mjs";

const REPLICAS = ["k1", "k2", "k3"];   // replicas up at the start
const FAILURE = 5000;                  // k2 goes silent at this instant (model parameter)
const NEW = 6000;                      // k4 comes up at this instant (model parameter)
const START = 5000, END = 11000, STEP = 10;   // request window and request interval (ms)
const CONSUMERS = 4;                   // number of calling services: gateway, billing, delivery-ops, batch worker

function run({ cache = 0, gracefulShutdown = false, static: isStatic = false }) {
  const r = registry();
  const resolve = resolver(r, "tracking", cache);
  let seq = 0, dropped = 0, wentToNew = 0, lastDropped = null, stillListed = false;
  for (let t = 0; t <= END; t += STEP) {
    for (const k of REPLICAS) {                     // heartbeats
      if (t % HEARTBEAT === 0 && (k !== "k2" || t <= FAILURE)) r.heartbeat("tracking", k, t);
    }
    if (t % HEARTBEAT === 0 && t >= NEW) r.heartbeat("tracking", "k4", t);
    if (gracefulShutdown && t === FAILURE) r.deleteRecord("k2");
    if (t < START) continue;
    const list = isStatic ? [...REPLICAS] : resolve(t);   // static list never learns about the failure
    stillListed = list.includes("k2");
    const chosen = list[seq++ % list.length];
    if (chosen === "k2") { dropped += 1; lastDropped = t; }
    if (chosen === "k4") wentToNew += 1;
  }
  const window = stillListed ? "manual" : lastDropped === null ? "0" : String(lastDropped - FAILURE + STEP);
  return { dropped, wentToNew, window };
}

const PLACEMENT = [
  ["static list", { static: true }, CONSUMERS],
  ["registry", {}, 1],
  ["registry + 800 cache", { cache: 800 }, 1],
  ["registry + graceful shutdown", { gracefulShutdown: true }, 1],
];

console.log(`heartbeat ${HEARTBEAT} ms, time to live ${TTL} ms; k2 goes silent at ${FAILURE} ms, ` +
  `k4 registers at ${NEW} ms`);
console.log(`request window ${START}-${END} ms, one request every ${STEP} ms = ${(END - START) / STEP + 1} requests\n`);
console.log(`${"placement".padEnd(30)}${"detection window(ms)".padStart(23)}` +
  `${"to dead address".padStart(17)}${"to new replica".padStart(19)}${"configuration points".padStart(22)}`);
for (const [name, option, point] of PLACEMENT) {
  const r = run(option);
  console.log(`${name.padEnd(30)}${r.window.padStart(23)}${String(r.dropped).padStart(17)}` +
    `${String(r.wentToNew).padStart(19)}${String(point).padStart(22)}`);
}
```

```
heartbeat 1000 ms, time to live 3000 ms; k2 goes silent at 5000 ms, k4 registers at 6000 ms
request window 5000-11000 ms, one request every 10 ms = 601 requests

placement                        detection window(ms)  to dead address     to new replica  configuration points
static list                                    manual              200                  0                     4
registry                                         2980               83                150                     1
registry + 800 cache                             3180               93                134                     1
registry + graceful shutdown                        0                0                167                     1
```

#### Measured Effect

These numbers come out of the model and are deterministic: the logical clock and the fixed request
interval give the same result on every run.

**The static list never detects anything.** 200 of the 601 requests go to the dead address and the
window never closes; closing it requires one person to change four separate configurations. The
`to new replica` column gives the second half of the picture: `k4` came up and, in the static
placement, received no requests at all. The benefit of the added capacity waits on the same delay.

**The registry closes the window, but not instantly.** The window is 2980 milliseconds — nearly
the entire time to live. Eighty-three requests going to the dead address is not a defect, it is the
direct consequence of the chosen time to live: a registration sits in the list for three more
seconds after the heartbeat stops. `k4` receives 150 requests in this placement — the registry does
not only remove what dropped, it also introduces what arrived.

**The resolution cache extends the window.** When the caller holds the list in hand for 800
milliseconds, the window grows from 2980 to 3180 and requests to the dead address rise from 83 to
93. The cache lowers the registry's load, and it does this by growing the window; the two numbers
are two faces of the same decision.

**Graceful shutdown resets the window.** The replica deletes its own registration before it shuts
down, and no request ever reaches the dead address. This row matters because most replicas shut
down **planned** — a release, a scale-down, a migration. Planned shutdowns do not have to pay the
window; the time to live is only insurance for unplanned silence.

The last column is a separate measure: the configuration point count is 4 in the static placement,
1 with the registry. The number grows with the number of calling services; repeating the replica
list in every consumer means the work of a replica-count change is multiplied by the number of
consumers.

#### Back to the Calculation

The window is a duration, and K01's and K02's numbers turn it into a request count. The Traffic
Layer course did the same calculation for the health check and found 34.26 requests per failure.

```js
// discovery/budget.mjs — the detection window's counterpart in K01's request rate and outage budget
const PEAK_EDGE = 513.89;                        // K01 Back-of-the-Envelope Estimation: peak edge req/s
const REPLICAS = 3;                              // K02 Role of the Load Balancer: minimum replica count
const FAILURES_MONTH = 2.82;                     // K01 Availability in Numbers: failures per month that fit the budget
const MONTH_REQUESTS = (12_000_000 + 2_800_000) * 30;  // K01: daily tracking query + state event
const HEALTH = 0.2, REGISTRY = 2.98;             // K02 assumption Y3's window and the window the model measured
const WINDOW = [
  ["health check (K02 Y3)", HEALTH], ["registry (U2)", REGISTRY],
  ["registry + 800 cache", 3.18], ["graceful shutdown", 0], ["manual intervention (K01)", 600],
];

const replicaRate = PEAK_EDGE / REPLICAS;
console.log(`rate per replica = ${PEAK_EDGE} / ${REPLICAS} = ${replicaRate.toFixed(2)} req/s\n`);
console.log(`${"window source".padEnd(26)}${"s".padStart(7)}${"per failure".padStart(20)}` +
  `${"monthly".padStart(13)}${"request-based availability".padStart(31)}`);
for (const [name, s] of WINDOW) {
  const one = replicaRate * s, monthly = one * FAILURES_MONTH;
  console.log(`${name.padEnd(26)}${String(s).padStart(7)}${one.toFixed(2).padStart(20)}` +
    `${monthly.toFixed(0).padStart(13)}${`${(100 * (1 - monthly / MONTH_REQUESTS)).toFixed(5)}%`.padStart(31)}`);
}

console.log(`\ncombined: effective window = min(${HEALTH}, ${REGISTRY}) = ${Math.min(HEALTH, REGISTRY)} s ` +
  `-> ${(replicaRate * Math.min(HEALTH, REGISTRY)).toFixed(2)} requests; discovery alone is ${(REGISTRY / HEALTH).toFixed(1)}x worse`);
const heartbeatRate = (ttl) => (REPLICAS * 1000) / (ttl / 3);
console.log(`sensitivity: if U2's time to live went 3000 -> 1000 ms, window would be ${(REGISTRY - 2).toFixed(2)} s, ` +
  `${(replicaRate * (REGISTRY - 2)).toFixed(2)} requests per failure`);
console.log(`  the registry's own load ${heartbeatRate(3000).toFixed(2)} -> ${heartbeatRate(1000).toFixed(2)} beats/s ` +
  `(K02's health check load is 15 req/s)`);
```

```
rate per replica = 513.89 / 3 = 171.30 req/s

window source                   s         per failure      monthly     request-based availability
health check (K02 Y3)         0.2               34.26           97                      99.99998%
registry (U2)                2.98              510.46         1440                      99.99968%
registry + 800 cache         3.18              544.72         1536                      99.99965%
graceful shutdown               0                0.00            0                     100.00000%
manual intervention (K01)     600           102778.00       289834                      99.93472%

combined: effective window = min(0.2, 2.98) = 0.2 s -> 34.26 requests; discovery alone is 14.9x worse
sensitivity: if U2's time to live went 3000 -> 1000 ms, window would be 0.98 s, 167.87 requests per failure
  the registry's own load 3.00 -> 9.00 beats/s (K02's health check load is 15 req/s)
```

These numbers are in the **computed value** class. The table gives a ranking: the health check
34.26, the registry 510.46, manual intervention 102,778 requests. Discovery is two hundred times
better than a hand-held list, and fifteen times worse than the health check. The two comparisons
have to be read together, because this is where the lesson's real argument sits: **service
discovery is not a failure-detection tool.** Its job is making sure the list comes from the
correct source; detecting a failure quickly is still the health check's job.

When both are used together, the effective window is set by the smaller one. The balancer removes
a dropped replica from its pool in 200 milliseconds; the registry's registration sits for three
more seconds, but that registration never reaches the balancer's pool in the first place. One
layer does not stand in for the other; one produces the other's input.

The sensitivity row shows U2's price. If the time to live is cut from 3000 milliseconds to 1000,
the window falls to 0.98 seconds and the requests per failure fall to 167.87, but the heartbeat
interval also has to shrink to a third: the registry's own load rises from 3.00 beats to
9.00 beats/s. The comparison point is K02's health check load — 15 req/s. Discovery's observation
load stays below the health check's load, and shortening the window grows it linearly.

#### Summary

- Service discovery and load balancing are separate jobs: the balancer distributes the request,
  discovery finds the address; the list discovery produces is the balancer's pool.
- In the static list, 200 of 601 requests went to the dead address and the window never closed on
  its own; the newly booted replica received no requests, and the configuration point count was as
  large as the consumer count (4).
- In the registry, the window was 2980 ms with 83 requests going to the dead address; the new
  replica received 150 requests and the configuration point count fell to 1.
- The resolution cache lowers the registry's load while raising the window to 3180 ms and the
  dropped requests to 93; graceful shutdown brings the window to 0, because the replica deletes its
  own registration.
- Back to K01: requests per failure are 34.26 for the health check, 510.46 for the registry, and
  102,778 for manual intervention; discovery is two hundred times better than a hand-held list and
  fifteen times worse than the health check.
- U2 sensitivity: if the time to live falls to 1000 ms, the window falls to 0.98 seconds and the
  registry's own load rises from 3.00 to 9.00 beats/s (K02's health check load is 15 req/s).

#### Next Step

The address is now found at runtime and the list keeps itself current. That was the last piece
needed for two services to reach each other; but what they say once they reach each other was
never chosen. The gateway called the delivery operations service and the billing service in
parallel and merged their responses, and the shape of those calls was never discussed anywhere.
The next lesson takes up that choice: the same internal call is written three separate ways and
compared with three measures — the number of internal calls made for one external request, the
bytes crossing the internal boundary, and the number of names the caller has to know.

### Communication Styles

Source: https://academia.sh/en/courses/application-layer/communication-styles

Writing the same internal call in remote procedure call, resource-based, and query-based styles and comparing them with three measures specific to the service boundary: the number of internal calls made for one external request, the bytes crossing the internal boundary, and the number of names the caller has to know; multiplying the internal byte volume by the introductory course's peak read rate to convert it into internal boundary bandwidth.

The address is now found at runtime and the list keeps itself current. The pieces needed for two
services to reach each other are complete; but what they say once they reach each other was never
chosen. The Traffic Layer course's gateway called the delivery operations service and the billing
service in parallel and merged their responses into a single body, and the shape of those two
calls was never discussed anywhere.

Three styles were established in the Web API Design course: **remote procedure call**,
**resource-based**, and **query-based**. That course compared the three at the **client
boundary**, and its measures belonged there — request count, the over-fetching rate, which side
has to change when a requirement changes. The service boundary is a different place: both ends are
ours, over-fetching is a defect that can be fixed by hand, and there is no client-version-lag
problem. Protocol and schema language are not retold in this lesson; what is measured is the
choice's three consequences on the system — the **internal call count** for one external request,
the **bytes crossing the internal boundary**, and the number of **names** the caller has to know.

#### Setup

Two domain services are separate processes and serve the same data in three styles. The domain
sets carry the two contexts established in M18: delivery operations carries the route and carrier
information, billing carries the tariff and contract information. The fields the gateway reads
from the response come from K01's V5 assumption — state, zone, update time, the last route steps —
plus the fee's net amount added to that.

```js
// communication/service.mjs — two domain services (delivery-ops 8471, billing 8472),
// each serving the same data three ways: resource-based, remote procedure call, and query-based.
import { createServer } from "node:http";

const SHIPMENT = { "TR-4821": { trackingNo: "TR-4821", state: "out-for-delivery", zone: "35",
  updatedAt: "2026-03-11T08:24:00Z", route: ["34", "41", "35"], carrier: "T3",
  weightGrams: 2400, volumeDm3: 12, contractNo: "S7", originBranch: "34-021", destinationBranch: "35-114" } };
const FEE = { "TR-4821": { trackingNo: "TR-4821", tariff: 9600, discount: 1440, net: 8160,
  contractNo: "S7", period: "2026-03", lineItems: 3, taxRate: 0.2 } };

const PROCEDURE = {                                // purpose-built, server-fixed responses
  trackingSummary: (r) => ({ state: r.state, zone: r.zone, updatedAt: r.updatedAt, route: r.route }),
  netFee: (r) => ({ net: r.net }),
};

const readBody = (request) => new Promise((resolve) => {
  let m = ""; request.on("data", (p) => (m += p)); request.on("end", () => resolve(m));
});

const serve = (port, name, table) => createServer(async (request, response) => {
  const path = new URL(request.url, "http://local").pathname;
  let body;
  if (path === "/rpc") {
    const { procedure, no } = JSON.parse(await readBody(request));
    body = PROCEDURE[procedure]?.(table[no]) ?? { error: "unknown_procedure" };
  } else if (path === "/query") {
    const { no, fields } = JSON.parse(await readBody(request));
    body = Object.fromEntries(fields.map((f) => [f, table[no][f]]));
  } else {
    body = table[path.split("/").pop()] ?? { error: "not_found" };
  }
  const text = JSON.stringify(body);
  response.writeHead(200, { "content-type": "application/json" }).end(text);
}).listen(port, "127.0.0.1", () => console.log(`${name} 127.0.0.1:${port}`));

serve(8471, "delivery-ops ", SHIPMENT);
serve(8472, "billing      ", FEE);
```

The measurement stands in for the gateway: it builds the same tracking response in each style and
collects four numbers. The byte counts count only the body; header and protocol overhead were left
out of K01's calculation too. The name count is the counterpart-side names that appear in the
caller's code — path name, procedure name, and read field name. The service names are the same in
all three styles and come from the previous lesson's discovery layer, so they are not counted.

```js
// communication/measure.mjs — measures the same internal call three ways, then applies K01's peak read rate
const READ = ["state", "zone", "updatedAt", "route", "net"];   // fields the gateway reads from the response
let calls = 0, sent = 0, received = 0;

async function call(port, path, body) {
  calls += 1;
  const text = body === undefined ? undefined : JSON.stringify(body);
  if (text) sent += Buffer.byteLength(text);
  const y = await fetch(`http://127.0.0.1:${port}${path}`, text ? { method: "POST", body: text } : {});
  const reply = await y.text();
  received += Buffer.byteLength(reply);
  return JSON.parse(reply);
}
const reset = () => { calls = sent = received = 0; };

const STYLE = {
  "resource-based": { path: ["/shipment", "/fee"], procedure: [],
    run: async (n) => [await call(8471, `/shipment/${n}`), await call(8472, `/fee/${n}`)] },
  "remote procedure call": { path: ["/rpc"], procedure: ["trackingSummary", "netFee"],
    run: async (n) => [await call(8471, "/rpc", { procedure: "trackingSummary", no: n }),
      await call(8472, "/rpc", { procedure: "netFee", no: n })] },
  "query-based": { path: ["/query"], procedure: [],
    run: async (n) => [await call(8471, "/query", { no: n, fields: READ.slice(0, 4) }),
      await call(8472, "/query", { no: n, fields: ["net"] })] },
};

const MEASURE = ["internal call", "sent bytes", "received bytes", "total internal bytes",
  "names caller knows", "  path name", "  procedure name", "  field name"];
const result = [];
for (const [name, style] of Object.entries(STYLE)) {
  reset();
  const [a, c] = await style.run("TR-4821");
  const read = READ.filter((k) => a[k] !== undefined || c[k] !== undefined).length;
  result.push([name, { "internal call": calls, "sent bytes": sent, "received bytes": received,
    "total internal bytes": sent + received, "  path name": style.path.length, "  procedure name": style.procedure.length,
    "  field name": read, "names caller knows": style.path.length + style.procedure.length + read }]);
}
console.log(`${"measure".padEnd(22)}${result.map(([n]) => n.padStart(22)).join("")}`);
for (const m of MEASURE) console.log(`${m.padEnd(22)}${result.map(([, s]) => String(s[m]).padStart(22)).join("")}`);

// K01 Back-of-the-Envelope Estimation: peak read 416.67 req/s; K02 gateway aggregation preserves this rate
const PEAK_READ = 416.67, V5 = 480, K01_EGRESS = 1.6;
console.log(`\n${"style".padEnd(22)}${"internal req/s".padStart(17)}${"internal boundary Mbit/s".padStart(27)}` +
  `${"ratio to K01 read egress".padStart(27)}`);
for (const [name, s] of result) {
  const mbit = (PEAK_READ * s["total internal bytes"] * 8) / 1e6;
  console.log(`${name.padEnd(22)}${(PEAK_READ * s["internal call"]).toFixed(2).padStart(17)}` +
    `${mbit.toFixed(3).padStart(27)}${(mbit / K01_EGRESS).toFixed(2).padStart(27)}`);
}
const resource = result[0][1]["total internal bytes"], rpc = result[1][1]["total internal bytes"];
console.log(`resource-based / remote procedure call byte ratio = ${(resource / rpc).toFixed(2)};` +
  ` K01's outbound tracking response is ${V5} bytes (V5)`);
```

```bash
node communication/service.mjs & p=$!
curl -s --retry 20 --retry-connrefused --retry-delay 0 -o /dev/null http://127.0.0.1:8472/fee/TR-4821
node communication/measure.mjs
kill $p
```

```
delivery-ops  127.0.0.1:8471
billing       127.0.0.1:8472
measure                       resource-based remote procedure call           query-based
internal call                              2                     2                     2
sent bytes                                 0                    83                    95
received bytes                           373                   112                   112
total internal bytes                     373                   195                   207
names caller knows                         7                     8                     6
  path name                                2                     1                     1
  procedure name                           0                     2                     0
  field name                               5                     5                     5

style                    internal req/s   internal boundary Mbit/s   ratio to K01 read egress
resource-based                   833.34                      1.243                       0.78
remote procedure call            833.34                      0.650                       0.41
query-based                      833.34                      0.690                       0.43
resource-based / remote procedure call byte ratio = 1.91; K01's outbound tracking response is 480 bytes (V5)
```

#### The Number That Does Not Change

The table's first row is 2 in all three columns. This is the lesson's most important result: **the
call style does not change the internal call count.** Two calls are made not because of the
protocol, but because the data sits in two separate contexts — the state in delivery operations,
the amount in billing. Choosing a style neither builds this split nor removes it.

The number's importance surfaces in the next lesson: the number of steps in the chain is the
number of pieces the latency budget divides into, and that number is set here, by the domain
split. It cannot be shrunk by choosing a style.

#### Bytes and Names

The second group of rows sets two measures against each other, and they move in opposite
directions.

**Bytes.** The resource-based style carries 373 bytes, remote procedure call 195, query-based 207.
The resource-based style brings back two complete records: the carrier, weight, volume, origin
branch, and destination branch also cross the wire, because the address corresponds to a
**record**. In the remote procedure call style, the server has fixed the response to the call's
purpose; the query-based style brings back the same fields, but because the requested field names
are also written into the request, the sent bytes climb to 95. Ratio 1.91: nearly twice the bytes
for the same job.

**Names.** The number of names the caller has to know is 6 in the query-based style, 7 in
resource-based, 8 in remote procedure call. The ranking does not match the byte ranking. The remote
procedure call style carries the fewest bytes but requires knowing the most names, because it adds
a layer of procedure names on top of the field names: `trackingSummary` and `netFee` are two names
written into the caller's code, and both of them can change.

The name count is a coupling measure and is the direct input to the next lesson: a name changing
means every consumer that reads that name changes. But the names' **owner** also has to be
counted. The resource-based style's five field names belong to the record itself and are shared by
every consumer; the `trackingSummary` procedure is written for a single caller. The same count of
names does not mean the same difficulty of change.

#### Back to the Calculation

The table below ties the measured bytes to K01's rate. The inputs: peak read 416.67 req/s (K01)
and the gateway's aggregation decision, which did not change the external request rate (K02).

The internal request rate is 833.34 req/s in all three styles — twice the external rate, because
every external request spawns two internal calls. This number was not in K01's table; K01 counted
the rate at the edge, not the application's own internal calls. This is the first consequence of
splitting the service on the system: **the internal request rate is a number independent of the
external request rate, and it is multiplied by the split.**

The internal boundary bandwidth is 1.243 Mbit/s in the resource-based style, 0.650 in remote
procedure call, 0.690 in query-based. The comparison point is K01's read egress: 1.60 Mbit/s. So
the traffic crossing the internal boundary, even in the most expensive style, stays at 0.78x the
outbound traffic. The number says two things. First, the bandwidth between the two services is not
a constraint at this scale; the style choice cannot be made by looking at bytes alone. Second, the
ratio grows with scale — once the split deepens and every external request spawns four or six
internal calls, internal traffic overtakes external traffic, and at that point the same table
dictates a different decision.

#### Summary

- The three styles were compared at the client boundary in the Web API Design course; at the
  service boundary both ends are ours, so the measures change: internal call count, internal
  bytes, and the number of names the caller knows.
- The internal call count came out to 2 in all three styles: what sets the call count is not the
  protocol, it is the data sitting in two separate contexts.
- Internal bytes are 373 in the resource-based style, 195 in remote procedure call, 207 in
  query-based; the ratio is 1.91, and the source of the difference is that the address corresponds
  to a record.
- The number of names the caller knows is 6 in query-based, 7 in resource-based, 8 in remote
  procedure call; remote procedure call carries the fewest bytes while requiring the most names,
  because it adds a layer of procedure names on top of the field names.
- The owner of the names matters as much as their count: record fields are shared with every
  consumer, a purpose-built procedure name belongs to a single caller.
- Back to K01: the internal request rate is 833.34 req/s (twice the external rate) and the internal
  boundary is 0.650–1.243 Mbit/s, that is, 0.41–0.78x K01's 1.60 Mbit/s read egress; the ratio
  grows as the split deepens.

#### Next Step

This lesson counted the names the caller has to know and showed the count moves between six and
eight. The number alone says nothing; its meaning surfaces once one of those names changes. The
gateway is not the only caller: the same two services are also called by the end-of-day billing
job, the interface built for the front end, and the reporting stream, and each of them reads a
different subset of the domain. The next lesson counts how many consumers break when one domain
name changes, shows that the same change can be split into a sequence to bring the number of broken
consumers to zero, and computes what that costs the internal boundary during the transition window.

### Inter-Service Contracts

Source: https://academia.sh/en/courses/application-layer/inter-service-contracts

Measuring the cost of a schema change by the number of broken consumers: counting how many services a single-step release forces to publish together across five internal consumers' read sets, splitting the same change into an expand–contract sequence to bring broken consumers to zero, and computing what carrying two names at once during the transition window costs in record bytes and internal boundary bandwidth.

The previous lesson counted the names the caller has to know and showed the count moves between
six and eight. The number alone says nothing; its meaning surfaces once one of those names
changes. The gateway is not the only caller either — the same shipment record is also read by the
end-of-day billing job, the interface built for the front end, the reporting stream, and the
delivery planner, and each of them looks at a different subset of the domain.

The definition of a breaking change, the schema's two directions, and deriving the version number
from the difference were established in the Web API Design course; they are not retold here. That
course measured the cost on **a single old client**. The measure changes at the service boundary:
the consumers are our own services, their count is known, and their read sets can be read. What
this lesson measures is **how many consumers break**; what it measures in exchange is **how many
units are forced to publish together**.

#### Schema, Consumers, and the Change

The schema under discussion is the previous lesson's shipment record. The proposed change corrects
four names and adds a field: the zone field did not say which zone it was, the weight was in
grams, the route carried only zone codes, and the carrier field's being a code was not clear from
its name.

```js
// contract/schema.mjs — two schemas for the shipment record, five internal consumers, and the breaking rule
export const V1 = { trackingNo: "string", state: "string", zone: "string", updatedAt: "string",
  route: "array-of-string", carrier: "string", weightGrams: "integer", volumeDm3: "integer",
  contractNo: "string", originBranch: "string", destinationBranch: "string" };

// V2 renames four names and adds a field; zone->destinationZone, weightGrams->weightKg,
// route->routeSteps (the type also changes), carrier->carrierCode.
export const V2 = { trackingNo: "string", state: "string", destinationZone: "string", updatedAt: "string",
  routeSteps: "array-of-object", carrierCode: "string", weightKg: "number", volumeDm3: "integer",
  contractNo: "string", originBranch: "string", destinationBranch: "string", estimatedDelivery: "string" };

export const MAPPING = { zone: "destinationZone", weightGrams: "weightKg",
  route: "routeSteps", carrier: "carrierCode" };

export const CONSUMER = {
  gateway: ["state", "zone", "updatedAt", "route"],
  "front-end-ui": ["state", "updatedAt", "destinationBranch"],
  "report-stream": ["trackingNo", "state", "zone", "carrier"],
  "batch-billing": ["trackingNo", "contractNo", "weightGrams", "volumeDm3"],
  "delivery-planner": ["route", "carrier", "originBranch", "destinationBranch"],
};

// A consumer breaks if a name it reads is missing from the schema, or its type changed.
export const broken = (schema, reads) => Object.entries(reads)
  .filter(([, fields]) => fields.some((f) => schema[f] === undefined || schema[f] !== V1[f]))
  .map(([name]) => name);

export const EXAMPLE = {
  trackingNo: "TR-4821", state: "out-for-delivery", zone: "35", updatedAt: "2026-03-11T08:24:00Z",
  route: ["34", "41", "35"], carrier: "T3", weightGrams: 2400, volumeDm3: 12,
  contractNo: "S7", originBranch: "34-021", destinationBranch: "35-114",
  destinationZone: "35", routeSteps: [{ zone: "34", at: "T08" }, { zone: "41", at: "T14" },
    { zone: "35", at: "T20" }], carrierCode: "T3", weightKg: 2.4, estimatedDelivery: "2026-03-12",
};
export const record = (schema) => Object.fromEntries(Object.keys(schema).map((f) => [f, EXAMPLE[f]]));
```

**Expand–contract** is a release sequence that splits the same change into three steps. In the
first step, the provider adds the new names **next to** the old ones and fills both. In the second
step, consumers move to the new names one by one. In the third step, the provider removes the old
names. The run below compares the two release arrangements on the same change.

```js
// contract/measure.mjs — the same change under two release arrangements: in one step, and in expand-contract sequence
import { V1, V2, MAPPING, CONSUMER, broken, record } from "./schema.mjs";

const TRANSITION = { ...V1, ...V2 };                  // expand step: old and new names together
const migrated = Object.fromEntries(Object.entries(CONSUMER)
  .map(([name, f]) => [name, f.map((x) => MAPPING[x] ?? x)]));
const consumers = Object.keys(CONSUMER).length;

console.log(`schema fields V1 ${Object.keys(V1).length}, transition ${Object.keys(TRANSITION).length}, ` +
  `V2 ${Object.keys(V2).length}; internal consumers ${consumers}`);
const count = {};
for (const fields of Object.values(CONSUMER)) for (const f of fields) count[f] = (count[f] ?? 0) + 1;
const shared = Object.entries(count).sort((x, y) => y[1] - x[1]).slice(0, 4);
console.log(`most-read fields: ${shared.map(([f, n]) => `${f}=${n}`).join(", ")}\n`);

const broken1 = broken(V2, CONSUMER);
console.log(`single-step release -> broken consumers ${broken1.length}/${consumers}: ${broken1.join(", ")}`);
console.log(`  units that must be published together = ${broken1.length + 1} (provider + broken)\n`);

const STEP = [
  ["1 expand (both names at once)", TRANSITION, CONSUMER, 1],
  ["2 consumers move to new name", TRANSITION, migrated, Object.entries(CONSUMER)
    .filter(([name, f]) => f.some((x) => MAPPING[x])).length],
  ["3 contract (old names drop)", V2, migrated, 1],
];
console.log(`${"step".padEnd(30)}${"fields in schema".padStart(18)}${"broken consumers".padStart(18)}` +
  `${"units published".padStart(18)}${"record bytes".padStart(14)}`);
let units = 0;
for (const [name, schema, reads, n] of STEP) {
  units += n;
  const brokenCount = Object.entries(reads).filter(([, fields]) =>
    fields.some((f) => schema[f] === undefined)).length;
  console.log(`${name.padEnd(30)}${String(Object.keys(schema).length).padStart(18)}` +
    `${`${brokenCount}/${consumers}`.padStart(18)}${String(n).padStart(18)}` +
    `${String(Buffer.byteLength(JSON.stringify(record(schema)))).padStart(14)}`);
}
console.log(`total: 3 release steps, ${units} unit releases, 0 broken consumers\n`);

// K01 Back-of-the-Envelope Estimation: peak read 416.67 req/s and read egress 1.60 Mbit/s
const PEAK_READ = 416.67, K01_EGRESS = 1.6;
const mbit = (bytes) => (PEAK_READ * bytes * 8) / 1e6;
console.log(`${"schema".padEnd(10)}${"record bytes".padStart(14)}${"internal boundary Mbit/s".padStart(27)}` +
  `${"ratio to K01 read egress".padStart(27)}`);
for (const [name, s] of [["V1", V1], ["transition", TRANSITION], ["V2", V2]]) {
  const b = Buffer.byteLength(JSON.stringify(record(s)));
  console.log(`${name.padEnd(10)}${String(b).padStart(14)}${mbit(b).toFixed(3).padStart(27)}` +
    `${(mbit(b) / K01_EGRESS).toFixed(2).padStart(27)}`);
}
const transitionBytes = Buffer.byteLength(JSON.stringify(record(TRANSITION)));
const v1Bytes = Buffer.byteLength(JSON.stringify(record(V1)));
console.log(`transition window cost = ${transitionBytes - v1Bytes} bytes/record, ` +
  `${(mbit(transitionBytes) - mbit(v1Bytes)).toFixed(3)} Mbit/s (record grows ${(transitionBytes / v1Bytes).toFixed(2)}x)`);
```

```
schema fields V1 11, transition 16, V2 12; internal consumers 5
most-read fields: state=3, zone=2, updatedAt=2, route=2

single-step release -> broken consumers 4/5: gateway, report-stream, batch-billing, delivery-planner
  units that must be published together = 5 (provider + broken)

step                            fields in schema  broken consumers   units published  record bytes
1 expand (both names at once)                 16               0/5                 1           423
2 consumers move to new name                  16               0/5                 4           423
3 contract (old names drop)                   12               0/5                 1           352
total: 3 release steps, 6 unit releases, 0 broken consumers

schema      record bytes   internal boundary Mbit/s   ratio to K01 read egress
V1                   243                      0.810                       0.51
transition           423                      1.410                       0.88
V2                   352                      1.173                       0.73
transition window cost = 180 bytes/record, 0.600 Mbit/s (record grows 1.74x)
```

#### Number of Broken Consumers

These numbers are in the **computed value** class: they come out of the read sets and schema
definitions by arithmetic.

In a single-step release, four of five consumers break. The one consumer that does not break is
the front-end interface, which reads none of the four changed fields. This row's result is not a
version number, it is a **release constraint**: the provider and four consumers have to be
published **at the same time**. Five units moving together in a single release takes back the
reason for splitting into services — the point of the split was that units could be published
separately.

The second row shows where the cost comes from: `state` is read by three consumers, `zone`,
`updatedAt`, and `route` are each read by two. The cost of changing a field is the number of
consumers that read it. The previous lesson's distinction turns into a number here — a shared
record field is more expensive than a procedure name written for a single caller.

#### The Result of Splitting Into a Sequence

The table below shows that splitting the same change into three steps keeps broken consumers at 0
in every step. In the first step, the schema grows from eleven fields to sixteen, and because the
old names stay in place, no consumer changes. In the second step, four consumers move to the new
names and can do it **separately**; their order does not matter, because both names are populated.
In the third step, the old names drop and the schema falls to twelve fields.

The two arrangements are compared with two numbers. Single-step release: 1 release step, 5 units
at once, 4 broken consumers. Expand–contract: 3 release steps, 6 unit releases, 0 broken consumers.
One extra unit gets published, and in exchange, no unit waits on another.

The sequence carries a rule that does not show up in the table: **once the first step is released,
someone has to know that consumers have actually migrated.** Otherwise the third step turns into
the single-step release on its own. Knowing this is the contract tests established in the Web API
Design course; if consumer expectations are written down, whether the third step is safe can be
tested before release.

#### Back to the Calculation

The table below gives the bill for the transition. The shipment record is 243 bytes in V1; because
the transition window carries both names at once, it climbs to 423 bytes, a 1.74x increase.
Settling at V2 leaves it at 352 bytes — above V1, because the route steps now carry objects and a
field was added.

Multiplied by K01's peak read rate, the bandwidth crossing the internal boundary climbs from 0.810
Mbit/s to 1.410 Mbit/s: the transition window's cost is 0.600 Mbit/s. The comparison point is K01's
read egress (1.60 Mbit/s); during the transition window the internal boundary approaches 0.88x that
number, while in V1 it was at 0.51x.

The result that follows is that the transition window is a **duration decision**. As long as the
window stays open, every record carries seventy-four percent extra bytes; closing it requires
waiting for every consumer to migrate. A long window keeps the risk of breakage at zero and pays a
continuous bandwidth tax; a short window cuts the tax and, if a consumer has not finished migrating,
brings the breakage back. Which of the two extremes is chosen depends on the test that shows the
third step is safe.

#### Summary

- At the service boundary, the measure of a schema change is the number of broken consumers and
  the number of units forced to publish together; the definition of breaking and the version
  derivation were established in the Web API Design course and are not repeated here.
- A proposal that changes four names breaks four of five internal consumers and requires
  publishing five units at the same time as the provider.
- The cost of changing a field is the number of consumers that read it: `state` is read by three,
  `zone`, `updatedAt`, and `route` by two each.
- The expand–contract sequence carries out the same change with 3 release steps and 6 unit
  releases, with 0 broken consumers; in the second step consumers migrate without waiting on each
  other.
- The safety of the sequence depends on the third step: if the old names are removed before the
  migration is verified as complete, the arrangement reverts to a single-step release.
- Back to K01: the record climbs from 243 bytes to 423 bytes in the transition window (a 1.74x
  increase), and the internal boundary rises from 0.810 to 1.410 Mbit/s — a 0.600 Mbit/s window
  tax, 0.88x K01's 1.60 Mbit/s read egress.

#### Next Step

After four lessons, a call can be made: the services are stateless, their addresses are found, the
conversation style is chosen, and their contracts can evolve without breaking. One question is
still open, and it returns to this topic's very first measure. When the gateway calls two services
and one of them does not answer, how long does it wait, how many times does it retry, and what does
that waiting add to the end-to-end response. K01 set a 200-millisecond threshold for the tracking
response, and the third lesson measured the number of steps in the chain at 2. The next lesson
takes up that threshold as a budget: it measures what the product of timeout and retry count gives
in the worst case, how the budget is split across the steps, and how much the internal request rate
multiplies by when no retry budget is set.

### Timeout and Retry Budget

Source: https://academia.sh/en/courses/application-layer/timeout-and-retry-budget

Splitting the end-to-end latency threshold across the steps in the chain: computing how far the product of timeout and retry count inflates the budget in the worst case, deadline propagation turning a threshold overrun into an early failure, a short timeout absorbing transient slowness, and using a model to measure how far the internal request rate multiplies under widespread slowdown when no retry budget is set.

After four lessons, a call can be made: the services are stateless, their addresses are found at
runtime, the conversation style is chosen, and their contracts can evolve without breaking. One
question is still open, and it returns to this topic's first measure. When the gateway calls two
services and one of them does not answer, how long does it wait, how many times does it retry, and
what does that waiting add to the end-to-end response.

The retry decision, exponential backoff, jitter, and the circuit breaker were established in the
Application Architecture course; the algorithms are not retold here. What this lesson takes up is
a **budget**. The budget's source is K01: the tracking read's hard threshold is **200
milliseconds**, and the source of that threshold is the response time the recipient expects from
the tracking page. The number of steps in the chain was also already measured — the third lesson
counted two internal calls for one external request and showed this comes from the domain split,
not the protocol.

#### Splitting the Budget

The chain is this: the gateway calls two services in parallel, and each service calls its own
store. Depth 2, arms 2, four step calls in total. The gateway's own work also asks for a share; this
is **this course's assumption**.

**U3 — the gateway's own share is 20 ms.** Rationale: validation, gate checks, and merging the two
responses were moved to the edge in K02, and that work takes up space in the budget. The
sensitivity is computed below; it is not added to K01's table.

```js
// chain/budget.mjs — splitting the end-to-end budget across the chain; deterministic arithmetic
const END_TO_END = 200;     // K01 Non-Functional Requirements: read threshold (hard) 200 ms
const GATEWAY_SHARE = 20;   // assumption U3: the gateway's own share of the budget (ms)
const STEP_MEDIAN = 8;      // model parameter: a step call's usual duration (ms)
const remaining = END_TO_END - GATEWAY_SHARE;

console.log(`end-to-end ${END_TO_END} ms, gateway's share ${GATEWAY_SHARE} ms, remaining for steps ${remaining} ms\n`);
console.log(`${"depth".padStart(9)}${"retries".padStart(8)}${"no-budget worst(ms)".padStart(22)}` +
  `${"budget ratio".padStart(14)}${"allocated timeout(ms)".padStart(31)}${"ratio to median".padStart(16)}`);
for (const depth of [1, 2, 3, 4]) {
  for (const tries of [1, 3]) {
    const noBudget = depth * tries * END_TO_END + GATEWAY_SHARE;
    const allocated = remaining / (depth * tries);
    console.log(`${String(depth).padStart(9)}${String(tries).padStart(8)}${String(noBudget).padStart(22)}` +
      `${`x${(noBudget / END_TO_END).toFixed(1)}`.padStart(14)}${allocated.toFixed(1).padStart(31)}` +
      `${`x${(allocated / STEP_MEDIAN).toFixed(2)}`.padStart(16)}`);
  }
}
const maxDepth = Math.floor(remaining / (3 * STEP_MEDIAN));
console.log(`\nat 3 retries and a step median of ${STEP_MEDIAN} ms, the timeout can go no more than ` +
  `${maxDepth} steps deep before falling below the median`);
console.log(`sensitivity: if the gateway's share went ${GATEWAY_SHARE} -> ${GATEWAY_SHARE * 2} ms, at depth 2, retries 3, ` +
  `the timeout would go ${(remaining / 6).toFixed(1)} -> ${((END_TO_END - 2 * GATEWAY_SHARE) / 6).toFixed(1)} ms`);
```

```
end-to-end 200 ms, gateway's share 20 ms, remaining for steps 180 ms

    depth retries   no-budget worst(ms)  budget ratio          allocated timeout(ms) ratio to median
        1       1                   220          x1.1                          180.0          x22.50
        1       3                   620          x3.1                           60.0           x7.50
        2       1                   420          x2.1                           90.0          x11.25
        2       3                  1220          x6.1                           30.0           x3.75
        3       1                   620          x3.1                           60.0           x7.50
        3       3                  1820          x9.1                           20.0           x2.50
        4       1                   820          x4.1                           45.0           x5.63
        4       3                  2420         x12.1                           15.0           x1.88

at 3 retries and a step median of 8 ms, the timeout can go no more than 7 steps deep before falling below the median
sensitivity: if the gateway's share went 20 -> 40 ms, at depth 2, retries 3, the timeout would go 30.0 -> 26.7 ms
```

These numbers are in the **computed value** class. The third column shows the most common mistake:
if every step is given the end-to-end threshold itself as its timeout, then at depth 2 and 3
retries, the worst case is 1220 milliseconds — 6.1x the budget. Setting the timeout equal to the
threshold does not protect the budget; **every step becomes able to spend the entire budget on its
own.**

The fifth column gives the correct split, and it comes with a limit. The timeout allotted to a
step is divided by the product of depth and retry count: 30 milliseconds at depth 2 and 3 retries,
15 milliseconds at depth 4. The last column says why this is a limit — as the timeout approaches
the step's usual duration, ordinary requests start timing out too. At a step median of eight
milliseconds and three retries, the budget can reach a depth of at most seven steps.

This is the bond between service splitting and the latency threshold: **every step added to the
chain divides the budget, and it cannot be divided forever.**

#### Model

The run below is a **model**: it uses a logical clock, step durations are read from a cycle, and
the slowdown is triggered by a parameter. It runs in two regimes — in the normal regime most steps
are fast and a step fails to answer once in a while; in the widespread slowdown regime most steps
fail to answer. No real service fleet or network is set up.

```js
// chain/run.mjs — the chain's MODEL: logical clock, step durations as a parameter, two regimes.
// Chain: gateway -> (delivery-ops, billing) parallel -> each calls its own store. Depth 2, arms 2.
const N = 40, THRESHOLD = 200, GATEWAY_SHARE = 20, DEPTH = 2, ARM = 2, STEP = DEPTH * ARM;
const REGIME = {                                  // model parameter: duration of a step call (ms)
  normal: [6, 7, 6, 8, 6, 7, 6, 45, 6, 7, 6, 8, 6, 7, 6, 1000],
  "widespread slowdown": [6, 1000, 1000, 1000, 6, 1000, 1000, 1000],
};

function run({ timeout, tries, deadline, retryBudget }, regime) {
  let s = 0, primary = 0, extra = 0;
  const count = { withinThreshold: 0, outsideThreshold: 0, dropped: 0 }, endToEnd = [];
  const duration = () => REGIME[regime][s++ % REGIME[regime].length];
  const allowed = () => retryBudget === null || extra < retryBudget * primary;

  const step = (remaining) => {                    // one step: timeout and retry
    let elapsed = 0;
    for (let i = 0; i < tries; i += 1) {
      const stepTimeout = deadline ? Math.min(timeout, remaining - elapsed) : timeout;
      if (stepTimeout <= 0) return { ok: false, elapsed };
      if (i === 0) primary += 1; else extra += 1;
      const d = duration();
      if (d <= stepTimeout) return { ok: true, elapsed: elapsed + d };
      elapsed += stepTimeout;
      if (i + 1 < tries && allowed() === false) return { ok: false, elapsed };
    }
    return { ok: false, elapsed };
  };
  const arm = () => {                               // one arm: DEPTH consecutive steps
    let elapsed = 0, ok = true;
    for (let i = 0; i < DEPTH && ok; i += 1) {
      const r = step(THRESHOLD - GATEWAY_SHARE - elapsed);
      elapsed += r.elapsed; ok = r.ok;
    }
    return { ok, elapsed };
  };

  for (let i = 0; i < N; i += 1) {
    const arms = Array.from({ length: ARM }, arm);   // parallel: duration is the arms' maximum
    const elapsed = GATEWAY_SHARE + Math.max(...arms.map((a) => a.elapsed));
    endToEnd.push(elapsed);
    if (arms.every((a) => a.ok) === false) count.dropped += 1;
    else if (elapsed > THRESHOLD) count.outsideThreshold += 1;
    else count.withinThreshold += 1;
  }
  const sorted = [...endToEnd].sort((a, b) => a - b);
  return { ...count, median: sorted[N >> 1], worst: sorted[N - 1],
    multiplier: (primary + extra) / (N * STEP) };
}

const CONFIGURATION = [
  ["no budget (200/3)", { timeout: 200, tries: 3, deadline: false, retryBudget: null }],
  ["+ deadline propagation", { timeout: 200, tries: 3, deadline: true, retryBudget: null }],
  ["+ allocated (30/3)", { timeout: 30, tries: 3, deadline: true, retryBudget: null }],
  ["+ retry budget 0.30", { timeout: 30, tries: 3, deadline: true, retryBudget: 0.3 }],
];
const PEAK_READ = 416.67, INTERNAL_BASE = PEAK_READ * STEP;   // K01: peak read; lesson 03: step count

console.log(`${N} external requests, depth ${DEPTH}, arms ${ARM}, steps ${STEP}; ` +
  `end-to-end threshold ${THRESHOLD} ms (K01), gateway share ${GATEWAY_SHARE} ms (U3)`);
console.log(`internal req/s base = ${PEAK_READ} x ${STEP} = ${INTERNAL_BASE.toFixed(2)}`);
for (const regime of Object.keys(REGIME)) {
  console.log(`\n-- ${regime}: ${REGIME[regime].filter((v, i, a) => a.indexOf(v) === i).join("/")} ms cycle --`);
  console.log(`${"configuration".padEnd(24)}${"within threshold".padStart(18)}${"outside threshold".padStart(19)}` +
    `${"dropped".padStart(9)}${"median".padStart(9)}${"worst".padStart(9)}${"multiplier".padStart(12)}` +
    `${"internal req/s".padStart(16)}`);
  for (const [name, y] of CONFIGURATION) {
    const r = run(y, regime);
    console.log(`${name.padEnd(24)}${String(r.withinThreshold).padStart(18)}${String(r.outsideThreshold).padStart(19)}` +
      `${String(r.dropped).padStart(9)}${String(r.median).padStart(9)}${String(r.worst).padStart(9)}` +
      `${`x${r.multiplier.toFixed(2)}`.padStart(12)}${(INTERNAL_BASE * r.multiplier).toFixed(2).padStart(16)}`);
  }
}
console.log(`\nunlimited-retry upper bound: 3 tries per step -> ${(INTERNAL_BASE * 3).toFixed(2)} req/s`);
```

```
40 external requests, depth 2, arms 2, steps 4; end-to-end threshold 200 ms (K01), gateway share 20 ms (U3)
internal req/s base = 416.67 x 4 = 1666.68

-- normal: 6/7/8/45/1000 ms cycle --
configuration             within threshold  outside threshold  dropped   median    worst  multiplier  internal req/s
no budget (200/3)                       30                 10        0       71      233       x1.06         1770.85
+ deadline propagation                  30                  0       10       71      200       x1.00         1666.68
+ allocated (30/3)                      40                  0        0       62       63       x1.14         1895.85
+ retry budget 0.30                     40                  0        0       62       63       x1.14         1895.85

-- widespread slowdown: 6/1000 ms cycle --
configuration             within threshold  outside threshold  dropped   median    worst  multiplier  internal req/s
no budget (200/3)                        0                  0       40      626      626       x2.00         3333.36
+ deadline propagation                   0                  0       40      200      200       x0.67         1114.59
+ allocated (30/3)                       0                  0       40      116      116       x2.00         3333.36
+ retry budget 0.30                      0                  0       40       86       86       x0.97         1614.60

unlimited-retry upper bound: 3 tries per step -> 5000.04 req/s
```

#### Reading the Four Rows

**Deadline propagation turns a threshold overrun into an early failure.** In the normal regime,
the no-budget arrangement answers thirty of forty requests within the threshold, ten outside it —
worst case 233 milliseconds. Once the caller passes the remaining budget along with the call, the
worst case is cut off at exactly 200 milliseconds and those ten requests stop being outside the
threshold and become dropped requests. The trade-off is explicit: a choice is made between a late
answer and an on-time failure. The user who received an answer at 233 milliseconds was already not
getting a threshold-meeting response; the work spent producing it was also unrewarded.

**A short timeout absorbs transient slowness.** In the allocated arrangement, when the timeout
falls to 30 milliseconds, all forty of forty requests answer within the threshold and the worst
case is 63 milliseconds. The reason is the model's assumption: the slowness is transient, and the
same step's second call returns fast. A long timeout carries this transient slowness through to
the end-to-end response; a short timeout cuts it off and retries. The cost shows up as the call
multiplier rising from 1.00 to 1.14: the internal request rate climbs from 1666.68 to
1895.85 req/s.

**The retry budget does not engage in the normal regime.** The fourth row is identical to the
third. The observed multiplier is 1.14 and the budget is 1.30, so the limit never comes into play.
A budget being invisible under normal conditions is the sign that it is sized correctly.

#### What Changes Under Widespread Slowdown

The second table runs the same configurations in a regime where most steps do not answer, and the
ranking reverses.

In the no-budget arrangement, all forty of forty requests drop, and the caller waits 626
milliseconds to learn this. Deadline propagation cuts this off at 200 milliseconds; the multiplier
falls to 0.67, because there are also primary calls that never get made once the budget runs out.
The internal request rate falls from 3333.36 to 1114.59 req/s.

The third row is the lesson's most important number. When the timeout is cut to 30 milliseconds,
the end-to-end duration falls to 116 milliseconds — but the multiplier climbs back to 2.00 and the
internal request rate becomes 3333.36 req/s. A short timeout fits more retries into the same
budget; **a service that is already slow receives twice the requests from the system that is
slowing it down.** A decision that was good in the normal regime is harmful in the failure regime.

The fourth row cuts this off. Once the retry budget limits extra retries to thirty percent of
primary calls, the multiplier falls to 0.97 and the internal request rate to 1614.60 req/s: a
below-baseline load instead of a doubling. The end-to-end duration also falls, from 116 to 86
milliseconds, because wasted retries are not made. The unlimited-retry upper bound is the
comparison point — if every step is tried three times, 5000.04 req/s, twelve times K01's peak rate
at the edge as internal load.

All of this holds only for calls that can be retried. A state-changing call being retryable
depends on processing the same call twice giving the same result as processing it once; this
property is called **idempotent**, the same concept as the idempotency property of HTTP methods. A
call that is not idempotent can only be retried if the client generates an **idempotency key** that
is written into the contract; both were established in the Web API Design course.

#### Summary

- The end-to-end threshold is a budget and it is split across the chain; if every step is given the
  full threshold as its timeout, the worst case at depth 2 and 3 retries climbs to 6.1x the budget.
- The allocated timeout is divided by the product of depth and retry count (30 ms at depth 2,
  retries 3) and hits a limit as it approaches the step's usual duration: at an 8 ms median and 3
  retries, the budget can reach a depth of at most 7 steps.
- Deadline propagation cuts the worst case off at the threshold: in the normal regime, 233 ms and
  10 outside-threshold responses turn into 200 ms and 10 dropped requests — an on-time failure
  instead of a late answer.
- A short timeout absorbs transient slowness (40/40 within threshold, worst case 63 ms) at the cost
  of the call amplification rising from 1.00 to 1.14 and the internal request rate climbing from
  1666.68 to 1895.85 req/s.
- Under widespread slowdown the same decision reverses: a short timeout drives the multiplier to
  2.00 and the internal request rate to 3333.36 req/s; a retry budget of 0.30 brings this to 0.97
  and 1614.60 req/s and cuts the end-to-end duration from 116 to 86 ms.
- Retrying is free only for idempotent calls; a call that is not idempotent can be retried only if
  the contract carries an idempotency key.

#### Next Step

This topic designed **synchronous** calling: the caller waits, the budget does its work, and in the
end either an answer arrives or a failure does. The budget calculation made a limit visible — as
the chain lengthens, either the budget splits into pieces and every step narrows, or the end-to-end
duration grows. Both options sit inside the same assumption: the caller is waiting for the result
of the work. For some jobs there is a third path, and that path removes the assumption — the caller
never waits at all. A state event arriving from the carrier being written into the store does not
have to finish in order to answer the request. The next topic measures what dropping a request into
a queue changes, and it asks this as a capacity question: on the synchronous path, capacity has to
be chosen for the instantaneous peak; once a queue is added, it can be brought closer to the
average. What that gain costs in pending work and waiting time, and which of K01's numbers change,
is computed there.

## Queues and Workflows

Spreading load over time and running multi-step jobs: buffering with message queues, moving long jobs to the background, raising throughput with competing consumers, leveling a spike with a queue, priority for different service levels, managing jobs that require ordering, choreography and orchestration models for a distributed workflow, the durable workflow pattern, leader election for a single point of responsibility, and returning an asynchronous response for a long-running operation.

### Message Queues

Source: https://academia.sh/en/courses/application-layer/message-queues

Treating the queue as a capacity decision: choosing capacity by the instantaneous peak on the synchronous write path, moving processing capacity toward the average once a queue is introduced, paying for the gain in backlog and wait time, comparing the number of events two designs lose during a brief store outage, and recalculating the request rate reaching the store.

The previous topic designed the synchronous call: the caller waits, a timeout budget runs, and
at its end either a response or an error arrives. Its last lesson worked out that budget's
arithmetic and exposed a limit: as the chain grows, either the budget splits into pieces and
each step narrows, or the end-to-end duration grows. Both options share one assumption — the
caller waits for the result. For some jobs a third path exists, and it drops that assumption:
the caller never waits at all. Writing the carrier's state event into the store does not have
to finish before the request can be answered.

This topic takes up that path. **Message queue** mechanics were built and measured in the
Caching, Queues and Asynchronous Processing course: producer writes, consumer receives,
acknowledgement, visibility timeout, delivery guarantees, consumer groups, ordering, the dead
letter queue. None of it is retold here. This course asks a different question, in one
sentence: **a queue is a capacity decision.** What is gained, what is paid, and which number
from the introductory course changes when load spreads over time.

#### Which Number Sets Capacity

On the synchronous write path one rule is not up for debate: capacity is chosen by the
**instantaneous peak** rate. If the carrier sends a state event and waits for a response, every
request arriving at that instant needs somewhere to be processed; one that does not get it is
rejected. The Introduction to System Design course's Back-of-the-Envelope Estimation lesson
gives this number: **peak writes of 97.22 requests/s** — a computed value that follows
arithmetically from 2,800,000 daily state events, a daily average of 32.41 requests/s, and V8's
peak factor of 3.

A queue splits that rule in two. The endpoint the carrier talks to still has to accept the peak,
but its work there shrinks to a single append: write the event, return the acknowledgement.
Actually processing it — updating the shipment record, writing the route step, preparing the
pricing line — moves behind the queue, where capacity becomes **something you choose**. That
freedom is the queue's gain; its limit is the queue itself, since every unconsumed event piles
up.

Calculating the backlog needs a number missing from K01's table. V8 says how **high** the peak
is, not how **long** it lasts.

**K1 — peak window: 3 hours a day.** This course's own assumption; not added to K01's table.
Rationale: carriers' delivery rounds produce the state events, and a round is packed into one
slice of the day. Since the daily total comes from K01, the off-peak rate is no longer free:
three hours at 97.22 requests/s produces 1,050,000 events, the remaining 1,750,000 spread over
the other twenty-one hours, and the off-peak rate works out to 23.15 requests/s. Its sensitivity
is measured in the second table below.

#### Backlog

The setup below builds the queue, producer, and consumer as in-process modules. A tick is one
second; it runs a day second by second and tracks the backlog against the chosen processing
capacity. **This is a model**, not a measurement: every number in it follows arithmetically from
K01 and K1, so results are deterministic and machine-independent — the computed-value class. It
runs four days so the starting backlog becomes periodic, and the numbers are taken from the
third day.

```js
// queue/spread.mjs — feeding a fixed-capacity consumer from the daily event profile.
// MODEL: queue and consumer are in-process modules, one tick is one second. Results are
// deterministic (computed-value class) and machine-independent; there is no measured duration.
const DAY = 86_400;
const DAILY_EVENTS = 2_800_000;    // K01 Back-of-the-Envelope Estimation: daily state events
const PEAK_FACTOR = 3;             // K01 assumption V8
const PEAK_START = 9 * 3600;       // where the peak window sits inside the day
const THRESHOLD = 300;             // "delayed event" threshold (s)

function profile(peakHours) {
  const average = DAILY_EVENTS / DAY;
  const peak = average * PEAK_FACTOR;
  const peakSec = peakHours * 3600;
  const offPeak = (DAILY_EVENTS - peak * peakSec) / (DAY - peakSec);
  return { average, peak, offPeak,
    arrival: (t) => (t >= PEAK_START && t < PEAK_START + peakSec ? peak : offPeak) };
}

function run(capacity, p) {         // runs four days; the first two days make the starting
  let backlog = 0;                  // backlog periodic, numbers are taken from the third day
  let highest = 0, total = 0, delayed = 0, drain = -1;
  const MEASURE = 2 * DAY;          // start of the measurement day
  for (let t = 0; t < 4 * DAY; t += 1) {
    const arriving = p.arrival(t % DAY);
    if (t >= MEASURE && t < MEASURE + DAY) {
      if (backlog / capacity > THRESHOLD) delayed += arriving;
      total += arriving;
    }
    backlog += arriving;
    backlog -= Math.min(backlog, capacity);
    if (t >= MEASURE && t < MEASURE + DAY && backlog > highest) highest = backlog;
    if (drain < 0 && t > MEASURE + PEAK_START && backlog < 1) drain = t - MEASURE - PEAK_START;
  }
  return { highest, drain, delayed, rate: delayed / total,
    wait: highest / capacity };
}

function table(peakHours) {
  const p = profile(peakHours);
  console.log(`K1 = ${peakHours} hour peak window -> peak ${p.peak.toFixed(2)}/s, ` +
    `off-peak ${p.offPeak.toFixed(2)}/s, average ${p.average.toFixed(2)}/s`);
  console.log("capacity  ratio to peak  highest backlog  longest wait  drain  delayed events  delayed rate");
  for (const ratio of [1, 2 / 3, 0.5, 0.4, 0.35, 1 / 3]) {
    const k = p.peak * ratio;
    const r = run(k, p);
    const w = r.wait < 3600 ? `${(r.wait / 60).toFixed(1)} min` : `${(r.wait / 3600).toFixed(2)} h`;
    const d = r.drain < 0 ? "none" : `${(r.drain / 3600).toFixed(2)} h`;
    console.log(`${k.toFixed(2).padStart(8)}  ${ratio.toFixed(2).padStart(13)}  ` +
      `${r.highest.toFixed(0).padStart(16)}  ${w.padStart(12)}  ${d.padStart(5)}  ` +
      `${r.delayed.toFixed(0).padStart(14)}  ${r.rate.toFixed(3).padStart(12)}`);
  }
}

console.log(`${DAILY_EVENTS} state events a day (K01), peak factor ${PEAK_FACTOR} (V8)`);
console.log(`"delayed event" = an event waiting longer than ${THRESHOLD} seconds\n`);
table(3);
console.log();
table(6);
```

```
2800000 state events a day (K01), peak factor 3 (V8)
"delayed event" = an event waiting longer than 300 seconds

K1 = 3 hour peak window -> peak 97.22/s, off-peak 23.15/s, average 32.41/s
capacity  ratio to peak  highest backlog  longest wait  drain  delayed events  delayed rate
   97.22           1.00                 0       0.0 min  0.00 h               0         0.000
   64.81           0.67            350000        1.50 h  5.33 h         1175324         0.420
   48.61           0.50            525000        3.00 h  8.73 h         1484861         0.530
   38.89           0.40            630000        4.50 h  14.12 h         1939884         0.693
   34.03           0.35            682500        5.57 h  20.43 h         2464667         0.880
   32.41           0.33            700000        6.00 h  24.00 h         2761134         0.986

K1 = 6 hour peak window -> peak 97.22/s, off-peak 10.80/s, average 32.41/s
capacity  ratio to peak  highest backlog  longest wait  drain  delayed events  delayed rate
   97.22           1.00                 0       0.0 min  0.00 h               0         0.000
   64.81           0.67            700000        3.00 h  9.60 h         2177789         0.778
   48.61           0.50           1050000        6.00 h  13.71 h         2366670         0.845
   38.89           0.40           1260000        9.00 h  18.46 h         2560693         0.915
   34.03           0.35           1365000       11.14 h  22.33 h         2714390         0.969
   32.41           0.33           1400000       12.00 h  24.00 h         2780556         0.993
```

#### What Is Gained, What Is Paid

The first row of the first table is the synchronous design itself: capacity equal to the peak
means zero backlog and zero wait. Every row below gives up capacity and buys wait time instead.

**The gain is linear.** At half the peak, the processing side shrinks by half; at a third, it
drops to a third. The lower bound is no coincidence: a third of the peak is the daily average,
and V8 defines the peak as three times the average. **The upper bound on a queue's capacity
gain is the peak factor itself** — sizing to the average instead of the peak gains exactly V8,
and no more.

**The cost is not linear.** As capacity drops from 1.00 to 0.67, the highest backlog rises from
0 to 350,000; at 0.50 it reaches 525,000, at 0.40 it reaches 630,000. Giving up the last
seventeen percent of capacity adds 105,000 to the backlog, while the first thirty-three percent
added 350,000 — but wait time reverses that order: the longest wait climbs from 1.50 hours to
3.00, then 4.50. Since wait is backlog divided by capacity, the numerator grows as the
denominator shrinks, and the two effects collide. In the row closest to the average, the wait is
six hours.

**The drain column draws a line.** At 0.35 of peak capacity, the backlog reaches zero 20.43
hours after the peak; at 0.33 it takes exactly 24.00 hours — zero at the exact moment the next
day's peak begins. This is a knife's edge: sizing capacity to the average leaves a single busy
day's backlog carrying into the next, growing day after day. **Sizing to the average is a
theoretical limit, not a design you can actually choose.**

**The delayed-events column says who pays.** Events waiting longer than five minutes number
1,175,324 at 0.67 of peak, 42 percent of the day's events. That climbs to 53 percent at half
capacity, 69 percent at 0.40. This number is directly the freshness the tracking query sees:
when the carrier sends "out for delivery," the recipient's tracking page shows it only once the
backlog drains.

**The second table is K1's sensitivity.** Growing the peak window from three hours to six does
not change the peak rate — that comes from V8 — but the off-peak rate drops from 23.15 to 10.80
requests/s, and backlogs exactly double: 1,050,000 instead of 525,000 at 0.50 of peak, six hours
of wait instead of three. The assumption's effect on the design is direct and one-directional:
**the peak's duration matters as much as its height**, and K01's table did not carry this
information.

#### Two Designs Facing an Outage

Looking only at the table, the decision could read against the queue. One measurement is
missing: a queue does not just spread load over time, it decouples the producer from the
consumer's failures. The second setup adds a case where the store refuses writes for fifteen
minutes mid-peak. The outage is a parameter.

```js
// queue/outage.mjs — comparing two designs when the store refuses writes for 15 minutes.
// MODEL: one tick is one second, the outage is triggered by a parameter. Results are
// deterministic (computed-value class).
const DAY = 86_400;
const DAILY_EVENTS = 2_800_000;    // K01: daily state events
const PEAK = (DAILY_EVENTS / DAY) * 3;              // K01 calculation + V8: peak write events/s
const PEAK_START = 9 * 3600, PEAK_SEC = 3 * 3600;   // K1: 3-hour peak window
const OFF_PEAK = (DAILY_EVENTS - PEAK * PEAK_SEC) / (DAY - PEAK_SEC);
const OUTAGE_START = 10 * 3600, OUTAGE_SEC = 900;   // outage sits in the middle of the peak window

const arrival = (t) => (t >= PEAK_START && t < PEAK_START + PEAK_SEC ? PEAK : OFF_PEAK);
const isOutage = (t) => t >= OUTAGE_START && t < OUTAGE_START + OUTAGE_SEC;

function run(capacity, hasQueue, hasOutage) {
  let backlog = 0, lost = 0, highest = 0, drain = -1;
  for (let t = 0; t < 2 * DAY; t += 1) {
    const day = t % DAY;
    const arriving = arrival(day);
    const cap = hasOutage && isOutage(day) ? 0 : capacity;
    if (hasQueue) backlog += arriving;
    else if (arriving > cap) { lost += (arriving - cap) * (t >= DAY ? 1 : 0); }
    backlog -= Math.min(backlog, cap);
    if (t >= DAY) {
      if (backlog > highest) highest = backlog;
      if (drain < 0 && day > PEAK_START && backlog < 1) drain = day - PEAK_START;
    }
  }
  return { lost, highest, drain, wait: highest / capacity };
}

const format = (r, capacity) => `${capacity.toFixed(2).padStart(8)}  ` +
  `${r.lost.toFixed(0).padStart(13)}  ${r.highest.toFixed(0).padStart(16)}  ` +
  `${(r.wait / 3600).toFixed(2).padStart(11)}  ` +
  `${(r.drain < 0 ? NaN : r.drain / 3600).toFixed(2).padStart(11)}`;

console.log(`outage: ${OUTAGE_SEC} s, inside the peak window; events arriving during it: ` +
  `${(PEAK * OUTAGE_SEC).toFixed(0)}\n`);
console.log("design                   capacity  lost events  highest backlog  wait (h)  drain (h)");
console.log(`${"synchronous write".padEnd(22)} ${format(run(PEAK, false, false), PEAK)}`);
console.log(`${"synchronous + outage".padEnd(22)} ${format(run(PEAK, false, true), PEAK)}`);
for (const ratio of [0.5, 0.4]) {
  const k = PEAK * ratio;
  console.log(`${`queue ${ratio.toFixed(2)}x peak`.padEnd(22)} ${format(run(k, true, false), k)}`);
  console.log(`${`queue ${ratio.toFixed(2)}x + outage`.padEnd(22)} ${format(run(k, true, true), k)}`);
}

const BEHIND_CACHE_READ = 41.67;   // K01: reads behind cache/s
const STORE_BASELINE = 138.89;     // K01: requests reaching the store/s
console.log(`\nK01 revisited: requests reaching the store/s, baseline ${STORE_BASELINE} (reads ${BEHIND_CACHE_READ} + writes ${PEAK.toFixed(2)})`);
for (const ratio of [1, 0.5, 0.4]) {
  const k = PEAK * ratio, store = BEHIND_CACHE_READ + k;
  console.log(`  processing capacity ${k.toFixed(2).padStart(6)}/s -> reaching the store ${store.toFixed(2).padStart(6)}/s, ` +
    `${(store / STORE_BASELINE).toFixed(2)}x the baseline, write/read at the store ${(k / BEHIND_CACHE_READ).toFixed(2)}`);
}
```

```
outage: 900 s, inside the peak window; events arriving during it: 87500

design                   capacity  lost events  highest backlog  wait (h)  drain (h)
synchronous write         97.22              0                 0         0.00         0.00
synchronous + outage      97.22          87500                 0         0.00         0.00
queue 0.50x peak          48.61              0            525000         3.00         8.73
queue 0.50x + outage      48.61              0            568750         3.25         9.20
queue 0.40x peak          38.89              0            630000         4.50        14.12
queue 0.40x + outage      38.89              0            665000         4.75        14.73

K01 revisited: requests reaching the store/s, baseline 138.89 (reads 41.67 + writes 97.22)
  processing capacity  97.22/s -> reaching the store 138.89/s, 1.00x the baseline, write/read at the store 2.33
  processing capacity  48.61/s -> reaching the store  90.28/s, 0.65x the baseline, write/read at the store 1.17
  processing capacity  38.89/s -> reaching the store  80.56/s, 0.58x the baseline, write/read at the store 0.93
```

The fifteen-minute outage drops **87,500 events** in the synchronous design — straight
arithmetic: 900 seconds × 97.22 requests/s. Even if the carrier retries, those retries are a
separate design decision, addressed in the Resilience and Reliability course; what is measured
here is what the design itself loses on its own.

The same outage drops **zero events** in the queued design. Instead it pays 43,750 events added
to the backlog and fifteen minutes added to the wait: at half capacity, the highest backlog
rises from 525,000 to 568,750, the wait from 3.00 to 3.25 hours, the drain from 8.73 to 9.20
hours. This is not a comparison but a **conversion**: the queue turns loss into delay. Where
loss is unacceptable and delay tolerable, that is a gain; otherwise, a loss.

#### Back to the Numbers

The last three lines rewrite one of K01's calculations. The Back-of-the-Envelope Estimation
lesson found the peak request rate reaching the store at 138.89 requests/s: 41.67 reads behind
the cache plus 97.22 writes. That write share was the synchronous design's assumption — every
event written to the store the moment it arrives.

A queue makes the write share something you choose. At half the peak, the request rate reaching
the store drops to 90.28 requests/s, 0.65 of the baseline; at 0.40 of peak, 80.56 requests/s,
0.58 of the baseline. The write/read ratio at the store shifts too: K01's 2.33 becomes 1.17 and
0.93. K01 found the system write-heavy where the store sees it; a queue can reverse that
balance, and sizing the store starts from this new number.

The decision comes down to one line. A queue gains capacity only when three conditions hold
together: a real gap between peak and average (V8 greater than one), the caller not waiting for
the result, and a tolerable delay in the work's effect. The third is skipped most often; the
"delayed events" column is its cost.

#### Summary

- On the synchronous write path, capacity is chosen by the instantaneous peak; K01's peak write
  rate of 97.22 requests/s is this design's requirement.
- A queue splits the capacity decision in two: the acceptance path still takes the peak,
  processing capacity becomes a choice; the gain's upper bound is the peak factor V8, three
  times.
- The gain is linear, the cost is not: as capacity drops to 0.67 / 0.50 / 0.40 of the peak, the
  highest backlog becomes 350,000 / 525,000 / 630,000, and the longest wait becomes 1.50 / 3.00 /
  4.50 hours.
- Capacity equal to the average is a knife's edge: the backlog drains in exactly 24.00 hours, and
  98.6 percent of events wait longer than five minutes.
- The K1 assumption is sensitive: growing the peak window from three to six hours doubles every
  backlog and wait, while the peak rate stays the same.
- A queue converts loss into delay: a fifteen-minute store outage drops 87,500 events in the
  synchronous design but zero in the queued one, at the cost of the wait rising from 3.00 to
  3.25 hours.

#### Next Step

Every event in this lesson was the equal of every other: each state event was a record of the
same size, doing the same work, lasting milliseconds. The backlog calculation rested on that too
— capacity is a single number because the jobs are all one type. In the same system one job
breaks this assumption: the period report a seller requests scans thousands of shipment records
and takes seconds. Run inside a request, such a job does not just spend its own time — it holds
up the short requests behind it. The next lesson separates these two kinds of work: it measures
what a long job on the request path does to short requests, and counts which measurement
improves once it moves into a task queue.

### Task Queues and Background Jobs

Source: https://academia.sh/en/courses/application-layer/task-queues-and-background-jobs

Measuring the removal of a long job from the request path: deriving the report job's duration from the introductory course's numbers, counting how much queuing a short request accumulates when it shares a slot pool with a long job, which measurement improves once the job moves into a separate task queue, and the scaling unit shifting from replica to worker.

Every event in the previous lesson was the equal of every other: each state event was a record
of the same size, doing the same work, lasting milliseconds. The backlog calculation rested on
that too — capacity can be expressed as a single number because the jobs are all one type. In
the same system there is a job that breaks this assumption: the period report a seller requests
scans thousands of shipment records and takes seconds. Run inside a request, such a job does not
just spend its own time — it also holds up the short requests behind it.

This lesson takes that job off the request path. **Task queues** use the same mechanics as
message queues; what differs is content — what gets written to the queue is not an event
notification but a **job** someone wants done. Worker processes, pull-based dispatch,
supervision, graceful shutdown, and recording a long job's progress in pieces were built in the
Caching, Queues and Asynchronous Processing course; none of it is retold here. What is measured
here is only the separation's **capacity payoff**: how much the short request's wait improves,
and what the design's scaling unit becomes.

#### Where the Job's Length Comes From

The long job's length is not invented; it follows from K01's numbers. The Back-of-the-Envelope
Estimation lesson found 4,000 invoice lines a day, each line one seller's one day, so the seller
count is 4,000. With 400,000 daily shipments, that is 100 shipments a day per seller. Per V11
the pricing period is 30 days, so a period report scans **3,000 shipment records**. The same
lesson gives the scan rate too: 833.33 records/s, or 1.20 ms per record — the report job takes
**3.60 seconds**. All these numbers belong to the computed-value class.

The short job being compared is the tracking query, reading a single record — one scan unit. The
ratio between them is directly 3,000. The model's tick unit follows from this: a tick is 1.20
ms, and no duration inside the model is measured — each is converted from ticks.

Two assumptions are needed, and both are this course's own; neither is added to K01's table.

**K2 — morning window: 30 minutes.** Rationale: K01's V10 row says "the seller wants the report
in the morning"; report requests bunch up at the start of the business day.

**K3 — share of sellers requesting a report in the window: 0.50.** Rationale: period reports are
requested in bulk on certain days of the month, when half the sellers are assumed to check in.
Sensitivity is measured below in the 0.25 rows.

The read load comes from K01 and K02: peak reads of 416.67 requests/s, replica count 4 — the
Traffic Layer course's Session Stickiness lesson raised that number from 3 to 4. Reads per
replica come to 104.17 requests/s.

#### Two Jobs in the Same Pool

The setup builds a single replica as an in-process model. The replica has a fixed number of
**slots**; each slot holds one request at a time, and once free takes the next from the front of
the queue. Arrivals come from two independent fixed-seed generators, so every design faces the
**same** request sequence. Being a model, the results are deterministic and machine-independent.

```js
// task/pool.mjs — comparing a long job staying on the request path against moving it into a
// task queue. MODEL: tick = 1.20 ms (K01: bulk job scan 833.33 records/s), arrivals come from a
// fixed-seed generator. Results are deterministic; duration is not measured, it is converted
// from ticks.
const TICK_MS = 1.2;               // K01: 1.20 ms per record
const READ_PEAK = 416.67;          // K01: peak read requests/s
const REPLICAS = 4;                // K02 session stickiness lesson: 4 replicas
const SELLERS = 4000;              // K01: daily invoice lines = number of sellers
const REPORT_RECORDS = 3000;       // K01: 100 shipments/day per seller x V11 = 30 days
const K2_WINDOW_MIN = 30;          // this course's assumption K2: morning window
const K3_RATIO = 0.5;              // this course's assumption K3: share of sellers requesting a report in the window
const SEED = 20260730;

const TICKS = Math.round((K2_WINDOW_MIN * 60 * 1000) / TICK_MS);
const trackingP = (READ_PEAK / REPLICAS) * (TICK_MS / 1000);
const reportCount = (ratio) => (SELLERS * ratio) / REPLICAS;

function generator(seed) {         // splitmix32: independent arrival sequences for two streams
  let a = seed >>> 0;
  return () => {
    a = (a + 0x9e3779b9) >>> 0;
    let z = Math.imul(a ^ (a >>> 16), 0x21f0aaad) >>> 0;
    z = Math.imul(z ^ (z >>> 15), 0x735a2d97) >>> 0;
    return ((z ^ (z >>> 15)) >>> 0) / 4294967296;
  };
}

function run({ slots, separate, workers, ratio = K3_RATIO }) {
  const reportP = reportCount(ratio) / TICKS;
  const randTracking = generator(SEED), randReport = generator(SEED + 1);
  const busy = new Array(slots).fill(0), reportBusy = new Array(workers).fill(0);
  const queue = [], tasks = [], wait = [], reportDuration = [];
  let reportsArrived = 0;
  for (let t = 0; t < TICKS; t += 1) {
    if (randTracking() < trackingP) queue.push({ t, duration: 1 });
    if (randReport() < reportP) { reportsArrived += 1; (separate ? tasks : queue).push({ t, duration: REPORT_RECORDS, report: true }); }
    for (let i = 0; i < slots; i += 1) {
      if (busy[i] > 0) busy[i] -= 1;
      if (busy[i] === 0 && queue.length > 0) {
        const job = queue.shift();
        if (job.report) reportDuration.push(t - job.t + job.duration); else wait.push(t - job.t);
        busy[i] = job.duration;
      }
    }
    for (let i = 0; i < workers; i += 1) {
      if (reportBusy[i] > 0) reportBusy[i] -= 1;
      if (reportBusy[i] === 0 && tasks.length > 0) {
        const job = tasks.shift();
        reportDuration.push(t - job.t + job.duration);
        reportBusy[i] = job.duration;
      }
    }
  }
  const sorted = [...wait].sort((a, b) => a - b);
  const percentile = (p) => sorted[Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length))];
  return { tracking: wait.length, p50: percentile(50), p99: percentile(99), longest: sorted[sorted.length - 1],
    late: wait.filter((w) => w * TICK_MS > 100).length,
    reportsArrived, completed: reportDuration.length,
    reportAvg: reportDuration.reduce((a, b) => a + b, 0) / reportDuration.length };
}

console.log(`tick = ${TICK_MS} ms, window ${K2_WINDOW_MIN} min = ${TICKS} ticks, ${REPLICAS} replicas`);
console.log(`per replica tracking ${(READ_PEAK / REPLICAS).toFixed(2)} requests/s, ` +
  `K3 = ${K3_RATIO} -> expected reports per replica ${reportCount(K3_RATIO)}`);
console.log(`report job = ${REPORT_RECORDS} records = ${(REPORT_RECORDS * TICK_MS / 1000).toFixed(2)} s\n`);

const row = (name, r) => console.log(`${name.padEnd(24)} ${String(r.tracking).padStart(7)} ` +
  `${(r.p50 * TICK_MS).toFixed(2).padStart(9)} ${(r.p99 * TICK_MS).toFixed(2).padStart(9)} ` +
  `${(r.longest * TICK_MS).toFixed(0).padStart(11)} ${String(r.late).padStart(9)} ` +
  `${(r.late / r.tracking).toFixed(4).padStart(6)} ${`${r.completed}/${r.reportsArrived}`.padStart(9)} ` +
  `${(r.reportAvg * TICK_MS / 1000).toFixed(2).padStart(9)}`);

console.log("design                    tracking  p50 (ms)  p99 (ms)  longest(ms)  >100 ms     rate  done/arrived  report (s)");
for (const slots of [2, 4, 8]) row(`shared s=${slots}`, run({ slots, separate: false, workers: 0 }));
for (const slots of [2, 4]) row(`shared s=${slots}, K3=0.25`, run({ slots, separate: false, workers: 0, ratio: 0.25 }));
for (const slots of [2, 4]) row(`separate s=${slots}, w=2`, run({ slots, separate: true, workers: 2 }));

const fewest = (make) => { for (let s = 1; s <= 12; s += 1) if (make(s).late === 0) return s; return 0; };
const sharedSlots = fewest((slots) => run({ slots, separate: false, workers: 0 }));
const separateSlots = fewest((slots) => run({ slots, separate: true, workers: 2 }));
const reportSeconds = reportCount(K3_RATIO) * REPLICAS * (REPORT_RECORDS * TICK_MS / 1000);
const workers = reportSeconds / (K2_WINDOW_MIN * 60);
console.log(`\nfewest slots that zero out tracking wait above 100 ms: shared ${sharedSlots}, separate ${separateSlots}`);
console.log(`scaling unit -> shared ${sharedSlots * REPLICAS} slots (all on the read path), ` +
  `separate ${separateSlots * REPLICAS} slots + task workers`);
console.log(`system-wide report work = ${reportSeconds.toFixed(0)} worker-seconds / ` +
  `${K2_WINDOW_MIN * 60} s = ${workers.toFixed(2)} workers (utilization 1.00)`);
```

```
tick = 1.2 ms, window 30 min = 1500000 ticks, 4 replicas
per replica tracking 104.17 requests/s, K3 = 0.5 -> expected reports per replica 500
report job = 3000 records = 3.60 s

design                    tracking  p50 (ms)  p99 (ms)  longest(ms)  >100 ms     rate  done/arrived  report (s)
shared s=2                187459      0.00   5887.20        9398     54392 0.2902   457/457      4.24
shared s=4                187459      0.00    634.80        2808      3088 0.0165   457/457      3.61
shared s=8                187459      0.00      0.00           0         0 0.0000   457/457      3.60
shared s=2, K3=0.25       187459      0.00   2766.00        4984     15204 0.0811   223/223      3.77
shared s=4, K3=0.25       187459      0.00      0.00        1240       293 0.0016   223/223      3.60
separate s=2, w=2         187459      0.00      0.00           0         0 0.0000   457/457      4.14
separate s=4, w=2         187459      0.00      0.00           0         0 0.0000   457/457      4.14

fewest slots that zero out tracking wait above 100 ms: shared 7, separate 1
scaling unit -> shared 28 slots (all on the read path), separate 4 slots + task workers
system-wide report work = 7200 worker-seconds / 1800 s = 4.00 workers (utilization 1.00)
```

In thirty minutes, 187,459 tracking requests arrived, and against an expected 500 reports the
seed produced 457 — one realization of the Bernoulli arrival, the same across every row.

#### Low Utilization, Long Queue

The shared pool's numbers look inconsistent at first glance. In a two-slot replica, report work
uses less than half the slots: 457 reports × 3.60 seconds = 1,645 slot-seconds, 46 percent of
the 3,600 slot-seconds two slots provide over a thirty-minute window. The tracking stream wants
only 12.5 percent. Total utilization is under 60 percent, so the pool has enough capacity.

Yet **29.02 percent** of tracking requests wait longer than 100 ms, p99 wait is 5887 ms, the
longest wait 9398 ms. Why sufficient capacity can still produce this table is clear: the queue
drains in order, and if the job ahead takes 3,000 ticks, a one-tick job behind it waits 3,000
ticks too. When two reports run at once, both slots close, and every tracking request arriving
during that span joins the line. **A queue can grow long while average utilization stays low
because of how job durations are distributed; a single utilization number hides that
distribution.**

Adding slots fixes it, but expensively. At four slots the rate drops to 1.65 percent, p99 to
634.80 ms; at eight it hits zero. The threshold is measured: the fewest slots that zero out
waits above 100 ms is **seven**. The short request alone needs one slot — that is the number
the separate design measures. So six slots exist purely to keep it from queuing behind the long
job.

K3's sensitivity points the same way. Halving the share of sellers requesting a report drops the
affected-request rate from 29.02 to 8.11 percent at two slots, and from 1.65 to 0.16 percent at
four — but it does **not reach zero**. Lower demand reduces how often the problem occurs, not
its structure; even one long job holds up the request behind it for its own duration.

#### The Scaling Unit

In the separate design, the report job goes into the queue and the caller returns with a job
ID. Tracking measurements come out the same at every slot count: p99 zero, longest wait zero,
nothing waiting above 100 ms. Pull the long job off the path, and the short request's queue
disappears with it.

The cost falls on the report side, and it is measured: with two workers, the average report
duration is **4.14 seconds** instead of 3.60 — it now waits in its own queue. The gap is 15
percent and closes by adding workers if wanted, because worker count is now independent of the
tracking stream.

The real result is in the last three lines. In the shared design, zeroing out the tracking queue
needs seven slots per replica — **28 slots** across four replicas. All of them sit on the read
path, each configured to accept tracking requests in the replicas where read traffic is
distributed. In the separate design, one slot per replica suffices for tracking — **4 slots**
across four replicas — and the report job moves to its own workforce. That workforce's size is
arithmetic too: system-wide, 2,000 reports × 3.60 seconds = 7,200 worker-seconds, which over the
thirty-minute window comes to **4.00 workers** — utilization of 1.00, so a real design would
leave some margin.

**Separation's gain is not a number, it is independence.** In the shared design, growing report
demand means growing the replica, and a bigger replica also grows the read path, though the
read path never needed that capacity. In the separate design, report demand moves worker count,
read demand moves replica count, and the two never look at each other. The peak read rate from
the introductory course, 416.67 requests/s, has not changed; what changed is how large the
replicas serving that rate have to be — from seven slots per replica down to one.

#### Summary

- The long job's length is derived from K01: 100 shipments/day per seller × 30 days = 3,000
  records, which at a scan rate of 833.33 records/s takes 3.60 seconds; the tracking request is,
  on that same scale, a single record.
- In the shared pool, even with utilization under 60 percent, 29.02 percent of tracking requests
  wait longer than 100 ms, with a p99 of 5887 ms: a single utilization number hides the
  distribution of job durations.
- The cost of zeroing out the queue is slots: the fewest slots that eliminate waits above 100 ms
  is seven in the shared design, one in the separate design.
- Halving the K3 assumption drops the affected-request rate from 29.02 to 8.11 percent, but it
  does not reach zero; demand changes how often the problem occurs, not its structure.
- In the separate design, tracking wait is zero at every slot count; the cost paid is the report
  duration rising from 3.60 to 4.14 seconds with two workers.
- The scaling unit splits apart: the shared design needs 28 slots across four replicas, the
  separate design needs 4 slots plus 4.00 workers system-wide, and the two numbers move
  independently of each other.

#### Next Step

The "4.00 workers" in the last line is not a capacity but a lower bound; at utilization of
exactly one, the queue never drains. The number itself leaves a question open: does adding a
worker really translate into addition? The previous lesson's backlog table treated consumer
capacity as a single number and never asked where it came from — yet capacity is the product of
consumer count and per-consumer rate, and one of those factors does not grow forever. The next
lesson takes up multiple consumers pulling from the same queue: it measures each added
consumer's contribution to drain time, where that contribution stops, and what breaks in
exchange for the throughput gained.

### Competing Consumers

Source: https://academia.sh/en/courses/application-layer/competing-consumers

Measuring how many consumers pull from the same queue as a capacity decision: the shrinking contribution each added consumer makes to drain time, calculating the asymptotic ceiling a shared resource sets, counting the shipment-order breakage and wrong final states traded for throughput, and converting the ceiling into the rate required from a single consumer.

The "4.00 workers" in the previous lesson's last line was not a capacity but a lower bound; at
utilization of exactly one, the queue never drains. The number left a question open: does
adding a worker really translate into addition? The first lesson's backlog table treated
consumer capacity as a single number and never asked where it came from — yet capacity is the
product of consumer count and per-consumer rate, and one factor does not grow forever.

This lesson measures that product. **Competing consumers** are a set of consumers pulling from
the same queue, where each message goes to only one of them. The same arrangement was built and
measured in the Caching, Queues and Asynchronous Processing course, where each message also went
to a single side; the pattern is the same, this course just uses the catalog name for it. The
difference from the consumer groups built there is one sentence: a group decides how many
separate sides the same message gets **copied** to; competing consumers decide how many units
one side's work gets **split** into. This lesson's measure is exactly that split's cost: the
tradeoff between throughput and order.

#### Work That Splits, a Resource That Does Not

The setup puts 5,000 shipments' state events into one queue. Events per shipment come from K01's
V4 assumption: 7, for a total of 35,000 events. The same shipment's consecutive events land next
to each other in the queue, because a carrier sends several scans in one transfer — the only way
order can break, and the model's focus.

There is one thing consumers cannot split: the store where an event's effect gets written. In
the model this is a parameter named the **store slot** — at most S events apply per tick. **S is
not a measured number**, it is a model parameter, and its job is to show that consumer count
alone is not a capacity. It runs with two values: S = 2 and S = 4.

A tick is an abstract step. Processing one event takes 1–3 ticks drawn from the seed; a
shipment's final event ("delivered") takes 6 ticks, because it also prepares the invoice line.
Every run faces the same event sequence, so the results are deterministic and machine-independent.

```js
// competing/consumer.mjs — how many consumers pulling from the same queue affect drain time and
// the final state of shipments. MODEL: a tick is an abstract step; store slot S is a model
// parameter. Results are deterministic (computed-value class) and machine-independent.
const SHIPMENTS = 5000;            // shipments in the workload
const STEPS = 7;                   // K01 assumption V4: state events per shipment
const GAP = 1;                     // queue spacing between the same shipment's consecutive events
const SEED = 20260731;

function generator(seed) {
  let a = seed >>> 0;
  return () => {
    a = (a + 0x9e3779b9) >>> 0;
    let z = Math.imul(a ^ (a >>> 16), 0x21f0aaad) >>> 0;
    z = Math.imul(z ^ (z >>> 15), 0x735a2d97) >>> 0;
    return ((z ^ (z >>> 15)) >>> 0) / 4294967296;
  };
}

function workload() {              // event sequence in the order it lands in the queue
  const r = generator(SEED);
  const events = [];
  for (let ship = 0; ship < SHIPMENTS; ship += 1) {
    const start = r() * SHIPMENTS * GAP;
    for (let step = 0; step < STEPS; step += 1) {
      events.push({ ship, step, t: start + step * GAP + r() * GAP, duration: step === STEPS - 1 ? 6 : 1 + Math.floor(r() * 3) });
    }
  }
  return events.sort((a, b) => a.t - b.t);
}

function run(events, N, S) {
  const consumers = Array.from({ length: N }, () => null);
  const lastApplied = new Array(SHIPMENTS).fill(-1);
  const outOfOrder = new Set();
  let cursor = 0, tick = 0, applied = 0;
  while (applied < events.length) {
    for (const c of consumers) if (c && c.remaining > 0) c.remaining -= 1;
    let slotsLeft = S;              // store write slots: at most S events applied per tick
    for (let i = 0; i < N; i += 1) {
      const c = consumers[i];
      if (c && c.remaining === 0 && slotsLeft > 0) {
        slotsLeft -= 1;
        if (c.job.step < lastApplied[c.job.ship]) outOfOrder.add(c.job.ship);
        lastApplied[c.job.ship] = c.job.step;
        applied += 1;
        consumers[i] = null;
      }
    }
    for (let i = 0; i < N; i += 1) {
      if (consumers[i] === null && cursor < events.length) {
        const job = events[cursor]; cursor += 1;
        consumers[i] = { job, remaining: job.duration };
      }
    }
    tick += 1;
  }
  const wrongFinal = lastApplied.filter((k) => k !== STEPS - 1).length;
  return { tick, outOfOrder: outOfOrder.size, wrongFinal, throughput: events.length / tick };
}

const events = workload();
console.log(`workload: ${SHIPMENTS} shipments x ${STEPS} events (V4) = ${events.length} events`);
console.log(`store slots S: at most S events applied per tick (model parameter)\n`);
for (const S of [2, 4]) {
  console.log(`S = ${S}`);
  console.log("  N  drain ticks  speedup  added consumer's contribution  throughput(events/tick)  out of order  wrong final state");
  let previous = null;
  const baseline = run(events, 1, S).tick;
  for (const N of [1, 2, 3, 4, 6, 8]) {
    const r = run(events, N, S);
    const h = baseline / r.tick;
    const contribution = previous === null ? "-" : ((h - previous.h) / (N - previous.N)).toFixed(3);
    console.log(`${String(N).padStart(3)}  ${String(r.tick).padStart(11)}  ${h.toFixed(2).padStart(7)}  ` +
      `${contribution.padStart(29)}  ${r.throughput.toFixed(3).padStart(23)}  ${String(r.outOfOrder).padStart(13)}  ` +
      `${String(r.wrongFinal).padStart(18)}`);
    previous = { h, N };
  }
  console.log();
}

const PEAK = 97.22;                // K01: peak write requests/s
const PROCESSING = 48.61;          // lesson 01's chosen processing capacity (half the peak)
const soloThroughput = run(events, 1, 2).throughput;
console.log(`K01 revisited: peak write ${PEAK} events/s, lesson 01's processing capacity ${PROCESSING} events/s`);
for (const S of [2, 4]) {
  const ceiling = S / soloThroughput;
  console.log(`  S = ${S}: asymptotic ceiling ${ceiling.toFixed(2)} consumers -> a single consumer needs at least ` +
    `${(PROCESSING / ceiling).toFixed(2)} events/s (queued) / ${(PEAK / ceiling).toFixed(2)} events/s (synchronous)`);
}
const a = run(events, 6, 2), b = run(events, 8, 2);
console.log(`  S = 2, N 6 -> 8: drain ${((b.tick / a.tick - 1) * 100).toFixed(1)}%, ` +
  `out of order +${((b.outOfOrder / a.outOfOrder - 1) * 100).toFixed(0)}%, ` +
  `wrong final state ${a.wrongFinal} -> ${b.wrongFinal}`);
```

```
workload: 5000 shipments x 7 events (V4) = 35000 events
store slots S: at most S events applied per tick (model parameter)

S = 2
  N  drain ticks  speedup  added consumer's contribution  throughput(events/tick)  out of order  wrong final state
  1        89972     1.00                              -                    0.389              0                   0
  2        44987     2.00                          1.000                    0.778            102                   0
  3        30620     2.94                          0.938                    1.143            248                   0
  4        23845     3.77                          0.835                    1.468            511                   0
  6        18470     4.87                          0.549                    1.895           1590                  30
  8        17551     5.13                          0.128                    1.994           2279                 432

S = 4
  N  drain ticks  speedup  added consumer's contribution  throughput(events/tick)  out of order  wrong final state
  1        89972     1.00                              -                    0.389              0                   0
  2        44987     2.00                          1.000                    0.778            102                   0
  3        29993     3.00                          1.000                    1.167            236                   0
  4        22496     4.00                          1.000                    1.556            429                   0
  6        15099     5.96                          0.980                    2.318            915                   0
  8        11591     7.76                          0.902                    3.020           1543                   0

K01 revisited: peak write 97.22 events/s, lesson 01's processing capacity 48.61 events/s
  S = 2: asymptotic ceiling 5.14 consumers -> a single consumer needs at least 9.45 events/s (queued) / 18.91 events/s (synchronous)
  S = 4: asymptotic ceiling 10.28 consumers -> a single consumer needs at least 4.73 events/s (queued) / 9.45 events/s (synchronous)
  S = 2, N 6 -> 8: drain -5.0%, out of order +43%, wrong final state 30 -> 432
```

#### Where the Contribution Stops

The "added consumer's contribution" column gives speedup's per-consumer increase; a contribution
of one means the added consumer did one consumer's worth of work.

In the S = 2 column, contribution is 1.000 at N = 2, 0.938 at N = 3, 0.835 at N = 4, 0.549 at
N = 6, and 0.128 at N = 8. The eighth consumer adds about an eighth of what the first one
did. Drain time goes from 18,470 to 17,551 ticks, a drop of only 5 percent. The throughput
column says why: 1.994 events/tick, the S = 2 limit itself. **Adding a consumer is a decision on
the queue's side, but the ceiling is set by a resource outside the queue.**

In the S = 4 column, the same consumer counts behave differently: contribution stays above 0.98
through N = 6, and is 0.902 at N = 8. Speedup is 7.76, throughput 3.020 events/tick. This time
the ceiling is far away, and adding a consumer still pays off. The difference between the two
columns is a single parameter, and **nothing changed on the queue's side**.

The ceiling itself can be calculated. Since a single consumer's throughput is 0.389 events/tick,
the asymptotic speedup ceiling is $S / 0.389$: 5.14 consumers for S = 2, 10.28 for S = 4. The
measured values 5.13 and 7.76 are consistent with this ceiling. The design result is direct:
**the upper bound on useful consumers behind a queue is a single consumer's rate divided by the
capacity of the narrowest resource they share.**

#### What Breaks in Exchange for Throughput

The two right-hand columns are the cost. "Out of order" means at least one of a shipment's
events is applied **after** a later event; "wrong final state" means a shipment's last-applied
event is not the seventh step, so it does not appear finished in the record.

With one consumer, both are zero: queue order is application order. With two consumers, 102 of
5,000 shipments lose their order. At four, 511; at six, 1,590; at eight, 2,279 — 45.6 percent of
shipments. The breakage accelerates with consumer count, because as more events process at once,
the odds of two of the same shipment's events colliding rise.

The relationship between the two columns is this lesson's most important result. At S = 4 and
N = 8, 1,543 shipments lose their order, yet **none** end in a wrong final state. At S = 2 and
N = 8, 432 of the 2,279 reordered shipments do. The difference is that the final event takes six
ticks, and a narrow store slot delays applying it. So **whether reordering turns into a visible
error depends on the coincidence of job durations and resource scarcity.** A design cannot rely
on that coincidence: change the load distribution, and a number that was 0 becomes 432.

The last line reduces this tradeoff to a single line. At S = 2, going from six to eight
consumers shortens drain time by 5 percent, raises reordered shipments by 43 percent, and raises
wrong-final-state shipments from 30 to 432. **The gain is five percent, the cost fourteen times
that.** Consumers beyond the ceiling add no throughput, only breakage.

The fix for preserving order — routing the same key's events to the same partition — was built
in the Caching, Queues and Asynchronous Processing course, which measured how partition count
affects parallelism and within-key order there. It is not repeated here; this lesson's
contribution is placing reordering's **cost at each consumer count** next to the capacity
decision.

#### Back to the Numbers

The ceiling turns into a requirement once translated into K01's numbers. The first lesson set
processing capacity at half the peak, 48.61 events/s; the synchronous design needed 97.22
events/s.

At S = 2, the highest capacity the group can deliver is 5.14 times a single consumer's rate.
Reaching 48.61 events/s needs a single consumer processing at least **9.45 events/s**; the
synchronous design's 97.22 events/s needs **18.91 events/s**. If one consumer falls below that,
no consumer count is enough, and the fix is not on the queue's side. At S = 4, the same
thresholds drop to 4.73 and 9.45 events/s.

The rule that follows: a capacity target is met not by "how many consumers" but by **the product
of two numbers**, one of which depends on the shared resource's capacity. In a design
discussion, "we add consumers" said before the ceiling is calculated is not a solution — it is
an assumption.

#### Summary

- Competing consumers decide how many units one side's work splits into; consumer groups decide
  how many sides the same message gets copied to.
- An added consumer's contribution shrinks: at S = 2, it is 1.000 at N = 2, 0.835 at N = 4,
  0.128 at N = 8; going from six to eight consumers shortens drain time by only 5 percent.
- The ceiling is set by the shared resource, not the queue: the asymptotic speedup ceiling is
  5.14 consumers for S = 2, 10.28 for S = 4, and the measured 5.13 and 7.76 are consistent with
  it.
- Reordering accelerates with consumer count: of 5,000 shipments, 102 lose their order at two
  consumers, 511 at four, 2,279 at eight.
- Whether reordering turns into a visible error is a matter of chance: at S = 4 and N = 8, none
  of the 1,543 reordered shipments produce a wrong final state, while at S = 2, 432 of 2,279 do.
- Back to K01: a processing capacity of 48.61 events/s requires at least 9.45 events/s from a
  single consumer at S = 2, and 4.73 at S = 4; if one consumer falls below that, consumer count
  is not the fix.

#### Next Step

This lesson's workload was an event sequence flowing at the same rate all day, with consumer
count chosen against that rate. The first lesson's profile carried the same assumption: the
peak is known, when it arrives is known, its duration is known. In a real stream none of the
three is guaranteed. When the carrier's connection drops for twenty minutes, no events arrive;
then everything that piled up arrives at once, at a rate with nothing to do with the day's peak.
The next lesson takes up this spike: it measures what a queue spreading load over time means for
an unpredicted peak, how large the buffer has to be, and what gets dropped when it is not
enough.

### Queue-Based Load Leveling

Source: https://academia.sh/en/courses/application-layer/queue-based-load-leveling

Buffering an unpredicted spike with a queue: a bulk resend after an outage raising the arrival rate tenfold while never changing the buffer requirement, an extra write stream at the same arrival rate growing the buffer by sixty-seven percent, calculating the buffer limit from volume, and counting the events an unbuffered design drops.

The workload in the previous two lessons quietly carried one assumption: the peak is known,
when it arrives is known, its duration is known. The first lesson built this from K01's V8
factor and its own K1 assumption. In a real stream, none of the three is guaranteed. When the
carrier's connection drops for twenty minutes, no events arrive; once it returns, everything
piled up on the handheld devices arrives at once, at a rate with nothing to do with the day's
peak.

This lesson takes up that spike. **Queue-based load leveling** is the decision to place a buffer
between producer and consumer, decoupling the rate the consumer sees from the rate the producer
produces. The name recalls the **load balancer** from the Traffic Layer course, and the two are
different things: a load balancer spreads one request across several replicas, splitting load
**in space**; queue-based load leveling spreads load headed to the same consumer **over time**.
The difference from the first lesson fits one sentence: there, what got leveled was a
**predicted** daily profile; here, what gets leveled is an **unpredicted** spike.

#### Two Different Kinds of Spike

A spike is not one phenomenon, and this lesson's claim is that the two kinds must be separated.
Two assumptions follow; both are this course's own, and neither is added to K01's table.

**K4 — carrier outage: 20 minutes.** Rationale: when a carrier's handheld devices lose coverage,
scans pile up on the device and get sent in bulk once the connection returns. The outage falls
inside the peak window; the number of events resent is 1200 × 97.22 = 116,667. This is a
**rate** event: volume does not change, the same events arrive in less time. How much less time
is measured by the resend multiplier R, run with the values two, four, and ten.

**K5 — extra write stream's duration: 1 hour.** Rationale: when a new carrier connects, or a
campaign period opens, an extra one-hour stream enters the peak window. The extra stream's rate
is set equal to the peak; the number of events added is 3600 × 97.22 = 350,000. This is a
**volume** event: the arrival rate only doubles, but the total work the system processes that
day grows.

Their sensitivities are linear: doubling the outage doubles the events resent, doubling the
extra stream doubles the events added. Why the distinction matters will come out of the
measurement.

#### The Model

The setup uses the first lesson's daily profile exactly: peak 97.22 events/s, three hours per
K1, off-peak 23.15 events/s. One of the two events is added on top. Consumer capacity is the
first lesson's chosen value, half the peak. The queue limit is also a parameter; a limit of zero
means the **unbuffered** design, where every event above capacity is turned away — this behavior
was built under the name **load shedding** in the Caching, Queues and Asynchronous Processing
course and is used here only as a comparison baseline. Being a model, the results are
deterministic and machine-independent.

```js
// leveling/spike.mjs — two shapes of spike: a bulk resend after an outage (rate rises, volume
// fixed) and an extra write stream (volume rises). MODEL: one tick is one second, the daily
// profile is the same as lesson 01's. Results are deterministic (computed-value class) and
// machine-independent.
const DAY = 86_400;
const DAILY_EVENTS = 2_800_000;          // K01: daily state events
const PEAK = (DAILY_EVENTS / DAY) * 3;   // K01 calculation + V8: peak write events/s
const PEAK_START = 9 * 3600, PEAK_SEC = 3 * 3600;   // K1: 3-hour peak window
const OFF_PEAK = (DAILY_EVENTS - PEAK * PEAK_SEC) / (DAY - PEAK_SEC);
const K4 = 20 * 60;                      // this course's assumption K4: carrier outage (s)
const K5 = 3600;                         // this course's assumption K5: extra stream's duration (s)
const START = PEAK_START + 2400;         // both events sit inside the peak window

const normal = (t) => (t >= PEAK_START && t < PEAK_START + PEAK_SEC ? PEAK : OFF_PEAK);

function run({ capacity, limit, R = 1, mode = "none" }) {
  let pending = 0, backlog = 0, dropped = 0, highest = 0, drain = -1, totalArrived = 0;
  const MEASURE = 2 * DAY;
  for (let t = 0; t < 4 * DAY; t += 1) {
    const day = t % DAY;
    let arriving = normal(day);
    if (mode === "retry") {
      if (day >= START && day < START + K4) { pending += arriving; arriving = 0; }
      else { const extra = Math.min(pending, (R - 1) * PEAK); pending -= extra; arriving += extra; }
    } else if (mode === "volume" && day >= START && day < START + K5) arriving += PEAK;
    const room = limit === 0 ? capacity : limit - backlog;
    const accepted = Math.min(arriving, Math.max(0, room));
    if (t >= MEASURE && t < MEASURE + DAY) { dropped += arriving - accepted; totalArrived += arriving; }
    backlog += limit === 0 ? Math.min(accepted, capacity) : accepted;
    backlog -= Math.min(backlog, capacity);
    if (t >= MEASURE && t < MEASURE + DAY && backlog > highest) highest = backlog;
    if (drain < 0 && t > MEASURE + PEAK_START && backlog < 1 && pending < 1) drain = t - MEASURE - PEAK_START;
  }
  return { dropped, totalArrived, highest, drain, wait: highest / capacity };
}

const arrivalRate = (o) => (o.mode === "retry" ? o.R * PEAK : o.mode === "volume" ? 2 * PEAK : PEAK);
const row = (name, o) => {
  const r = run(o), v = arrivalRate(o);
  console.log(`${name.padEnd(24)} ${v.toFixed(0).padStart(7)} ${o.capacity.toFixed(2).padStart(8)} ` +
    `${r.dropped.toFixed(0).padStart(7)} ${r.highest.toFixed(0).padStart(9)} ` +
    `${(r.wait / 3600).toFixed(2).padStart(11)} ` +
    `${(r.drain < 0 ? NaN : r.drain / 3600).toFixed(2).padStart(11)} ` +
    `${(v / o.capacity).toFixed(2).padStart(7)}`);
};

console.log(`K01 peak write ${PEAK.toFixed(2)} events/s; K4 outage ${K4} s -> ` +
  `${(PEAK * K4).toFixed(0)} events resent`);
console.log(`K5 extra stream ${K5} s x ${PEAK.toFixed(2)} events/s -> ${(PEAK * K5).toFixed(0)} events added\n`);
console.log("design                   arrival/s  capacity  dropped  backlog  wait(h)  drain(h)  leveling");
row("unbuffered, no event", { capacity: PEAK, limit: 0 });
row("unbuffered retry R=4", { capacity: PEAK, limit: 0, R: 4, mode: "retry" });
row("unbuffered volume", { capacity: PEAK, limit: 0, mode: "volume" });
row("queue 0.5x, no event", { capacity: PEAK / 2, limit: Infinity });
for (const R of [2, 4, 10]) row(`queue 0.5x retry R=${R}`, { capacity: PEAK / 2, limit: Infinity, R, mode: "retry" });
row("queue 0.5x volume", { capacity: PEAK / 2, limit: Infinity, mode: "volume" });

const smallestLimit = (o) => {
  let low = 0, high = 2_000_000;
  while (high - low > 500) { const mid = (low + high) / 2; if (run({ ...o, limit: mid }).dropped < 1) high = mid; else low = mid; }
  return high;
};
console.log("\nsmallest queue limit needed (capacity 0.5x peak):");
for (const [name, o] of [["no event", {}], ["retry R=4", { R: 4, mode: "retry" }],
  ["retry R=10", { R: 10, mode: "retry" }], ["volume", { mode: "volume" }]]) {
  console.log(`  ${name.padEnd(12)} ${smallestLimit({ capacity: PEAK / 2, ...o }).toFixed(0).padStart(9)}`);
}
console.log("\ndropped events with limit 525000 (chosen for the event-free day):");
for (const [name, o] of [["retry R=4", { R: 4, mode: "retry" }], ["retry R=10", { R: 10, mode: "retry" }],
  ["volume", { mode: "volume" }]]) {
  console.log(`  ${name.padEnd(12)} ${run({ capacity: PEAK / 2, limit: 525_000, ...o }).dropped.toFixed(0).padStart(9)}`);
}
```

```
K01 peak write 97.22 events/s; K4 outage 1200 s -> 116667 events resent
K5 extra stream 3600 s x 97.22 events/s -> 350000 events added

design                   arrival/s  capacity  dropped  backlog  wait(h)  drain(h)  leveling
unbuffered, no event          97    97.22       0         0        0.00        0.00    1.00
unbuffered retry R=4         389    97.22  116667         0        0.00        0.00    4.00
unbuffered volume            194    97.22  350000         0        0.00        0.00    2.00
queue 0.5x, no event          97    48.61       0    525000        3.00        8.73    2.00
queue 0.5x retry R=2         194    48.61       0    525000        3.00        8.73    4.00
queue 0.5x retry R=4         389    48.61       0    525000        3.00        8.73    8.00
queue 0.5x retry R=10        972    48.61       0    525000        3.00        8.73   20.00
queue 0.5x volume            194    48.61       0    875000        5.00       12.55    4.00

smallest queue limit needed (capacity 0.5x peak):
  no event        525391
  retry R=4       525391
  retry R=10      525391
  volume          875488

dropped events with limit 525000 (chosen for the event-free day):
  retry R=4           49
  retry R=10          49
  volume          350049
```

#### A Rate Spike Does Not Write to the Buffer

The four middle rows of the table are this lesson's most counterintuitive result. As the resend
multiplier climbs from two to four to ten, the arrival rate climbs from 194 to 389 to 972
events/s — ten times K01's peak write rate. In those same rows, the highest backlog **stays
fixed at 525,000**, the longest wait at 3.00 hours, the drain at 8.73 hours. All three numbers
are identical to the event-free day's.

The reason is volume conservation. During the outage no events arrive while the consumer keeps
working, so the queue **drains** for those twenty minutes; the resend puts the same events back.
The pit the outage digs and the peak the spike builds cancel out, because their sum has not
changed. All the consumer ever sees is the same amount of work within the same day.

The **leveling** column on the right gives the queue's work as a single number: arrival rate
over consumer capacity. It is 2.00 on the event-free day, then 4.00, 8.00, and **20.00** on the
resend rows. A twentyfold arrival spike leaves no trace on the consumer's side. The queue's real
job fits one line: **it levels rate, not volume.**

In the unbuffered design, the same event drops 116,667 events, and the number dropped is
independent of R — the rate it arrives at makes no difference, everything above capacity goes.
In this design the only fix is sizing capacity to the arrival peak; for R = 10 that requires 972
events/s, ten times K01's peak write rate, and a capacity that sits entirely idle for the rest
of the day.

Slowing the producer down looks like an option too — the path built under the name
**backpressure** in the Caching, Queues and Asynchronous Processing course. It does not apply
here, because the producer is the carrier's handheld device: slowed down, it just buffers
events in its own memory, and the spike is only postponed. Backpressure works in chains where
the producer can genuinely be paused.

#### A Volume Spike Does Write to the Buffer

The last row behaves very differently with the same queue at the same capacity. The extra write
stream only doubles the arrival rate — 194 events/s, the same as the gentlest resend case — yet
the highest backlog rises from 525,000 to **875,000**, 1.67 times. The longest wait stretches
from 3.00 to 5.00 hours, the drain from 8.73 to 12.55 hours.

Comparing the two events' arrival rates, the result is unambiguous: a 972-event/s spike adds
nothing to the buffer while a 194-event/s stream adds 350,000 events. **What fills the buffer is
not arrival rate but unprocessed volume.** In a design discussion, "how many requests a second
arrived" does not determine buffer size; the question that does is how much of the incoming work
stayed unprocessed that day.

#### Where the Limit Is Set

An unbounded queue is not a design; every buffer is finite, and once full it either stops the
producer or drops events. Where the limit goes is a measurable question, and two groups of
output answer it.

The smallest limit needed is 525,391 on the event-free day, and — independent of R — still
**525,391** for the resend event, but **875,488** for the volume event. A buffer sized to the
predicted daily profile absorbs a tenfold arrival spike without dropping one event; the same
buffer falls short against a volume increase.

The last group counts this directly. With the limit set to 525,000, chosen for the event-free
day, the resend event drops 49 events — the result of rounding the limit 391 short of the true
requirement, in practice zero. With the same limit, the volume event drops **350,049** events, a
number that is the added volume itself.

The rule follows: **the buffer limit is calculated from a volume number, not a rate number** —
from the worst day's gap between work produced and work processed. What drop behavior to use
once the limit is chosen is a separate decision, and its options (blocking the producer,
dropping the newest, dropping the oldest) were measured in the Caching, Queues and Asynchronous
Processing course.

#### Back to the Numbers

K01's table does not have the number this lesson needs, and that absence is itself a finding.
V8's peak factor is 3, making the peak write rate 97.22 events/s; that number is the
**predicted** peak. The arrival rate the acceptance path actually has to meet came out to 194,
389, and 972 events/s in the measurement — two, four, and ten times what V8 produces. If a
design sizes its acceptance path to V8, a single twenty-minute outage saturates it.

Processing capacity never changed, though: the 48.61 events/s the first lesson chose is enough
even at a tenfold arrival spike. K01's "peak write 97.22 requests/s" line thus splits into two
numbers — the **instantaneous arrival peak** the acceptance path has to see, and the **volume
rate** the processing path has to meet — and the ratio between them is the queue's leveling
factor.

#### Summary

- Queue-based load leveling spreads load over time; a load balancer spreads load across
  replicas, that is, in space. The profile the first lesson leveled was predicted; the spike
  here is not.
- The 116,667 events arriving after a twenty-minute outage push the arrival rate to 972 events/s,
  yet the highest backlog stays at 525,000, the wait at 3.00 hours, the drain at 8.73 hours.
- A one-hour extra stream raised the arrival rate to only 194 events/s, yet moved the backlog to
  875,000 (1.67 times), the wait to 5.00 hours, the drain to 12.55 hours.
- What fills the buffer is volume, not arrival rate: the smallest queue limit needed came out to
  525,391 for the resend event, independent of R, and 875,488 for the volume increase.
- A limit of 525,000, chosen for the predicted profile, drops 49 events at a tenfold arrival
  spike and 350,049 events at the volume increase.
- K01's peak write rate splits into two numbers: the instantaneous arrival peak the acceptance
  path sees can climb to ten times V8's 97.22, while the volume rate the processing path meets
  stays at 48.61 events/s.

#### Next Step

Every queue so far carried one type of work, and consumers took whatever was next without
distinguishing it. In the same system, the jobs entering a queue are not each other's equal: the
carrier's state event shows up on the recipient's tracking page and its delay is felt directly,
the end-of-day billing record can wait until morning, and the seller's report job can wait
longer than either. This lesson's 525,000-event backlog, treated as a single line, makes all
three wait the same three hours. The next lesson turns the line itself into a decision: it
measures what separating jobs by service level gains each class, what it costs each class, and
what unlimited priority stops completely.

### Priority Queue

Source: https://academia.sh/en/courses/application-layer/priority-queue

Separating three work classes that share the same consumer group by service level: a single queue giving all three classes the same wait, strict priority stopping the lowest class entirely, weighted share redistributing the shortfall, a share acting as both a floor and a ceiling at once, and the total shortfall staying independent of the policy.

Every queue up to this point held a single kind of job, and consumers took whatever was next
without distinguishing it. Jobs entering the same system are not each other's equals: the
carrier's state event shows up on the recipient's tracking page and its delay is felt directly;
the end-of-day billing record has until morning; the seller's report job can wait for either.
When the previous lesson's backlog of 525,000 jobs is treated as a single queue, all three wait
the same three hours.

This lesson turns the queue itself into a decision. Here, a **priority queue** is a service-level
decision: choosing which job goes ahead of which by looking at its work class. The Data
Structures curriculum uses the same English term for a data structure — a heap-based structure
that returns the smallest element by key. The two are separate things, and this lesson does not
cover the data structure; it measures the capacity cost of the decision.

#### Three Classes and One Capacity

The measurement window is K01's V10 assumption: four hours. Three classes enter the window, and
all three rates come from K01.

**A — state event.** 97.22 jobs/s (K01's peak write rate, via the V8 multiplier). Service level:
shows up on the recipient's tracking page, its wait must be short.

**B — billing record.** 833.33 jobs/s (K01's batch job scan rate). Service level: its individual
wait does not matter, but it must finish by the end of the window.

**C — report job.** A hundred reports, each 3,000 records — a size derived from K01 in the
previous lesson. Service level: must finish sometime during the day.

The unit of work is one record processed, so all three classes share the same unit. Their
combined requirement is 951.39 jobs/s. Capacity is an assumption:

**K6 — capacity of the shared consumer group: 900 jobs/s.** This is this course's own assumption
and is not added to K01's table. Rationale: the group was sized to the end-of-day job's
requirement (833.33 records/s) with a small margin, and the event stream and report jobs were
added to it later. Its sensitivity appears at the end of the output.

The three classes fully overlapping is the worst case; separating their windows is a design
decision that removes this lesson's problem, and the measurement shows that decision's value.

Three policies are compared: **single queue**, which takes jobs in arrival order; **strict
priority**, which gives capacity to A first, the remainder to B, and what is left to C; **weighted
share**, which reserves a fraction of capacity for each class (A 0.15, B 0.75, C 0.10) and passes
unused share to the others in order. In the fourth and fifth runs, the previous lesson's spike is
added to A: four times its rate for twenty minutes.

```js
// priority/work-class.mjs — three work classes sharing the same consumer group: single
// queue, strict priority, and weighted share. MODEL: a tick is one second, the work
// unit is one record processed. Results are deterministic (calculation class) and
// machine-independent.
const WINDOW = 4 * 3600;           // K01 assumption V10: end-of-day job window
const A_RATE = (2_800_000 / 86_400) * 3;   // K01 + V8: peak write events/s
const B_RATE = 833.3333;           // K01: batch job scan records/s
const C_COUNT = 100, C_SIZE = 3000;   // report jobs: 3000 records each (lesson 02)
const K6 = 900;                    // this course's assumption K6: shared group capacity (jobs/s)

class Queue {
  constructor() { this.buf = []; this.head = 0; this.backlog = 0; this.arrived = 0; }
  add(t, n) { if (n > 0) { this.buf.push({ t, n }); this.backlog += n; this.arrived += n; } }
  headTime() { return this.head < this.buf.length ? this.buf[this.head].t : Infinity; }
  take(t, s) {
    let remaining = s;
    while (remaining > 1e-9 && this.head < this.buf.length) {
      const batch = this.buf[this.head];
      const taken = Math.min(remaining, batch.n);
      batch.n -= taken; remaining -= taken; this.backlog -= taken;
      this.totalWait = (this.totalWait ?? 0) + taken * (t - batch.t);
      this.completedJobs = (this.completedJobs ?? 0) + taken;
      if (t - batch.t > (this.longestWait ?? 0)) this.longestWait = t - batch.t;
      if (batch.n <= 1e-9) this.head += 1;
    }
    return s - remaining;
  }
}

function run(policy, capacity, aMultiplier) {
  const q = [new Queue(), new Queue(), new Queue()];
  const SHARE = [0.15, 0.75, 0.10];
  for (let t = 0; t < WINDOW; t += 1) {
    const burst = aMultiplier > 1 && t >= 3600 && t < 3600 + 1200;
    q[0].add(t, A_RATE * (burst ? aMultiplier : 1));
    q[1].add(t, B_RATE);
    if (t % Math.floor(WINDOW / C_COUNT) === 0) q[2].add(t, C_SIZE);
    let remaining = capacity;
    if (policy === "single-queue") {
      while (remaining > 1e-9) {
        const i = [0, 1, 2].reduce((a, b) => (q[b].headTime() < q[a].headTime() ? b : a), 0);
        if (q[i].headTime() === Infinity) break;
        remaining -= q[i].take(t, Math.min(remaining, q[i].buf[q[i].head].n));
      }
    } else if (policy === "strict-priority") {
      for (const i of [0, 1, 2]) remaining -= q[i].take(t, remaining);
    } else {
      for (const i of [0, 1, 2]) remaining -= q[i].take(t, Math.min(remaining, capacity * SHARE[i]));
      for (const i of [0, 1, 2]) remaining -= q[i].take(t, remaining);
    }
  }
  return q;
}

const LABEL = ["A state event", "B billing", "C report"];
console.log(`window ${WINDOW} s (V10), capacity K6 = ${K6} jobs/s`);
console.log(`required rate = ${((A_RATE * WINDOW + B_RATE * WINDOW + C_COUNT * C_SIZE) / WINDOW).toFixed(2)} jobs/s\n`);
for (const [label, pol, cap, mult] of [["single queue", "single-queue", K6, 1], ["strict priority", "strict-priority", K6, 1],
  ["weighted share", "share", K6, 1], ["strict priority + A burst", "strict-priority", K6, 4],
  ["weighted share + A burst", "share", K6, 4]]) {
  console.log(`${label} (capacity ${cap})`);
  console.log("  class             arrived  completed  completed/arrived  avg. wait(min)  longest(min)  remaining backlog");
  const queues = run(pol, cap, mult);
  for (const [i, qu] of queues.entries()) {
    const completed = qu.completedJobs ?? 0;
    console.log(`  ${LABEL[i].padEnd(16)} ${qu.arrived.toFixed(0).padStart(8)} ${completed.toFixed(0).padStart(9)} ` +
      `${(completed / qu.arrived).toFixed(3).padStart(13)} ${((qu.totalWait ?? 0) / Math.max(completed, 1) / 60).toFixed(1).padStart(17)} ` +
      `${((qu.longestWait ?? 0) / 60).toFixed(1).padStart(12)} ${Math.max(0, qu.backlog).toFixed(0).padStart(14)}`);
  }
  const total = queues.reduce((a, qu) => a + Math.max(0, qu.backlog), 0);
  console.log(`  total remaining backlog: ${total.toFixed(0)}\n`);
}

const required = (A_RATE * WINDOW + B_RATE * WINDOW + C_COUNT * C_SIZE) / WINDOW;
console.log(`shortfall = (${required.toFixed(2)} - ${K6}) x ${WINDOW} = ` +
  `${((required - K6) * WINDOW).toFixed(0)} jobs; same in every policy`);
console.log(`closing the shortfall: capacity ${K6} -> ${required.toFixed(2)} jobs/s (x${(required / K6).toFixed(3)}) ` +
  `or window 4.00 -> ${(4 * required / K6).toFixed(2)} h (V10)`);
console.log("K6 sensitivity (weighted share, total remaining backlog):");
for (const cap of [850, 900, 950]) {
  const total = run("share", cap, 1).reduce((a, qu) => a + Math.max(0, qu.backlog), 0);
  console.log(`  capacity ${cap} -> ${total.toFixed(0)}`);
}
```

```
window 14400 s (V10), capacity K6 = 900 jobs/s
required rate = 951.39 jobs/s

single queue (capacity 900)
  class             arrived  completed  completed/arrived  avg. wait(min)  longest(min)  remaining backlog
  A state event     1400000   1324264         0.946               6.5         13.0          75736
  B billing        12000000  11350736         0.946               6.5         13.0         649263
  C report           300000    285000         0.950               6.5         12.9          15000
  total remaining backlog: 740000

strict priority (capacity 900)
  class             arrived  completed  completed/arrived  avg. wait(min)  longest(min)  remaining backlog
  A state event     1400000   1400000         1.000               0.0          0.0              0
  B billing        12000000  11560000         0.963               4.4          8.8         440000
  C report           300000         0         0.000               0.0          0.0         300000
  total remaining backlog: 740000

weighted share (capacity 900)
  class             arrived  completed  completed/arrived  avg. wait(min)  longest(min)  remaining backlog
  A state event     1400000   1400000         1.000               0.0          0.0              0
  B billing        12000000  11260000         0.938               7.4         14.8         740000
  C report           300000    300000         1.000               0.3          0.6              0
  total remaining backlog: 740000

strict priority + A burst (capacity 900)
  class             arrived  completed  completed/arrived  avg. wait(min)  longest(min)  remaining backlog
  A state event     1750000   1750000         1.000               0.0          0.0              0
  B billing        12000000  11210000         0.934               9.5         15.8         790000
  C report           300000         0         0.000               0.0          0.0         300000
  total remaining backlog: 1090000

weighted share + A burst (capacity 900)
  class             arrived  completed  completed/arrived  avg. wait(min)  longest(min)  remaining backlog
  A state event     1750000   1750000         1.000               3.5         18.1              0
  B billing        12000000  10910000         0.909              12.0         21.8        1090000
  C report           300000    300000         1.000               0.3          0.6              0
  total remaining backlog: 1090000

shortfall = (951.39 - 900) x 14400 = 740000 jobs; same in every policy
closing the shortfall: capacity 900 -> 951.39 jobs/s (x1.057) or window 4.00 -> 4.23 h (V10)
K6 sensitivity (weighted share, total remaining backlog):
  capacity 850 -> 1460000
  capacity 900 -> 740000
  capacity 950 -> 20000
```

#### Single Queue Is Fair and Wrong

In the first block, all three classes' numbers are nearly identical: completion ratio 0.946 /
0.946 / 0.950, average wait 6.5 minutes for all three, longest wait 13.0 minutes. The single
queue treats them equally.

The cost of that equality is that the classes' service levels are not equal. The state event
shown on the recipient's tracking page waits 6.5 minutes; the billing record, which has until
morning, waits the same 6.5 minutes. The single queue harms no class and helps none; **it is a
policy with no concept of service level.**

#### Strict Priority Produces Starvation

In the second block, A was given absolute priority. The result for A is flawless: all 1,400,000
jobs finish, average and longest wait both 0.0 minutes. B also improves — wait drops from 6.5 to
4.4 minutes, completion rises from 0.946 to 0.963 — because all the capacity A leaves over goes
to it.

For C, the result is different: **zero jobs complete.** None of the hundred reports finish; all
300,000 jobs are still sitting in the queue at the end of the window. This is **starvation**: as
long as the higher classes' combined demand fills capacity, nothing is left for the lower class,
and its wait time cannot be expressed as a number, because the job never starts. Strict priority
is not an ordering — it is a conditional **cancellation**.

#### Weighted Share Redistributes the Shortfall

In the third block, each class received a share. A is again flawless (0.0 minutes wait), because
its reserved share is 135 jobs/s against its need of 97.22. All of C finishes: the whole 300,000
jobs, average wait 0.3 minutes.

B pays the cost: completion falls from 0.963 to 0.938, remaining backlog rises from 440,000 to
740,000. The difference is exactly **300,000** — C's own job volume. The policy change created no
new work; it took C's job at B's expense, and the accounting balances to the penny.

#### A Share Is Both a Floor and a Ceiling

The last two blocks show the same policies under A's spike. Under strict priority, A is again
unaffected: all 1,750,000 jobs, 0.0 minutes wait. Under weighted share, A **waits**: 3.5 minutes
on average, 18.1 minutes at the longest. The reason is direct — during the spike, A's rate rises
to 388.9 jobs/s, while its reserved share is 135 jobs/s. Unused shares are passed to A, but they
are not enough.

**A share is a floor and a ceiling at the same time.** The mechanism that saves C from starvation
is the same one that limits A during the spike. The choice of policy therefore does not answer
"which one is better" but "what do we want to happen on each class's bad day." Strict priority
makes the other classes pay for the top class's bad day; weighted share limits every class's bad
day to its own share.

#### Back to the Estimate

The last three lines mark this lesson's boundary. The three classes' combined requirement is
951.39 jobs/s, capacity is 900; the shortfall over the four-hour window is 740,000 jobs. **This
number is the same in all five runs.** In the single queue it is 75,736 + 649,263 + 15,000; under
strict priority, 0 + 440,000 + 300,000; under weighted share, 740,000 + 0 + 0. The policy chooses
who the shortfall is charged to; it does not shrink the shortfall. In the runs with the spike, the
total rises to 1,090,000, and the difference is exactly the 350,000 jobs that were added.

Two ways close the shortfall, both outside the queue: raising capacity from
900 to 951.39 jobs/s (1.057 times), or extending the V10 window from 4.00 hours to 4.23 hours.
K6's sensitivity confirms this: at capacity 850, the total remaining is 1,460,000; at 900, it is
740,000; at 950, it is 20,000. A six-percent increase in capacity erases 97 percent of the
shortfall.

The result that flows back to K01 is this: V10's four-hour window holds when the end-of-day job
runs **alone**. Once another stream enters the same group, the window is no longer a constraint
but a shared resource, and whether V10 is met can no longer be judged from B's rate alone.

#### Summary

- A priority queue is a service-level decision; the priority queue in the Data Structures
  curriculum is a data structure, and it is not covered in this lesson.
- In the single queue, all three classes get the same numbers — completion 0.946 / 0.946 / 0.950,
  wait 6.5 minutes — so there is no concept of service level.
- Strict priority serves the top class flawlessly (0.0 minutes wait) and produces starvation in
  the bottom class: zero of a hundred reports finished, 300,000 jobs stayed in the queue.
- Weighted share finishes all of C and B pays the cost to the penny: B's remaining backlog rises
  from 440,000 to 740,000, an increase of exactly 300,000.
- A share is both a floor and a ceiling: under A's spike, strict priority never makes A wait,
  while weighted share makes it wait 3.5 minutes on average and 18.1 minutes at the longest.
- The total shortfall is independent of the policy: 740,000 jobs in three of the five runs;
  closing it means raising capacity by a factor of 1.057 or extending the V10 window to 4.23 hours.

#### Next Step

This lesson treated order as a **preference**: a priority queue chose which order jobs were taken
in, letting the fast one go ahead of the slow one. A job's wait here was a service-level decision,
not a correctness problem. For some jobs, though, order is not something
that can be chosen. The same shipment's state events overwrite each other; if "left the transfer
hub" and "delivered" are processed in reverse order, the shipment appears back in transit after it
has already been delivered, and the result is corrupted — this is why the order of 2,279
shipments broke across eight consumers in the third lesson. For jobs like these, order is not a
preference but a **constraint**. The next lesson takes up jobs whose order must be preserved, and
examines how that constraint can be kept alongside concurrency.

### Sequential Convoy

Source: https://academia.sh/en/courses/application-layer/sequential-convoy

Managing jobs whose order is a constraint, not a preference: measuring free consumption, sequential convoy — which routes the same shipment's state events into a single lane per key — and a version-stamped handler on the same event set; separating backward writes, wrong final states, and steps missing from history; lane granularity setting the throughput ceiling, and applying the cost to K01's peak write rate.

The previous lesson treated order as a **preference**: a priority queue chose which order to take
jobs in, letting the fast one go ahead of the slow one. A job's wait there was a service-level
decision, not a correctness problem. For some jobs, though, order is not
something that can be chosen. The same shipment's state events — accepted, transferred, out for
delivery, delivered — overwrite each other; if they are processed in reverse order, the shipment
appears back in transfer after it has already been delivered.

**Sequential convoy** is an arrangement that routes jobs carrying an order constraint into a
single lane by key, while preserving concurrency across lanes. Queue mechanics, delivery
guarantees, and counting order violations by inverted pairs were established in the Caching,
Queues and Asynchronous Processing course; delivery guarantees are not discussed here.
This lesson's question is a capacity question: how much concurrency does preserving order give
back, and is there a way to escape this constraint.

#### The Event That Carries an Order Constraint

Not every event belonging to a shipment demands order. The information "arrived at the zone" is
a state transition; a "passed through the scanner" record does not overwrite the state, it is
only appended to the history. The criterion for the distinction fits in one sentence: **if an
event erases a field's previous value, it carries an order constraint.**

The model builds this distinction with a state machine. Four steps are ordered, and each event
carries a step number. Three arrangements are run on the same event set: the **free** arrangement,
which distributes events with no constraint; the **convoy** arrangement, which opens a
lane per key; and the **version-stamp** arrangement, which releases events freely and checks the
step number in the handler. A fourth column shifts the convoy's key from shipment to line.

```js
// convoy/event.mjs — a deterministic set of state events and the in-process model of three
// arrangements. A tick is an abstract step; duration is a model parameter, not a measured
// time.
export const STEP = { accepted: 1, transferred: 2, outForDelivery: 3, delivered: 4 };
export const EVENTS = [           // [tracking no, line, state, duration]
  ["TR-4821", "L1", "accepted", 1], ["TR-9007", "L1", "accepted", 3],
  ["TR-4821", "L1", "transferred", 4], ["TR-5533", "L2", "accepted", 1],
  ["TR-4821", "L1", "outForDelivery", 1], ["TR-9007", "L1", "transferred", 1],
  ["TR-2214", "L2", "accepted", 2], ["TR-4821", "L1", "delivered", 1],
  ["TR-5533", "L2", "transferred", 3], ["TR-9007", "L1", "outForDelivery", 1],
  ["TR-7788", "L3", "accepted", 1], ["TR-5533", "L2", "outForDelivery", 1],
  ["TR-2214", "L2", "transferred", 4], ["TR-9007", "L1", "delivered", 1],
  ["TR-5533", "L2", "delivered", 1], ["TR-7788", "L3", "transferred", 2],
  ["TR-2214", "L2", "outForDelivery", 1], ["TR-7788", "L3", "outForDelivery", 1],
  ["TR-2214", "L2", "delivered", 1], ["TR-7788", "L3", "delivered", 1],
];

// Events are distributed to consumers; when a consumer frees up, it takes the next job.
// laneOf: the lane name by the convoy key, null if there is no order constraint.
function run(events, consumers, laneOf) {
  const free = new Array(consumers).fill(0);   // the tick each consumer will be free at
  const laneFree = new Map();                  // the tick each lane will be free at
  return events.map((e) => {
    const c = free.indexOf(Math.min(...free));
    const lane = laneOf ? laneOf(e) : null;
    const start = Math.max(free[c], lane === null ? 0 : (laneFree.get(lane) ?? 0));
    const finish = start + e.duration;
    free[c] = finish;
    if (lane !== null) laneFree.set(lane, finish);
    return { ...e, consumer: c, finish, lane };
  });
}

// Events applied in finish order: last writer wins, or a version stamp.
function apply(finished, stamped) {
  const state = new Map(), s = { backwardWrite: 0, skipped: 0 };
  for (const e of [...finished].sort((a, b) => a.finish - b.finish || a.order - b.order)) {
    const current = state.get(e.trackingNo) ?? 0;
    if (STEP[e.state] < current) {
      s.backwardWrite += 1;
      if (stamped) { s.skipped += 1; continue; }   // the stamp rejects backward writes
    }
    state.set(e.trackingNo, STEP[e.state]);
  }
  const wrong = [...state.entries()].filter(([, a]) => a !== STEP.delivered).map(([n]) => n);
  return { ...s, wrongFinalState: wrong.length, wrong, tick: Math.max(...finished.map((e) => e.finish)) };
}

export function arrangement(name, consumers) {
  const events = EVENTS.map(([trackingNo, line, state, duration], order) => ({ trackingNo, line, state, duration, order }));
  const LANE = { free: null, convoy: (e) => e.trackingNo, "convoy-line": (e) => e.line,
    "version-stamp": null };
  const finished = run(events, consumers, LANE[name]);
  const r = apply(finished, name === "version-stamp");
  const load = new Map();
  for (const e of finished) if (e.lane !== null) load.set(e.lane, (load.get(e.lane) ?? 0) + e.duration);
  return { ...r, lanes: load.size, busiestLane: load.size ? Math.max(...load.values()) : 0,
    workload: EVENTS.reduce((t, e) => t + e[3], 0) };
}
```

```js
// convoy/measure.mjs — four arrangements on the same event set: order breakage, ticks, and lane load
import { arrangement, EVENTS } from "./event.mjs";

const CONSUMERS = 4;
const FIELD = [["backwardWrite", "backward write"], ["wrongFinalState", "wrong final state"],
  ["skipped", "skipped event"], ["tick", "tick"], ["lanes", "lane"], ["busiestLane", "busiest lane"]];
const D = ["free", "convoy", "convoy-line", "version-stamp"].map((a) => [a, arrangement(a, CONSUMERS)]);

console.log(`${EVENTS.length} state events, ${new Set(EVENTS.map((e) => e[0])).size} shipments, ` +
  `${CONSUMERS} consumers, total workload ${D[0][1].workload} ticks`);
console.log();
console.log(`${"measure".padEnd(18)}${D.map(([a]) => a.padStart(15)).join("")}`);
for (const [k, label] of FIELD)
  console.log(`${label.padEnd(18)}${D.map(([, r]) => String(r[k]).padStart(15)).join("")}`);

const [free, convoy, line, stamp] = D.map(([, r]) => r);
console.log(`\nshipments with a wrong final state under free = ${free.wrong.join(", ")}`);
console.log(`convoy / free tick ratio = ${(convoy.tick / free.tick).toFixed(3)}, ` +
  `line convoy / free = ${(line.tick / free.tick).toFixed(3)}`);
console.log(`effective concurrency = workload / ticks: free ${(free.workload / free.tick).toFixed(2)}, ` +
  `convoy ${(convoy.workload / convoy.tick).toFixed(2)}, line convoy ${(line.workload / line.tick).toFixed(2)}`);
console.log(`steps missing from state history under the stamp arrangement = ${stamp.skipped}`);

console.log(`\n${"consumers".padEnd(10)}${["free", "convoy", "convoy-line"].map((a) => a.padStart(12)).join("")}`);
for (const c of [1, 2, 4, 8]) {
  const row = ["free", "convoy", "convoy-line"].map((a) => String(arrangement(a, c).tick).padStart(12));
  console.log(`${String(c).padEnd(10)}${row.join("")}`);
}
```

```
20 state events, 5 shipments, 4 consumers, total workload 32 ticks

measure                      free         convoy    convoy-line  version-stamp
backward write                  5              0              0              5
wrong final state               1              0              0              0
skipped event                   0              0              0              5
tick                            8             13             19              8
lane                            0              5              3              0
busiest lane                    0              8             14              0

shipments with a wrong final state under free = TR-4821
convoy / free tick ratio = 1.625, line convoy / free = 2.375
effective concurrency = workload / ticks: free 4.00, convoy 2.46, line convoy 1.68
steps missing from state history under the stamp arrangement = 5

consumers         free      convoy convoy-line
1                   32          32          32
2                   16          17          23
4                    8          13          19
8                    5           9          14
```

#### Reading the Numbers

The first two rows separate two numbers that should not be confused. Under the
free arrangement, 5 of the 20 events overwrote a step ahead of themselves, but only **one**
shipment ended up in a wrong final state. The difference is that most of the backward writes
close themselves: a correct event arriving later erases the wrong value. Only the last backward
write applied stays broken. The measurement names it directly: the broken shipment is `TR-4821`.
A mechanism that reports order breakage only by violation count shows damage five times larger
than reality; a mechanism that checks only the final state misses that the problem happened five
times.

The convoy column zeroes both numbers and collects the cost in one place: tick count rises from 8
to 13, effective concurrency falls from 4.00 to 2.46. The ratio is 1.625 — the order constraint
takes back roughly two-fifths of the concurrency. This is the pattern's direct cost, and it is
not unavoidable — it **depends on the key**.

The fourth column shows a way to escape the constraint. The version-stamped handler releases
events freely, and tick count stays at 8, concurrency at 4.00; backward writes are still 5, but
because the handler rejects them, wrong final state is 0. The cost shows up in a new row: 5
skipped events. These shipments' state history is missing five
steps — the final state is correct, the path is not. The two arrangements buy the same correctness in different
currencies: the convoy pays in throughput, the version stamp pays in history.

#### Lane Granularity Sets the Ceiling

The difference between the third and fourth columns is only the key. When the convoy key is the
tracking number, 5 lanes open and the busiest lane carries 8 ticks of work; when the key is the
line, lanes drop to 3 and the busiest lane carries 14 ticks. Tick count rises from 13 to 19,
concurrency falls from 2.46 to 1.68. A coarse key puts unrelated shipments in the same lane:
shipment `TR-2214`'s out-for-delivery event waits behind shipment `TR-5533`'s four-tick transfer
event. There is no order relationship between them.

The table below shows why this is a capacity question. As consumer count rises from 1 to 8, the
free arrangement falls from 32 ticks to 5. The convoy stalls at 9 ticks, the line convoy at 14 —
both at the busiest lane's load. **Lane load is the ceiling on throughput, and adding consumers
does not raise that ceiling.** The real question when designing a sequential convoy is not
consumer count, but what the busiest lane carries.

#### Back to the Estimate

The model's 20 events are not a volume figure. What is measured is a ratio, applied to K01's peak write rate: 97.22 state events per
second, a **computed value** from the Back-of-the-Envelope Estimation lesson. This lesson adds
one assumption:

**KK1 — share of state events carrying an order constraint: 4/7.** Its rationale is written in
K01's V4 line: seven events per shipment were justified as "acceptance, transfer, out for
delivery, delivery, and intermediate states"; four are named state transitions, three are
intermediate records. This number is not added to K01's assumption table; its sensitivity is
calculated below with 5/7.

```js
// convoy/cost.mjs — applies the model's concurrency cost to K01's peak write rate
import { arrangement } from "./event.mjs";

const PEAK_WRITE = 97.22;          // K01 Back-of-the-Envelope Estimation, calculation class
const C = 4;
const concurrency = (a) => arrangement(a, C).workload / arrangement(a, C).tick;
const cost = { convoy: concurrency("free") / concurrency("convoy"),
  "convoy-line": concurrency("free") / concurrency("convoy-line") };

console.log(`${"KK1".padStart(5)}${"constrained ev/s".padStart(18)}${"unconstrained ev/s".padStart(20)}` +
  `${"convoy x".padStart(11)}${"line convoy x".padStart(15)}`);
for (const KK1 of [4 / 7, 5 / 7]) {          // KK1: share of state events carrying an order constraint
  const constrained = PEAK_WRITE * KK1, unconstrained = PEAK_WRITE - constrained;
  const selective = (name) => KK1 * cost[name] + (1 - KK1);
  console.log(`${KK1.toFixed(3).padStart(5)}${constrained.toFixed(2).padStart(18)}` +
    `${unconstrained.toFixed(2).padStart(20)}${selective("convoy").toFixed(3).padStart(11)}` +
    `${selective("convoy-line").toFixed(3).padStart(15)}`);
}
console.log(`\nif the entire stream is routed through the convoy: convoy x${cost.convoy.toFixed(3)}, ` +
  `line convoy x${cost["convoy-line"].toFixed(3)}`);
console.log(`lane load ceiling (8 consumers): convoy ${arrangement("convoy", 8).tick} ticks, ` +
  `line convoy ${arrangement("convoy-line", 8).tick} ticks, free ${arrangement("free", 8).tick} ticks`);
```

```
  KK1  constrained ev/s  unconstrained ev/s   convoy x  line convoy x
0.571             55.55               41.67      1.357          1.786
0.714             69.44               27.78      1.446          1.982

if the entire stream is routed through the convoy: convoy x1.625, line convoy x2.375
lane load ceiling (8 consumers): convoy 9 ticks, line convoy 14 ticks, free 5 ticks
```

55.55 events/s of the peak write rate carries an order constraint, 41.67 events/s does not. If
the entire write stream is routed through the convoy, the consumer count required for the same
completion rate rises to 1.625 times. Applying the constraint only to the events that carry it —
a **selective convoy** — brings the multiplier down to 1.357; with the line key it is 1.786
instead of 2.375. If KK1 rises to 5/7, the multipliers climb to 1.446 and 1.982: the selective
convoy's gain tracks the share of events without an order constraint, and shrinks as that share
shrinks.

The decision is among three options, all measured on the same event set. The
version-stamped handler gives up nothing in throughput but leaves a gap in the state history;
because the carrier's history is used in contract disputes, that gap is not free. The
shipment-keyed selective convoy keeps the history complete and pays a 1.357 multiplier. The
line-keyed convoy adds no extra correctness — it only reduces the lane count — and raises the
multiplier to 1.786: a coarse key is a cost with nothing to show for it.

#### Summary

- Order is a constraint, not a preference: an event that erases a field's previous value must
  keep its order; four of the seven events per shipment fall in this class.
- Backward writes and wrong final states are separate numbers: free produced 5 backward writes,
  but only 1 shipment ended up in a wrong final state, because later correct events closed most
  of them.
- Sequential convoy zeroes both numbers and takes the cost out of throughput: ticks rise from 8 to
  13, effective concurrency falls from 4.00 to 2.46 (a ratio of 1.625).
- The version-stamped handler holds concurrency at 4.00 and zeroes wrong final states, but skips
  5 events; the state history is missing five steps.
- Lane load is the ceiling on throughput: with eight consumers, the free arrangement falls to 5
  ticks while the convoy stalls at 9 and the line convoy at 14 — both at the busiest lane's load.
- Back to K01: 55.55 events/s of the peak write rate is constrained; the selective convoy raises
  the consumer requirement by 1.357 times, by 1.625 if the entire stream is routed through the
  convoy, and by 1.786 with the coarse key.

#### Next Step

Every event in this lesson was single-step: a state is written and the job is done. End-of-day
billing is not like that. For a seller's one day, a tariff is calculated, a discount is applied,
an invoice line is produced, and a notification is sent; four steps wait on each other, and each
step lives in a separate service. A job like this cannot have its order preserved by a key the
way it was here, because the problem is not two writes to the same key — it is **a single job
running in four separate places**. The next lesson builds this job in two forms — an arrangement
where steps trigger each other with events, and an arrangement where a single orchestrator calls
the steps in sequence — and measures both in terms of step count, message count, compensated
steps, and unfinished jobs.

### Choreography and Orchestration

Source: https://academia.sh/en/courses/application-layer/choreography-and-orchestration

Building the end-of-day billing job — four steps spread across four services — in two models: choreography, where steps trigger each other with events, and orchestration, where a single orchestrator calls the steps; showing that the shared transaction boundary is lost, building the compensating step, and measuring both modes in messages, compensated steps, uncompensated steps, and half-finished jobs.

Every event in the previous lesson was single-step: a state is written and the job is done.
End-of-day billing is not like that. For a seller's one day, a tariff is calculated, a discount is
applied, an invoice line is produced, and a notification is sent. Four steps wait on each other,
and each step runs in a separate service — tariff and invoice in the billing context, discount in
the contract context, notification on the delivery operations side.

There are two models for building this job. In the **choreography** arrangement, there is no
orchestrator in the middle: when a step finishes, it publishes an event, and the next step's
service is subscribed to that event. In the **orchestration** arrangement, a single orchestrator
calls the steps in sequence and keeps track of where the job stands itself. Choreography is the
workflow form of the event-driven architecture covered in the Architectural Styles course; the
style itself and the publish-subscribe mechanics were established there and in the Caching,
Queues and Asynchronous Processing course, and are not repeated here. What is measured is the
numbers the two modes leave behind when they run the same job through the same failures.

#### No Shared Transaction Boundary

The Data Access Layer and Business Logic course established the unit of work and the transaction
boundary: the writes inside one boundary either commit together or none of them commit. When four
steps run in four separate services, this foundation is gone. Each service opens and closes its
own boundary; when the third step fails, the writes from the first and second steps **have
already committed**, and there is no call that rolls them back together.

What replaces it is a **compensating step**: a second step is defined for each step that undoes
its effect in business terms, and when an error occurs, the completed steps are compensated in
reverse order. Compensation is not rollback; the write stays visible for a while, then is closed
with an opposite write. The fourth step — notification — cannot be compensated, because a
notification that has been sent cannot be taken back; that is why it is placed at the end of the
sequence. This ordering rule is the same for both modes, and in the model it is written as the
`compensation` field being `null`.

```js
// workflow/job.mjs — the in-process model of the end-of-day billing workflow. Four steps
// run in four separate services; each service closes its own transaction boundary, and
// there is no shared boundary.
export const STEP = [
  { name: "tariff", service: "billing", compensation: "tariff-cancel" },
  { name: "discount", service: "contract", compensation: "discount-cancel" },
  { name: "invoice", service: "invoicing", compensation: "invoice-cancel" },
  { name: "notification", service: "notification", compensation: null },   // cannot be undone, so it is last
];
export const JOBS = ["S1", "S2", "S3", "S4", "S5", "S6", "S7", "S8"];
export const FAILURE = { S3: "invoice", S7: "discount" };      // deterministic failure points

// mode: "choreography" triggers steps with events, "orchestration" calls them with a single
// orchestrator. missingSubscriber: a service in choreography that has not subscribed to the
// cancellation event (null means the chain is complete).
export function run({ mode, missingSubscriber = null }) {
  const s = { completed: 0, steps: 0, messages: 0, compensatedSteps: 0, uncompensatedSteps: 0, halfFinishedJobs: 0 };
  const committed = new Map(STEP.map((a) => [a.name, new Set()]));   // committed writes
  const trace = [];

  for (const job of JOBS) {
    if (mode === "choreography") s.messages += 1;                  // the event that starts the workflow
    const done = [];
    let failure = null;
    for (const a of STEP) {
      if (mode === "orchestration") s.messages += 1;                // orchestrator -> service command
      if (FAILURE[job] === a.name) { failure = a.name; s.messages += 1; break; }   // error reply or event
      s.steps += 1;
      committed.get(a.name).add(job);                               // step committed in its own boundary
      done.push(a);
      s.messages += 1;                 // choreography: the event the step publishes; orchestration: the reply
    }
    if (failure === null) { s.completed += 1; continue; }
    let open = 0;
    for (const a of [...done].reverse()) {
      if (a.compensation === null) continue;
      const subscribed = mode === "orchestration" || a.service !== missingSubscriber;
      s.messages += mode === "orchestration" ? 2 : (subscribed ? 1 : 0);  // compensation command+reply / cancel event
      if (subscribed) { committed.get(a.name).delete(job); s.compensatedSteps += 1; }
      else { s.uncompensatedSteps += 1; open += 1; }                 // no one listened, the write stayed
    }
    if (open > 0) s.halfFinishedJobs += 1;
    trace.push(`${job}: stopped at the ${failure} step, committed steps ${done.length}, left open ${open}`);
  }
  return { ...s, trace, remainingWrites: STEP.map((a) => `${a.name}=${committed.get(a.name).size}`).join(" ") };
}

// Structural measures: fixed once the mode is chosen, independent of the run.
export function structure(mode) {
  const n = STEP.length, compensable = STEP.filter((a) => a.compensation !== null).length;
  const o = mode === "orchestration";
  return {
    "units carrying the workflow definition": o ? 1 : n,
    "units holding workflow state": o ? 1 : 0,
    "units queried to read status": o ? 1 : n,
    "units responsible for compensation": o ? 1 : compensable,
    "units changed when a step is appended": o ? 2 : 1,
    "transaction boundaries": n,
    "atomically reversible steps": 0,
  };
}
```

The third run tests a question: in choreography, compensation is also triggered by an event,
meaning every service must be subscribed to the cancellation event. `missingSubscriber` produces
the case where one service has not set up that subscription.

```js
// workflow/measure.mjs — two modes on the same job set: steps, messages, compensation, half-finished jobs, and structure
import { run, structure, JOBS, STEP, FAILURE } from "./job.mjs";

const K = [["choreography", run({ mode: "choreography" })],
  ["chor+missing sub", run({ mode: "choreography", missingSubscriber: "billing" })],
  ["orchestration", run({ mode: "orchestration" })]];
const MEASURE = [["completed", "completed job"], ["steps", "steps run"], ["messages", "message"],
  ["compensatedSteps", "compensated step"], ["uncompensatedSteps", "uncompensated step"], ["halfFinishedJobs", "half-finished job"]];

console.log(`${JOBS.length} seller-days, ${STEP.length} steps, ` +
  `${Object.keys(FAILURE).length} jobs fail: ${Object.entries(FAILURE).map(([j, a]) => `${j}@${a}`).join(", ")}`);
console.log();
console.log(`${"measure".padEnd(22)}${K.map(([a]) => a.padStart(18)).join("")}`);
for (const [k, label] of MEASURE)
  console.log(`${label.padEnd(22)}${K.map(([, r]) => String(r[k]).padStart(18)).join("")}`);

console.log(`\nfailure trace (same in all three runs):`);
for (const line of K[2][1].trace) console.log(`  ${line}`);
console.log(`committed remaining writes:`);
for (const [label, r] of K) console.log(`  ${label.padEnd(18)}${r.remainingWrites}`);

const STRUCT = Object.keys(structure("orchestration"));
console.log(`\n${"structural measure".padEnd(40)}${"choreography".padStart(13)}${"orchestration".padStart(14)}`);
for (const a of STRUCT)
  console.log(`${a.padEnd(40)}${String(structure("choreography")[a]).padStart(13)}` +
    `${String(structure("orchestration")[a]).padStart(14)}`);

const [chor, , orch] = K.map(([, r]) => r);
console.log(`\nmessage ratio orchestration/choreography = ${(orch.messages / chor.messages).toFixed(3)}`);
console.log(`messages per job: choreography ${(chor.messages / JOBS.length).toFixed(2)}, ` +
  `orchestration ${(orch.messages / JOBS.length).toFixed(2)}`);
```

```
8 seller-days, 4 steps, 2 jobs fail: S3@invoice, S7@discount

measure                     choreography  chor+missing sub     orchestration
completed job                          6                 6                 6
steps run                             27                27                27
message                               40                38                64
compensated step                       3                 1                 3
uncompensated step                     0                 2                 0
half-finished job                      0                 2                 0

failure trace (same in all three runs):
  S3: stopped at the invoice step, committed steps 2, left open 0
  S7: stopped at the discount step, committed steps 1, left open 0
committed remaining writes:
  choreography      tariff=6 discount=6 invoice=6 notification=6
  chor+missing sub  tariff=8 discount=6 invoice=6 notification=6
  orchestration     tariff=6 discount=6 invoice=6 notification=6

structural measure                       choreography orchestration
units carrying the workflow definition              4             1
units holding workflow state                        0             1
units queried to read status                        4             1
units responsible for compensation                  3             1
units changed when a step is appended               1             2
transaction boundaries                              4             4
atomically reversible steps                         0             0

message ratio orchestration/choreography = 1.600
messages per job: choreography 5.00, orchestration 8.00
```

#### Reading the Numbers

Both modes do the same job: 6 of 8 seller-days completed, 27 steps ran, compensated steps 3 in
both. There is no difference in the job outcome. The difference collects in two other places.

The first is message count: choreography 40, orchestration 64 — a ratio of 1.600. Five per job
against eight. The reason is structural: in choreography, a completed step publishes a single
event and the next step picks it up itself; in orchestration, every step needs a command and a
reply, and so does compensation. The orchestrator's knowledge of where the job stands is not
free; it charges itself for the messages that carry it.

The second is in the third column, and this is where the real decision sits. In choreography,
compensation is also triggered by an event; when the billing service is not subscribed to the
cancellation event, compensated steps fall from 3 to 1, uncompensated steps rise from 0 to 2, and
2 jobs are left half-finished. The committed write in the `tariff` row rises from 6 to 8: the
tariff calculation for two sellers keeps sitting in the system, even though their invoices were
never produced. These two writes produce no error message; they only show up when month-end
reconciliation does not balance.

The missed subscription has no counterpart in orchestration, because the compensation list lives
in the orchestrator's code. The structural table names this: compensation responsibility is
spread across 3 units in choreography, gathered into 1 unit in orchestration. The same pattern
holds for the workflow definition and its state. In choreography, the workflow's definition is
not written down as a whole anywhere; reading it means looking at four services' subscription
lists, and you cannot ask a single unit "where is this job now."

One row in the table runs the other way: when a step is appended at the end, 1 unit changes in
choreography (the new service subscribes to the last event), 2 in orchestration (the new service
and the orchestrator). Choreography's gain is concrete here, and it must be read together with
the 1.600 multiplier in messages.

#### Back to the Estimate

The model's 8 jobs and 2 failures are not a volume; what is measured is a ratio per job. Volume
comes from K01: 4,000 invoice lines a day is a **computed value**, and since each line is one
seller's one day, 4,000 workflows run per day; the end-of-day window of 4 hours is K01's V10
**assumption**.

**KK2 — end-of-day workflow failure rate: 0.02.** Its rationale is missing contract records,
tariff changes that arrive after the window has closed, and shipments not yet closed producing a
steady baseline. The model's 2/8 ratio is not realistic, and it is not
presented as such or added to K01's table. Its sensitivity is calculated together with 0.04.

```js
// workflow/cost.mjs — applies the model's ratios to K01's end-of-day job volume
import { run, structure, JOBS, STEP } from "./job.mjs";

const INVOICE_LINES = 4000;      // K01: daily invoice lines (computed value) — one workflow per seller
const WINDOW = 4 * 3600;         // K01 V10: end-of-day window 4 hours (assumption)
const chor = run({ mode: "choreography" }), orch = run({ mode: "orchestration" });
const gap = run({ mode: "choreography", missingSubscriber: "billing" });
const messagesPerJob = { choreography: chor.messages / JOBS.length, orchestration: orch.messages / JOBS.length };
const failingJobs = 2;                                       // number of jobs that fail in the model
const compensatedPerFailure = chor.compensatedSteps / failingJobs;
const uncompensatedPerFailure = gap.uncompensatedSteps / failingJobs;

console.log(`workflow ${INVOICE_LINES}/day, window ${WINDOW} s -> ` +
  `${(INVOICE_LINES / WINDOW).toFixed(4)} jobs/s, ${(INVOICE_LINES * STEP.length / WINDOW).toFixed(4)} steps/s`);
for (const [label, n] of Object.entries(messagesPerJob))
  console.log(`  ${label.padEnd(13)} ${n.toFixed(2)} messages/job -> ${(INVOICE_LINES * n / WINDOW).toFixed(4)} messages/s`);

console.log(`\n${"KK2".padStart(5)}${"failed jobs/day".padStart(17)}${"comp steps/day".padStart(16)}` +
  `${"uncomp steps/day".padStart(18)}${"half-finished/day".padStart(20)}`);
for (const KK2 of [0.02, 0.04]) {                          // KK2: end-of-day workflow failure rate
  const failed = INVOICE_LINES * KK2;
  console.log(`${KK2.toFixed(2).padStart(5)}${failed.toFixed(0).padStart(17)}` +
    `${(failed * compensatedPerFailure).toFixed(0).padStart(16)}` +
    `${(failed * uncompensatedPerFailure).toFixed(0).padStart(18)}${failed.toFixed(0).padStart(20)}`);
}
console.log(`\nratios from the model: compensated steps per failure = ${compensatedPerFailure.toFixed(2)}, ` +
  `uncompensated remaining with a missing subscriber = ${uncompensatedPerFailure.toFixed(2)}`);
console.log(`comparison with K01's peak write of 97.22 req/s: orchestration message flow ` +
  `${(100 * (INVOICE_LINES * messagesPerJob.orchestration / WINDOW) / 97.22).toFixed(2)}%`);
console.log(`units to open to read the workflow definition: choreography ` +
  `${structure("choreography")["units carrying the workflow definition"]}, orchestration ` +
  `${structure("orchestration")["units carrying the workflow definition"]}`);
```

```
workflow 4000/day, window 14400 s -> 0.2778 jobs/s, 1.1111 steps/s
  choreography  5.00 messages/job -> 1.3889 messages/s
  orchestration 8.00 messages/job -> 2.2222 messages/s

  KK2  failed jobs/day  comp steps/day  uncomp steps/day   half-finished/day
 0.02               80             120                80                  80
 0.04              160             240               160                 160

ratios from the model: compensated steps per failure = 1.50, uncompensated remaining with a missing subscriber = 1.00
comparison with K01's peak write of 97.22 req/s: orchestration message flow 2.29%
units to open to read the workflow definition: choreography 4, orchestration 1
```

The message difference shrinks when it is scaled up. Over the four-hour window, workflow rate is
0.2778 per second, step rate is 1.1111; choreography produces 1.3889 messages per second,
orchestration 2.2222. Orchestration's excess is 0.83 messages per second, about 2.29 percent of
K01's peak write flow. The 1.600 multiplier in message count is not a capacity problem system-wide.

The other column behaves the opposite way under the same scaling. With KK2 = 0.02, 80 workflows
fail per day; at the model's ratio, 120 compensating steps run per day. A single missing
subscription in the compensation chain means 80 uncompensated steps and 80 half-finished jobs per
day. If KK2 doubles, it becomes 160. These numbers produce no outage and raise no error counter;
they only surface at month-end reconciliation. The order-of-magnitude gap between the two
measured costs settles the decision: choreography's message gain is about two percent of the peak
flow, while the cost of one gap in the compensation chain is 80 wrong records a day. For workflows
with more than three steps that need compensation, orchestration is the defensible choice, because
it gathers compensation responsibility into a single unit; for notification-style flows with no
compensation, expected to grow steps that know nothing of each other, choreography is the right
fit.

#### Summary

- When four steps run in four services, there is no shared transaction boundary: transaction
  boundaries number 4, atomically reversible steps 0; when an error occurs, the earlier steps
  stay committed.
- A compensating step replaces it; the step that cannot be compensated (notification) is placed
  at the end of the sequence.
- Both modes produce the same result — 6 completed jobs, 27 steps, 3 compensated — but
  choreography spends 40 messages against orchestration's 64 (a ratio of 1.600; 5 per job
  against 8).
- In choreography, compensation responsibility is spread across 3 units: a single missing
  subscription dropped compensated steps from 3 to 1, left 2 steps uncompensated, and raised
  `tariff` writes from 6 to 8.
- The structural difference collects in the workflow's definition: in choreography, the
  definition is spread across 4 units and the unit holding state is 0; in orchestration, both
  are 1. The one row that runs the other way is extension — appending a step changes 1 unit in
  choreography, 2 in orchestration.
- Back to K01: 4,000 workflows and a 4-hour window give 0.2778 jobs/s, 1.1111 steps/s; the
  message gap is 0.83 per second (2.29 percent of the peak write), while a missing subscription
  with KK2 = 0.02 leaves 80 uncompensated steps and 80 half-finished jobs per day.

#### Next Step

Every failure in this lesson announced itself clearly: a step returned an error, and the
compensation chain either worked or did not. Most failures in a real workflow do not speak up like
that. The orchestrator crashes after sending a command, a step hangs without replying, a
compensation call is tried once and dropped. If no one asks about such a job's status, it appears
to be "running" forever, and the window closes. The next lesson builds the arrangement that closes
this gap: a scheduler that durably records the job and starts its steps, an agent that runs a step
remotely, and a supervisor that finds steps whose time has expired and either re-drives or
compensates them — and it measures the trade-off between scan frequency, recovery delay, and the
number of records the supervisor reads.

### Scheduler–Agent–Supervisor

Source: https://academia.sh/en/courses/application-layer/scheduler-agent-supervisor

Making resilient a workflow whose half-finished job no one asks about: a scheduler that durably records the job, an agent that runs the step remotely, and a supervisor that finds an expired step and either re-drives or compensates it; measuring half-finished jobs, re-driven steps, and wasted steps in unsupervised and supervised runs, and the trade-off between the scan interval and recovery delay.

Every failure in the previous lesson announced itself clearly: a step returned an error, and the
compensation chain either worked or did not. Most failures in a real workflow do not speak up
like that. The orchestrator crashes after sending a command, a step stalls without replying, a
compensation call is tried once and dropped. Such a job appears to be "running" forever, and
because no one asks, no one notices.

**Scheduler–agent–supervisor** closes this gap with three roles. The **scheduler** writes the
workflow's state to a durable record and starts the next step; where the job stands lives not in
memory but in a record that can still be read after the process crashes. The **agent** runs the
step on the remote service and writes the result to the record. The **supervisor** scans the
record at regular intervals; it finds steps that were started but have not recorded a result
within the allotted time, re-drives them, and starts compensation once the retry threshold is
exceeded. The time allotted to each started step is called the **lease**.

#### Why the Three Roles Are Separated

The rationale for the separation fits in one sentence: **no role can notice its own failure.**
When the agent crashes, there is no party left to say "I crashed"; when the scheduler crashes, no
one is left to wait for the result of the steps it started. Noticing is handed to a third party
that does no work itself and looks only at the durable record.

The model separates two kinds of failure. **Stall**: the agent has run the job but failed to
record the result — the call went unanswered, or the process stopped before recording.
**Persistent failure**: the step never passes on any attempt, because the contract record is
missing or the tariff is undefined. The two look identical — the record has not landed — and the
supervisor can only tell them apart by retrying.

```js
// supervisor/run.mjs — scheduler, agent, and supervisor; a round-based in-process model. A round
// is an abstract step. Stall and persistent failure are triggered by parameters, not measured durations.
export const STEP = ["tariff", "discount", "invoice", "notification"];
export const JOBS = Array.from({ length: 12 }, (_, i) => `S${i + 1}`);
export const STALLED = { S2: "discount", S5: "tariff", S9: "invoice" };  // agent did the work, record did not land
export const PERSISTENT = { S7: "discount" };                          // step never passes on any attempt

export function run({ supervisor = true, scan = 2, lease = 2, retryLimit = 3, round = 80 }) {
  const jobs = JOBS.map((job) => ({ job, step: 0, state: "waiting", started: 0, attempt: 0 }));
  const s = { completed: 0, steps: 0, redriven: 0, compensatedSteps: 0, wasted: 0,
    recordsRead: 0, recovery: [], finishedRound: 0 };

  for (let t = 1; t <= round; t++) {
    for (const k of jobs) {                                // agent: the step that started the previous round
      if (k.state !== "running" || t !== k.started + 1) continue;
      const st = STEP[k.step];
      if (PERSISTENT[k.job] === st) continue;              // the record never lands
      if (STALLED[k.job] === st && k.attempt === 0) continue; // the work was done, the record did not land
      s.steps += 1; k.step += 1; k.attempt = 0; k.state = "waiting";
    }
    if (supervisor && t % scan === 0) {                    // supervisor: finds steps whose time has expired
      s.recordsRead += jobs.length;
      for (const k of jobs) {
        if (k.state !== "running" || t - k.started < lease) continue;
        s.redriven += 1;
        s.recovery.push(t - (k.started + 1));              // rounds elapsed since the expected record
        if (STALLED[k.job] === STEP[k.step]) s.wasted += 1;    // the step will run a second time
        k.attempt += 1; k.state = "waiting";
      }
    }
    for (const k of jobs) {                                // scheduler: starts the next step
      if (k.state !== "waiting") continue;
      if (k.step >= STEP.length) { k.state = "done"; s.completed += 1; s.finishedRound = t; continue; }
      if (k.attempt >= retryLimit) {                        // threshold exceeded: completed steps are compensated
        s.compensatedSteps += k.step; k.state = "compensated"; s.finishedRound = t; continue;
      }
      k.state = "running"; k.started = t;
    }
    if (jobs.every((k) => k.state === "done" || k.state === "compensated")) break;  // all jobs stopped
  }
  const avg = s.recovery.length ? s.recovery.reduce((a, b) => a + b, 0) / s.recovery.length : 0;
  return { ...s, halfFinished: jobs.filter((k) => k.state === "running").length,
    compensatedJobs: jobs.filter((k) => k.state === "compensated").length,
    avgRecovery: avg, worstRecovery: s.recovery.length ? Math.max(...s.recovery) : 0 };
}
```

```js
// supervisor/measure.mjs — a run without and with the supervisor, then a scan of the scan interval
import { run, JOBS, STEP, STALLED, PERSISTENT } from "./run.mjs";

const D = [["no supervisor", run({ supervisor: false })], ["supervisor on", run({ supervisor: true })]];
const MEASURES = [["completed", "completed jobs"], ["steps", "steps run"],
  ["redriven", "steps re-driven"], ["wasted", "wasted steps"],
  ["compensatedJobs", "compensated jobs"], ["compensatedSteps", "compensated steps"],
  ["halfFinished", "half-finished jobs"], ["finishedRound", "last job round"]];

console.log(`${JOBS.length} jobs, ${STEP.length} steps; ${Object.keys(STALLED).length} step(s) stalled ` +
  `(${Object.entries(STALLED).map(([i, a]) => `${i}@${a}`).join(", ")}), ` +
  `${Object.keys(PERSISTENT).length} step(s) with a persistent failure (${Object.entries(PERSISTENT).map(([i, a]) => `${i}@${a}`).join(", ")})`);
console.log();
console.log(`${"measure".padEnd(24)}${D.map(([a]) => a.padStart(14)).join("")}`);
for (const [k, label] of MEASURES)
  console.log(`${label.padEnd(24)}${D.map(([, r]) => String(r[k]).padStart(14)).join("")}`);

console.log(`\n${"scan".padStart(7)}${"jobs recovered".padStart(15)}${"avg recovery".padStart(14)}` +
  `${"worst".padStart(9)}${"last round".padStart(11)}${"records read".padStart(14)}`);
for (const scan of [1, 2, 4, 8]) {
  const r = run({ supervisor: true, scan });
  console.log(`${String(scan).padStart(7)}${String(r.completed).padStart(15)}` +
    `${r.avgRecovery.toFixed(2).padStart(14)}${String(r.worstRecovery).padStart(9)}` +
    `${String(r.finishedRound).padStart(11)}${String(r.recordsRead).padStart(14)}`);
}
```

```
12 jobs, 4 steps; 3 step(s) stalled (S2@discount, S5@tariff, S9@invoice), 1 step(s) with a persistent failure (S7@discount)

measure                  no supervisor supervisor on
completed jobs                       8            11
steps run                           36            45
steps re-driven                      0             6
wasted steps                         0             3
compensated jobs                     0             1
compensated steps                    0             1
half-finished jobs                   4             0
last job round                       5             8

   scan jobs recovered  avg recovery    worst last round  records read
      1             11          1.00        1          8            96
      2             11          1.33        2          8            48
      4             11          2.33        4         12            36
      8             11          5.67        7         24            36
```

#### Reading the Numbers

The no-supervisor column shows what a silent failure means. Eight of twelve jobs completed, 36
steps ran, and the last job finished on round 5. On the surface everything looks fine: the error
counter is zero, compensation is zero, no one saw an exception. The bottom row carries the real
result: **4 jobs are half-finished** and sit in the "running" state forever. The scheduler started
the step, the agent did not reply, and there is no one to ask the question.

In the supervisor-on column, 3 of the 4 jobs are recovered, one is compensated, and half-finished
jobs drop to 0. Completed jobs rise from 8 to 11 — not 12, because job `S7` has a persistent
failure, and the supervisor counted it as past the threshold after driving it three times; the
one completed step of that job was compensated. The supervisor does not turn a failed job into a
successful one; **it turns an uncertain state into a determined one.** A job either finishes or is
compensated; the count of jobs left in the "running" state is zero.

Two rows show the cost. Steps run rises from 36 to 45, and 3 of those are **wasted** steps: the
agent had already done the work, it had only failed to record it. The supervisor cannot know
this; the only way to tell a stall apart from a persistent failure is to retry. This is why the
scheduler–agent–supervisor arrangement has a mandatory precondition: **steps must be
idempotent.** Running the same step a second time must not produce an additional effect; if a
step does not carry this property naturally, it is protected with an idempotency key. Idempotency
and the idempotency key were established in the Web API Design and The Data Access Layer and
Business Logic courses; here they precondition the supervisor's work. A non-idempotent `invoice`
step produces three extra invoice lines across three wasted runs.

The table below shows what the scan interval buys. Recovered jobs are 11 across all four
intervals: the supervisor finds them sooner or later. What changes is how late it finds them.
Average recovery delay rises from 1.00 to 5.67 rounds, the worst case from 1 to 7; the batch's
last job shifts from round 8 to round 24. In exchange, the records the supervisor reads drop from
96 to 36. When the interval doubles, the delay roughly doubles and the reads are halved; frequent
scanning speeds up recovery, infrequent scanning reduces the read load on the durable record. This
trade-off is deterministic, read directly from the model, not a duration measurement dependent on
the environment.

#### Back to the Estimate

The model's 12 jobs are not a volume. The volume comes from K01: 4,000 workflows a day and four
steps per job, that is, 16,000 steps a day. The previous lesson's KK2 assumption (the end-of-day
workflow's persistent failure ratio of 0.02) carries forward. This lesson adds one more
assumption.

**KK3 — a step's stall ratio is 0.005.** The rationale is that agent process restarts, calls cut
off before recording, and steps that time out and lose their reply produce a steady baseline. It
is not added to K01's table; its sensitivity is calculated at 0.01.

```js
// supervisor/cost.mjs — applies the model's ratios to K01's end-of-day job volume
import { run, STEP } from "./run.mjs";

const INVOICE_LINES = 4000;                 // K01: daily invoice lines (computed value) = daily workflows
const STEPS_PER_DAY = INVOICE_LINES * STEP.length;
const KK2 = 0.02;                           // previous lesson: end-of-day workflow's persistent failure ratio
const RETRY_LIMIT = 3;                      // the supervisor's threshold (model parameter)
const r = run({ supervisor: true, scan: 2 });
const wastedRatio = r.wasted / r.redriven;   // the share of re-drives that go to waste

console.log(`daily workflows ${INVOICE_LINES}, steps/day ${STEPS_PER_DAY}, ` +
  `model's wasted re-drive ratio ${wastedRatio.toFixed(3)}`);
console.log();
console.log(`${"KK3".padStart(6)}${"stalled steps/day".padStart(19)}${"re-drives/day".padStart(19)}` +
  `${"wasted steps/day".padStart(20)}${"half-finished, no supervisor".padStart(31)}${"job share".padStart(10)}`);
for (const KK3 of [0.005, 0.01]) {          // KK3: a step's stall ratio
  const stalled = STEPS_PER_DAY * KK3;
  const persistent = INVOICE_LINES * KK2;
  const halfFinished = stalled + persistent;
  console.log(`${KK3.toFixed(3).padStart(6)}${stalled.toFixed(0).padStart(19)}` +
    `${(stalled + persistent * RETRY_LIMIT).toFixed(0).padStart(19)}${stalled.toFixed(0).padStart(20)}` +
    `${halfFinished.toFixed(0).padStart(31)}${`${(100 * halfFinished / INVOICE_LINES).toFixed(2)}%`.padStart(10)}`);
}

console.log(`\nthe cost of the scan interval (model measurement, deterministic):`);
for (const scan of [1, 8]) {
  const k = run({ supervisor: true, scan });
  console.log(`  scan ${scan}: avg recovery ${k.avgRecovery.toFixed(2)} rounds, ` +
    `worst ${k.worstRecovery}, last round ${k.finishedRound}, records read ${k.recordsRead}`);
}
```

```
daily workflows 4000, steps/day 16000, model's wasted re-drive ratio 0.500

   KK3  stalled steps/day      re-drives/day    wasted steps/day   half-finished, no supervisor job share
 0.005                 80                320                  80                            160     4.00%
 0.010                160                400                 160                            240     6.00%

the cost of the scan interval (model measurement, deterministic):
  scan 1: avg recovery 1.00 rounds, worst 1, last round 8, records read 96
  scan 8: avg recovery 5.67 rounds, worst 7, last round 24, records read 36
```

Once the number is read, the decision makes itself. A stall ratio of five in a thousand means 80
stalled steps a day; together with persistent failures, an unsupervised arrangement leaves **160
workflows half-finished** a day, which is 4 percent of the daily 4,000 invoice lines. If the ratio
rises to 0.01, that becomes 240 and 6 percent. None of these jobs increment an error counter;
their first symptom is that month-end reconciliation does not balance.

The same table carries the supervisor's cost: 320 re-drives a day, 80 of them wasted steps. The
wasted-run ratio in the model is 0.500, because half of the re-drives come from
retrying a persistently failed job. The eighty are absorbed by idempotency; if they are not, 80
extra invoice lines appear a day — that is, the supervisor produces new errors equal to exactly
half of the error it set out to fix. The scheduler–agent–supervisor arrangement is therefore not a
pattern on its own; together with idempotent steps, it is a single decision.

#### Summary

- The three roles are separated because no role can notice its own failure: the scheduler writes
  the job to a durable record and starts the step, the agent runs the step, the supervisor only
  looks at the record.
- In the unsupervised run, 4 of 12 jobs stayed in the "running" state without producing a single
  error; the error counter was zero and completed jobs stopped at 8.
- The supervisor turns an uncertain state into a determined one: 3 jobs were recovered, 1 job was
  compensated, half-finished jobs dropped from 4 to 0, and completed jobs rose from 8 to 11.
- The cost sits in steps run (36 → 45), 3 of which are wasted runs; this is why idempotent steps
  are a precondition of the arrangement.
- The scan interval trades recovery delay against read load: as the interval rises from 1 to 8,
  average recovery climbs from 1.00 to 5.67 rounds and the last job shifts from round 8 to round
  24, while records read drop from 96 to 36.
- Back to K01: at 16,000 steps/day and KK3 = 0.005, the unsupervised arrangement leaves 160
  half-finished jobs a day (4 percent of invoice lines); the supervisor zeroes this out and in
  exchange brings 320 re-drives and 80 wasted runs a day.

#### Next Step

This lesson's supervisor was treated as a single party. In reality, the supervisor, and the
scheduler that starts the end-of-day job, both run on more than one node — because a supervisor
running on a single node becomes, when it goes down, exactly the kind of workflow no one notices.
But the scheduler's job is singular: end-of-day billing must be started exactly once. If two nodes
start it at the same time, two invoice lines are produced for the same seller-day, and that can
only be fixed by hand, not by compensation. The next lesson builds the arrangement that gives this
singular responsibility to a single node: one of the nodes taking over responsibility for the
duration of a lease, how many rounds a handover takes when that node goes down, and counting the
rounds in which a misconfigured lease runs two nodes at once.

### Leader Election

Source: https://academia.sh/en/courses/application-layer/leader-election

Assigning a task that requires singular responsibility to a single node: building lease-based leader election with an in-process model, measuring the handover round and the number of rounds two leaders run at once, the leaderless round growing as the lease lengthens, fencing removing double starts, the persistent trigger recovering the missed batch, and translating the results into K01's daily invoice lines.

The previous lesson's supervisor was treated as a single party. In reality, the supervisor, and
the scheduler that starts the end-of-day job, both run on more than one node — because a
supervisor running on a single node becomes, when it goes down, exactly the kind of workflow no
one notices. But the scheduler's job is singular: end-of-day billing must be started exactly
once. If two nodes start it at the same time, two invoice lines are produced for the same
seller-day, and that can only be fixed by hand, not by compensation.

**Leader election** is the arrangement that gives responsibility to exactly one node from a set
of nodes for a defined period. It is related to the handover measured in the Introduction to
System Design course, but the question differs: there, handover was done to **sustain the
service** — a replica accepting writes had to remain; here, election is done for **singular
responsibility** — exactly one party must do the work. The first kind's failure is an outage; the
second kind's is a double job.

#### Lease and Fencing Token

Responsibility is not granted indefinitely, because no one can know for certain that the node
holding it has gone down. Instead, a **lease** is granted: the leader signs the shared durable
record at regular intervals. If the record has not been refreshed for the length of the lease,
another node takes over responsibility and increments the record's **fencing token** counter by
one. The token says which ordinal a leadership was granted in; it has nothing to do with
billing's thirty-day period.

The lease does not distinguish a node that is down from one that is up; it only distinguishes one
that **cannot write.** A leader that has paused, dropped off the network, or slowed down is still
up, and keeps believing itself to be the leader. This is why the number that matters is not the
handover round, but **the number of rounds in which two nodes believe themselves to be the leader
at the same time.** The model has 5 nodes; the first leader is
unreachable for five rounds starting on round 6, and the end-of-day batch must be started every
six rounds.

```js
// leader/election.mjs — lease-based leader election; a round-based in-process model. A round
// is an abstract step; outage, lease, and renewal are interval parameters, not measured durations.
export const TRIGGER = [6, 12, 18, 24];           // rounds at which the end-of-day batch must start
export const START = 6;                            // first round the leader becomes unreachable

export function run({ nodes = 5, lease = 3, renewal = 3, outage = 5,
  fencing = false, persistentTrigger = false, rounds = 24 }) {
  const end = START + outage - 1;
  const record = { leader: 0, lastRenewal: 0, token: 1 };     // shared durable record
  const believes = new Array(nodes).fill(false); believes[0] = true;
  const nodeToken = new Array(nodes).fill(0); nodeToken[0] = 1;
  const s = { started: 0, missed: 0, doubleStarts: 0, rejected: 0, delayedRounds: 0,
    twoLeaderRounds: 0, leaderlessRounds: 0, handoverRounds: 0, handovers: 0 };
  let pending = 0, pendingRound = 0;
  const reachable = (d, t) => d !== 0 || t < START || t > end;

  for (let t = 1; t <= rounds; t++) {
    if (reachable(record.leader, t)) record.lastRenewal = t;    // the leader renews its lease
    if (t - record.lastRenewal >= lease) {                      // lease expired: a node takes over
      const newLeader = [...Array(nodes).keys()].find((d) => d !== record.leader && reachable(d, t));
      s.handoverRounds += t - record.lastRenewal; s.handovers += 1;
      record.leader = newLeader; record.lastRenewal = t; record.token += 1;
      believes[newLeader] = true; nodeToken[newLeader] = record.token;
    }
    const active = [...Array(nodes).keys()].filter((d) => believes[d] && reachable(d, t));
    if (active.length > 1) s.twoLeaderRounds += 1;
    if (active.length === 0) s.leaderlessRounds += 1;
    let accepted = 0;
    for (const d of active) {
      if (fencing && nodeToken[d] < record.token) { s.rejected += 1; continue; } // stale token
      accepted += 1;
    }
    if (TRIGGER.includes(t)) {                                  // the end-of-day batch must start
      if (accepted === 0) { if (persistentTrigger) { pending += 1; pendingRound = t; } else s.missed += 1; }
      else { s.started += 1; s.doubleStarts += accepted - 1; }
    } else if (pending > 0 && accepted > 0) {                   // persistent trigger: the new leader takes over
      s.started += 1; s.delayedRounds += t - pendingRound; pending = 0;
    }
    if (t % renewal === 0) for (const d of active)               // node reads the record, drops belief if needed
      if (record.leader !== d) believes[d] = false;
  }
  return s;
}
```

The model has two add-on options. **Fencing** is doing every piece of work together with this
token: the record rejects the work of a node whose token has fallen behind. Fencing was
established in the Relational Database Administration course; here it is applied to singular
responsibility. The **persistent trigger** is recording a start request that arrives in a
leaderless round, so that the next leader takes it over.

```js
// leader/measure.mjs — a run without and with fencing, then a scan of the lease duration
import { run, TRIGGER, START } from "./election.mjs";

const D = [["plain", run({})], ["fencing", run({ fencing: true })],
  ["fencing+trigger", run({ fencing: true, persistentTrigger: true })]];
const MEASURES = [["started", "batches started"], ["missed", "batches missed"],
  ["delayedRounds", "delayed start rounds"], ["doubleStarts", "double-started batches"],
  ["rejected", "rejected by fencing"], ["twoLeaderRounds", "two-leader rounds"],
  ["leaderlessRounds", "leaderless rounds"], ["handoverRounds", "handover rounds"]];

console.log(`5 nodes, 24 rounds, lease 3, renewal 3; the leader node is unreachable in rounds ` +
  `${START}-${START + 4}. End-of-day batch trigger: rounds ${TRIGGER.join(", ")}.`);
console.log();
console.log(`${"measure".padEnd(24)}${D.map(([a]) => a.padStart(22)).join("")}`);
for (const [k, label] of MEASURES)
  console.log(`${label.padEnd(24)}${D.map(([, r]) => String(r[k]).padStart(22)).join("")}`);

const S = ["handoverRounds", "leaderlessRounds", "missed", "twoLeaderRounds", "doubleStarts"];
const heads = ["handover", "leaderless", "missed", "2-leader", "double"];
console.log(`\n${"".padStart(6)}${"long outage (5 rounds)".padStart(50)}${"short outage (2 rounds)".padStart(50)}`);
console.log(`${"lease".padStart(6)}${heads.map((a) => a.padStart(11)).join("").repeat(2)}`);
for (const lease of [2, 3, 5, 8]) {
  const row = [5, 2].flatMap((outage) => {
    const r = run({ lease, outage, fencing: false });
    return S.map((k) => String(r[k]).padStart(11));
  });
  console.log(`${String(lease).padStart(6)}${row.join("")}`);
}
```

```
5 nodes, 24 rounds, lease 3, renewal 3; the leader node is unreachable in rounds 6-10. End-of-day batch trigger: rounds 6, 12, 18, 24.

measure                                  plain               fencing       fencing+trigger
batches started                              3                     3                     4
batches missed                               1                     1                     0
delayed start rounds                         0                     0                     2
double-started batches                       1                     0                     0
rejected by fencing                          0                     2                     2
two-leader rounds                            2                     2                     2
leaderless rounds                            2                     2                     2
handover rounds                              3                     3                     3

                                  long outage (5 rounds)                           short outage (2 rounds)
 lease   handover leaderless     missed   2-leader     double   handover leaderless     missed   2-leader     double
     2          2          1          1          2          1          2          1          1          2          0
     3          3          2          1          2          1          0          2          1          0          0
     5          5          4          1          2          1          0          2          1          0          0
     8          0          5          1          0          0          0          2          1          0          0
```

#### Reading the Numbers

The plain run could start only three of the four triggers. On round 6 the leader became
unreachable, and because the lease had not yet expired, no one took over: **the batch never
started.** The handover took three rounds to complete, and two rounds passed leaderless. On round
11 the old leader came back and still believed itself to be the leader; both it and the new
leader saw the trigger on round 12, and the batch was started twice. The plain arrangement's
balance sheet is this: one batch was missed, one batch ran twice, and one ran correctly.

Three rows must be kept separate. `two-leader rounds` is the number of rounds in which two nodes
**believe themselves to be the leader**, and it is 2 in all three columns; fencing does not
change this number, because fencing does not correct a node's belief. What changes is the
`double-started batches` row: it drops from 1 to 0, and in exchange `rejected by fencing` rises
from 0 to 2. The record rejected the work of the node whose token had fallen behind. This
distinction between belief and effect is the core of the pattern: **leader election determines
who is the leader; fencing determines who can write.** The first is not enough without the
second.

The third column closes the second failure. With the persistent trigger, missed batches drop to
0, batches started rise from 3 to 4, and the cost is a two-round delay. The trigger being
momentary ties it to a point in time; recording it turns it into a job. This is the same rule as
the previous lesson's scheduler: an intent that sits only in memory cannot survive a failure.

The table below shows what the lease buys. As the lease rises from 2 to 5, the handover round
climbs from 2 to 5 and leaderless rounds from 1 to 4; at lease 8, no handover happens at all, and
5 rounds pass leaderless across the whole outage. By contrast, the `2-leader` column is the same
at lease 2, 3, and 5 — 2. **A longer lease does not narrow the two-leader window**, because that
window arises not from the lease but from when the returning node next reads the record. The
column reading 0 at lease 8 is not a win: no handover happened there, so a second leader never
appeared, and the system stayed unaccountable for all five rounds.

The block on the right shows the short outage: at a two-round slowdown, a lease of 3 or higher
produces no handover at all, while a lease of 2 produces an unnecessary handover and two rounds of
two leaders. A short lease mistakes a temporary slowdown for a failure; a long lease notices a
real failure late. The correct setting sits between the two, and the measure for the decision is
the sum of leaderless rounds and two-leader rounds.

#### Back to the Estimate

The model's rounds connect to K01's numbers in two steps. What the model gives is a **window
share**: in a run with one handover, 2/24 of the rounds passed leaderless and 2/24 passed with
two leaders. The rate at which the trigger falls into these windows is the same share as reading
that the trigger is spread evenly across rounds.

**KK4 — 6 handovers a year.** This assumption is not new: the same number was used when the
Introduction to System Design course's Availability Patterns lesson converted the handover window
into a request rate, and it carries forward here for the same reason. It is not added to K01's
assumption table; its sensitivity is calculated at 12. One end-of-day batch produces 4,000 invoice
lines by K01's accounting.

```js
// leader/cost.mjs — applies the model's window ratios to K01's end-of-day batch
import { run } from "./election.mjs";

const INVOICE_LINES = 4000;          // K01: daily invoice lines (computed value) = one end-of-day batch
const ROUNDS = 24;
const plain = run({}), fenced = run({ fencing: true, persistentTrigger: true });
const leaderlessShare = plain.leaderlessRounds / ROUNDS, twoLeaderShare = plain.twoLeaderRounds / ROUNDS;

console.log(`model window: leaderless ${plain.leaderlessRounds}/${ROUNDS} = ${leaderlessShare.toFixed(4)}, ` +
  `two-leader ${plain.twoLeaderRounds}/${ROUNDS} = ${twoLeaderShare.toFixed(4)}`);
console.log(`one batch = ${INVOICE_LINES} invoice lines (K01, computed value)`);
console.log();
console.log(`${"KK4".padStart(5)}${"missed batches/yr".padStart(21)}${"lines not produced".padStart(19)}` +
  `${"double batches/yr".padStart(18)}${"double lines".padStart(13)}`);
for (const KK4 of [6, 12]) {         // KK4: handovers per year (M19/K01 Availability Patterns)
  const missed = KK4 * leaderlessShare, double = KK4 * twoLeaderShare;
  console.log(`${String(KK4).padStart(5)}${missed.toFixed(2).padStart(21)}` +
    `${(missed * INVOICE_LINES).toFixed(0).padStart(19)}${double.toFixed(2).padStart(18)}` +
    `${(double * INVOICE_LINES).toFixed(0).padStart(13)}`);
}
console.log(`\nwith fencing + persistent trigger: missed batches ${fenced.missed}, ` +
  `double-started batches ${fenced.doubleStarts}, delayed start ${fenced.delayedRounds} rounds`);
console.log(`with lease 8 (long): handover rounds ${run({ lease: 8 }).handoverRounds}, ` +
  `leaderless rounds ${run({ lease: 8 }).leaderlessRounds}, two-leader rounds ${run({ lease: 8 }).twoLeaderRounds}`);
```

```
model window: leaderless 2/24 = 0.0833, two-leader 2/24 = 0.0833
one batch = 4000 invoice lines (K01, computed value)

  KK4    missed batches/yr lines not produced double batches/yr double lines
    6                 0.50               2000              0.50         2000
   12                 1.00               4000              1.00         4000

with fencing + persistent trigger: missed batches 0, double-started batches 0, delayed start 2 rounds
with lease 8 (long): handover rounds 0, leaderless rounds 5, two-leader rounds 0
```

At six handovers a year, the expected missed batch is 0.50 and the expected double batch is 0.50.
Converted to invoice lines: once every two years a 4,000-line day is never produced, and once
every two years one is produced twice; the annual expectation is 2,000 lines either way. If the
handover count doubles, that becomes one event a year and 4,000 lines each.

The size of these numbers says where the decision should be anchored. Measured in service
availability, two leaderless rounds are nothing; they do not show up in K01's outage budget.
Measured in invoice lines, the same two rounds mean 2,000 duplicate lines a year, each one posting
the wrong amount to a seller. For work that requires singular responsibility, the unit of measure
is not duration but **the job**. Fencing brings the duplicate line to zero, the persistent trigger
brings the unproduced line to zero; together the two cost a two-round delay. Leader election on
its own removes neither of these two failures.

#### Summary

- Leader election and the handover that sustains a service are separate questions: in one, the
  goal is that a replica accepting writes remains; in the other, that the work is done exactly
  once.
- The lease distinguishes a node that cannot write, not one that is down; a paused leader still
  believes itself to be the leader when it comes back.
- In the plain run, 3 of 4 triggers were started: 1 batch was missed (it arrived in a leaderless
  round), 1 batch was started twice (it arrived in a two-leader round), and the handover took 3
  rounds to complete.
- Fencing corrects effect, not belief: two-leader rounds stayed at 2, while double-started
  batches dropped from 1 to 0 and rejected work rose from 0 to 2. The persistent trigger brought
  missed batches to 0 and added a two-round delay.
- As the lease lengthens, the handover round climbs from 2 to 5 and leaderless rounds from 1 to
  4; the two-leader window does not change, because its source is not the lease but the moment
  the returning node re-reads the record.
- Back to K01: at KK4 = 6 handovers a year, the expected missed batch is 0.50 and the double
  batch 0.50, that is, 2,000 unproduced and 2,000 duplicate invoice lines a year; at 12
  handovers, these become 4,000 each.

#### Next Step

Every lesson in this course had the work run detached from the party that made the request: the
state event was written to a queue, the workflow ran in the background, the end-of-day batch was
started by a leader. One question always stayed outside — **how the requesting party learns the
result.** When a seller requests a thirty-day billing report, the job is queued and finishes
seconds later; but the seller's browser is waiting for a response, and the connection cannot be
held open that long. The next lesson builds the arrangement that closes this gap: the request is
answered immediately with a job id, the result is fetched from a separate address, and how many
times the client asks before it sees the result is measured.

### Asynchronous Request–Reply

Source: https://academia.sh/en/courses/application-layer/async-request-reply

How a reply is returned for a long-running operation: a 3.60-second period report that does not fit a 200-millisecond threshold, the request being answered immediately with a job id, separating the status and result addresses, measuring the poll interval's trade-off between poll count and time-to-see, and exponential-backoff polling cutting fifteen polls to four.

Every lesson in this course had the work run detached from the party that made the request: the
state event was written to a queue, the workflow ran in the background, the end-of-day batch was
started by a leader. One question always stayed outside — how the requesting party learns the
result. When a seller requests a thirty-day billing report, the job is queued and finishes seconds
later; but the seller's client is waiting for a reply.

**Asynchronous request–reply** is the arrangement that answers the request immediately with a job
id and serves the result from a separate address. This lesson measures the cost of that
arrangement: how many times the client asks before it sees the result, how late it sees it, and
what those questions add to the peak request rate at the edge.

#### Why the Synchronous Path Does Not Work

The last lesson of the Service Design topic treated the end-to-end threshold as a budget: the
tight threshold on a tracking read is 200 milliseconds, and every step in the chain takes a share
of that budget. The period report's duration was likewise derived from K01 in this course's
second lesson: 3,000 records per seller, a scan rate of 833.33 records/s, that is, **3.60
seconds**.

Once the two numbers sit side by side, the decision is arithmetic: the report job is 18 times the
threshold. Splitting the budget does not help either, because splitting **narrows** every step;
if the step itself is larger than the entire budget, there is nothing left to split. The product
of timeout and retry count does not produce a third option here. The third option is that the
caller does not wait.

#### Three Addresses

The arrangement splits the work a single call would do into three addresses, and each address
answers exactly one question. The **request address** asks for the report and returns
immediately; the reply carries not the result but confirmation that the job was accepted, plus a
job id. The **status address** says whether the job for that id has finished. The **result
address** returns the output of a finished job.

The split has three consequences. First, the request address is now a short write and fits the
200-millisecond threshold; the status address is a single record read and fits the same
threshold. Second, once the job id is written, the client can close the connection and fetch the
result later; the client crashing does not drop the job. Third, sending the same request twice
must not start a second job — the request address must be idempotent, and the same idempotency
key must return the same job id. These three consequences are the arrangement itself; what
remains to measure is how many times the client asks the status address.

```js
// async/polling.mjs — poll count and time-to-see in a long-running report request.
// Durations are computed values derived from K01; the run is arithmetic, not an environment-dependent measurement.
export const JOB_DURATION = 3.6;      // K01: 3000 records / 833.33 records/s (Task Queues and Background Jobs lesson)
export const THRESHOLD = 0.2;         // K01: the tight threshold on a tracking read is 200 ms

// The client leaves the request, then polls the status address. With exponential backoff the
// interval doubles up to the cap; the algorithm is established in M14/K06, applied here only to poll count.
export function poll({ interval, exponential = false, cap = 2, duration = JOB_DURATION }) {
  let t = 0, p = interval, polls = 0, empty = 0;
  for (;;) {
    t += p; polls += 1;
    if (t >= duration) break;
    empty += 1;
    if (exponential) p = Math.min(p * 2, cap);
  }
  return { polls, empty, seen: t, extra: t - duration,
    requests: 1 + polls, emptyRatio: empty / polls };
}
```

```js
// async/measure.mjs — the poll interval is scanned, then tied to K01's volume and threshold
import { poll, JOB_DURATION, THRESHOLD } from "./polling.mjs";

const S = [...[0.25, 0.5, 1, 2].map((a) => [`fixed ${a} s`, poll({ interval: a })]),
  ["exp 0.25 s (cap 2 s)", poll({ interval: 0.25, exponential: true })]];

console.log(`job duration ${JOB_DURATION.toFixed(2)} s (K01, computed value), end-to-end threshold ${THRESHOLD * 1000} ms (K01)`);
console.log(`if kept synchronous: 1 request, connection open ${JOB_DURATION.toFixed(2)} s, ` +
  `${(JOB_DURATION / THRESHOLD).toFixed(0)}x the threshold`);
console.log();
console.log(`${"strategy".padEnd(26)}${"polls".padStart(9)}${"empty".padStart(7)}` +
  `${"empty ratio".padStart(12)}${"seen s".padStart(10)}${"extra s".padStart(10)}${"requests".padStart(10)}`);
for (const [label, r] of S)
  console.log(`${label.padEnd(26)}${String(r.polls).padStart(9)}${String(r.empty).padStart(7)}` +
    `${r.emptyRatio.toFixed(3).padStart(12)}${r.seen.toFixed(2).padStart(10)}` +
    `${r.extra.toFixed(2).padStart(10)}${String(r.requests).padStart(10)}`);

// K01: daily invoice lines 4000 = seller count (computed value); peak multiplier V8 = 3 (assumption);
// edge peak 513.89 requests/s (computed value).
const SELLERS = 4000, V8 = 3, EDGE_PEAK = 513.89, DAY = 86_400;
console.log(`\n${"KK5".padStart(5)}${"reports/day".padStart(13)}${"strategy".padStart(27)}` +
  `${"requests/day".padStart(13)}${"peak requests/s".padStart(16)}${"share of edge".padStart(15)}`);
for (const KK5 of [0.25, 0.5]) {          // KK5: period report requests per seller per day
  const reports = SELLERS * KK5;
  for (const [label, r] of [S[0], S[4]]) {
    const peak = (reports * r.requests / DAY) * V8;
    console.log(`${KK5.toFixed(2).padStart(5)}${reports.toFixed(0).padStart(13)}${label.padStart(27)}` +
      `${(reports * r.requests).toFixed(0).padStart(13)}${peak.toFixed(4).padStart(16)}` +
      `${`${(100 * peak / EDGE_PEAK).toFixed(3)}%`.padStart(15)}`);
  }
}
const sync = (reports) => ((reports / DAY) * V8 * JOB_DURATION);
console.log(`\nif kept synchronous, open connections at peak: KK5 = 0.25 -> ` +
  `${sync(SELLERS * 0.25).toFixed(3)}, KK5 = 0.50 -> ${sync(SELLERS * 0.5).toFixed(3)}`);
```

```
job duration 3.60 s (K01, computed value), end-to-end threshold 200 ms (K01)
if kept synchronous: 1 request, connection open 3.60 s, 18x the threshold

strategy                      polls  empty empty ratio    seen s   extra s  requests
fixed 0.25 s                     15     14       0.933      3.75      0.15        16
fixed 0.5 s                       8      7       0.875      4.00      0.40         9
fixed 1 s                         4      3       0.750      4.00      0.40         5
fixed 2 s                         2      1       0.500      4.00      0.40         3
exp 0.25 s (cap 2 s)              4      3       0.750      3.75      0.15         5

  KK5  reports/day                   strategy requests/day peak requests/s  share of edge
 0.25         1000               fixed 0.25 s        16000          0.5556         0.108%
 0.25         1000       exp 0.25 s (cap 2 s)         5000          0.1736         0.034%
 0.50         2000               fixed 0.25 s        32000          1.1111         0.216%
 0.50         2000       exp 0.25 s (cap 2 s)        10000          0.3472         0.068%

if kept synchronous, open connections at peak: KK5 = 0.25 -> 0.125, KK5 = 0.50 -> 0.250
```

#### Reading the Numbers

The four fixed-interval rows show a trade-off. When the interval grows fourfold (0.25 → 1
second), poll count drops from 15 to 4; in exchange, time-to-see rises from 3.75 to 4.00 seconds.
Extra wait rises from 0.15 to 0.40 seconds, that is, 2.67 times. The trade-off is asymmetric: four
times fewer questions, only 0.25 seconds later a result.

The empty-poll ratio names the actual waste. At the quarter-second interval, 14 of 15 polls
receive a "still running" reply: the ratio is 0.933. At the two-second interval, the ratio drops
to 0.500. A fixed interval does not know how long the job will take, so it asks at the same
frequency at the start of the job and at the end; yet the knowledge that a job will finish grows
over time.

The last row puts this to use. Exponential-backoff polling starts at a quarter second, doubles the
interval up to a two-second cap, and sees the result **in 4 polls at 3.75 seconds** — close to the
fixed two-second interval's poll count, and the same as the fixed quarter-second interval's
time-to-see. Extra wait stays at 0.15 seconds. None of the fixed-interval rows deliver both
numbers together. The backoff algorithm was established in the Resilience and Reliability course;
here it is applied not to retrying but to polling, and it behaves the same way.

#### Back to the Estimate

Seeing what polling requests add to the peak rate at the edge requires the report volume. K01
computed 4,000 invoice lines a day, and since each line is one seller's one day, the seller count
is 4,000. Report request frequency is not in K01.

**KK5 — 0.25 period report requests per seller per day.** The rationale is that sellers request
the period report not every day but once or twice a week. It is not added to K01's table; its
sensitivity is calculated at 0.50. The peak multiplier is K01's V8 assumption (3).

Read the result two ways. At the fixed quarter-second poll, 16,000 requests result a day and
amount to 0.5556 requests/s at the peak — 0.108 percent of K01's 513.89 requests/s at the edge. At
exponential backoff, 5,000 requests, 0.1736 requests/s, and 0.034 percent. If KK5 doubles, the
shares become 0.216 percent and 0.068 percent. That is, **polling's cost at the edge is a few
thousandths in every case.**

The synchronous alternative costs little either: connections open at once at the peak amount to
0.125 (0.250 at KK5 = 0.50). Read together, the two numbers make the pattern's rationale clear:
asynchronous request–reply is not a **resource** decision, it is a **budget** decision. Both paths
are cheap in resources; the distinction is that a 3.60-second job can never drop below a
200-millisecond threshold. In a system with a threshold, the only way out for work that does not
fit it is to not complete the request with a reply.

#### Summary

- The 3.60-second period report is 18 times K01's 200-millisecond end-to-end threshold;
  splitting the budget narrows a step, it does not enlarge it.
- The arrangement sets up three addresses — request, status, result; the request address is a
  short write and fits the threshold, and a second request with the same idempotency key does
  not start a new job.
- At a fixed interval, poll count and time-to-see trade off asymmetrically: as the interval rises
  from 0.25 to 1 second, polls drop from 15 to 4 while extra wait rises from 0.15 to 0.40 seconds.
- The empty-poll ratio is 0.933 at a fixed quarter second and 0.500 at two seconds; a fixed
  interval does not use the fact that the probability of a job finishing grows over time.
- Exponential-backoff polling delivers both numbers together: 4 polls and 3.75 seconds to see the
  result, extra wait 0.15 seconds.
- Back to K01: at KK5 = 0.25, polling amounts to 0.5556 requests/s at the peak (0.108 percent of
  the edge rate), 0.1736 requests/s with exponential backoff (0.034 percent); kept synchronous,
  that would be 0.125 open connections at the peak. The pattern is a budget decision, not a
  resource decision.

#### Course Wrap-Up

The course designed what happens after the request reaches the application, across two topics,
and asked the same three questions in every lesson: what is this decision, which number in the
introductory courses' accounting does it move, and which number grows in exchange. The measures
the lessons leave behind are collected in the table below.

| Lesson | Decision | Number moved | Grows in exchange |
|---|---|---|---|
| Stateless Services | moving state out of the service | replicas needed 4 → 3; per-replica 200.93 → 171.30 requests/s | 1,027.78 operations/s to the state store, 7.40 times K01's store load |
| Service Discovery | finding the address at runtime | requests per failure 102,778 → 510.46 | list window 2,980 ms; the registry's own load 3.00 heartbeats/s |
| Communication Styles | the shape of the internal call | internal bytes 373 / 195 / 207; internal requests 833.34 requests/s | names the caller knows 6 → 8; internal bound 0.41–0.78 times 1.60 Mbit/s |
| Inter-Service Contracts | the schema's evolution order | broken consumers 4/5 → 0 | record 243 → 423 bytes; internal bound 0.810 → 1.410 Mbit/s |
| Timeout and Retry Budget | distributing the budget across the chain | worst-case duration from 6.1 times the budget to 200 ms | internal requests 1,666.68 → 1,895.85 requests/s; 3,333.36 under a widespread slowdown |
| Message Queues | spreading the load over time | processing capacity 97.22 → 48.61 events/s | backlog peak 525,000, longest wait 3.00 hours |
| Task Queues and Background Jobs | taking the long job out of the request path | tracking requests waiting over 100 ms 29.02 percent → 0 | report duration 3.60 → 4.14 s; 4 slots and 4.00 workers |
| Competing Consumers | how many consumers the work is split across | at least 9.45 events/s per consumer for 48.61 events/s; contribution 0.128 at N = 8 | order breakage 102 → 2,279 shipments |
| Queue-Based Load Leveling | buffering an unforeseen spike | arrivals climbing to 972 events/s while backlog holds at 525,000, wait at 3.00 hours | queue bound 525,391; 350,049 dropped events under the volume increase |
| Priority Queue | splitting jobs by service level | the top class's wait drops from 6.5 minutes to 0.0 | 300,000 jobs stay queued in the bottom class; under weighted share, B's backlog 440,000 → 740,000 |
| Sequential Convoy | putting the order constraint into a lane | wrong final state 1 → 0; constrained peak write 55.55 events/s | consumer requirement ×1.357, ×1.786 with a coarse key |
| Choreography and Orchestration | how the workflow is assembled | compensation responsibility 3 → 1 unit; uncompensated steps 80 → 0/day | messages 1.3889 → 2.2222 messages/s, ratio 1.600 |
| Scheduler–Agent–Supervisor | finding the half-finished job | half-finished jobs 160 → 0/day, 4 percent of invoice lines | 320 re-drives and 80 wasted runs/day |
| Leader Election | giving singular responsibility to a single node | duplicate and unproduced invoice lines 2,000 + 2,000 → 0/yr | a two-round delay; two-leader rounds stay at 2 |
| Asynchronous Request–Reply | the caller not waiting | a 3.60 s job moves outside the 200 ms threshold | polling at 0.1736 requests/s at the peak, 0.034 percent of the edge rate |

The table's rule deserves its own name. This course has **two measures, and the choice is made by
whether the caller waits.** If the caller waits, the measure is **chain latency**: step count, the
product of timeout and retry count, worst-case end-to-end duration, and requests dropped once the
budget is exceeded. If the caller does not wait, the measure is **backlog and drain**: the peak of
waiting work, drain duration, each added consumer's contribution, and work dropped at the buffer's
bound. The two measures do not substitute for each other; a queue decision cannot be defended with a
latency budget, nor a chain decision with a backlog peak. The course's own assumptions — the
state store's load, the registry's time to live, the gateway's share, the peak window, long-job
demand, carrier outages, the extra stream's duration, the consumer group's capacity, the
order-constraint ratio, the workflow's failure ratio, the stall ratio, handover frequency, and
report request frequency — never mixed into K01's table; each was written down with its own name,
rationale, and sensitivity.

The question the course leaves behind sits one layer deeper. Services became stateless, found their
addresses, evolved without breaking their contracts, and split their budgets; work was queued,
given priority, kept in order, coordinated as a flow, and had its half-finished jobs found. But
this entire design rests on the assumption of **a single data store**. When state was moved out of
the service, it was put somewhere, and how that place would scale was never asked: replicating the
store, partitioning the data, federating contexts into their own stores, and which layer the cache
sits in were not designed in this course. The `GB of stored data` row in K01's accounting never
moved across this entire course; the `requests/s reaching the store` row was moved by the
queue-capacity decision in the Message Queues lesson, but the store itself was never scaled. The
next course, **Scaling the Data Layer**, takes up those rows.