Backpressure

The previous lesson set the budget to a single request’s duration and taught how to reclaim its resource on time. One thing the budget never touches: the rate at which requests arrive. Facing a slowing dependency, the budget cuts each request off, but the next one still arrives and spends the same budget on the same slow step. The budget says how long a request waits; it says nothing to whoever feeds the system work.

This lesson asks that missing notice on the write path. Status events from the carrier flow in at a fixed rate; if the consumer reading the event log and feeding the read model falls behind, work piles up somewhere. The Caching, Queues and Asynchronous Processing course established the single-queue case — a bounded queue as a latency-budget decision, overflow policies, blocking passing the slowdown to the producer, and an unaware layer in front emptying out the limit — not retold here. The question here is the chain: how many links back the signal is carried, where work collects where it is not, and where propagation must stop.

Failure Mode and Chain

The write path has four parts: the carrier’s notification endpoint sends the event, the entry service accepts it, the log writer persists it, the projection consumer updates the read model. The source sits outside the system, beyond its control — the carrier belongs to another organization.

AY11 — the projection consumer falls behind. It does not go down or write incorrectly; its speed drops from 120 events per round to 60. Duration 20 minutes, frequency three times a month. Rationale: consumer slowdown usually comes from a projection schema change or a rebuild job overlap, lasting until that job finishes. Its sensitivity is below, not added to K01’s table.

AY11 is not an outage; the tracking query keeps answering, only the projection it reads falls behind, so it is not written to K01’s monthly 43.2-minute downtime budget. A lost write goes into a different measure, its threshold set in this course’s recovery topic.

A round is an abstract step, corresponding in the model to a one-second window, and this cycle coefficient is an assumption. The source rate is K01’s peak write count: 97.22 events per round. Arrival is not flat but oscillating — a multiplier cycle averaging 1.00 represents fluctuation within the peak hour.

Propagation Depth

The signal is a single piece of information carried backward from the consumer: “my successor is full, I cannot take your work.” Each link can heed this or ignore it. In the model this is a parameter — depth, how many links back it travels from the consumer. At depth 0 no link heeds it; at depth 3 it reaches the service boundary and the carrier’s event is not accepted.

// backpressure/chain.mjs — MODEL of the write path. A round is an abstract step; the cycle
// coefficient is an assumption (one round = one second). No real queue, log, or consumer is set up.
export const K01 = { writePeak: 97.22 };      // K01: carrier state event, peak write/s

// Links are ordered from source to tip. `rate` is the number of events processed in one round.
export const LINK = [
  { name: "entry", capacity: 200, rate: 250 },       // event admission endpoint
  { name: "log", capacity: 150, rate: 150 },         // event log writer
  { name: "projection", capacity: 100, rate: 120 },  // consumer feeding the read model
];

// Oscillation of arrivals within the peak hour: a multiplier cycle averaging 1.00.
export const OSCILLATION = [0.6, 0.6, 0.6, 2.2];

// Link i pauses if it heeds its successor's fullness; if not, it drops the event.
// depth: how many links back from the consumer the signal is carried. 3 = to the limit.
const heeds = (i, depth) => depth >= LINK.length - 1 - i;

export function run({ roundCount, depth, consumptionRate, oscillate = true, unlimited = false }) {
  const rate = LINK.map((h, i) => (i === LINK.length - 1 ? consumptionRate : h.rate));
  const capacity = LINK.map((h) => (unlimited ? Infinity : h.capacity));
  const buffer = LINK.map(() => []);
  const s = {
    produced: 0, completed: 0, rejected: 0, finishedAge: 0, sourceStopped: 0,
    lost: LINK.map(() => 0), paused: LINK.map(() => 0),
  };
  let carry = 0;

  for (let t = 1; t <= roundCount; t += 1) {
    for (let i = LINK.length - 1; i >= 0; i -= 1) {          // drain from the tip toward the source
      let n = Math.min(rate[i], buffer[i].length);
      while (n > 0) {
        if (i === LINK.length - 1) {                         // last link: work completes
          s.finishedAge = Math.max(s.finishedAge, t - buffer[i].shift());
          s.completed += 1; n -= 1; continue;
        }
        if (buffer[i + 1].length < capacity[i + 1]) { buffer[i + 1].push(buffer[i].shift()); n -= 1; continue; }
        if (heeds(i, depth)) { s.paused[i] += 1; break; }   // successor full: pause
        s.lost[i + 1] += 1; buffer[i].shift(); n -= 1;      // no signal: lose it
      }
    }
    const multiplier = oscillate ? OSCILLATION[(t - 1) % OSCILLATION.length] : 1;
    carry += K01.writePeak * multiplier;
    while (carry >= 1) {                                    // source: carrier notification endpoint
      carry -= 1; s.produced += 1;
      if (buffer[0].length < capacity[0]) { buffer[0].push(t); continue; }
      if (depth >= LINK.length) {                            // rejected at the limit: event not accepted
        s.rejected += 1; s.sourceStopped += 1;
      } else s.lost[0] += 1;                                // accepted, then lost
    }
  }
  s.backlog = buffer.reduce((a, b) => a + b.length, 0);
  const pending = buffer.flat();
  s.pendingAge = pending.length === 0 ? 0 : roundCount - Math.min(...pending);
  s.totalLost = s.lost.reduce((a, b) => a + b, 0);
  return s;
}

Two counters are kept separate, and the lesson’s whole distinction is here. A lost event is one the system accepted, then dropped; the carrier counts it sent and never resends it. A rejected event is never accepted at the boundary; the carrier knows this and resends it under its own policy.

// backpressure/run.mjs — signal propagation depth: five settings across two days
import { K01, LINK, run } from "./chain.mjs";

const ROUNDS = 1200;                       // AY11: 20 minutes, one round = one second
const CONSUMPTION_NORMAL = 120, CONSUMPTION_SLOW = 60;   // projection consumer's rate per round
const SETTING = [
  ["unlimited buffer", { depth: 0, unlimited: true }],
  ["no signal", { depth: 0 }],
  ["1 link", { depth: 1 }],
  ["2 links", { depth: 2 }],
  ["to the limit", { depth: 3 }],
];

console.log(`${ROUNDS} rounds, source K01 peak write ${K01.writePeak}/round (oscillating, average multiplier 1.00)`);
console.log(`links: ${LINK.map((h, i) => `${h.name}(cap ${h.capacity}, rate ${i === 2 ? "consumption" : h.rate})`).join(" -> ")}\n`);

for (const [regime, consumptionRate] of [["failure-free: consumption 120/round", CONSUMPTION_NORMAL], ["AY11: consumption 60/round", CONSUMPTION_SLOW]]) {
  console.log(`-- ${regime} --`);
  console.log(`${"setting".padEnd(16)}${"completed".padStart(11)}${"rejected".padStart(11)}${"lost".padStart(9)}` +
    `${"loss location".padStart(20)}${"backlog".padStart(8)}${"pending age".padStart(13)}${"finished age".padStart(13)}`);
  for (const [label, y] of SETTING) {
    const r = run({ roundCount: ROUNDS, consumptionRate, ...y });
    console.log(`${label.padEnd(16)}${String(r.completed).padStart(11)}${String(r.rejected).padStart(11)}` +
      `${String(r.totalLost).padStart(9)}${r.lost.join("/").padStart(20)}${String(r.backlog).padStart(8)}` +
      `${String(r.pendingAge).padStart(13)}${String(r.finishedAge).padStart(13)}`);
  }
  console.log();
}

const unlimited = run({ roundCount: ROUNDS, depth: 0, unlimited: true, consumptionRate: CONSUMPTION_NORMAL });
const signaled = run({ roundCount: ROUNDS, depth: 3, consumptionRate: CONSUMPTION_NORMAL });
const flat = run({ roundCount: ROUNDS, depth: 3, consumptionRate: CONSUMPTION_NORMAL, oscillate: false });
console.log(`-- failure-free day's cost (against the unlimited buffer) --`);
console.log(`produced ${signaled.produced}, valid events rejected at the limit ${signaled.rejected}` +
  ` = ${((signaled.rejected / signaled.produced) * 100).toFixed(2)}%`);
console.log(`same setting with oscillation off: ${flat.rejected} rejections -> the source of rejection is not average load but oscillation`);
console.log(`unlimited buffer's pending age ${unlimited.pendingAge} rounds, signal to the limit ${signaled.pendingAge} rounds`);
console.log(`paused link-rounds: ${signaled.paused.join("/")}; events where the source was stopped ${signaled.sourceStopped}`);
console.log(`signal check cost: ${LINK.length} links x ${K01.writePeak} events/s = ${(LINK.length * K01.writePeak).toFixed(2)} checks/s`);

console.log(`\n-- AY11's sensitivity: consumer rate (1200 rounds) --`);
console.log(`${"consumption/round".padStart(19)}${"gap/round".padStart(10)}${"unlimited backlog".padStart(19)}` +
  `${"no-signal lost".padStart(16)}${"rejected at limit".padStart(19)}${"lost/rejected".padStart(14)}`);
for (const h of [90, 60, 30]) {
  const a = run({ roundCount: ROUNDS, depth: 0, unlimited: true, consumptionRate: h });
  const y = run({ roundCount: ROUNDS, depth: 0, consumptionRate: h });
  const t = run({ roundCount: ROUNDS, depth: 3, consumptionRate: h });
  console.log(`${String(h).padStart(19)}${(K01.writePeak - h).toFixed(2).padStart(10)}${String(a.backlog).padStart(19)}` +
    `${String(y.totalLost).padStart(16)}${String(t.rejected).padStart(19)}` +
    `${(y.totalLost / t.rejected).toFixed(3).padStart(14)}`);
}

1200 rounds, source K01 peak write 97.22/round (oscillating, average multiplier 1.00)
links: entry(cap 200, rate 250) -> log(cap 150, rate 150) -> projection(cap 100, rate consumption)

-- failure-free: consumption 120/round --
setting           completed   rejected     lost       loss location backlog  pending age finished age
unlimited buffer     116301          0        0               0/0/0     362            3            4
no signal             82282          0    34064    4164/14950/14950     317            2            3
1 link                97224          0    19114        4164/14950/0     325            3            4
2 links              112124          0     4164            4164/0/0     375            3            4
to the limit         112124       4164        0               0/0/0     375            3            4

-- AY11: consumption 60/round --
setting           completed   rejected     lost       loss location backlog  pending age finished age
unlimited buffer      71814          0        0               0/0/0   44849          460          460
no signal             71814          0    44494    4164/14950/25380     355            3            4
1 link                71814          0    44404        4164/40240/0     445            5            6
2 links               71814          0    44399           44399/0/0     450            7            8
to the limit          71814      44399        0               0/0/0     450            7            8

-- failure-free day's cost (against the unlimited buffer) --
produced 116663, valid events rejected at the limit 4164 = 3.57%
same setting with oscillation off: 0 rejections -> the source of rejection is not average load but oscillation
unlimited buffer's pending age 3 rounds, signal to the limit 3 rounds
paused link-rounds: 598/897/0; events where the source was stopped 4164
signal check cost: 3 links x 97.22 events/s = 291.66 checks/s

-- AY11's sensitivity: consumer rate (1200 rounds) --
  consumption/round gap/round  unlimited backlog  no-signal lost  rejected at limit lost/rejected
                 90      7.22               9029           34064               8579         3.971
                 60     37.22              44849           44494              44399         1.002
                 30     67.22              80753           80394              80303         1.001

All these numbers belong to the computed value class: they are counted over a deterministic arrival sequence.

Partial Propagation Moves Loss, Does Not Remove It

The second table is the lesson’s main finding. Under AY11, the lost-event count is 44,494 at depth 0, 44,404 at depth 1, 44,399 at depth 2 — nearly identical. Only loss location changes: first the projection’s gate (25,380), then the log’s gate (40,240), then the entry’s gate (44,399).

Partial backpressure rescues no event; it only relocates the loss. Two things are paid for this: the backlog climbs from 355 to 450, and the pending event’s age rises from 3 rounds to 7 — holding work back is itself a cost.

In the fourth row the number zeroes out: once the signal reaches the service boundary, 44,399 events are not lost, they are rejected. The completed count is unchanged (71,814) — whatever the consumer’s capacity is; what changes is what happens to the events that do not fit. A lost event counts as sent for the carrier and never comes again; a rejected one was never sent and can be resent.

Backpressure rescues no event until it reaches the service boundary. The background jobs topic said that unless applied across the whole chain, it counts as applied nowhere in it; this table adds which link the boundary is — the only place the signal can stop is where the system talks to the outside.

Where Propagation Must Stop

The signal cannot reach the carrier — another organization’s system this one cannot slow down, only decline. Propagation therefore ends not at the chain’s fourth link but at the third one’s outer gate, and there it changes form: inside it is a hold, at the boundary a rejection.

The first table shows the first cost. On a failure-free day — consumer faster than producer — the signal carried to the limit rejects 4164 valid events: 3.57 percent of production. With oscillation off, the same setting rejects nothing — so rejection’s source is not average load but the momentary fluctuation of arrivals. When the buffer cannot absorb that fluctuation, backpressure rounds an absorbable peak into a rejection. The same row has a second cost: 598 and 897 paused link-rounds, where a link had work but stopped because its successor was full — unused capacity.

The unlimited-buffer row pays neither cost: on a failure-free day it completes 116,301 events and rejects none. Its cost shows up under AY11 instead: no event is lost, but 44,849 pile up, and the oldest pending one is 460 rounds old — about seven and a half minutes. The projection is that far behind, and the tracking query shows a state that old. An unlimited buffer does not eliminate loss; it rounds it into staleness.

The three options write to three places: memory and staleness (unlimited buffer), loss (signal-free limited buffer), or the source (signal to the limit). The choice depends on the data — a lost status event permanently shortens the tracking chain by one entry, so the second option is unacceptable here.

As the Gap Grows, the Gain Shrinks

The last table is AY11’s sensitivity, drawing backpressure’s limit. At a consumer rate of 90 the gap is only 7.22 events per round; the signal-free arrangement loses 34,064 events while the signal to the limit rejects only 8579 — a ratio of 3.971. Here backpressure rescues three-quarters of the work, because the work piling up between peaks gets drained away.

At a rate of 60 the ratio falls to 1.002; at 30, to 1.001. As the gap grows, backpressure rescues no work at all, only turning loss into rejection. This still has value — a rejected event can be resent — but the two gains differ and must be stated explicitly. At small gaps backpressure is an absorption mechanism; at large gaps a labeling one.

Summary

• The backpressure signal carries a propagation depth: how many links back from the consumer heed their successor’s fullness. Depth 3 means the signal reaches the service boundary.
• Partial propagation rescues no event: under AY11 the lost-event count at depths 0, 1, and 2 is 44,494, 44,404, and 44,399 — the same number. Only the loss location changes, with the backlog (355 → 450) and pending age (3 → 7 rounds).
• Once the signal reaches the limit, 44,399 events stop being lost and become rejected; a lost event was sent for the carrier, a rejected one was not and can be resent.
• The failure-free cost is the peaks oscillation absorbs: 4164 valid events rejected (3.57 percent of production), zeroing out with oscillation off; plus 598 and 897 paused link-rounds and 291.66 signal checks per second.
• An unlimited buffer does not eliminate loss, it turns it into staleness: under AY11, 44,849 events pile up and the oldest pending event is 460 rounds old.
• The gain’s size depends on the gap: at a 7.22-event gap, backpressure cuts loss from 34,064 to 8579 (ratio 3.971); at 37.22 events the ratio is 1.002 — labeling, not rescuing.

Next Step

The signal reached the limit and turned into a rejection there. That was unavoidable, but the rejection itself was not designed here: when 4164 valid events were rejected, which ones, on what basis, and what was told to the other side all came down to one blunt rule — buffer full, reject. On the read side the question is sharper. At the edge there are 513.89 requests per second, and three flows share one gate: the tracking query, the carrier’s status event, the end-of-day billing request. When capacity falls short, are all three throttled together, or is one kept and the others cut. Two questions tangle here — cutting what exceeds a known rate is not the same as rejecting by current capacity. The next lesson separates the two and counts which request is kept at K01’s 513.89 requests/s peak.