--- title: Backpressure source: 'https://academia.sh/en/courses/resilience-patterns/backpressure' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:30+00:00' license: 'CC BY-SA 4.0' --- # Backpressure The overload signal propagating backward from the service boundary: measuring propagation depth across four settings from zero to the limit, partial propagation only moving the loss one link back, the signal reaching the limit turning an accepted-then-lost write into a never-accepted write, and an unlimited buffer turning loss into staleness. The previous lesson set the budget to a single request's duration and taught how to reclaim its resource on time. One thing the budget never touches: the **rate at which requests arrive**. Facing a slowing dependency, the budget cuts each request off, but the next one still arrives and spends the same budget on the same slow step. The budget says how long a request waits; it says nothing to whoever feeds the system work. This lesson asks that missing notice on the write path. Status events from the carrier flow in at a fixed rate; if the consumer reading the event log and feeding the read model falls behind, work piles up somewhere. The Caching, Queues and Asynchronous Processing course established the **single-queue** case — a bounded queue as a latency-budget decision, overflow policies, blocking passing the slowdown to the producer, and an unaware layer in front emptying out the limit — not retold here. The question here is the chain: how many **links back** the signal is carried, where work collects where it is not, and where propagation must stop. ## Failure Mode and Chain The write path has four parts: the carrier's notification endpoint sends the event, the **entry** service accepts it, the **log** writer persists it, the **projection** consumer updates the read model. The source sits outside the system, beyond its control — the carrier belongs to another organization. **AY11 — the projection consumer falls behind.** It does not go down or write incorrectly; its speed drops from 120 events per round to 60. Duration **20 minutes**, frequency **three times a month**. Rationale: consumer slowdown usually comes from a projection schema change or a rebuild job overlap, lasting until that job finishes. Its sensitivity is below, not added to K01's table. AY11 is not an outage; the tracking query keeps answering, only the projection it reads falls behind, so it is not written to K01's monthly 43.2-minute downtime budget. A lost write goes into a different measure, its threshold set in this course's recovery topic. A round is an abstract step, corresponding in the model to a one-second window, and this cycle coefficient is an assumption. The source rate is K01's peak write count: 97.22 events per round. Arrival is not flat but oscillating — a multiplier cycle averaging 1.00 represents fluctuation within the peak hour. ## Propagation Depth The signal is a single piece of information carried backward from the consumer: "my successor is full, I cannot take your work." Each link can heed this or ignore it. In the model this is a parameter — **depth**, how many links back it travels from the consumer. At depth 0 no link heeds it; at depth 3 it reaches the service boundary and the carrier's event **is not accepted**. ```js // backpressure/chain.mjs — MODEL of the write path. A round is an abstract step; the cycle // coefficient is an assumption (one round = one second). No real queue, log, or consumer is set up. export const K01 = { writePeak: 97.22 }; // K01: carrier state event, peak write/s // Links are ordered from source to tip. `rate` is the number of events processed in one round. export const LINK = [ { name: "entry", capacity: 200, rate: 250 }, // event admission endpoint { name: "log", capacity: 150, rate: 150 }, // event log writer { name: "projection", capacity: 100, rate: 120 }, // consumer feeding the read model ]; // Oscillation of arrivals within the peak hour: a multiplier cycle averaging 1.00. export const OSCILLATION = [0.6, 0.6, 0.6, 2.2]; // Link i pauses if it heeds its successor's fullness; if not, it drops the event. // depth: how many links back from the consumer the signal is carried. 3 = to the limit. const heeds = (i, depth) => depth >= LINK.length - 1 - i; export function run({ roundCount, depth, consumptionRate, oscillate = true, unlimited = false }) { const rate = LINK.map((h, i) => (i === LINK.length - 1 ? consumptionRate : h.rate)); const capacity = LINK.map((h) => (unlimited ? Infinity : h.capacity)); const buffer = LINK.map(() => []); const s = { produced: 0, completed: 0, rejected: 0, finishedAge: 0, sourceStopped: 0, lost: LINK.map(() => 0), paused: LINK.map(() => 0), }; let carry = 0; for (let t = 1; t <= roundCount; t += 1) { for (let i = LINK.length - 1; i >= 0; i -= 1) { // drain from the tip toward the source let n = Math.min(rate[i], buffer[i].length); while (n > 0) { if (i === LINK.length - 1) { // last link: work completes s.finishedAge = Math.max(s.finishedAge, t - buffer[i].shift()); s.completed += 1; n -= 1; continue; } if (buffer[i + 1].length < capacity[i + 1]) { buffer[i + 1].push(buffer[i].shift()); n -= 1; continue; } if (heeds(i, depth)) { s.paused[i] += 1; break; } // successor full: pause s.lost[i + 1] += 1; buffer[i].shift(); n -= 1; // no signal: lose it } } const multiplier = oscillate ? OSCILLATION[(t - 1) % OSCILLATION.length] : 1; carry += K01.writePeak * multiplier; while (carry >= 1) { // source: carrier notification endpoint carry -= 1; s.produced += 1; if (buffer[0].length < capacity[0]) { buffer[0].push(t); continue; } if (depth >= LINK.length) { // rejected at the limit: event not accepted s.rejected += 1; s.sourceStopped += 1; } else s.lost[0] += 1; // accepted, then lost } } s.backlog = buffer.reduce((a, b) => a + b.length, 0); const pending = buffer.flat(); s.pendingAge = pending.length === 0 ? 0 : roundCount - Math.min(...pending); s.totalLost = s.lost.reduce((a, b) => a + b, 0); return s; } ``` Two counters are kept separate, and the lesson's whole distinction is here. A **lost** event is one the system accepted, then dropped; the carrier counts it sent and never resends it. A **rejected** event is never accepted at the boundary; the carrier knows this and resends it under its own policy. ```js // backpressure/run.mjs — signal propagation depth: five settings across two days import { K01, LINK, run } from "./chain.mjs"; const ROUNDS = 1200; // AY11: 20 minutes, one round = one second const CONSUMPTION_NORMAL = 120, CONSUMPTION_SLOW = 60; // projection consumer's rate per round const SETTING = [ ["unlimited buffer", { depth: 0, unlimited: true }], ["no signal", { depth: 0 }], ["1 link", { depth: 1 }], ["2 links", { depth: 2 }], ["to the limit", { depth: 3 }], ]; console.log(`${ROUNDS} rounds, source K01 peak write ${K01.writePeak}/round (oscillating, average multiplier 1.00)`); console.log(`links: ${LINK.map((h, i) => `${h.name}(cap ${h.capacity}, rate ${i === 2 ? "consumption" : h.rate})`).join(" -> ")}\n`); for (const [regime, consumptionRate] of [["failure-free: consumption 120/round", CONSUMPTION_NORMAL], ["AY11: consumption 60/round", CONSUMPTION_SLOW]]) { console.log(`-- ${regime} --`); console.log(`${"setting".padEnd(16)}${"completed".padStart(11)}${"rejected".padStart(11)}${"lost".padStart(9)}` + `${"loss location".padStart(20)}${"backlog".padStart(8)}${"pending age".padStart(13)}${"finished age".padStart(13)}`); for (const [label, y] of SETTING) { const r = run({ roundCount: ROUNDS, consumptionRate, ...y }); console.log(`${label.padEnd(16)}${String(r.completed).padStart(11)}${String(r.rejected).padStart(11)}` + `${String(r.totalLost).padStart(9)}${r.lost.join("/").padStart(20)}${String(r.backlog).padStart(8)}` + `${String(r.pendingAge).padStart(13)}${String(r.finishedAge).padStart(13)}`); } console.log(); } const unlimited = run({ roundCount: ROUNDS, depth: 0, unlimited: true, consumptionRate: CONSUMPTION_NORMAL }); const signaled = run({ roundCount: ROUNDS, depth: 3, consumptionRate: CONSUMPTION_NORMAL }); const flat = run({ roundCount: ROUNDS, depth: 3, consumptionRate: CONSUMPTION_NORMAL, oscillate: false }); console.log(`-- failure-free day's cost (against the unlimited buffer) --`); console.log(`produced ${signaled.produced}, valid events rejected at the limit ${signaled.rejected}` + ` = ${((signaled.rejected / signaled.produced) * 100).toFixed(2)}%`); console.log(`same setting with oscillation off: ${flat.rejected} rejections -> the source of rejection is not average load but oscillation`); console.log(`unlimited buffer's pending age ${unlimited.pendingAge} rounds, signal to the limit ${signaled.pendingAge} rounds`); console.log(`paused link-rounds: ${signaled.paused.join("/")}; events where the source was stopped ${signaled.sourceStopped}`); console.log(`signal check cost: ${LINK.length} links x ${K01.writePeak} events/s = ${(LINK.length * K01.writePeak).toFixed(2)} checks/s`); console.log(`\n-- AY11's sensitivity: consumer rate (1200 rounds) --`); console.log(`${"consumption/round".padStart(19)}${"gap/round".padStart(10)}${"unlimited backlog".padStart(19)}` + `${"no-signal lost".padStart(16)}${"rejected at limit".padStart(19)}${"lost/rejected".padStart(14)}`); for (const h of [90, 60, 30]) { const a = run({ roundCount: ROUNDS, depth: 0, unlimited: true, consumptionRate: h }); const y = run({ roundCount: ROUNDS, depth: 0, consumptionRate: h }); const t = run({ roundCount: ROUNDS, depth: 3, consumptionRate: h }); console.log(`${String(h).padStart(19)}${(K01.writePeak - h).toFixed(2).padStart(10)}${String(a.backlog).padStart(19)}` + `${String(y.totalLost).padStart(16)}${String(t.rejected).padStart(19)}` + `${(y.totalLost / t.rejected).toFixed(3).padStart(14)}`); } ``` ``` 1200 rounds, source K01 peak write 97.22/round (oscillating, average multiplier 1.00) links: entry(cap 200, rate 250) -> log(cap 150, rate 150) -> projection(cap 100, rate consumption) -- failure-free: consumption 120/round -- setting completed rejected lost loss location backlog pending age finished age unlimited buffer 116301 0 0 0/0/0 362 3 4 no signal 82282 0 34064 4164/14950/14950 317 2 3 1 link 97224 0 19114 4164/14950/0 325 3 4 2 links 112124 0 4164 4164/0/0 375 3 4 to the limit 112124 4164 0 0/0/0 375 3 4 -- AY11: consumption 60/round -- setting completed rejected lost loss location backlog pending age finished age unlimited buffer 71814 0 0 0/0/0 44849 460 460 no signal 71814 0 44494 4164/14950/25380 355 3 4 1 link 71814 0 44404 4164/40240/0 445 5 6 2 links 71814 0 44399 44399/0/0 450 7 8 to the limit 71814 44399 0 0/0/0 450 7 8 -- failure-free day's cost (against the unlimited buffer) -- produced 116663, valid events rejected at the limit 4164 = 3.57% same setting with oscillation off: 0 rejections -> the source of rejection is not average load but oscillation unlimited buffer's pending age 3 rounds, signal to the limit 3 rounds paused link-rounds: 598/897/0; events where the source was stopped 4164 signal check cost: 3 links x 97.22 events/s = 291.66 checks/s -- AY11's sensitivity: consumer rate (1200 rounds) -- consumption/round gap/round unlimited backlog no-signal lost rejected at limit lost/rejected 90 7.22 9029 34064 8579 3.971 60 37.22 44849 44494 44399 1.002 30 67.22 80753 80394 80303 1.001 ``` All these numbers belong to the **computed value** class: they are counted over a deterministic arrival sequence. ## Partial Propagation Moves Loss, Does Not Remove It The second table is the lesson's main finding. Under AY11, the lost-event count is 44,494 at depth 0, 44,404 at depth 1, 44,399 at depth 2 — nearly identical. Only `loss location` changes: first the projection's gate (25,380), then the log's gate (40,240), then the entry's gate (44,399). Partial backpressure rescues no event; it only relocates the loss. Two things are paid for this: the backlog climbs from 355 to 450, and the pending event's age rises from 3 rounds to 7 — holding work back is itself a cost. In the fourth row the number zeroes out: once the signal reaches the service boundary, 44,399 events are not **lost**, they are **rejected**. The completed count is unchanged (71,814) — whatever the consumer's capacity is; what changes is what happens to the events that do not fit. A lost event counts as sent for the carrier and never comes again; a rejected one was never sent and can be resent. **Backpressure rescues no event until it reaches the service boundary.** The background jobs topic said that unless applied across the whole chain, it counts as applied nowhere in it; this table adds which link the boundary is — the only place the signal can stop is where the system talks to the outside. ## Where Propagation Must Stop The signal cannot reach the carrier — another organization's system this one cannot slow down, only decline. Propagation therefore ends not at the chain's fourth link but at the third one's outer gate, and there it changes form: **inside it is a hold, at the boundary a rejection.** The first table shows the first cost. On a failure-free day — consumer faster than producer — the signal carried to the limit rejects 4164 valid events: 3.57 percent of production. With oscillation off, the same setting rejects nothing — so rejection's source is not average load but the momentary fluctuation of arrivals. When the buffer cannot absorb that fluctuation, backpressure rounds an absorbable peak into a rejection. The same row has a second cost: 598 and 897 paused link-rounds, where a link had work but stopped because its successor was full — unused capacity. The unlimited-buffer row pays neither cost: on a failure-free day it completes 116,301 events and rejects none. Its cost shows up under AY11 instead: no event is lost, but 44,849 pile up, and the oldest pending one is **460 rounds** old — about seven and a half minutes. The projection is that far behind, and the tracking query shows a state that old. An unlimited buffer does not eliminate loss; it rounds it into **staleness**. The three options write to three places: memory and staleness (unlimited buffer), loss (signal-free limited buffer), or the source (signal to the limit). The choice depends on the data — a lost status event permanently shortens the tracking chain by one entry, so the second option is unacceptable here. ## As the Gap Grows, the Gain Shrinks The last table is AY11's sensitivity, drawing backpressure's limit. At a consumer rate of 90 the gap is only 7.22 events per round; the signal-free arrangement loses 34,064 events while the signal to the limit rejects only 8579 — a ratio of 3.971. Here backpressure **rescues** three-quarters of the work, because the work piling up between peaks gets drained away. At a rate of 60 the ratio falls to 1.002; at 30, to 1.001. As the gap grows, backpressure rescues no work at all, only turning loss into rejection. This still has value — a rejected event can be resent — but the two gains differ and must be stated explicitly. At small gaps backpressure is an **absorption** mechanism; at large gaps a **labeling** one. ## Summary - The backpressure signal carries a propagation depth: how many links back from the consumer heed their successor's fullness. Depth 3 means the signal reaches the service boundary. - Partial propagation rescues no event: under AY11 the lost-event count at depths 0, 1, and 2 is 44,494, 44,404, and 44,399 — the same number. Only the loss location changes, with the backlog (355 → 450) and pending age (3 → 7 rounds). - Once the signal reaches the limit, 44,399 events stop being lost and become rejected; a lost event was sent for the carrier, a rejected one was not and can be resent. - The failure-free cost is the peaks oscillation absorbs: 4164 valid events rejected (3.57 percent of production), zeroing out with oscillation off; plus 598 and 897 paused link-rounds and 291.66 signal checks per second. - An unlimited buffer does not eliminate loss, it turns it into staleness: under AY11, 44,849 events pile up and the oldest pending event is 460 rounds old. - The gain's size depends on the gap: at a 7.22-event gap, backpressure cuts loss from 34,064 to 8579 (ratio 3.971); at 37.22 events the ratio is 1.002 — labeling, not rescuing. ## Next Step The signal reached the limit and turned into a rejection there. That was unavoidable, but the rejection **itself** was not designed here: when 4164 valid events were rejected, which ones, on what basis, and what was told to the other side all came down to one blunt rule — buffer full, reject. On the read side the question is sharper. At the edge there are 513.89 requests per second, and three flows share one gate: the tracking query, the carrier's status event, the end-of-day billing request. When capacity falls short, are all three throttled together, or is one kept and the others cut. Two questions tangle here — cutting what exceeds a known rate is not the same as rejecting by current capacity. The next lesson separates the two and counts which request is kept at K01's 513.89 requests/s peak.