--- title: 'Poison Messages and Retry' source: 'https://academia.sh/en/courses/asynchronous-processing/poison-messages-and-retry' course: 'Caching, Queues and Asynchronous Processing' language: en updated: '2026-08-23T07:00:23+00:00' license: 'CC BY-SA 4.0' --- # Poison Messages and Retry Isolating the message that brings down the consumer, and calculating wait durations: the necessity of incrementing the attempt counter at delivery time, comparing immediate, fixed, and exponential schedules by total wait and retry budget, jitter lowering the peak of the retry wave, and reconciling backoff with the visibility timeout. The previous lesson noted that the threshold said how many times to retry, but not when. This lesson's two topics come out of that gap. First, a failure type the threshold does not handle: a message that does not leave the consumer processing it standing. Second, how the wait between attempts gets calculated. Both will be measured on the same job: overdue notifications taken from the outbox. ## Poison Message The malformed messages in the previous lesson behaved politely: they threw an error, the consumer caught it, incremented the attempt counter, and moved on to the next message. A **poison message** does not do this. Processing it terminates the consumer process — a body that exhausts memory, a parse that enters an infinite loop, a failure that cannot be caught. Because the process dies, the error-handling code never runs. The result is a measurable vicious cycle. The process restarts, receives the same message from the queue, and dies again. If the attempt counter is incremented when the error is caught, it never gets incremented at all; the threshold is never reached; the dead-letter queue never kicks in. The run below makes the third message in a five-message queue lethal, and compares two different positions for the counter: one that increments when the error is caught, and one that increments at delivery time, before the work begins. ```js // poison-counter.mjs — the message that kills the process: which counter position makes isolation work const LETHAL = new Set([3]); // this message terminates the consumer process const THRESHOLD = 3; // A single process lifetime: it dies on the lethal message, the code behind it never runs. function processLifetime(queue, counterAtDelivery, dead) { let delivered = 0; for (const m of queue.filter((m) => m.status === "pending")) { delivered += 1; if (counterAtDelivery) { // counter increments at delivery time, before the work m.attempt += 1; if (m.attempt >= THRESHOLD) { m.status = "dead"; dead.push(m.id); continue; } } if (LETHAL.has(m.id)) return { delivered, died: true }; m.status = "processed"; // counter would have incremented on catch; it does not when the process dies } return { delivered, died: false }; } function run(counterAtDelivery, maxRestarts = 12) { const queue = [1, 2, 3, 4, 5].map((id) => ({ id, attempt: 0, status: "pending" })); const dead = []; let delivered = 0, deaths = 0, restarts = 0; for (let b = 1; b <= maxRestarts; b++) { restarts = b; const r = processLifetime(queue, counterAtDelivery, dead); delivered += r.delivered; if (r.died) deaths += 1; if (queue.every((m) => m.status !== "pending")) break; } return { restarts, delivered, deaths, deadLetters: dead.length, processed: queue.filter((m) => m.status === "processed").length, remaining: queue.filter((m) => m.status === "pending").length }; } console.log("counter position".padEnd(22) + "restarts delivered process deaths processed dead letters remaining"); for (const [name, atDelivery] of [["on catch", false], ["at delivery", true]]) { const s = run(atDelivery); console.log(name.padEnd(22) + `${String(s.restarts).padStart(8)}${String(s.delivered).padStart(11)}${String(s.deaths).padStart(16)}` + `${String(s.processed).padStart(11)}${String(s.deadLetters).padStart(15)}${String(s.remaining).padStart(11)}`); } ``` ```sh node poison-counter.mjs ``` ``` counter position restarts delivered process deaths processed dead letters remaining on catch 12 14 12 2 0 3 at delivery 3 7 2 4 1 0 ``` In the first row, all twelve of twelve restarts ended in death, only two messages were processed, and three stayed in the queue. The number twelve is not a result, it is the run's upper bound: the loop stopped because it was stopped there, not on its own. The dead-letter column is zero — a threshold existed, but because the counter never incremented, it was never reached. In the second row, three restarts were enough. The poison message was delivered twice; on the third delivery, because the counter had reached the threshold, it was isolated before the work even started. The two messages behind it were both processed. The queue emptied. The rule fits in one sentence: **the attempt counter must be incremented at delivery time, independent of the work's outcome.** Placing the counter on the error path only protects against cases where the error path actually runs. This is why the `attempt` field built in the previous lesson was written before the work; the same reasoning holds for the queue's own delivery count too — the delivery count we measured in lesson 02 incremented on the `receive` call, not when the work finished. Isolation bounds the poison message's cost but does not zero it out. Two process deaths happened, and whatever other work those processes were holding at the time was left half-done. This is why a second defense is used for jobs carrying poison-message risk: validating the body before it is processed. The criterion set in the Validation Layers lesson applies here too — a malformed body never enters the work at all, it is taken straight to the dead-letter queue. ## Calculating the Wait Duration In the runs so far, attempts were made back to back, with no wait between them. Most transient failures come from a resource being overloaded, and retrying without waiting loads that resource even more. The rule that gives the wait duration is called a **backoff schedule**. ```js // backoff.mjs — wait schedules, and how many attempts fit in a retry budget const BASE = 500, CAP = 30_000; // ms const SCHEDULES = { "immediate": () => 0, "fixed 2 s": () => 2000, "exponential": (n) => BASE * 2 ** (n - 1), "exponential + cap": (n) => Math.min(CAP, BASE * 2 ** (n - 1)), }; const NAMES = Object.keys(SCHEDULES); console.log("attempt" + NAMES.map((a) => a.padStart(19)).join("")); for (let n = 1; n <= 8; n++) console.log(String(n).padStart(7) + NAMES.map((a) => String(SCHEDULES[a](n)).padStart(19)).join("")); const total = (name, count) => Array.from({ length: count }, (_, i) => SCHEDULES[name](i + 1)) .reduce((t, x) => t + x, 0); console.log("total" + NAMES.map((a) => String(total(a, 8)).padStart(19)).join("")); const BUDGET = 5 * 60 * 1000; // 5-minute time budget console.log("\nschedule".padEnd(19) + "attempts fit in 5 min time elapsed when budget runs out"); for (const name of NAMES) { let elapsed = 0, attempt = 0; while (elapsed + SCHEDULES[name](attempt + 1) <= BUDGET && attempt < 1000) { attempt += 1; elapsed += SCHEDULES[name](attempt); } console.log(name.padEnd(19) + String(attempt === 1000 ? "unlimited" : attempt).padStart(23) + String(elapsed).padStart(28) + " ms"); } ``` ```sh node backoff.mjs ``` ``` attempt immediate fixed 2 s exponential exponential + cap 1 0 2000 500 500 2 0 2000 1000 1000 3 0 2000 2000 2000 4 0 2000 4000 4000 5 0 2000 8000 8000 6 0 2000 16000 16000 7 0 2000 32000 30000 8 0 2000 64000 30000 total 0 16000 127500 91500 schedule attempts fit in 5 min time elapsed when budget runs out immediate unlimited 0 ms fixed 2 s 150 300000 ms exponential 9 255500 ms exponential + cap 14 271500 ms ``` Four schedules produce four separate behaviors. Immediate retry never waits: the time budget never runs out, so only the threshold can put a limit on attempts. Far from reducing the resource's load, it concentrates the load at the moment of failure. A fixed interval is predictable, but wrong at both ends: waiting two seconds for a failure that would clear in one second is unnecessary, and retrying every two seconds during an outage that lasts a minute wears the resource out with a hundred and fifty pointless attempts. **Exponential backoff** solves both by doubling the wait duration on every attempt: if the failure is short, early attempts catch it; if it is long, attempts spread thin. Its cost shows up in the seventh row — the wait has climbed to 32 seconds. This is why a **cap** is imposed; in the capped version, total wait drops from 127.5 seconds to 91.5 seconds, and fourteen attempts fit into the five-minute budget instead of nine. The cap keeps the thinning-out from becoming unbounded. The same schedules were built for the client side in the Retry and Backoff lesson of the Application Architecture: Routing, State and Data course. The difference on the consumer side is that the party waiting is not a user but a message: queue depth keeps rising during the wait, and the delay lands not just on that message but on all the work behind it. ## Jitter Even when a backoff schedule is correct for a single consumer, it creates a new problem once a large number of consumers fail at the same time. Because they all use the same schedule, they all retry at the same time; the resource, trying to recover, meets wave after wave arriving together. The fix is adding randomness to the wait duration. This is called **jitter**. **Full jitter** picks the wait randomly between zero and the calculated duration; **decorrelated jitter** picks it, at every step, from a wider range that depends on the previous wait. The run below tracks two hundred consumers that fail at the same instant, over four attempts, and buckets the retry moments into hundred-millisecond slots. The generator is fixed-seed; the numbers are the same on every run. ```js // jitter.mjs — the retry wave of 200 consumers that fail at the same instant const generator = (seed) => { let x = seed; return () => (x = (x * 1103515245 + 12345) % 2147483648) / 2147483648; }; const CONSUMERS = 200, ATTEMPTS = 4, BUCKET = 100; // bucket width 100 ms const BASE = 500, CAP = 30_000; function wave(mode, seed = 20250720) { const rand = generator(seed); const moments = []; for (let c = 0; c < CONSUMERS; c++) { let moment = 0, previous = BASE; for (let n = 1; n <= ATTEMPTS; n++) { const base = Math.min(CAP, BASE * 2 ** (n - 1)); if (mode === "no jitter") moment += base; else if (mode === "full jitter") moment += Math.round(rand() * base); // [0, base] else { // decorrelated previous = Math.min(CAP, Math.round(BASE + rand() * (previous * 3 - BASE))); moment += previous; } moments.push(moment); } } const buckets = new Map(); for (const m of moments) { const b = Math.floor(m / BUCKET); buckets.set(b, (buckets.get(b) ?? 0) + 1); } return { requests: moments.length, peak: Math.max(...buckets.values()), filledBuckets: buckets.size, lastBucket: Math.max(...buckets.keys()) }; } console.log("schedule".padEnd(20) + "total requests peak per 100 ms filled buckets last bucket"); for (const mode of ["no jitter", "full jitter", "decorrelated jitter"]) { const s = wave(mode); console.log(mode.padEnd(20) + String(s.requests).padStart(15) + String(s.peak).padStart(17) + String(s.filledBuckets).padStart(17) + String(s.lastBucket).padStart(14)); } ``` ```sh node jitter.mjs ``` ``` schedule total requests peak per 100 ms filled buckets last bucket no jitter 800 200 4 75 full jitter 800 69 65 67 decorrelated jitter 800 28 163 389 ``` Across all three schedules, the total request count is the same: eight hundred. The only thing that changes is how those requests spread out over time. In the no-jitter schedule, eight hundred requests fit into just four buckets, with two hundred requests per bucket — all two hundred consumers knock on the door in the same hundred milliseconds. Full jitter brings the peak down to 69 and spreads the requests over sixty-five buckets. Decorrelated jitter brings the peak down to 28, but the last bucket is 389 — meaning the spread stretches out to thirty-nine seconds. This is where the trade-off sits. Jitter lowers the burst load the resource sees; in exchange, some messages' retries run late. If the peak of the load is large enough to bring the resource down again, jitter is mandatory; if the job is highly sensitive to delay, the spread is kept narrow. ## Retry Budget and Visibility Timeout A retry rule is defined together with two limits. The **retry budget** looks at count: the threshold. The time budget looks at duration: how long after the first delivery a message gets taken to the dead-letter queue. Both are needed together, because the threshold alone, combined with backoff, makes the duration unpredictable — in the capped exponential schedule, fourteen attempts took longer than four and a half minutes; uncapped, nine attempts reached the same duration. The second reconciliation is with the queue itself. The visibility timeout built in lesson 02 said how long a received message stays withheld from others. If the consumer holds onto the message and waits through the backoff duration, the moment that wait exceeds the visibility timeout, the message becomes visible to another consumer and the same job runs concurrently. The correct placement is for the consumer not to hold the message while waiting: the message is put back into the queue, and the moment it becomes visible again is pushed forward by the backoff duration. This way the wait is kept on the queue's own timeline, and the consumer process does other work in the meantime. A third limit concerns the failure rate. When a resource becomes wholly unreachable, every message gets the same error, and retries by themselves turn into a source of load. In this case, what needs to stop is not the schedule of individual messages but the whole consumer; this is the problem the **circuit breaker** pattern, introduced in the Retry and Backoff lesson mentioned above, solves. ## Summary - A poison message is not merely a message that fails; because it terminates the consumer process, it never runs the error-handling code at all. - With the counter incremented on catch, all twelve of twelve restarts ended in death, two messages were processed, and three stayed in the queue; incremented at delivery time, three restarts were enough, the message was isolated on its third delivery, and the queue emptied. - Capped exponential backoff brought total wait over eight attempts down from 127.5 seconds to 91.5 seconds, and fit fourteen attempts into the five-minute budget instead of nine. - Two hundred consumers' eight hundred retries piled into four buckets under the no-jitter schedule with a peak of 200; full jitter brought the peak down to 69, decorrelated jitter to 28, stretching the spread out to thirty-nine seconds. - The retry budget and the time budget are defined together; the backoff duration must not exceed the visibility timeout, and the wait must be kept on the queue's timeline, not the consumer's. ## Next Step The message is now durable. It is written to a persistent place, reaches more than one interested party, keeps its order where order is needed, has its repetition made idempotent, has what it cannot process isolated, and has its retries tied to a measured schedule. Across all these measurements, one side was always taken as a given: the consumer processing the message. How many worker processes are there, how many messages does one finish per second, what is gained by raising that number, what happens when the production rate exceeds their capacity? The next topic treats the consumer as a unit of scaling, and it starts with the Worker Processes lesson.