--- title: 'Dead-Letter Queues' source: 'https://academia.sh/en/courses/asynchronous-processing/dead-letter-queues' course: 'Caching, Queues and Asynchronous Processing' language: en updated: '2026-08-23T07:00:23+00:00' license: 'CC BY-SA 4.0' --- # Dead-Letter Queues Isolating messages that will never succeed: unbounded retry letting the wasted-attempt count grow without limit, an attempt counter and threshold draining the queue, the fields a dead-letter record must carry, redriving messages once their cause has been fixed, and the trade-off between the threshold value and wrongly isolating a transient failure. Every measurement so far quietly carried one assumption: a retried job eventually succeeds. The crash was transient, the second attempt finished the job. In reality, some messages fail on the second attempt too, and the tenth — their body cannot be parsed, the member record they reference has been deleted, the business rule they carry will never accept them. In at-least-once mode, a message like this never leaves the queue at all. This lesson measures how much damage that message causes, and builds the structure that removes it from the queue and isolates it. ## Permanent Failure versus Transient Failure The distinction rests on whether retrying is meaningful. A **transient failure** is one that disappears once conditions change: a database connection dropping, an outside service being briefly unreachable, two writes colliding. Retrying is the correct response to these failures. A **permanent failure** comes not from conditions but from the message itself. The same message, run through the same code, produces the same result on every attempt; retrying only repeats the failure. When both failure types share the same path, the only way to separate out the permanent one is to look at the attempt count. The measurement setup builds two relations: the job queue holding messages and attempt counters, and the dead-letter queue holding isolated messages. ```sh # setup.sh — job queue and dead-letter queue relations rm -f queue.db sqlite3 queue.db <<'SQL' CREATE TABLE job_queue (message_id INTEGER PRIMARY KEY, type TEXT NOT NULL, body TEXT NOT NULL, attempt INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'pending'); CREATE TABLE dead_letter (dead_letter_id INTEGER PRIMARY KEY, message_id INTEGER NOT NULL, type TEXT NOT NULL, body TEXT NOT NULL, attempt INTEGER NOT NULL, last_error TEXT NOT NULL, source TEXT NOT NULL); SQL sqlite3 queue.db "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name" ``` ``` dead_letter job_queue ``` ## The Cost of Unbounded Retry Twenty overdue notifications will be processed; two of them — messages 7 and 13 — reference a member record that does not exist, so they will fail on every attempt. The run below runs the same workload under three settings: eight rounds with no threshold, twenty-four rounds with no threshold, and eight rounds with a three-attempt threshold. ```js // threshold.mjs — comparing unbounded retry with threshold-based isolation on the same workload import { DatabaseSync } from "node:sqlite"; const PERMANENT = new Set([7, 13]); // messages with a malformed body: fail on every attempt const COUNT = 20; function reset(db) { db.exec("DELETE FROM job_queue; DELETE FROM dead_letter"); for (let n = 1; n <= COUNT; n++) db.prepare("INSERT INTO job_queue (message_id, type, body) VALUES (?,?,?)") .run(n, "overdue_notification", JSON.stringify({ loanId: n })); } function process(m) { if (PERMANENT.has(m.message_id)) throw new Error(`member not found (loan ${m.message_id})`); } function run(db, threshold, roundCount) { // threshold = 0 -> no isolation reset(db); let delivered = 0, succeeded = 0, wasted = 0; for (let round = 1; round <= roundCount; round++) { for (const m of db.prepare("SELECT * FROM job_queue WHERE status='pending' ORDER BY message_id").all()) { delivered += 1; const attempt = m.attempt + 1; db.prepare("UPDATE job_queue SET attempt=? WHERE message_id=?").run(attempt, m.message_id); try { process(m); db.prepare("UPDATE job_queue SET status='processed' WHERE message_id=?").run(m.message_id); succeeded += 1; } catch (e) { wasted += 1; if (threshold > 0 && attempt >= threshold) { db.prepare(`INSERT INTO dead_letter (message_id, type, body, attempt, last_error, source) VALUES (?,?,?,?,?,'job_queue')`) .run(m.message_id, m.type, m.body, attempt, e.message); db.prepare("UPDATE job_queue SET status='dead' WHERE message_id=?").run(m.message_id); } } } } const count = (s) => db.prepare(s).get().n; return { delivered, succeeded, wasted, depth: count("SELECT count(*) AS n FROM job_queue WHERE status='pending'"), dead: count("SELECT count(*) AS n FROM dead_letter") }; } const db = new DatabaseSync("queue.db"); console.log("mode".padEnd(13) + "rounds delivered succeeded wasted remaining depth dead letters"); for (const [name, threshold, rounds] of [["unbounded", 0, 8], ["unbounded", 0, 24], ["threshold=3", 3, 8]]) { const s = run(db, threshold, rounds); console.log(name.padEnd(13) + `${String(rounds).padStart(6)}${String(s.delivered).padStart(11)}${String(s.succeeded).padStart(11)}` + `${String(s.wasted).padStart(8)}${String(s.depth).padStart(17)}${String(s.dead).padStart(14)}`); } ``` ```sh node threshold.mjs ``` ``` mode rounds delivered succeeded wasted remaining depth dead letters unbounded 8 34 18 16 2 0 unbounded 24 66 18 48 2 0 threshold=3 8 24 18 6 0 2 ``` The first two rows are snapshots of the same workload at two different durations, and their only difference is the wasted-attempt count: 16 over eight rounds, 48 over twenty-four. Successful work stays fixed at 18 in both, remaining depth stays at 2 in both. Because the two malformed messages never finish, they get redelivered every round; the capacity spent grows directly proportional to round count, with no upper bound. The third row shows what the threshold does. Wasted attempts stopped at 6 — the two messages were tried three times each and removed from the queue. Total delivery is 24 instead of 34. The gain is not only in the count: remaining depth dropped from two to zero, meaning the queue is **finished**. The two malformed messages have not been lost — they have been moved somewhere separate. In the previous lesson's partitioned arrangement, the same message costs even more. If a malformed message sits at the head of a partition that preserves order, every message behind it waits in that partition; the ordering guarantee does not permit skipping the failed message. The threshold is the only mechanism that gets the partition flowing again. ## What a Dead-Letter Record Must Carry The destination a message goes to once it crosses the threshold is called the **dead-letter queue**. The reason it is a queue is that what sits there is not a log entry but still a processable **message**. This is why the record carries at least five fields: the message's body as-is, its type, how many times it was tried, the last error's text, and which queue it came from. Without the body, the message cannot be redriven; without the error text, investigating the cause requires repeating the attempt. ```js // redrive.mjs — the dead-letter record's contents, and redriving messages once the cause is fixed import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("queue.db"); let broken = new Set([7, 13]); const process = (m) => { if (broken.has(m.message_id)) throw new Error(`member not found (loan ${m.message_id})`); }; db.exec("DELETE FROM job_queue; DELETE FROM dead_letter"); for (let n = 1; n <= 20; n++) db.prepare("INSERT INTO job_queue (message_id, type, body) VALUES (?,?,?)") .run(n, "overdue_notification", JSON.stringify({ loanId: n })); function drain(threshold = 3) { let processed = 0; for (let round = 1; round <= threshold; round++) for (const m of db.prepare("SELECT * FROM job_queue WHERE status='pending' ORDER BY message_id").all()) { const attempt = m.attempt + 1; db.prepare("UPDATE job_queue SET attempt=? WHERE message_id=?").run(attempt, m.message_id); try { process(m); db.prepare("UPDATE job_queue SET status='processed' WHERE message_id=?").run(m.message_id); processed += 1; } catch (e) { if (attempt >= threshold) { // threshold crossed: message is isolated db.prepare(`INSERT INTO dead_letter (message_id, type, body, attempt, last_error, source) VALUES (?,?,?,?,?,'job_queue')`) .run(m.message_id, m.type, m.body, attempt, e.message); db.prepare("UPDATE job_queue SET status='dead' WHERE message_id=?").run(m.message_id); } } } return processed; } const count = (s) => db.prepare(`SELECT count(*) AS n FROM ${s}`).get().n; console.log(`first pass: processed=${drain()} pending=${count("job_queue WHERE status='pending'")}`); for (const r of db.prepare("SELECT * FROM dead_letter ORDER BY message_id").all()) console.log(` message=${r.message_id} type=${r.type} attempt=${r.attempt} source=${r.source} ` + `body=${r.body} last_error=${r.last_error}`); broken = new Set(); // cause fixed: missing member records were completed const redriven = db.prepare("SELECT * FROM dead_letter ORDER BY message_id").all(); for (const r of redriven) { db.prepare("UPDATE job_queue SET status='pending', attempt=0 WHERE message_id=?").run(r.message_id); db.prepare("DELETE FROM dead_letter WHERE dead_letter_id=?").run(r.dead_letter_id); } console.log(`redriven=${redriven.length} second pass: processed=${drain()}`); console.log(`result: processed=${count("job_queue WHERE status='processed'")} ` + `pending=${count("job_queue WHERE status='pending'")} dead letters=${count("dead_letter")}`); ``` ```sh node redrive.mjs ``` ``` first pass: processed=18 pending=0 message=7 type=overdue_notification attempt=3 source=job_queue body={"loanId":7} last_error=member not found (loan 7) message=13 type=overdue_notification attempt=3 source=job_queue body={"loanId":13} last_error=member not found (loan 13) redriven=2 second pass: processed=2 result: processed=20 pending=0 dead letters=0 ``` The dead-letter record itself gives the diagnosis of the failure: two messages of the same type, the same error text, three attempts each. After the missing member records were completed, the two messages were redriven back to the job queue, their attempt counters were reset, and they were processed on the second pass. All twenty of the twenty messages ended up processed. **Redrive** has two conditions. The attempt counter must be reset, or else the message, having already crossed the threshold, gets isolated again on its first failure. And the cause must have genuinely been fixed; a redrive done without fixing it sends the same messages back to the dead-letter queue with the same number of attempts. The second condition makes it mandatory that redriving be tied to a diagnosis, not a button. ## Choosing the Threshold Value The threshold cannot tell two failures apart; it only looks at the attempt count. Its value is therefore a choice between two flaws: a small threshold mistakes a transient failure that would clear on the fourth attempt for a permanent one and isolates it; a large threshold tries a permanently failing message more times than it needs to. The run below sets up two of twenty messages as permanently broken, five as clearing on the third attempt, and four as clearing on the fifth, and varies the threshold from one to eight. ```js // threshold-selection.mjs — the threshold value: choosing between wrongful isolation and wasted attempts const MESSAGES = []; // needed = the attempt number that succeeds, 0 = never for (let n = 1; n <= 20; n++) { const needed = n === 7 || n === 13 ? 0 : (n % 5 === 0 ? 5 : (n % 3 === 0 ? 3 : 1)); MESSAGES.push({ message_id: n, needed }); } function run(threshold) { let wasted = 0, wronglyIsolated = 0, permanentlyIsolated = 0, processed = 0; for (const m of MESSAGES) { for (let attempt = 1; attempt <= threshold; attempt++) { if (m.needed !== 0 && attempt >= m.needed) { processed += 1; break; } wasted += 1; if (attempt === threshold) { // threshold reached: message goes to the dead-letter queue if (m.needed === 0) permanentlyIsolated += 1; else wronglyIsolated += 1; } } } return { processed, wasted, wronglyIsolated, permanentlyIsolated }; } console.log(`messages=${MESSAGES.length} permanently broken=${MESSAGES.filter((m) => m.needed === 0).length} ` + `clears on attempt 3=${MESSAGES.filter((m) => m.needed === 3).length} ` + `clears on attempt 5=${MESSAGES.filter((m) => m.needed === 5).length}`); console.log("threshold processed wasted attempts wrongly isolated permanently isolated"); for (const threshold of [1, 2, 3, 5, 8]) { const s = run(threshold); console.log(`${String(threshold).padStart(9)}${String(s.processed).padStart(11)}${String(s.wasted).padStart(17)}` + `${String(s.wronglyIsolated).padStart(18)}${String(s.permanentlyIsolated).padStart(21)}`); } ``` ```sh node threshold-selection.mjs ``` ``` messages=20 permanently broken=2 clears on attempt 3=5 clears on attempt 5=4 threshold processed wasted attempts wrongly isolated permanently isolated 1 9 11 9 2 2 9 22 9 2 3 14 28 4 2 5 18 36 0 2 8 18 42 0 2 ``` The table moves monotonically in both directions. As the threshold grows, the wrongly isolated message count drops from 9 to 0, and the wasted-attempt count rises from 11 to 42. Processed work saturates at a threshold of five: raising it to eight gains not a single message, it only tries the two permanently broken messages three more times each. This also tells us how to choose the threshold. The right value sits a little above the highest attempt count the work genuinely needs; anything past that is pure waste. The "attempt count genuinely needed" is not guessed, it is measured — the distribution of attempt counts across successful jobs gives this value directly. ## Isolating Is Not Solving The danger of a dead-letter queue is that it makes the problem invisible. The messages are there, the queue looks clean, the metrics look healthy — and three members never received their notification. Isolation only stops the loss from being silent if it is monitored. Three rules provide that. Dead-letter queue depth is a monitored metric, and any value other than zero produces an alert; unlike ordinary queue depth, there is no "normal" baseline here. Every record has an owner: whichever team produces the message also examines the isolated one. And records have an age — a dead letter sitting unexamined for months means it is never going to be examined. **Clustering** the isolated messages is also a diagnostic tool. A large number of records carrying the same error text points to a single root cause and does not need to be examined one by one; scattered errors say each one is its own case. This is the real payoff of storing the error text in the dead-letter record. ## Summary - A transient failure disappears once conditions change and responds to retrying; a permanent failure comes from the message itself and produces the same result on every attempt. - With unbounded retry, the two malformed messages produced 16 wasted attempts over eight rounds and 48 over twenty-four; successful work stayed at 18 in both, and the capacity spent had no upper bound. - With a three-attempt threshold, wasted attempts stopped at 6, the queue emptied, and two messages moved to the dead-letter queue; in a partitioned arrangement, the threshold gets the partition a malformed message halted flowing again. - A dead-letter record carries the body, the type, the attempt count, the last error text, and the source queue; once the cause was fixed, two redriven messages with reset attempt counters were processed. - As the threshold grew, wrongly isolated messages dropped from 9 to 0 and wasted attempts rose from 11 to 42; the right value sits a little above the highest attempt count successful jobs actually need. ## Next Step The threshold said how many times to retry; it did not say when to retry. In earlier measurements, attempts were made back to back, with no wait in between. Most transient failures come from a resource being overloaded, and retrying without waiting loads that resource even more; worse, hundreds of consumers failing at the same moment and retrying at the same moment pile the load onto a single point. The next lesson establishes how wait durations are calculated: exponential backoff, an upper bound, and the jitter that spreads out the wave of retries.