--- title: 'Failure Modes' source: 'https://academia.sh/en/courses/observability/failure-modes' course: 'Observability and Reliability' language: en updated: '2026-08-23T07:00:27+00:00' license: 'CC BY-SA 4.0' --- # Failure Modes What failure modes correspond to in application code: the same endpoint breaking eight separate ways, producing five different error identities at the call site, the pattern stack seeing five of them and missing three entirely, and the same zero value reaching the call site under six modes. Six patterns were built in this topic. Each was written against a way of breaking, each has a place in the code, and each had its uncovered call sites counted. What the patterns do is clear; which failure they correspond to was never spelled out one by one. The failure-mode vocabulary — how many ways a dependency can break, and each one's effect on the system — was built in **Resilience and Reliability**. The question here is different: **what does each mode appear as in application code.** Does the call site get an error object, a timeout, an empty body, or a valid response showing no symptom at all. This is the topic's closing lesson, and it counts the patterns' total coverage. ## Producing Eight Modes in Code **RS15 — eight modes.** A single billing endpoint breaks eight separate ways: connection refusal, a delayed response, a server error, an empty body, a body cut off partway through, a field with the right shape but wrong type, a valid and correctly typed but wrong value, and slowness just under the timeout. The last three all return HTTP 200 and valid JSON. The actual balance is 12.5, and the late fee adds five to it. **RS16 — the stack's settings.** Timeout 120 ms, retry count 2, breaker threshold 3 consecutive failures, 5 calls per mode. These values were measured in earlier lessons; here they are held fixed. ```js // mode/mode.mjs — a dependency's eight failure modes and the patterns stacked on top. import http from "node:http"; export const ACTUAL = 12.5, TIMEOUT = 120, RETRIES = 2, THRESHOLD = 3, CALLS = 5; const send = (res, g) => { res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify(g)); }; // Every mode breaks the same endpoint a different way. The last two return HTTP 200 and valid JSON. export const MODE = { "connection-refused": null, // no listener "timeout": (res) => setTimeout(() => send(res, { balance: ACTUAL }), 400), "server-error": (res) => { res.writeHead(500, { "content-type": "application/json" }); res.end("{}"); }, "empty-body": (res) => { res.writeHead(200, { "content-type": "application/json" }); res.end(""); }, "truncated-body": (res) => { res.writeHead(200, { "content-type": "application/json", "content-length": "20" }); res.write('{"balance":'); res.socket.destroy(); }, "wrong-type": (res) => send(res, { balance: String(ACTUAL.toFixed(2)) }), "silent-wrong": (res) => send(res, { balance: 0 }), "slow-under-limit": (res) => setTimeout(() => send(res, { balance: ACTUAL }), 90), }; export function breaker() { let consecutive = 0; return { isOpen: () => consecutive >= THRESHOLD, success() { consecutive = 0; }, failure() { consecutive += 1; } }; } // Timeout + retry + circuit breaker in a single wrapper. Which one triggers is written to the trail. export async function guarded(port, k, trail) { if (k.isOpen()) { trail.add("breaker"); throw new Error("circuit open"); } let last; for (let d = 1; d <= RETRIES; d += 1) { try { const y = await fetch(`http://127.0.0.1:${port}/`, { signal: AbortSignal.timeout(TIMEOUT) }); if (y.ok === false) { const e = new Error(`HTTP ${y.status}`); e.name = `HTTP ${y.status}`; throw e; } const g = await y.json(); k.success(); return g; } catch (e) { last = e; trail.add(e.name === "TimeoutError" ? "expired" : "error"); if (d < RETRIES) trail.add("retry"); } } k.failure(); throw last; } export const symptom = (e) => (e.cause?.code ? `${e.name}/${e.cause.code}` : e.name); // Call site: the degradation fallback lives here. The fallback value is the same as what silent-wrong returns. export async function callSite(port, k, trail) { try { return { value: await guarded(port, k, trail), fallback: 0, label: "HTTP 200" }; } catch (e) { trail.add("fallback"); return { value: { balance: 0 }, fallback: 1, label: symptom(e) }; } } ``` ## The Pattern Stack and the Measurement The measurement produces each mode in turn and makes five calls. Four things get recorded: the error identity that shows at the call site, the patterns that trigger, how many calls pass without falling back, and the balance value reaching the call site along with the late fee calculated from it. ```js // mode/measurement.mjs — produces each mode one at a time, writes what shows at the call site and which pattern triggers. import http from "node:http"; import { MODE, ACTUAL, TIMEOUT, RETRIES, THRESHOLD, CALLS, breaker, callSite } from "./mode.mjs"; let current = "silent-wrong"; const s = http.createServer((req, res) => MODE[current](res)); await new Promise((r) => s.listen(0, "127.0.0.1", r)); const PORT = s.address().port; const dead = http.createServer(() => {}); // a port whose listener has been closed await new Promise((r) => dead.listen(0, "127.0.0.1", r)); const DEAD_PORT = dead.address().port; await new Promise((r) => dead.close(r)); console.log(`run-independent: timeout ${TIMEOUT} ms, retries ${RETRIES}, breaker threshold ${THRESHOLD}, ${CALLS} calls per mode`); console.log(`actual balance ${ACTUAL}; the late-fee calculation should be balance + 5 = ${ACTUAL + 5}\n`); console.log(`${"mode".padEnd(20)}${"seen at call site".padEnd(27)}${"pattern triggered".padEnd(33)}${"passed".padStart(6)}${"balance".padStart(9)}${"balance+5".padStart(11)}`); const rows = []; for (const mode of Object.keys(MODE)) { current = mode; const k = breaker(), trail = new Set(); let passed = 0, first = null; for (let c = 0; c < CALLS; c += 1) { const y = await callSite(mode === "connection-refused" ? DEAD_PORT : PORT, k, trail); if (y.fallback === 0) passed += 1; first ??= y; // the symptom is read from the first call } const seen = trail.size > 0, correct = first.value.balance === ACTUAL; rows.push({ mode, seen, correct, trail }); console.log(`${mode.padEnd(20)}${first.label.padEnd(27)}${([...trail].join("+") || "none").padEnd(33)}` + `${`${passed}/${CALLS}`.padStart(6)}${String(first.value.balance).padStart(9)}${String(first.value.balance + 5).padStart(11)}`); } s.close(); const unnoticed = rows.filter((x) => x.seen === false); console.log(`\n${rows.length - unnoticed.length} of ${rows.length} modes triggered at least one pattern, ` + `${unnoticed.length} triggered none: ${unnoticed.map((x) => x.mode).join(", ")}`); const wrong = rows.filter((x) => x.correct === false); console.log(`the actual value never reached the call site in ${wrong.length}/${rows.length} modes: a pattern reported it in ${wrong.filter((x) => x.seen).length} modes, ` + `${wrong.filter((x) => !x.seen).length} modes carried no report at all`); console.log("\ncoverage per pattern: " + ["expired", "error", "retry", "breaker", "fallback"] .map((kl) => `${kl} ${rows.filter((x) => x.trail.has(kl)).length}/${rows.length}`).join(", ")); console.log(`run-independent upper bound: a failed call waits at most ${RETRIES * TIMEOUT} ms, 0 ms once the circuit is open`); ``` ``` run-independent: timeout 120 ms, retries 2, breaker threshold 3, 5 calls per mode actual balance 12.5; the late-fee calculation should be balance + 5 = 17.5 mode seen at call site pattern triggered passed balance balance+5 connection-refused TypeError/ECONNREFUSED error+retry+fallback+breaker 0/5 0 5 timeout TimeoutError expired+retry+fallback+breaker 0/5 0 5 server-error HTTP 500 error+retry+fallback+breaker 0/5 0 5 empty-body SyntaxError error+retry+fallback+breaker 0/5 0 5 truncated-body TypeError/UND_ERR_SOCKET error+retry+fallback+breaker 0/5 0 5 wrong-type HTTP 200 none 5/5 12.50 12.505 silent-wrong HTTP 200 none 5/5 0 5 slow-under-limit HTTP 200 none 5/5 12.5 17.5 5 of 8 modes triggered at least one pattern, 3 triggered none: wrong-type, silent-wrong, slow-under-limit the actual value never reached the call site in 7/8 modes: a pattern reported it in 5 modes, 2 modes carried no report at all coverage per pattern: expired 1/8, error 4/8, retry 5/8, breaker 5/8, fallback 5/8 run-independent upper bound: a failed call waits at most 240 ms, 0 ms once the circuit is open ``` ## Five Identities at the Call Site, Two Paths The second column gives the eight modes' identities on the application side, and they do not resemble each other. Connection refusal produces a `TypeError`, its real cause only showing up in `cause.code`. Timeout is a separate class, `TimeoutError`. A server error throws nothing at all — a resolved response comes back, and checking the status code is left to the caller; skip that check, and a 500 counts as success. An empty body is a `SyntaxError`, a **parsing** error: nothing is wrong at the network layer, the body itself is invalid. A truncated body is again a `TypeError`, with a different `cause.code`. Five separate identities, read from five separate places: the error class, the error cause, the status code, the parse result. **Recognizing a failure mode in code requires looking at all four of these sources at once.** The behavior in the last columns, by contrast, is uniform. The same four patterns trigger on all five of those modes, and all five calls fall back. The pattern stack does not distinguish between modes; it reduces every one to a single category: **the call failed.** This is not a flaw in the patterns, it is the limit of their scope — but the difference between connection refusal, which says the service is down, and an empty body, which says it produced a broken response, disappears once it passes through the stack. Only a log entry recording that error identity can carry the difference forward. The coverage-per-pattern row puts a number on this. Timeout triggers on only one of eight modes: the delayed response. The remaining four failing modes already throw an error on their own and never need it. Retry, breaker, and the degradation fallback each trigger on five modes. The most expensive pattern — the breaker — is also the most commonly triggered; timeout, narrowest in coverage, is the only one whose absence leaves a call hanging indefinitely. **Coverage width and importance are not the same thing.** ## The Modes No Pattern Meets The bottom three rows are this lesson's real result. For wrong type, silent wrong value, and below-limit slowness, the `pattern triggered` column reads `none`, and all five calls pass. The pattern stack does not see these modes, because every pattern triggers on a symptom in the transport layer: the connection couldn't be made, time ran out, the status code was wrong, the body couldn't be parsed. In these three modes the transport layer works flawlessly: connection made, response on time, status 200, body valid JSON. The results, though, are not flawless. When the wrong-type field arrives, the late-fee calculation produces `12.505`: text arrives where a number was expected, and addition turns into concatenation. What comes out is not an error but a **wrong fee**, written to the member's account. Under the silent wrong value, the fee comes out `5` instead of the genuine `17.5`. Under below-limit slowness the value is correct, and the only cost is time spent; to the stack, this mode is indistinguishable from a healthy call. The reason the patterns do not meet these modes is not a design flaw, it is a **missing input.** A pattern needs a symptom to trigger on; in these three modes, the symptom is not in the transport layer, it is in the data itself. What would see it is not a resilience pattern but a validation that checks the response's schema and value — and that validation was never written in this topic. ## Zero's Three Meanings One last measure sits in the column of the value reaching the call site. In six of the eight modes, the `balance` value is zero. In five of those, zero is a **degradation fallback**: a pattern triggered, the failure was seen, and the value put in its place was zero. In the sixth, zero is the value the service **actually returned**, and behind it there is no pattern, no record, no symptom at all. The previous lesson measured that a fallback value is indistinguishable from a genuine one; the table here goes one level deeper. **The same number tells three stories: genuine data, the fallback a pattern put in place, and the wrong value a dependency silently produced.** Code reading the stack's output cannot tell these apart. The only way is to write down, next to the value, where it came from: a fallback marker provides that in five modes, and in the sixth there is nothing to provide it. Up to this point, the topic counted where the patterns sit: wrapped and uncovered call sites, paths left outside the limit, the behavior a wrong key produces. This last measurement gives the coverage as a whole. Of eight modes, five are seen and three are not, and two of the three unseen carry a wrong value to the call site. **Patterns do not make a failure visible; they only make visible the symptom that triggers them.** A failure with no symptom is just as invisible with patterns as without. ## Summary - The failure-mode vocabulary was built in the Resilience and Reliability course; the measure here is which identity each mode shows in application code. - Eight modes produce five separate identities at the call site, read from four places: error class (`TimeoutError`), error cause (`ECONNREFUSED`, `UND_ERR_SOCKET`), status code (HTTP 500), and parse result (`SyntaxError`). - The pattern stack does not distinguish these five identities; the same four patterns trigger on all five and every call falls back. The difference between modes disappears once it passes through the stack. - Coverage per pattern is not equal: timeout triggers on 1 of 8 modes, retry, breaker, and the degradation fallback on 5. Coverage width is not the same thing as importance. - 3 modes trigger no pattern — wrong type, silent wrong value, and below-limit slowness — because in all three the transport layer works flawlessly. Two carry a wrong value to the call site: the late fee comes out `12.505` and `5` instead of `17.5`. - In 6 of 8 modes, zero reaches the call site; in 5, it is a fallback value carrying a marker, and in 1, it is the dependency's silent wrong value, carrying no marker at all. ## Next Step This topic built the patterns, counted where they sit in the code, and measured what falls outside their coverage. Every number measured describes one system: two of fourteen call sites unwrapped, two of eight path-access pairs outside every limit, three of eight failure modes seen by no pattern. None of these gaps was learned when written; each was learned when a failure touched it. Learning itself, though, is not yet a mechanism. How a timeline gets built once an incident happens, which signals from the first three topics fill that timeline, what the outcome tells whom, and how the lesson learned makes its way back into the code — none of that has been designed. The next lesson takes up this mechanism and asks: which of this topic's numbers can the text written after an incident actually change.