Error and Retry Patterns

Cancellation and timeouts made failure well-defined: it is now known when a request will end. What remains is what to do after a failure.

Not every failure is the same. Some pass on their own — a brief overload, the loss of a single packet. Some give the same result when repeated — a wrong address, an unknown station. This lesson builds that distinction, writes the correct form of retrying, and determines which operations can be repeated safely.

Transient and Permanent Errors

There is exactly one reason to retry: the chance that the same request succeeds this time. If that chance does not exist, retrying both keeps the caller waiting and keeps the server busy for nothing.

Error	Class	Reason
Connection could not be established	Transient	Network conditions can change
408, 429	Transient	Time or rate limit; can be tried later
500, 502, 503, 504	Transient	Temporary state on the server side
400, 401, 403, 404	Permanent	The same result even if the request is resent

The distinction is not formal, it is semantic: 404 says “this resource does not exist”; asking again does not change the answer. 503 says “I cannot serve right now”; this is a time-dependent statement.

The distinction between a network error and a failed status code is decisive here too. As shown in the previous lesson, a network error is the case where it is not known whether the request reached the server. This uncertainty makes retrying dangerous for side-effecting operations.

Retrying With Growing Wait

Repeating a failed request immediately makes the problem worse: an overloaded server faces every client that gave up retrying at once. This is why the wait time is increased on every attempt — exponential backoff.

import { createServer } from "node:http";

let requestCount = 0;

const server = createServer((request, response) => {
  requestCount += 1;
  if (requestCount <= 2) {
    response.writeHead(503, { "content-type": "application/json" });
    response.end(JSON.stringify({ error: "service temporarily unavailable" }));
    return;
  }
  response.writeHead(200, { "content-type": "application/json" });
  response.end(JSON.stringify({ station: "A1", value: 21.4 }));
});

await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const base = `http://127.0.0.1:${server.address().port}`;

const TRANSIENT_STATUSES = new Set([408, 429, 500, 502, 503, 504]);

function wait(duration) {
  return new Promise((resolve) => setTimeout(resolve, duration));
}

async function fetchWithRetry(url, attempts, baseDelay) {
  let lastError = null;

  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    const response = await fetch(url);
    if (response.ok) {
      console.log("attempt", attempt, "— successful");
      return await response.json();
    }
    if (!TRANSIENT_STATUSES.has(response.status)) {
      throw new Error(`permanent error: ${response.status}`);
    }
    lastError = new Error(`transient error: ${response.status}`);
    console.log("attempt", attempt, "— transient error:", response.status);

    if (attempt < attempts) {
      const delay = baseDelay * 2 ** (attempt - 1);
      console.log("  planned wait:", delay, "ms");
      await wait(delay);
    }
  }

  throw new Error(`retry budget exhausted: ${attempts}`, { cause: lastError });
}

console.log("result:", await fetchWithRetry(`${base}/measurement/A1`, 4, 10));

server.close();

attempt 1 — transient error: 503
  planned wait: 10 ms
attempt 2 — transient error: 503
  planned wait: 20 ms
attempt 3 — successful
result: { station: 'A1', value: 21.4 }

The server was set up to turn away the first two requests; the third attempt succeeded. The printed wait values are not a measurement, they are the computed plan: the base value is multiplied by two on every attempt.

In real deployments, a random offset is also added to this plan. The reason is this: if clients that fail at the same moment all apply the same plan, their retries also land on the same moment, and the server faces a second wave. The offset spreads the wave across time. It was not used in this lesson so that the outputs stay deterministic.

Retry Budget and Early Exit on a Permanent Error

Two safeguards are required. The number of attempts has to be bounded, or the caller waits forever when the service never returns. On a permanent error, there should be no wait at all.

import { createServer } from "node:http";

const server = createServer((request, response) => {
  const path = new URL(request.url, "http://local").pathname;
  if (path === "/measurement/Z9") {
    response.writeHead(404, { "content-type": "application/json" });
    response.end(JSON.stringify({ error: "unknown station" }));
    return;
  }
  response.writeHead(503, { "content-type": "application/json" });
  response.end(JSON.stringify({ error: "service temporarily unavailable" }));
});

await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const base = `http://127.0.0.1:${server.address().port}`;

const TRANSIENT_STATUSES = new Set([408, 429, 500, 502, 503, 504]);

function wait(duration) {
  return new Promise((resolve) => setTimeout(resolve, duration));
}

async function fetchWithRetry(url, attempts, baseDelay) {
  let lastError = null;

  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    const response = await fetch(url);
    if (response.ok) return await response.json();
    if (!TRANSIENT_STATUSES.has(response.status)) {
      throw new Error(`permanent error: ${response.status}`);
    }
    lastError = new Error(`transient error: ${response.status}`);
    console.log("attempt", attempt, "— transient error:", response.status);
    if (attempt < attempts) await wait(baseDelay * 2 ** (attempt - 1));
  }

  throw new Error(`retry budget exhausted: ${attempts}`, { cause: lastError });
}

try {
  await fetchWithRetry(`${base}/measurement/Z9`, 4, 10);
} catch (error) {
  console.log("no wait on permanent error:", error.message);
}

try {
  await fetchWithRetry(`${base}/measurement/A1`, 3, 10);
} catch (error) {
  console.log("final error:", error.message);
  console.log("root cause:", error.cause.message);
}

server.close();

no wait on permanent error: permanent error: 404
attempt 1 — transient error: 503
attempt 2 — transient error: 503
attempt 3 — transient error: 503
final error: retry budget exhausted: 3
root cause: transient error: 503

A single request was made for the unknown station; three attempts were spent on the address giving a transient error, and once the budget ran out, the error was handed upward carrying its root cause. The cause pattern from the error-handling lesson is needed here for diagnosis: the caller sees both “how many attempts before giving up” and “what happened most recently.”

The total duration of retrying is also a limit. Once the number of attempts and the wait plan are combined, the worst-case elapsed time should be computable in advance; the abort signal can also be bound to this total duration.

Retrying Side-Effecting Operations

Reading a measurement is harmless: sending the same request twice just does extra work. Writing a measurement is not. When you get a network error, you cannot know whether the request reached the server; if you resend it, the same record can be created twice.

The fix is adding an operation key to the request that makes it unique. The server does not count a second request arriving with the same key as a new operation; it returns the first operation’s result.

import { createServer } from "node:http";

const records = [];

const server = createServer((request, response) => {
  let body = "";
  request.on("data", (chunk) => {
    body += chunk;
  });
  request.on("end", () => {
    const key = request.headers["x-operation-key"];
    const existing = key === undefined ? undefined : records.find((r) => r.key === key);

    if (existing === undefined) {
      records.push({ key, data: JSON.parse(body) });
    }
    response.writeHead(200, { "content-type": "application/json" });
    response.end(JSON.stringify({ totalRecords: records.length }));
  });
});

await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const base = `http://127.0.0.1:${server.address().port}`;

async function writeMeasurement(headers) {
  const response = await fetch(`${base}/record`, {
    method: "POST",
    headers: { "content-type": "application/json", ...headers },
    body: JSON.stringify({ station: "A1", value: 21.4 }),
  });
  return await response.json();
}

console.log("no-key attempt 1:", await writeMeasurement({}));
console.log("no-key attempt 2:", await writeMeasurement({}));
console.log("keyed attempt 1:", await writeMeasurement({ "x-operation-key": "write-77" }));
console.log("keyed attempt 2:", await writeMeasurement({ "x-operation-key": "write-77" }));

server.close();

no-key attempt 1: { totalRecords: 1 }
no-key attempt 2: { totalRecords: 2 }
keyed attempt 1: { totalRecords: 3 }
keyed attempt 2: { totalRecords: 3 }

The two requests without a key produced two records; the two requests with a key produced a single record. The property that repeating the same request does not change the result is called idempotence; the How the Internet Works course introduced the same property for HTTP methods. The addition here is that the property can be provided through a key in cases where it does not come from the method.

The rule is summarized as: retrying is safe only if the operation is idempotent. If it is not, it has to be made idempotent first, and only then retried.

The Limit of Retrying

Retrying rescues a single request; it does not rescue the system. If the service has genuinely gone down, every client continuing to retry adds to the load and delays recovery.

For this reason, applications bound retrying with a higher-level mechanism: once consecutive failures cross a certain threshold, requests are not sent at all for a while, then the state is tested with a single probe request. Another form of the same idea is capping the total number of attempts running at the same time.

For the measurement stream, the result is this: a few attempts are made per station, a permanent error gives up immediately, once the budget runs out the station is marked “no data,” and collection continues. Resilient aggregation built with allSettled is the natural home for this behavior.

Summary

• Retrying is meaningful only on transient errors; on a permanent error it both keeps the caller waiting and keeps the server busy for nothing.
• The wait time is increased on every attempt; fixed-interval retries hit an overloaded service as a second wave.
• The number of attempts and the total duration should be bounded, and once the budget runs out, the error should be handed upward together with its root cause.
• A side-effecting operation can be retried only if it is idempotent; idempotence is provided through an operation key.
• Retrying rescues a single request; a system-level bounding mechanism is also needed.

Next Step

The network topic is complete: a request was made, bounded, cancelled, and made resilient. These lessons said “stays in memory” a few times without explaining what, why, and for how long. The next topic covers this: the memory allocation and release cycle, reachability-based collection, what asynchronous code holds in memory, and diagnosing leaks with tools.