Fault Injection

The previous topic closed out security testing and left a boundary behind. Both load and attack were pressure from the outside: many valid requests in one case, a single deliberately malformed request in the other. In both cases, every part of the system kept working. What happens when one of the system’s own components breaks was never tested.

Fault injection is deliberately triggering a failure scenario in a running system and measuring the response it produces. The loan service depends on the catalog service, and that dependency already carries two patterns built in the Resilience and Reliability course: timeout, and graceful degradation via a fallback path. What those patterns do was explained there; the question here is different — does the pattern actually engage, and with what threshold can we say so.

The Injection Harness

The harness is a model and is written from scratch; no real chaos tool or cluster is set up. The catalog service is a local node:http process with an injection layer placed in front of it. The layer produces three fault types: delay (the response is held back), error (returns 503), and loss (the response is never written). Which request gets broken is derived from the request sequence with a seeded hash; the same seed breaks the same set of requests.

NF14 — 0.03 of healthy catalog requests stall for 150 ms. This is an assumption: even on a fault-free day a real service is not perfectly flat, and this stall sets the floor of the threshold.

// catalog.mjs — the catalog service and the injection layer. The plan comes from outside
// and defaults to closed; which request breaks is derived from the request sequence.
import { createServer } from 'node:http';

export const CLOSED = { type: 'closed', rate: 0, duration: 0 };
export const SEED = { injection: 20260731, noise: 19870415 };
export const NOISE = { rate: 0.03, duration: 150 };      // NF14: background stall

export function hash(seed, i) {                      // seeded, deterministic value per index
  let x = (seed ^ Math.imul(i + 1, 0x9e3779b1)) >>> 0;
  x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0;
  x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0;
  return ((x ^ (x >>> 16)) >>> 0) / 4294967296;
}

const RECORD = { 'K-903': { title: 'book K-903', author: 'Sabahattin Ali' } };

export function setupCatalog(plan = CLOSED) {
  const measure = { request: 0, injected: 0, stall: 0 };
  const server = createServer((request, response) => {
    const address = new URL(request.url, 'http://local');
    const index = Number(address.searchParams.get('index') ?? 0);
    const no = address.searchParams.get('no') ?? 'K-903';
    measure.request += 1;
    const injected = plan.type !== 'closed' && hash(SEED.injection, index) < plan.rate;
    if (injected) measure.injected += 1;
    const delayed = injected && plan.type === 'delay';
    const stall = delayed === false && hash(SEED.noise, index) < NOISE.rate;
    if (stall) measure.stall += 1;
    if (injected && plan.type === 'loss') return;             // response never written
    const send = () => {
      if (injected && plan.type === 'error') {
        response.writeHead(503, { 'content-type': 'application/json' });
        return response.end('{"error":"catalog cannot respond"}');
      }
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end(JSON.stringify({ no, ...RECORD[no] }));
    };
    const wait = delayed ? plan.duration : (stall ? NOISE.duration : 0);
    if (wait === 0) send(); else setTimeout(send, wait);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}

The loan flow produces one of three outcomes: a full response, a gracefully degraded response (record fields empty), or a dropped request; the second outcome disappears once the fallback path is closed.

// loan.mjs — the loan flow requests a record from the catalog. Timeout and the fallback
// path are patterns from the Resilience and Reliability course; here we test if they engage.
export function loanFlow({ base, timeout, fallbackPath }) {
  const s = { full: 0, degraded: 0, dropped: 0, timeoutCount: 0, errorCount: 0 };
  return {
    measure: () => ({ ...s }),
    async lend(index, no = 'K-903') {
      try {
        const y = await fetch(`${base}/record?no=${no}&index=${index}`,
          { signal: AbortSignal.timeout(timeout) });
        if (y.ok === false) { await y.text(); throw new Error(`status ${y.status}`); }
        const g = await y.json();
        s.full += 1;
        return { no, title: g.title, status: 'full' };
      } catch (h) {
        if (h.name === 'TimeoutError' || h.name === 'AbortError') s.timeoutCount += 1;
        else s.errorCount += 1;
        if (fallbackPath === false) { s.dropped += 1; return null; }
        s.degraded += 1;                       // graceful degradation: response without record fields
        return { no, title: null, status: 'degraded' };
      }
    },
  };
}

The Response Per Fault Type

Injection is run across three configurations. The healthy configuration is the system under test; loose and noFallback are two deliberately broken versions, and they measure whether the test can tell them apart.

NF13 — the run is 200 requests, 8 concurrent clients. NF15 — injection rate 0.20, delay 240 ms. NF16 — timeout is 100 ms in the healthy configuration, 400 ms in the loose one. All three are assumptions; the reasoning behind NF16 is this: 100 ms cuts off both the 150 ms stall and the 240 ms delay, while 400 ms swallows both.

// run.mjs — three configurations are run through four fault types and the measurement is
// written to measurement.json. Counts are deterministic; only wall-clock time depends on the run.
import { writeFileSync } from 'node:fs';
import { setupCatalog, CLOSED, NOISE } from './catalog.mjs';
import { loanFlow } from './loan.mjs';

const N = 200, CONCURRENCY = 8;                     // NF13
const PLAN = {
  closed: CLOSED,
  delay: { type: 'delay', rate: 0.20, duration: 240 },   // NF15
  error: { type: 'error', rate: 0.20, duration: 0 },
  loss: { type: 'loss', rate: 0.20, duration: 0 },
};
const CONFIG = {
  healthy: { timeout: 100, fallbackPath: true },           // NF16
  loose: { timeout: 400, fallbackPath: true },
  noFallback: { timeout: 100, fallbackPath: false },
};

async function run(configName, planName) {
  const { server, measure } = setupCatalog(PLAN[planName]);
  await new Promise((c) => server.once('listening', c));
  const base = `http://127.0.0.1:${server.address().port}`;
  const flow = loanFlow({ base, ...CONFIG[configName] });
  let index = 0;
  const worker = async () => { while (index < N) { const i = index; index += 1; await flow.lend(i); } };
  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return { ...flow.measure(), ...measure };
}

const result = {};
const p = (x, n) => String(x).padStart(n);
console.log(`${N} requests, ${CONCURRENCY} concurrent; injection rate ${PLAN.delay.rate}, delay ` +
  `${PLAN.delay.duration} ms, background stall ${NOISE.rate} / ${NOISE.duration} ms`);
console.log(`\n${'config'.padEnd(13)}${'fault'.padEnd(9)}${'injected'.padStart(9)}` +
  `${'full'.padStart(6)}${'degraded'.padStart(10)}${'dropped'.padStart(8)}` +
  `${'timeout'.padStart(13)}${'drop rate'.padStart(13)}`);
for (const config of Object.keys(CONFIG)) {
  for (const plan of Object.keys(PLAN)) {
    const r = await run(config, plan);
    result[`${config}/${plan}`] = r;
    console.log(`${config.padEnd(13)}${plan.padEnd(9)}${p(r.injected, 9)}${p(r.full, 6)}` +
      `${p(r.degraded, 10)}${p(r.dropped, 8)}${p(r.timeoutCount, 13)}` +
      `${p(((100 * r.dropped) / N).toFixed(2) + '%', 13)}`);
  }
}
writeFileSync('measurement.json', `${JSON.stringify({ N, result }, null, 1)}\n`);
console.log(`\nin the fault-free run the injection check ran on ${result['healthy/closed'].request} requests, ` +
  `injecting on ${result['healthy/closed'].injected}`);

200 requests, 8 concurrent; injection rate 0.2, delay 240 ms, background stall 0.03 / 150 ms

config       fault     injected  full  degraded dropped      timeout    drop rate
healthy      closed           0   190        10       0           10        0.00%
healthy      delay           45   148        52       0           52        0.00%
healthy      error           45   148        52       0           10        0.00%
healthy      loss            45   148        52       0           52        0.00%
loose        closed           0   200         0       0            0        0.00%
loose        delay           45   200         0       0            0        0.00%
loose        error           45   155        45       0            0        0.00%
loose        loss            45   155        45       0           45        0.00%
noFallback   closed           0   190         0      10           10        5.00%
noFallback   delay           45   148         0      52           52       26.00%
noFallback   error           45   148         0      52           10       26.00%
noFallback   loss            45   148         0      52           52       26.00%

in the fault-free run the injection check ran on 200 requests, injecting on 0

The healthy configuration’s three rows produce the same result: 45 injected requests, plus 10 stalled ones, turn into a degraded response — 52 in total — and dropped requests are zero. But the path that engages the pattern is not the same: delay and loss count 52 timeouts, error only 10. The error type never runs the timeout; the other half of the pattern, the status-code check, engages instead. The same drop rate can come from different code paths, which is why the measurement also counts which path ran.

The loose configuration’s second row is the lesson’s most important number: all 45 injected requests turn into full responses, degraded is zero, timeouts are zero. The pattern never engaged, and the system displayed it as a flawless result — 200 full responses, a 0.00% drop rate. The broken configuration looks cleaner than the healthy one.

The Threshold and Its Source

Calling something “successful” under a fault is choosing a threshold; the most natural threshold is the drop rate. The scan below compares two rules across five thresholds: at each threshold, how many defective configurations pass (false pass), and how many healthy ones fail (false fail).

// threshold.mjs — a threshold scan over the measurement.json written by run.mjs: at each
// threshold, how many defective configurations pass and how many healthy ones fail.
import { readFileSync } from 'node:fs';

const { N, result } = JSON.parse(readFileSync('measurement.json', 'utf8'));
const TRUTH = { healthy: 'healthy', loose: 'defective', noFallback: 'defective' };
const FAULT = ['delay', 'error', 'loss'];

function ruleA(config, e) {                       // drop rate threshold only
  const k = result[`${config}/closed`];
  if ((k.degraded + k.dropped) / N > e) return false;
  return FAULT.every((a) => result[`${config}/${a}`].dropped / N <= e);
}
function ruleB(config, e) {                       // drop rate + how often the pattern engaged
  if (ruleA(config, e) === false) return false;
  return FAULT.every((a) => result[`${config}/${a}`].degraded >= result[`${config}/${a}`].injected);
}

const count = (rule, e) => {
  let falsePass = 0, falseFail = 0;
  for (const [config, g] of Object.entries(TRUTH)) {
    const passes = rule(config, e);
    if (passes && g === 'defective') falsePass += 1;
    if (passes === false && g === 'healthy') falseFail += 1;
  }
  return { falsePass, falseFail };
};

const p = (x, n) => String(x).padStart(n);
console.log(`${'threshold e'.padEnd(12)}${'rule'.padEnd(9)}${'passing config'.padStart(26)}` +
  `${'false pass'.padStart(14)}${'false fail'.padStart(14)}`);
for (const e of [0, 0.02, 0.05, 0.10, 0.30])
  for (const [name, rule] of [['A', ruleA], ['B', ruleB]]) {
    const r = count(rule, e);
    const passing = Object.keys(TRUTH).filter((y) => rule(y, e));
    console.log(`${((100 * e).toFixed(0) + '%').padEnd(12)}${name.padEnd(9)}` +
      `${(passing.join(",") || "-").padStart(26)}${p(r.falsePass, 14)}${p(r.falseFail, 14)}`);
  }
console.log(`\nA: in the fault-free run the share going to the fallback path <= e, drop rate <= e in every fault`);
console.log(`B: A, and in every fault degraded responses >= injected requests ` +
  `(${result['healthy/delay'].injected}; source: the harness's own counter)`);

threshold e rule                 passing config    false pass    false fail
0%          A                             loose             1             1
0%          B                                 -             0             1
2%          A                             loose             1             1
2%          B                                 -             0             1
5%          A                     healthy,loose             1             0
5%          B                           healthy             0             0
10%         A                     healthy,loose             1             0
10%         B                           healthy             0             0
30%         A          healthy,loose,noFallback             2             0
30%         B                           healthy             0             0

A: in the fault-free run the share going to the fallback path <= e, drop rate <= e in every fault
B: A, and in every fault degraded responses >= injected requests (45; source: the harness's own counter)

Rule A cannot drive both error types to zero at any threshold. Below a 2% threshold, the healthy configuration fails — the background stall sends 10 requests to the fallback path even in the fault-free run; this is a false fail, and its source is not a flaw in the system but noise in the environment. At a 30% threshold the noFallback configuration also passes: two false passes. In the 5–10% range in between, false fail is zero, but the loose configuration passes at every threshold; the effect of the flaw is not a drop but a turn into delay.

Rule B adds a second threshold, and the source of that threshold is the injection itself: the harness counts that it broke 45 requests, so exactly 45 gracefully degraded responses can be demanded of the pattern — injection also produces the source of the threshold. This is the chosen threshold: e = 5% (its source is the measurement, the 10/200 share going to the fallback path in the fault-free run), plus one gracefully degraded response per injected request (its source is the harness’s own counter). At this pair, false pass and false fail are both zero.

Rule B’s limit was also measured: only the delay fault gives away the loose configuration; in the error and loss rows, degraded is 45 and the rule passes. What makes the flaw visible is not the threshold but the fault type injected. The outcome is a release decision: when rule B breaks, the resilience configuration has changed, and the release stops.

Injection’s Own Cost

The injection layer sits inside production code and runs on every request, even on a fault-free day.

// cost.mjs — the injection harness's own cost: code path, the default check, and how many
// paths can open the plan. measurement.json is produced by run.mjs.
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { setupCatalog } from './catalog.mjs';

const code = readFileSync('catalog.mjs', 'utf8').split('\n')
  .filter((s) => s.trim() !== '' && s.trim().startsWith('//') === false);
const injectionPattern = /plan|injected|delayed|NOISE|stall|hash|SEED/;
const decision = code.filter((s) => /if |\? /.test(s) && injectionPattern.test(s));

const { server, measure } = setupCatalog();               // no plan given: defaults to closed
await new Promise((c) => server.once('listening', c));
const base = `http://127.0.0.1:${server.address().port}`;
for (let i = 0; i < 50; i += 1) await (await fetch(`${base}/record?no=K-903&index=${i}`)).json();
server.closeAllConnections();
await new Promise((c) => server.close(c));
assert.equal(measure.injected, 0, 'the default plan must not inject');

const faultFree = JSON.parse(readFileSync('measurement.json', 'utf8')).result['healthy/closed'];
const injectionLines = code.filter((s) => injectionPattern.test(s)).length;
console.log(`catalog.mjs: ${code.length} lines of code, ${injectionLines} of them the injection layer ` +
  `(${((100 * injectionLines) / code.length).toFixed(1)}%); ${decision.length} decision points on the request path`);
console.log(`in the fault-free run the injection check ran on ${faultFree.request}/${faultFree.request} requests, ` +
  `injecting ${faultFree.injected} times`);
console.log(`paths that can open the plan: 1 (setupCatalog's plan parameter); default check ` +
  `${measure.injected} injections on 50 requests - passed`);

catalog.mjs: 38 lines of code, 13 of them the injection layer (34.2%); 5 decision points on the request path
in the fault-free run the injection check ran on 200/200 requests, injecting 0 times
paths that can open the plan: 1 (setupCatalog's plan parameter); default check 0 injections on 50 requests - passed

The cost is written in quantities independent of the run: 13 of the catalog code’s 38 lines (34.2%) are the injection layer, it adds 5 decision points on the request path, and these checks run on all 200 requests in the fault-free run. That is not the real cost — the real cost is what it means if the harness is accidentally left open in production: the loan flow breaking itself. The counterpart to that risk is the number of paths that can open the plan — here, 1 — tested against a claim that the default is closed. If the plan were also read from an environment variable, a header, and a configuration file, the number would be 4, and each would need its own check.

The Failure Class Injection Cannot See

Injection produces the symptom of a failure, not its cause. This distinction leaves three classes outside the harness. The first is a multi-component failure: the layer sits at a single boundary, but the catalog and the notification channel could break at once. The second is the layer underneath the symptom: a full disk produces the same delay but brings secondary effects with it, such as logging loss. The third is call sites where the layer is not placed: if a dependency has no layer, that path is never tested and the table still comes back green. The harness’s coverage is exactly as wide as the number of boundaries it covers.

Summary

• Fault injection deliberately triggers a failure scenario and measures the response it produces; three types were injected (delay, error, loss), the harness was written from scratch, and the seed is visible.
• In the healthy configuration all three fault types produced the same result (52 gracefully degraded responses, 0 dropped) but did not go through the same code path: 52 timeouts for delay and loss, 10 for error.
• The broken configuration whose timeout is longer than the injected delay produced 200 full responses and a 0.00% drop rate; a drop-rate threshold catches this flaw at no value at all.
• In the threshold scan, rule A produced 1 false fail below 2% and 2 false passes at 30%; rule B, which turns the injected request count itself into a threshold, zeroed out both.
• The chosen threshold is e = 5% (its source is the measurement: 10/200 going to the fallback path in the fault-free run) plus one gracefully degraded response per injected request (its source is the harness’s own counter); the release stops when the rule breaks.
• The cost is 13 of 38 lines, 5 decision points on the request path, and a 200/200 check; the real risk is the plan being left open in production, and the number of opening paths is kept at 1 and tested with a claim about the default.

Next Step

In this lesson, injection was a test harness: the scope was a single boundary, the environment was stood up only for this lesson, and nothing broken did anyone any harm. Doing the same injection where real traffic flows is a different question, and it comes before whether the pattern works: when is breakage allowed, and when does it stop. A deviation cannot be measured without first writing down the system’s normal behavior; blast radius cannot be controlled without bounding in advance how many requests the experiment will affect; if things go worse than expected, the number that stops the experiment must be clear. The next lesson builds these three decisions — the steady state hypothesis, bounding the blast radius, and the abort condition — and counts the cost of each being set late or wrong.