Chaos Experiments

The previous lesson used injection as a test harness: the scope was a single boundary, the environment was stood up only for that lesson, and nothing that broke harmed anyone. When the same injection is done in an environment where real traffic flows, three decisions move to the front: what is the system’s normal behavior, how many requests will the experiment affect, and by what number will the experiment be stopped.

A chaos experiment is performing a controlled breakage in a production-like environment and measuring whether it deviates from the system’s normal behavior. This lesson deals with the experiment’s design; the patterns that respond to an injected fault were built in the Resilience and Reliability course and are not retold here. Each of the three design decisions is given as a number, and the cost of getting each one wrong is measured.

The Steady State Hypothesis

The steady state hypothesis is a measurable claim about the system’s fault-free behavior, written before the experiment. “The system is healthy” is not a hypothesis; “in every 40-request window, the full-response share is above this threshold” is. Here the source of the threshold is not a requirement but a measurement: before the experiment opens, the same load is run once, and the window average is taken as the baseline.

The measurement harness is a model and is written from scratch. The catalog service is a local node:http process; injection is applied only to requests carrying the experiment tag. The loan service sends record lookups in batches of four, because a loan list requests several records on a single screen.

NF17 — batch size 4, injected delay 240 ms, loan service timeout 100 ms. NF18 — 0.03 of untagged requests stall for 150 ms. Both are assumptions; choosing a delay larger than the timeout makes the effect of injection deterministic.

// chaos.mjs — the catalog service scoped to the experiment, and the loan client that sends
// batches. THIS IS A MODEL: no real chaos tool or cluster is set up; injection applies only to
// a batch carrying the experiment tag, and which user is tagged comes from a seeded hash.
import { createServer } from 'node:http';

export const BATCH = 4, DELAY = 240, TIMEOUT = 100;    // NF17
export const NOISE = { rate: 0.03, duration: 150 };     // NF18

export function hash(seed, i) {
  let x = (seed ^ Math.imul(i + 1, 0x9e3779b1)) >>> 0;
  x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0;
  x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0;
  return ((x ^ (x >>> 16)) >>> 0) / 4294967296;
}

export function setupCatalog() {
  const measure = { batch: 0, tagged: 0, stall: 0 };
  const server = createServer((request, response) => {
    const a = new URL(request.url, 'http://local');
    const tag = a.searchParams.get('experiment') === '1';
    const no = Number(a.searchParams.get('batch'));
    measure.batch += 1;
    if (tag) measure.tagged += 1;
    const stall = tag === false && hash(31337, no) < NOISE.rate;
    if (stall) measure.stall += 1;
    const wait = tag ? DELAY : (stall ? NOISE.duration : 0);
    const send = () => {
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end('{"record":"full"}');
    };
    if (wait === 0) send(); else setTimeout(send, wait);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}

export function batchSender(base) {
  return async (no, tagged) => {
    try {
      await fetch(`${base}/record?batch=${no}&experiment=${tagged ? 1 : 0}`,
        { signal: AbortSignal.timeout(TIMEOUT) });
      return 'full';
    } catch { return 'degraded'; }
  };
}

Bounding the Blast Radius

The blast radius is the widest area the experiment could affect, and it is written down as a number before the experiment starts. The plan here is: 0.05 of users are tagged, and injection is applied only to them. A plan is an intent; what needs to be measured is whether the tag actually covers that many requests.

NF19 — the run is 400 requests, the window is 40 requests, the experiment opens after request 200, tagged user share is 0.05. This is an assumption; opening the experiment in the middle of the run lets the earlier windows serve as the control.

// experiment.mjs — the control run and the experiment run are made with real requests; each
// request's record is written to log.json. Counts are deterministic, only wall-clock time depends on the run.
import { writeFileSync } from 'node:fs';
import { setupCatalog, batchSender, hash, BATCH } from './chaos.mjs';

const N = 400, WINDOW = 40, EXPERIMENT_START = 200, SHARE = 0.05, CONCURRENCY = 4;   // NF19
const taggedUser = (i) => hash(20260731, i) < SHARE;

async function run(experimentOn) {
  const { server, measure } = setupCatalog();
  await new Promise((c) => server.once('listening', c));
  const send = batchSender(`http://127.0.0.1:${server.address().port}`);
  const log = [];
  let index = 0;
  const worker = async () => {
    while (index < N / BATCH) {
      const b = index; index += 1;
      const members = Array.from({ length: BATCH }, (_, k) => b * BATCH + k);
      const tagged = experimentOn && members[0] >= EXPERIMENT_START && members.some(taggedUser);
      const result = await send(b, tagged);
      for (const u of members) log.push({ request: u, tagged, result });
    }
  };
  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return { log: log.sort((a, b) => a.request - b.request), measure };
}

const control = await run(false);
const experiment = await run(true);
writeFileSync('log.json', `${JSON.stringify({ N, WINDOW, EXPERIMENT_START,
  control: control.log, experiment: experiment.log }, null, 0)}\n`);

const inWindow = experiment.log.filter((k) => k.request >= EXPERIMENT_START);
const planned = inWindow.filter((k) => taggedUser(k.request)).length;
const affected = inWindow.filter((k) => k.tagged).length;
const degraded = inWindow.filter((k) => k.result === 'degraded').length;
const y = (a, b) => `${a}/${b} (${((100 * a) / b).toFixed(2)}%)`;
console.log(`${N} requests, batches of ${BATCH}, experiment opens after request ${EXPERIMENT_START}`);
console.log(`planned blast radius          : ${y(planned, inWindow.length)} tagged users`);
console.log(`measured blast radius         : ${y(affected, inWindow.length)} tagged batch members`);
console.log(`blast radius multiplier       : ${(affected / planned).toFixed(2)}x`);
console.log(`degraded in experiment window : ${y(degraded, inWindow.length)}`);
console.log(`degraded in control run       : ${y(control.log.filter((k) => k.result === 'degraded').length, N)}`);

400 requests, batches of 4, experiment opens after request 200
planned blast radius          : 11/200 (5.50%) tagged users
measured blast radius         : 44/200 (22.00%) tagged batch members
blast radius multiplier       : 4.00x
degraded in experiment window : 52/200 (26.00%)
degraded in control run       : 16/400 (4.00%)

The plan said 11 users; the measurement found 44 requests: a blast radius multiplier of 4.00x. Its source is batching. The tag is placed on the user, but injection is applied to the batch; a single tagged member in a batch pulls the whole batch into the experiment. The multiplier’s upper bound is the batch size, and this run hit that upper bound, because all eleven tagged users happened to fall into separate batches. The blast radius is only as narrow as the tag is carried: every aggregation, queueing, and caching step that does not carry the tag silently widens the radius.

This is an experiment design flaw, and it is visible only by measuring. The way to actually narrow the radius is to carry the tag into the batch-forming rule as well; the cost of that is the number of call sites that must now carry the tag, and that number must be known before the experiment starts.

The Hypothesis, Falsification, and the Abort Condition

Once the experiment run ends, what is left is a request log. The hypothesis’s threshold, the abort condition’s sensitivity, and the cost of both are scanned over the same log; the experiment is not rerun.

// scan.mjs — two scans over log.json: the steady state hypothesis's share margin and the
// abort condition's consecutive window count. The experiment is not rerun, its log is read.
import { readFileSync } from 'node:fs';

const g = JSON.parse(readFileSync('log.json', 'utf8'));
const windows = (log) => {
  const p = [];
  for (let b = 0; b < g.N; b += g.WINDOW) {
    const d = log.slice(b, b + g.WINDOW);
    p.push({ fullShare: d.filter((k) => k.result === 'full').length / d.length,
      damage: d.filter((k) => k.tagged && k.result === 'degraded').length });
  }
  return p;
};
const C = windows(g.control), E = windows(g.experiment);
const BASELINE = C.reduce((a, x) => a + x.fullShare, 0) / C.length;   // source of threshold: control run
const falsifies = (p, m) => p.fullShare < BASELINE - m;
const p = (x, n) => String(x).padStart(n);

console.log(`steady state hypothesis: full-response share in every window >= baseline - m; baseline ` +
  `is the control run's window average = ${(100 * BASELINE).toFixed(2)}% (${C.length} windows, ` +
  `${g.WINDOW} requests per window)`);
console.log(`\n${'share margin m'.padEnd(15)}${'falsified in control'.padStart(22)}` +
  `${'falsified in experiment'.padStart(25)}${'first falsified window'.padStart(24)}`);
for (const m of [0, 0.05, 0.10, 0.20]) {
  const first = E.findIndex((x) => falsifies(x, m));
  console.log(`${((100 * m).toFixed(0) + '%').padEnd(15)}` +
    `${p(`${C.filter((x) => falsifies(x, m)).length}/${C.length}`, 22)}` +
    `${p(`${E.filter((x) => falsifies(x, m)).length}/${E.length}`, 25)}` +
    `${p(first < 0 ? 'none' : first + 1, 24)}`);
}

const M = 0.10;                            // chosen share margin: no false falsification in control
const TOTAL = E.reduce((a, x) => a + x.damage, 0);
const after = (w) => E.slice(w).reduce((a, x) => a + x.damage, 0);
console.log(`\nchosen share margin ${100 * M}%; the experiment's total damage is ${TOTAL} requests`);
console.log(`\n${'consecutive windows k'.padEnd(23)}${'abort window'.padStart(14)}` +
  `${'damage until abort'.padStart(20)}${'cost over k=1'.padStart(16)}`);
let base = null;
for (const k of [1, 2, 3, 4]) {
  let streak = 0, stop = -1;
  for (let i = 0; i < E.length; i += 1) {
    streak = falsifies(E[i], M) ? streak + 1 : 0;
    if (streak >= k) { stop = i + 1; break; }
  }
  const incurred = stop < 0 ? TOTAL : TOTAL - after(stop);
  if (base === null) base = incurred;
  console.log(`${String(k).padEnd(23)}${p(stop < 0 ? 'never' : stop, 14)}` +
    `${p(incurred, 20)}${p(`+${incurred - base}`, 16)}`);
}

steady state hypothesis: full-response share in every window >= baseline - m; baseline is the control run's window average = 96.00% (10 windows, 40 requests per window)

share margin m   falsified in control  falsified in experiment  first falsified window
0%                               4/10                     7/10                       2
5%                               4/10                     7/10                       2
10%                              0/10                     4/10                       6
20%                              0/10                     2/10                       8

chosen share margin 10%; the experiment's total damage is 44 requests

consecutive windows k    abort window  damage until abort   cost over k=1
1                                   6                   8              +0
2                                   9                  28             +20
3                                  10                  44             +36
4                               never                  44             +36

The first table scans the share margin. Below a 5% margin, the hypothesis is falsified in four of the control run’s ten windows — while the experiment was never even open. This is a false fail, and its consequence is serious: if the abort condition is tied to this hypothesis, the experiment stops in the second window before it has effectively begun. Raised to a 20% margin, the control clears, but the experiment is falsified in only two windows and the first falsification shifts to window 8; most of the damage passes unseen. The chosen threshold sits between the two: m = 10%, and its source is the control run’s measured distribution — the control’s lowest window sits 6 points below the baseline, and the margin is set above that noise.

The second table scans the abort condition. The abort condition is the rule by which the experiment terminates itself, and here it takes the form “stop if the hypothesis is falsified in k consecutive windows.” k = 1 stops the experiment at the sixth window, by which point 8 requests have taken damage. k = 2 waits until the ninth window: 28 requests, that is, 20 requests extra. k = 3 and k = 4 never stop before the experiment ends, and accept the full damage: 44 requests. The cost of a late abort is written not in run duration but in the number of requests damaged.

The seventh window shows why the second table can be misleading: even while the experiment was open, the hypothesis was not falsified in that window, because the window average stayed above the threshold. The experiment ran for five windows, and the hypothesis was falsified in four of them; the falsification rate is 4/5, not 5/5. The window average swallows sparse damage, and growing k makes it harder to bridge these gaps.

The outcome of this test is not a release decision; it belongs to the experiment itself. When the abort condition breaks, the experiment is stopped, the tag is withdrawn, and the falsified window is opened as a record.

The Class the Experiment Does Not Catch

The experiment can only falsify what the hypothesis measures. Because the hypothesis is built on the full-response share, damage that turns into delay is invisible: had the timeout been 400 ms, the requests would count as full responses and the hypothesis would stay silent — the same as the loose configuration in the previous lesson. The second class is damage that spills outside the window: work still queued after the experiment stops, or a retried request, never enters the log. The third is dependencies outside the scope: this experiment only broke the catalog boundary; the notification channel was never tested and the table would still come back green.

Summary

• A chaos experiment is defined by three decisions: the steady state hypothesis, the blast radius, and the abort condition; all three are written down as numbers before the experiment starts.
• The hypothesis’s threshold comes from measurement: the control run’s window average, 96.00%, was taken as the baseline, and the share margin is a share set below that baseline.
• The planned blast radius was 11 users; the measured one came out to 44 requests: a blast radius multiplier of 4.00x, whose upper bound is the batch size; the radius is only as narrow as the tag is carried.
• Below a 5% share margin, the hypothesis was falsified in 4/10 control-run windows (a false fail); at 20%, only 2/10 experiment windows were falsified and the first falsification shifted to window 8.
• At the chosen 10% margin, the abort condition’s damage is 8 requests at k = 1, 28 at k = 2, and 44 at k = 3 and beyond; the cost of a late abort is measured in the number of requests damaged.
• The experiment ran for five windows and the hypothesis was falsified in four of them; the window average swallows sparse damage.

Next Step

This experiment broke a part of the system that kept running: the catalog slowed down, the loan flow degraded gracefully, the experiment stopped, and everything returned to how it was. The return was automatic, because nothing was lost. That assumption collapses once what breaks is data: when a copy is lost, a record is corrupted, or a unit drops entirely, what brings the system back is not stopping the experiment but restoring the backup. A backup is taken every day, and every day it is taken is counted a success — but that is a claim, and it has never been tested anywhere. The next lesson tests that claim: a backup existing does not show that it is restorable, the recovery steps that were planned and the steps actually required are not the same, and what a verification set fails to check can be counted just as well as what it does check.