---
title: 'Chaos Experiments'
source: 'https://academia.sh/en/courses/non-functional-testing/chaos-experiments'
course: 'Non-Functional Testing'
language: en
updated: '2026-08-23T14:25:16+00:00'
license: 'CC BY-SA 4.0'
---

# Chaos Experiments

Designing controlled breakage in a production-like environment: measuring the steady state hypothesis from a control run, the blast radius bounded by the experiment tag turning out four times wider because of batching, scanning the share margin to count false fails against late detection, and measuring the damage that accumulates under the abort condition's consecutive window count.

The previous lesson used injection as a test harness: the scope was a single boundary, the
environment was stood up only for that lesson, and nothing that broke harmed anyone. When the
same injection is done in an environment where real traffic flows, three decisions move to the
front: what is the system's normal behavior, how many requests will the experiment affect, and
by what number will the experiment be stopped.

A **chaos experiment** is performing a controlled breakage in a production-like environment and
measuring whether it deviates from the system's normal behavior. This lesson deals with the
experiment's **design**; the patterns that respond to an injected fault were built in the
Resilience and Reliability course and are not retold here. Each of the three design decisions is
given as a number, and the cost of getting each one wrong is measured.

## The Steady State Hypothesis

The **steady state hypothesis** is a measurable claim about the system's fault-free behavior,
written before the experiment. "The system is healthy" is not a hypothesis; "in every 40-request
window, the full-response share is above this threshold" is. Here the source of the threshold is
not a requirement but a **measurement**: before the experiment opens, the same load is run once,
and the window average is taken as the baseline.

The measurement harness is a **model** and is written from scratch. The catalog service is a
local `node:http` process; injection is applied only to requests carrying the **experiment tag**.
The loan service sends record lookups in batches of four, because a loan list requests several
records on a single screen.

**NF17 — batch size 4, injected delay 240 ms, loan service timeout 100 ms.** **NF18 — 0.03 of
untagged requests stall for 150 ms.** Both are assumptions; choosing a delay larger than the
timeout makes the effect of injection deterministic.

```js
// chaos.mjs — the catalog service scoped to the experiment, and the loan client that sends
// batches. THIS IS A MODEL: no real chaos tool or cluster is set up; injection applies only to
// a batch carrying the experiment tag, and which user is tagged comes from a seeded hash.
import { createServer } from 'node:http';

export const BATCH = 4, DELAY = 240, TIMEOUT = 100;    // NF17
export const NOISE = { rate: 0.03, duration: 150 };     // NF18

export function hash(seed, i) {
  let x = (seed ^ Math.imul(i + 1, 0x9e3779b1)) >>> 0;
  x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0;
  x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0;
  return ((x ^ (x >>> 16)) >>> 0) / 4294967296;
}

export function setupCatalog() {
  const measure = { batch: 0, tagged: 0, stall: 0 };
  const server = createServer((request, response) => {
    const a = new URL(request.url, 'http://local');
    const tag = a.searchParams.get('experiment') === '1';
    const no = Number(a.searchParams.get('batch'));
    measure.batch += 1;
    if (tag) measure.tagged += 1;
    const stall = tag === false && hash(31337, no) < NOISE.rate;
    if (stall) measure.stall += 1;
    const wait = tag ? DELAY : (stall ? NOISE.duration : 0);
    const send = () => {
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end('{"record":"full"}');
    };
    if (wait === 0) send(); else setTimeout(send, wait);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}

export function batchSender(base) {
  return async (no, tagged) => {
    try {
      await fetch(`${base}/record?batch=${no}&experiment=${tagged ? 1 : 0}`,
        { signal: AbortSignal.timeout(TIMEOUT) });
      return 'full';
    } catch { return 'degraded'; }
  };
}
```

## Bounding the Blast Radius

The **blast radius** is the widest area the experiment could affect, and it is written down as a
number before the experiment starts. The plan here is: 0.05 of users are tagged, and injection
is applied only to them. A plan is an **intent**; what needs to be measured is whether the tag
actually covers that many requests.

**NF19 — the run is 400 requests, the window is 40 requests, the experiment opens after request
200, tagged user share is 0.05.** This is an assumption; opening the experiment in the middle of
the run lets the earlier windows serve as the control.

```js
// experiment.mjs — the control run and the experiment run are made with real requests; each
// request's record is written to log.json. Counts are deterministic, only wall-clock time depends on the run.
import { writeFileSync } from 'node:fs';
import { setupCatalog, batchSender, hash, BATCH } from './chaos.mjs';

const N = 400, WINDOW = 40, EXPERIMENT_START = 200, SHARE = 0.05, CONCURRENCY = 4;   // NF19
const taggedUser = (i) => hash(20260731, i) < SHARE;

async function run(experimentOn) {
  const { server, measure } = setupCatalog();
  await new Promise((c) => server.once('listening', c));
  const send = batchSender(`http://127.0.0.1:${server.address().port}`);
  const log = [];
  let index = 0;
  const worker = async () => {
    while (index < N / BATCH) {
      const b = index; index += 1;
      const members = Array.from({ length: BATCH }, (_, k) => b * BATCH + k);
      const tagged = experimentOn && members[0] >= EXPERIMENT_START && members.some(taggedUser);
      const result = await send(b, tagged);
      for (const u of members) log.push({ request: u, tagged, result });
    }
  };
  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return { log: log.sort((a, b) => a.request - b.request), measure };
}

const control = await run(false);
const experiment = await run(true);
writeFileSync('log.json', `${JSON.stringify({ N, WINDOW, EXPERIMENT_START,
  control: control.log, experiment: experiment.log }, null, 0)}\n`);

const inWindow = experiment.log.filter((k) => k.request >= EXPERIMENT_START);
const planned = inWindow.filter((k) => taggedUser(k.request)).length;
const affected = inWindow.filter((k) => k.tagged).length;
const degraded = inWindow.filter((k) => k.result === 'degraded').length;
const y = (a, b) => `${a}/${b} (${((100 * a) / b).toFixed(2)}%)`;
console.log(`${N} requests, batches of ${BATCH}, experiment opens after request ${EXPERIMENT_START}`);
console.log(`planned blast radius          : ${y(planned, inWindow.length)} tagged users`);
console.log(`measured blast radius         : ${y(affected, inWindow.length)} tagged batch members`);
console.log(`blast radius multiplier       : ${(affected / planned).toFixed(2)}x`);
console.log(`degraded in experiment window : ${y(degraded, inWindow.length)}`);
console.log(`degraded in control run       : ${y(control.log.filter((k) => k.result === 'degraded').length, N)}`);
```

```
400 requests, batches of 4, experiment opens after request 200
planned blast radius          : 11/200 (5.50%) tagged users
measured blast radius         : 44/200 (22.00%) tagged batch members
blast radius multiplier       : 4.00x
degraded in experiment window : 52/200 (26.00%)
degraded in control run       : 16/400 (4.00%)
```

The plan said 11 users; the measurement found 44 requests: a **blast radius multiplier of
4.00x**. Its source is **batching**. The tag is placed on the user, but injection is applied to
the batch; a single tagged member in a batch pulls the whole batch into the experiment. The
multiplier's upper bound is the batch size, and this run hit that upper bound, because all eleven
tagged users happened to fall into separate batches. **The blast radius is only as narrow as the
tag is carried**: every aggregation, queueing, and caching step that does not carry the tag
silently widens the radius.

This is an experiment design flaw, and it is visible only by measuring. The way to actually
narrow the radius is to carry the tag into the batch-forming rule as well; the cost of that is
the number of call sites that must now carry the tag, and that number must be known before the
experiment starts.

## The Hypothesis, Falsification, and the Abort Condition

Once the experiment run ends, what is left is a request log. The hypothesis's threshold, the
abort condition's sensitivity, and the cost of both are scanned over the same log; the experiment
is not rerun.

```js
// scan.mjs — two scans over log.json: the steady state hypothesis's share margin and the
// abort condition's consecutive window count. The experiment is not rerun, its log is read.
import { readFileSync } from 'node:fs';

const g = JSON.parse(readFileSync('log.json', 'utf8'));
const windows = (log) => {
  const p = [];
  for (let b = 0; b < g.N; b += g.WINDOW) {
    const d = log.slice(b, b + g.WINDOW);
    p.push({ fullShare: d.filter((k) => k.result === 'full').length / d.length,
      damage: d.filter((k) => k.tagged && k.result === 'degraded').length });
  }
  return p;
};
const C = windows(g.control), E = windows(g.experiment);
const BASELINE = C.reduce((a, x) => a + x.fullShare, 0) / C.length;   // source of threshold: control run
const falsifies = (p, m) => p.fullShare < BASELINE - m;
const p = (x, n) => String(x).padStart(n);

console.log(`steady state hypothesis: full-response share in every window >= baseline - m; baseline ` +
  `is the control run's window average = ${(100 * BASELINE).toFixed(2)}% (${C.length} windows, ` +
  `${g.WINDOW} requests per window)`);
console.log(`\n${'share margin m'.padEnd(15)}${'falsified in control'.padStart(22)}` +
  `${'falsified in experiment'.padStart(25)}${'first falsified window'.padStart(24)}`);
for (const m of [0, 0.05, 0.10, 0.20]) {
  const first = E.findIndex((x) => falsifies(x, m));
  console.log(`${((100 * m).toFixed(0) + '%').padEnd(15)}` +
    `${p(`${C.filter((x) => falsifies(x, m)).length}/${C.length}`, 22)}` +
    `${p(`${E.filter((x) => falsifies(x, m)).length}/${E.length}`, 25)}` +
    `${p(first < 0 ? 'none' : first + 1, 24)}`);
}

const M = 0.10;                            // chosen share margin: no false falsification in control
const TOTAL = E.reduce((a, x) => a + x.damage, 0);
const after = (w) => E.slice(w).reduce((a, x) => a + x.damage, 0);
console.log(`\nchosen share margin ${100 * M}%; the experiment's total damage is ${TOTAL} requests`);
console.log(`\n${'consecutive windows k'.padEnd(23)}${'abort window'.padStart(14)}` +
  `${'damage until abort'.padStart(20)}${'cost over k=1'.padStart(16)}`);
let base = null;
for (const k of [1, 2, 3, 4]) {
  let streak = 0, stop = -1;
  for (let i = 0; i < E.length; i += 1) {
    streak = falsifies(E[i], M) ? streak + 1 : 0;
    if (streak >= k) { stop = i + 1; break; }
  }
  const incurred = stop < 0 ? TOTAL : TOTAL - after(stop);
  if (base === null) base = incurred;
  console.log(`${String(k).padEnd(23)}${p(stop < 0 ? 'never' : stop, 14)}` +
    `${p(incurred, 20)}${p(`+${incurred - base}`, 16)}`);
}
```

```
steady state hypothesis: full-response share in every window >= baseline - m; baseline is the control run's window average = 96.00% (10 windows, 40 requests per window)

share margin m   falsified in control  falsified in experiment  first falsified window
0%                               4/10                     7/10                       2
5%                               4/10                     7/10                       2
10%                              0/10                     4/10                       6
20%                              0/10                     2/10                       8

chosen share margin 10%; the experiment's total damage is 44 requests

consecutive windows k    abort window  damage until abort   cost over k=1
1                                   6                   8              +0
2                                   9                  28             +20
3                                  10                  44             +36
4                               never                  44             +36
```

The first table scans the share margin. Below a 5% margin, the hypothesis is falsified in four
of the control run's ten windows — while the experiment was never even open. This is a **false
fail**, and its consequence is serious: if the abort condition is tied to this hypothesis, the
experiment stops in the second window before it has effectively begun. Raised to a 20% margin,
the control clears, but the experiment is falsified in only two windows and the first
falsification shifts to window 8; most of the damage passes unseen. The chosen threshold sits
between the two: **m = 10%**, and its source is the control run's measured distribution — the
control's lowest window sits 6 points below the baseline, and the margin is set above that noise.

The second table scans the **abort condition**. The abort condition is the rule by which the
experiment terminates itself, and here it takes the form "stop if the hypothesis is falsified in
k consecutive windows." k = 1 stops the experiment at the sixth window, by which point 8 requests
have taken damage. k = 2 waits until the ninth window: 28 requests, that is, **20 requests
extra**. k = 3 and k = 4 never stop before the experiment ends, and accept the full damage: 44
requests. The cost of a late abort is written not in run duration but in the number of requests
damaged.

The seventh window shows why the second table can be misleading: even while the experiment was
open, the hypothesis was not falsified in that window, because the window average stayed above
the threshold. The experiment ran for five windows, and the hypothesis was falsified in four of
them; the falsification rate is **4/5, not 5/5**. The window average swallows sparse damage, and
growing k makes it harder to bridge these gaps.

The outcome of this test is not a release decision; it belongs to the experiment itself. When the
abort condition breaks, the experiment is stopped, the tag is withdrawn, and the falsified window
is opened as a record.

## The Class the Experiment Does Not Catch

The experiment can only falsify what the hypothesis measures. Because the hypothesis is built on
the full-response share, **damage that turns into delay is invisible**: had the timeout been
400 ms, the requests would count as full responses and the hypothesis would stay silent — the
same as the loose configuration in the previous lesson. The second class is **damage that spills
outside the window**: work still queued after the experiment stops, or a retried request, never
enters the log. The third is **dependencies outside the scope**: this experiment only broke the
catalog boundary; the notification channel was never tested and the table would still come back
green.

## Summary

- A chaos experiment is defined by three decisions: the steady state hypothesis, the blast
  radius, and the abort condition; all three are written down as numbers before the experiment
  starts.
- The hypothesis's threshold comes from measurement: the control run's window average, 96.00%,
  was taken as the baseline, and the share margin is a share set below that baseline.
- The planned blast radius was 11 users; the measured one came out to 44 requests: a blast radius
  multiplier of 4.00x, whose upper bound is the batch size; the radius is only as narrow as the
  tag is carried.
- Below a 5% share margin, the hypothesis was falsified in 4/10 control-run windows (a false
  fail); at 20%, only 2/10 experiment windows were falsified and the first falsification shifted
  to window 8.
- At the chosen 10% margin, the abort condition's damage is 8 requests at k = 1, 28 at k = 2, and
  44 at k = 3 and beyond; the cost of a late abort is measured in the number of requests damaged.
- The experiment ran for five windows and the hypothesis was falsified in four of them; the
  window average swallows sparse damage.

## Next Step

This experiment broke a part of the system that kept running: the catalog slowed down, the loan
flow degraded gracefully, the experiment stopped, and everything returned to how it was. The
return was automatic, because nothing was lost. **That assumption collapses once what breaks is
data**: when a copy is lost, a record is corrupted, or a unit drops entirely, what brings the
system back is not stopping the experiment but **restoring the backup**. A backup is taken every
day, and every day it is taken is counted a success — but that is a claim, and it has never been
tested anywhere. The next lesson tests that claim: a backup existing does not show that it is
restorable, the recovery steps that were planned and the steps actually required are not the
same, and what a verification set fails to check can be counted just as well as what it does
check.