Finding the Capacity Limit

The three experiments in this topic were all built the same way: a part of the system was chosen and deliberately broken — the catalog slowed down, a batch was tagged, a table dropped out of the backup. None of them pushed the system to its own limit.

A breaking point test raises load in stages and searches for the step where the system starts to break down. The difficulty is not in measuring but in recognizing: the step where throughput stops rising and the step where delay steepens may not be the same place. The saturation and utilization terms from the Introduction to System Design course name these two readings; the capacity planning from the Performance Anti-Patterns and Monitoring course is this test’s input. The question is: which one gets declared the limit, and what do the two error types of that decision come to.

The Load Ladder and the Wait Round

The loan endpoint is built as a set of service slots with a fixed service duration and a bounded queue. Instead of duration, rounds are counted: one round is one service duration, and a request’s wait round comes from its queue position at the moment it is accepted. The numbers below do not change from run to run.

NF22 — one service round is 40 ms, 4 service slots. NF23 — queue length 16, 5 waves per step. Both are assumptions; the requests in a wave arrive before the first service finishes, which makes the queue position deterministic.

// capacity.mjs — the loan endpoint runs with a fixed service duration, S service slots, and a
// queue of length Q. Every response carries the wait round computed from the queue position at
// the moment the request was accepted; if the queue is full the request is rejected (load shedding).
import { createServer } from 'node:http';

export const ROUND = 40;                       // NF22: one service round is 40 ms

export function setupEndpoint({ slots, queue }) {
  const measure = { accepted: 0, rejected: 0 };
  const waiting = [];
  let active = 0;
  const start = (finish) => {
    active += 1;
    setTimeout(() => {
      active -= 1;
      finish();
      if (waiting.length > 0) start(waiting.shift());
    }, ROUND);
  };
  const server = createServer((request, response) => {
    const occupied = active + waiting.length;
    if (occupied >= slots + queue) {
      measure.rejected += 1;
      response.writeHead(503, { 'content-type': 'application/json' });
      return response.end('{"status":"rejected"}');
    }
    measure.accepted += 1;
    const round = Math.floor(occupied / slots);      // which round it will enter service in
    const finish = () => {
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end(`{"round":${round}}`);
    };
    if (active < slots) start(finish); else waiting.push(finish);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}

The Steps

The measure of acceptability comes from a number already measured in this topic: the timeout the loan service places on its catalog call in the first lesson is 100 ms; with a 40 ms round, that means a request can wait at most one round. One that waits longer is processed but never reaches the caller. NF24 — the acceptable wait is the number of rounds the timeout allows; it is not an assumption but a calculation derived from the first lesson’s measurement.

// ladder.mjs — load is raised step by step; at every step, throughput, wait round, and
// successful requests are counted. All numbers are integers and run-independent; duration is not printed.
import { writeFileSync } from 'node:fs';
import { setupEndpoint, ROUND } from './capacity.mjs';

const SLOTS = 4, QUEUE = 16, WAVE = 5;                 // NF22, NF23
const STEP = [1, 2, 4, 8, 12, 16, 20, 24, 32];
const TIMEOUT = 100;                                // lesson 01's NF16: 100 ms
const MAX_ROUND = Math.ceil(TIMEOUT / ROUND) - 2;    // NF24: wait round cap

async function runStep(base, c) {
  const rounds = [];
  let rejected = 0;
  for (let d = 0; d < WAVE; d += 1) {
    const s = await Promise.all(Array.from({ length: c }, async () => {
      const y = await fetch(base);
      const g = await y.json();
      return y.status === 503 ? null : g.round;
    }));
    for (const x of s) { if (x === null) rejected += 1; else rounds.push(x); }
  }
  const s = [...rounds].sort((a, b) => a - b);
  return { c, sent: c * WAVE, processed: rounds.length, rejected,
    p95: s.length > 0 ? s[Math.ceil(0.95 * s.length) - 1] : 0,
    successful: rounds.filter((x) => x <= MAX_ROUND).length };
}

async function ladder(queue, steps) {
  const { server } = setupEndpoint({ slots: SLOTS, queue });
  await new Promise((c) => server.once('listening', c));
  const base = `http://127.0.0.1:${server.address().port}/loan`;
  const r = [];
  for (const c of steps) r.push(await runStep(base, c));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return r;
}

const p = (x, n) => String(x).padStart(n);
const result = await ladder(QUEUE, STEP);
writeFileSync('ladder.json', `${JSON.stringify({ MAX_ROUND, result }, null, 0)}\n`);
console.log(`${SLOTS} service slots, ${QUEUE} queue, round ${ROUND} ms, ${WAVE} waves per step; ` +
  `timeout ${TIMEOUT} ms -> acceptable wait at most ${MAX_ROUND} rounds`);
console.log(`\n${'concurrent'.padStart(11)}${'sent'.padStart(10)}${'processed'.padStart(11)}` +
  `${'rejected'.padStart(10)}${'p95 wait rounds'.padStart(18)}${'successful'.padStart(12)}`);
for (const r of result)
  console.log(`${p(r.c, 11)}${p(r.sent, 10)}${p(r.processed, 11)}${p(r.rejected, 10)}` +
    `${p(r.p95, 18)}${p(r.successful, 12)}`);

const saturation = result.find((r, i) => i > 0 && r.processed === result[i - 1].processed);
const knee = result.find((r) => r.p95 > MAX_ROUND);
console.log(`\nsaturation step (where throughput stops rising): c=${saturation.c}, throughput ${saturation.processed} requests`);
console.log(`knee step (where p95 wait exceeds ${MAX_ROUND} rounds): c=${knee.c}`);

console.log(`\nsame step (c=32), two queue lengths:`);
for (const q of [QUEUE, 64]) {
  const [r] = await ladder(q, [32]);
  console.log(`queue ${p(q, 2)}: processed ${p(r.processed, 3)}, rejected ${p(r.rejected, 2)}, ` +
    `p95 ${r.p95} rounds, successful ${r.successful}`);
}

4 service slots, 16 queue, round 40 ms, 5 waves per step; timeout 100 ms -> acceptable wait at most 1 rounds

 concurrent      sent  processed  rejected   p95 wait rounds  successful
          1         5          5         0                 0           5
          2        10         10         0                 0          10
          4        20         20         0                 0          20
          8        40         40         0                 1          40
         12        60         60         0                 2          40
         16        80         80         0                 3          40
         20       100        100         0                 4          40
         24       120        100        20                 4          40
         32       160        100        60                 4          40

saturation step (where throughput stops rising): c=24, throughput 100 requests
knee step (where p95 wait exceeds 1 rounds): c=12

same step (c=32), two queue lengths:
queue 16: processed 100, rejected 60, p95 4 rounds, successful 40
queue 64: processed 160, rejected  0, p95 7 rounds, successful 40

Saturation or Knee

The three columns tell three different stories. Processed requests — throughput — rise up to c = 24 and stop there: this is the saturation step. p95 wait rounds moves much earlier and crosses the threshold at c = 12: this is the knee step. Three steps sit between them, and across those three steps the system looks like it is “processing more requests” while most of its responses are timing out.

The third column settles it. Successful requests never rise after c = 8: fixed at 40. Every added request only joins the queue, lengthens the wait of the ones already there, and times out. The real capacity is 40 successful requests, and it was found far earlier than where throughput flattens out.

The last two lines say why this cannot be seen from throughput alone. The same step (c = 32) was run at two queue lengths: at queue 16, throughput is 100 with 60 rejected; at queue 64, throughput is 160 with none rejected — 160/100, that is, 1.60x. Successful requests are 40 in both, that is, 1.00x. Lengthening the queue shifts the saturation step to the right and changes nothing about what reaches the user; a throughput criterion makes the breaking point something a queue setting can hide.

The Rule for Declaring the Limit

The limit is written down as a number, and the rule by which it was chosen is the threshold. The scan compares two families of rules: one that looks at delay (p95 wait ≤ t rounds) and one that looks at throughput (the last step before saturation).

NF25 — a step is acceptable as long as the successful-request share does not drop below 99%. This is an assumption and is kept outside the rules; the rules decide without knowing this share.

// limit.mjs — a scan of limit-declaration rules over ladder.json. The actual acceptance
// criterion is the successful-request share; the rules decide without knowing it.
import { readFileSync } from 'node:fs';

const { result } = JSON.parse(readFileSync('ladder.json', 'utf8'));
const SHARE = 0.99;                                        // NF25
const actual = (r) => r.successful / r.sent >= SHARE;
const saturation = result.findIndex((r, i) => i > 0 && r.processed === result[i - 1].processed);

const count = (accept) => {
  let falsePass = 0, falseFail = 0;
  result.forEach((r, i) => {
    if (accept(r, i) && actual(r) === false) falsePass += 1;
    if (accept(r, i) === false && actual(r)) falseFail += 1;
  });
  return { falsePass, falseFail, limit: result.filter(accept).at(-1)?.c };
};

const p = (x, n) => String(x).padStart(n);
console.log(`actual acceptance criterion: successful-request share >= ${100 * SHARE}%; ` +
  `the highest step meeting this criterion is c=${result.filter(actual).at(-1).c}`);
console.log(`\n${'rule'.padEnd(30)}${'declared limit'.padStart(18)}` +
  `${'false pass'.padStart(14)}${'false fail'.padStart(14)}`);
for (const t of [0, 1, 2, 3, 4]) {
  const r = count((x) => x.p95 <= t);
  console.log(`${`delay: p95 <= ${t} rounds`.padEnd(30)}${p(`c=${r.limit}`, 18)}${p(r.falsePass, 14)}${p(r.falseFail, 14)}`);
}
const v = count((_, i) => i < saturation);
console.log(`${'throughput: before saturation'.padEnd(30)}${p(`c=${v.limit}`, 18)}${p(v.falsePass, 14)}${p(v.falseFail, 14)}`);

actual acceptance criterion: successful-request share >= 99%; the highest step meeting this criterion is c=8

rule                              declared limit    false pass    false fail
delay: p95 <= 0 rounds                       c=4             0             1
delay: p95 <= 1 rounds                       c=8             0             0
delay: p95 <= 2 rounds                      c=12             1             0
delay: p95 <= 3 rounds                      c=16             2             0
delay: p95 <= 4 rounds                      c=32             5             0
throughput: before saturation               c=20             3             0

The last line is this lesson’s outcome. The throughput rule declares the limit at c = 20 and lets three steps through falsely — 12, 16, and 20. In all three, the system processes more requests; in none of them do more users get an answer. When the delay rule’s threshold is set to one round, it declares the limit at c = 8 and zeroes out both error types.

The other end of the scan shows the cost of tightening the threshold: the p95 ≤ 0 rounds rule accepts no wait at all and eliminates c = 8 too — a false fail; at that step the wait is below the timeout, and half the capacity is written off for nothing. The chosen threshold is p95 wait ≤ 1 round, and its source is the timeout the first lesson measured. The decision belongs to the capacity plan: this number does not stop a release, it feeds into a scaling decision.

Cost and the Limit Not Seen

The run-independent cost: nine steps, 595 requests sent, 515 accepted, and 80 rejected. That is not the real cost — the only way to find the limit is to go past it. The last five steps deliberately cause harm: of the 400 requests processed there, 240 timed out. In a production-like environment, this cannot be done without bounding the blast radius, and it needs the previous lesson’s abort condition.

There are three classes the ladder cannot see. The first is a limit tied to the workload mix: the ladder sent the same request to a single endpoint. The second is cumulative degradation: a five-wave step is not the same window as load sustained for hours. The third is collapse behavior: because this endpoint does load shedding, throughput stayed flat past saturation; in a system without shedding, throughput would drop.

Summary

• A breaking point test raises load step by step; across nine steps, throughput, p95 wait round, and successful requests were read separately, and every number is an integer.
• The saturation step is c = 24, the knee step is c = 12; three steps sit between them.
• Successful requests stayed fixed at 40 after c = 8: every added request only lengthened the queue and timed out.
• When the queue was raised from 16 to 64, throughput grew 1.60x while successful requests stayed at 1.00x.
• The chosen threshold is p95 wait ≤ 1 round, its source is the first lesson’s 100 ms timeout, and it gives 0 / 0; the throughput rule lets three steps through falsely.
• The cost is 595 requests sent, and it requires going past the limit: of the 400 requests processed in the last five steps, 240 timed out.

Course Wrap-Up

This course’s fifteen lessons tested the same system with three questions — what happens under load, under attack, and under failure — and every lesson tied its outcome to a threshold.

Lesson	Threshold tested	Source of threshold	False pass / false fail	Cost
Performance Testing Types	p95 ≤ 20 ms, across four decision windows	NF1 assumption+calculation: the 60 ms budget split three ways	1 / 0; at 3 ms 0 / 0, at 1 ms 0 / 3	9 processes, 55,000 requests, 1.00 billion row steps
Test Scenario Modeling	same 20 ms, as the scenario changes	inherited from NF1	1 / 0; at 5 ms 0 / 1; at 100 ms 3 / 0	3,000 requests per run; the scan step spreads 10.7x
Metric Selection	at least $1/(1-p)$ samples per percentile	NF5 calculation	at 20 samples 22/200 / 0; at 100 samples 0 / 0	8,496 and 8,494 latency values, 1,068,000 samples drawn
Bottleneck Analysis	layer share ≥ 0.50	NF6 assumption	1 misattribution / 0 unattributable; at 0.80 0 / 1	5 processes, 4 clock readings per request
Benchmarking Discipline	candidate/baseline ±5%, 8 rounds	NF9 assumption; lower bound is the measured spread	0 / 0; at 2% and one round 0 / 179; at 20% 500 / 0	30 processes, 2.4 million rows of fill, 20,940 requests
Static Application Security Testing	finding score ≥ 40	NF8 assumption: a missed defect = 5 false positives	1 / 2; at 50 2 / 0	75 comparisons, 0 processes, 7 manual reviews
Dynamic Application Security Testing	finding score ≥ 50	measurement: lowest cost at 50	1 / 0	1 process, 8 requests, a hand-built route table
Dependency and Component Scanning	severity ≥ 80	measurement; weight from NF8	1 / 0	5 of 10 packages upgraded, each with a regression test
The Role of Penetration Testing	chain length ≤ 4 steps	budget (NF12), not a requirement	at k=3 2/0; at k=4 0/2; at k=5 0/8	40 steps at k=3, 84 at k=4, 169 at k=5
Accessibility Testing	zero automated findings and 6 manual criteria	measurement: the automated check comes back green with three flaws	3 / 1	1 tree traversal, 14 elements; 6 criteria per screen
Compatibility Testing	covered-user share ≥ 90%	fallacy: the share does not represent the defect distribution	3 / 3; with the profile-and-layout criterion 0 / 5	8 environment runs, a hand-built support table, multiplicative growth
Fault Injection	drop rate ≤ 5% and one degraded response per injected request	measurement: 10/200; the second threshold is the harness’s own counter	0 / 0; with the drop rate alone 1 / 0	13 of 38 lines, 5 decision points, 1 opening path
Chaos Experiments	window full-response share ≥ baseline − 10%	measurement: the control run’s average, 96.00%	0 / 0; at 5% 0 / 4; at 20% detection shifts to window 8	blast radius multiplier 4.00; 44 damaged requests at k = 3
Recovery Verification	the whole of D1+D2+D3+D4 must pass	measurement: the set scan	0 / 0; with D1 alone 2 / 0; with D5 0 / 1	restore grows 10.00x, verification stays at 7 queries
Finding the Capacity Limit	p95 wait ≤ 1 round	measurement: the first lesson’s 100 ms timeout	0 / 0; with the throughput criterion 3 / 0	595 requests, 9 steps, 240 timeouts

The third column is this course’s rule: a threshold without a source carries no decision. The fourth column is the other half of that rule. In functional testing, the outcome is a pass/fail, because the expected value is known. In non-functional testing, the outcome is a distribution, and calling it “successful” is choosing a threshold; that threshold can only be defended with two error types — how many real defects it lets through, and how many healthy runs it turns red. None of the rows in the table that zeroed out both did so by tightening the threshold; every one of them changed what it measured.

Across four courses, testing itself was measured: which test catches which class of defect, which it misses, and what it costs in return. But one question was never asked — when will each test run. Some tests here are cheap enough to run on every change; some are expensive enough to run once per release; one needs a production-like environment. How limited time gets divided among these tests, and how a red result turns into a team decision, was never addressed in any lesson. The Testing Process and Automation Infrastructure course opens with that question.