Synchronous I/O

The previous lesson’s every measurement held one number fixed: that the pool processes 540 units per round. Where that number comes from was never asked. The work a process can carry is bounded not by its processor’s speed but by how many requests it can advance at the same time.

The symptom is this: throughput capped at 0.36 requests per round, and adding workers does not grow that number. Worker utilization is 0.95 — the pool looks saturated. This lesson shows that appearance can be misleading.

Same Symptom, Two Causes

First cause: the downstream dependency is saturated. Throughput is whatever number of calls the dependency can satisfy per round; adding workers changes nothing, because the bottleneck is not in the caller.

Second cause: the workers are waiting on a blocking call. The dependency sits idle, but each worker cannot advance any other request while it waits for a blocking call’s response. This is called synchronous I/O.

In both cases worker utilization comes out close to 1, and adding workers looks like a remedy either way. The measurement that separates them is not on the workers but on the dependency: close to 1, the cause is the first; low, the second. Every explanation made without this measurement is a guess.

What a Blocking Call Actually Does

The block below is not a model but a real measurement running on this machine. Two hundred files are read; duration is not measured, because duration depends on this machine. Both measured quantities are deterministic: the count of other work that advances while the read is underway, and the count of operations concurrently in flight.

// synchronous/real.mjs — a real node run: during a blocking call, no other work can
// advance. The measured quantity is not duration, but the count of advancing work and
// the count of operations concurrently in flight.
import { mkdtempSync, writeFileSync, readFileSync, rmSync } from "node:fs";
import { readFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";

const N = 200;
const dir = mkdtempSync(join(tmpdir(), "sync09-"));
for (let i = 0; i < N; i += 1) writeFileSync(join(dir, `k${i}`), "x".repeat(1024));
const path = (i) => join(dir, `k${i}`);

// A: blocking call. Work queued before the loop cannot advance until the loop ends.
let advanced = 0, inFlight = 0, maxInFlightA = 0;
const pendingWork = Promise.resolve().then(() => { advanced += 1; });
for (let i = 0; i < N; i += 1) {
  inFlight += 1; maxInFlightA = Math.max(maxInFlightA, inFlight);
  readFileSync(path(i));
  inFlight -= 1;
}
console.log(`blocking: other work advanced during the loop = ${advanced},` +
  ` max concurrent in-flight = ${maxInFlightA}, files read = ${N}`);
await pendingWork;

// B: non-blocking call. The same work advances while the calls are underway.
let advancedB = 0, inFlightB = 0, maxInFlightB = 0;
const pendingWorkB = Promise.resolve().then(() => { advancedB += 1; });
const result = await Promise.all(Array.from({ length: N }, async (_, i) => {
  inFlightB += 1; maxInFlightB = Math.max(maxInFlightB, inFlightB);
  const b = await readFile(path(i));
  inFlightB -= 1;
  return b.length;
}));
await pendingWorkB;
console.log(`non-blocking: other work advanced while calls were underway = ${advancedB},` +
  ` max concurrent in-flight = ${maxInFlightB}, files read = ${result.length}`);
console.log(`concurrent in-flight operation ratio = ${maxInFlightB / maxInFlightA}`);
rmSync(dir, { recursive: true, force: true });

blocking: other work advanced during the loop = 0, max concurrent in-flight = 1, files read = 200
non-blocking: other work advanced while calls were underway = 1, max concurrent in-flight = 200, files read = 200
concurrent in-flight operation ratio = 200

Two numbers give the whole mechanism. In the blocking regime, work queued before entering the loop does not advance at all until the two hundred reads finish, and the count of operations in flight at once stays at 1. In the non-blocking regime the same work advances while the calls are underway, and in-flight operations rise to 200: 200-fold. What is measured here is not parallelism — how many jobs the runtime actually executes in parallel is a separate question. What is measured is whether the worker is blocked.

The language-level asynchronous model was built in the Asynchronous JavaScript and the Runtime course; it is not retold here. The question here is what this difference does to capacity.

Two Measurements That Separate the Two Causes

The run below is a model: a round is an abstract step and no real process or thread is set up. KK13 — call profile. The concurrency limit is 8 workers, local work is 1 round, a blocking call’s wait is 20 rounds. Rationale: a remote dependency’s response is two orders of magnitude longer than local work; its sensitivity is given in the last table by varying the wait time. KK14 — cycle coefficient. One round counts as 1 ms; this is only to convert throughput to K01’s requests/s measure. These assumptions are not added to K01’s table.

// synchronous/capacity.mjs — MODEL of a worker pool. A round is an abstract step; KK14
// counts one round as 1 ms (cycle coefficient assumption). No real process or thread is
// set up.
export const K01 = { peakEdge: 513.89 };

export function run({ round = 20_000, limit = 8, work = 1, wait = 20, dependency = 2.0,
                       blocking = true, offer = 3 }) {
  let idleWorkers = limit, queue = 0, tBacklog = 0, jBacklog = 0;
  const active = [];                      // { stage: "work" | "token" | "wait", remaining }
  const s = { done: 0, working: 0, waiting: 0, tokens: 0, maxInFlight: 0 };

  for (let t = 1; t <= round; t += 1) {
    tBacklog += offer;
    while (tBacklog >= 1) { tBacklog -= 1; queue += 1; }
    jBacklog += dependency;
    let token = Math.floor(jBacklog); jBacklog -= token;

    while (idleWorkers > 0 && queue > 0) { queue -= 1; idleWorkers -= 1; active.push({ stage: "work", remaining: work }); }

    for (let i = active.length - 1; i >= 0; i -= 1) {
      const r = active[i];
      if (r.stage === "work") {
        r.remaining -= 1; s.working += 1;
        if (r.remaining === 0) r.stage = "token";
      } else if (r.stage === "token") {
        if (token > 0) {
          token -= 1; s.tokens += 1; r.stage = "wait"; r.remaining = wait;
          if (!blocking) idleWorkers += 1;       // the worker starts the call and is freed
        } else if (blocking) s.waiting += 1;     // with no token available, the worker is held
      } else {
        r.remaining -= 1;
        if (blocking) s.waiting += 1;
        if (r.remaining === 0) {
          s.done += 1; if (blocking) idleWorkers += 1;
          active.splice(i, 1);
        }
      }
    }
    s.maxInFlight = Math.max(s.maxInFlight, active.length);
  }

  const workerRounds = s.working + s.waiting;
  return {
    throughput: s.done / round,
    workerUtilization: workerRounds / (limit * round),
    waitShare: workerRounds === 0 ? 0 : s.waiting / workerRounds,
    dependencyUtilization: s.tokens / (dependency * round),
    inFlight: s.maxInFlight,
  };
}

// synchronous/measure.mjs — two causes of the same symptom, two measurements that tell them apart, and the cost
import { K01, run } from "./capacity.mjs";

const WIDTHS = [38, 10, 12, 13, 13, 8];
const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(WIDTHS[i]) : String(x).padEnd(WIDTHS[i]))).join(" "));
const line = () => console.log(WIDTHS.map((n) => "-".repeat(n)).join(" "));
const row = (name, r) => write([name, r.throughput.toFixed(4), r.workerUtilization.toFixed(4),
  r.waitShare.toFixed(4), r.dependencyUtilization.toFixed(4), r.inFlight]);

const [LIMIT, WORK, WAIT] = [8, 1, 20];
console.log(`concurrency limit ${LIMIT} workers, local work ${WORK} round, blocking call's wait ${WAIT} rounds`);
console.log(`arithmetic ceiling: blocking ${LIMIT}/(${WAIT}+${WORK}) = ${(LIMIT / (WAIT + WORK)).toFixed(4)} requests/round,` +
  ` non-blocking min(${LIMIT}/${WORK}, dependency) requests/round\n`);

write(["case", "throughput", "worker util.", "wait share", "dep. util.", "flight"]);
line();
row("cause 1: dependency saturated (D=0.35)", run({ dependency: 0.35 }));
row("cause 2: blocking call (D=2.0)", run({ dependency: 2.0 }));
console.log();
row("cause 1, non-blocking", run({ dependency: 0.35, blocking: false }));
row("cause 2, non-blocking", run({ dependency: 2.0, blocking: false }));

const blocking = run({ dependency: 2.0 });
const nonBlocking = run({ dependency: 2.0, blocking: false });
console.log(`\ncause 2's gain from the fix: ${blocking.throughput.toFixed(4)} -> ${nonBlocking.throughput.toFixed(4)} requests/round` +
  ` = ${(nonBlocking.throughput / blocking.throughput).toFixed(2)}x`);
console.log(`workers needed to carry the same throughput with the blocking design = ${nonBlocking.throughput} x (${WAIT}+${WORK})` +
  ` = ${(nonBlocking.throughput * (WAIT + WORK)).toFixed(0)}, that is ${((WAIT + WORK) / WORK).toFixed(0)}x`);
console.log(`in-flight requests growing in exchange: ${blocking.inFlight} -> ${nonBlocking.inFlight} (${(nonBlocking.inFlight / blocking.inFlight).toFixed(2)}x)` +
  ` and dependency utilization ${blocking.dependencyUtilization.toFixed(4)} -> ${nonBlocking.dependencyUtilization.toFixed(4)}`);

// KK14: one round counts as 1 ms; throughput is converted to requests/s and compared with K01's peak rate.
const ROUND_MS = 1;
const perSecond = (r) => (r.throughput * 1000) / ROUND_MS;
console.log(`\none round is ${ROUND_MS} ms (KK14): a single process carries blocking ${perSecond(blocking).toFixed(2)} requests/s,` +
  ` non-blocking ${perSecond(nonBlocking).toFixed(2)} requests/s`);
console.log(`K01's peak edge is ${K01.peakEdge} requests/s -> processes needed ` +
  `${Math.ceil(K01.peakEdge / perSecond(blocking))} and ${Math.ceil(K01.peakEdge / perSecond(nonBlocking))}`);

console.log(`\nthe effect of adding workers in the blocking design (D=2.0):`);
write(["case", "throughput", "worker util.", "wait share", "dep. util.", "flight"]);
line();
for (const limit of [8, 16, 42, 84]) row(`limit ${limit} workers`, run({ limit, dependency: 2.0 }));

console.log(`\nKK13's sensitivity: as wait shortens, the blocking design's cost (D=2.0)`);
for (const wait of [20, 5, 2, 1]) {
  const a = run({ wait, dependency: 2.0 }), b = run({ wait, dependency: 2.0, blocking: false });
  console.log(`  wait ${String(wait).padStart(2)} round(s) -> blocking ${a.throughput.toFixed(4)},` +
    ` non-blocking ${b.throughput.toFixed(4)}, ratio ${(b.throughput / a.throughput).toFixed(2)}`);
}

concurrency limit 8 workers, local work 1 round, blocking call's wait 20 rounds
arithmetic ceiling: blocking 8/(20+1) = 0.3810 requests/round, non-blocking min(8/1, dependency) requests/round

case                                   throughput worker util.    wait share    dep. util.   flight
-------------------------------------- ---------- ------------ ------------- ------------- --------
cause 1: dependency saturated (D=0.35)     0.3496       0.9562        0.9542        0.9999        8
cause 2: blocking call (D=2.0)             0.3635       0.9545        0.9524        0.1819        8

cause 1, non-blocking                      0.3496       0.0438        0.0000        0.9999       15
cause 2, non-blocking                      1.9979       0.2500        0.0000        1.0000       46

cause 2's gain from the fix: 0.3635 -> 1.9979 requests/round = 5.50x
workers needed to carry the same throughput with the blocking design = 1.9979 x (20+1) = 42, that is 21x
in-flight requests growing in exchange: 8 -> 46 (5.75x) and dependency utilization 0.1819 -> 1.0000

one round is 1 ms (KK14): a single process carries blocking 363.50 requests/s, non-blocking 1997.90 requests/s
K01's peak edge is 513.89 requests/s -> processes needed 2 and 1

the effect of adding workers in the blocking design (D=2.0):
case                                   throughput worker util.    wait share    dep. util.   flight
-------------------------------------- ---------- ------------ ------------- ------------- --------
limit 8 workers                            0.3635       0.9545        0.9524        0.1819        8
limit 16 workers                           0.7267       0.9544        0.9524        0.3636       16
limit 42 workers                           1.9071       0.9542        0.9524        0.9545       42
limit 84 workers                           1.9979       0.9755        0.9756        1.0000       82

KK13's sensitivity: as wait shortens, the blocking design's cost (D=2.0)
  wait 20 round(s) -> blocking 0.3635, non-blocking 1.9979, ratio 5.50
  wait  5 round(s) -> blocking 1.1426, non-blocking 1.9994, ratio 1.75
  wait  2 round(s) -> blocking 1.9997, non-blocking 1.9997, ratio 1.00
  wait  1 round(s) -> blocking 1.9998, non-blocking 1.9998, ratio 1.00

The in-flight counts and the count of advancing work are measurement; the throughput and utilizations coming out of the model’s deterministic run are computed; the call profile and cycle coefficient are assumption.

The Measurement Tells the Cause

The first table’s two rows give the same symptom: throughput 0.3496 versus 0.3635, worker utilization 0.9562 versus 0.9545, wait share 0.9542 versus 0.9524. The workers look saturated, and in both cases ninety-five percent of their time passes waiting.

The distinction is in the column before the last. In the first cause, dependency utilization is 0.9999 — the dependency is genuinely saturated. In the second, 0.1819 — throughput hits its ceiling while four-fifths of the dependency’s capacity sits idle. Worker utilization alone cannot separate the two cases; the dependency’s own utilization does.

The confirmation is in the second table. When the blocking call is removed, nothing changes for the first cause (0.3496 → 0.3496), while for the second, throughput climbs from 0.3635 to 1.9979 — 5.50 times. The same fix works for one cause and not the other — this is why measurement comes before the fix.

The ceiling’s source is arithmetic and is written in the first line: in the blocking design, throughput cannot exceed the concurrency limit divided by wait plus work time — 8 / (20 + 1) = 0.3810. The model measures 0.3635; the difference comes from the dependency’s token distribution being spread across rounds.

The third table shows what adding workers does. When the limit rises from 8 to 16, throughput reaches 0.7267; at 42 it reaches 1.9071 — the blocking design carries the same throughput only with 21 times the workers. Worker utilization stays around 0.95 across all these rows — utilization never reveals the problem, it only says the workers are busy.

What Grows in Exchange for the Fix

The non-blocking design is not free, and its cost can be counted.

In-flight requests. The count of requests standing in the system at once rises from 8 to 46: 5.75 times. Every in-flight request holds memory — buffer, context, partial response. The concurrency limit bounded this count in the blocking design; the non-blocking design has no such limit, so admission control must be added separately. The Caching, Queues and Asynchronous Processing course’s admission control and concurrency limit lessons fill exactly this gap.

The bottleneck relocates. Dependency utilization climbs from 0.1819 to 1.0000. Once the caller is fixed, the load lands on the dependency, and the next diagnosis will have to happen there.

Carried requests. With KK14 counting one round as 1 ms, a single process carries 363.50 requests/s in the blocking design and 1997.90 requests/s in the non-blocking design. The process count needed for K01’s 513.89 requests/s peak rate ranges between 2 and 1: the same hardware, a twofold difference.

The Condition Under Which a Blocking Call Is Correct

The last table gives the limit. The cost arises from the ratio of wait to local work. At a 20-round wait the ratio is 5.50; at 5 rounds, 1.75; at 2 rounds, 1.00. Once the wait drops to the same order as local work, the blocking design costs nothing, because the time the worker holds is already spent working.

The second condition follows from the first cause: if the dependency is saturated, the gain from not blocking is zero (0.3496 → 0.3496). If the bottleneck is downstream, complicating the caller only grows the in-flight request count.

Summary

• The symptom is throughput hitting a ceiling and adding workers not helping; it has two causes — the downstream dependency is saturated, or the workers are waiting on a blocking call.
• Worker utilization cannot separate the two causes: 0.9562 versus 0.9545, wait share 0.9542 versus 0.9524. The measurement that separates them is the dependency’s own utilization — 0.9999 versus 0.1819.
• In the blocking design, throughput cannot exceed the concurrency limit divided by wait plus work time: 8 / (20 + 1) = 0.3810 requests/round; carrying the same throughput requires 21 times the workers.
• The fix raises throughput from 0.3635 to 1.9979 (5.50 times) for the second cause and changes nothing for the first; with one round counted as 1 ms, a single process carries 1997.90 requests/s instead of 363.50.
• In exchange, in-flight requests rise from 8 to 46 (5.75 times) and dependency utilization climbs from 0.1819 to 1.0000: with the limit gone, admission control must be added separately.
• A blocking call is correct when the wait is the same order as local work: at a 2-round wait the ratio drops to 1.00.

Next Step

All nine patterns rest on one assumption: that a request is made once. The load coming from the edge is set from outside, and the system only carries it. This assumption falls the moment something breaks. When calls start failing, the caller retries, the retries multiply at every layer of the chain, and the load reaching the dependency now comes not from outside but from the system itself. The next lesson is the topic’s final diagnosis, and its question is this: how does such a loop look from the outside. A doubling in the call rate reaching a dependency can be the symptom of either a genuine demand increase or a retry storm; the metric that separates the two is not the call count.