Latency and Throughput

The previous topic covered how a design is built and communicated: separating components, splitting the requirement into functional and non-functional halves, roughly computing the resource need, and presenting every decision with a rationale. What the rationale rests on, however, went unnamed. “This design is fast,” “this design scales,” and “this design stays up” contain no metric; all three start an argument rather than end one, because once the other side reverses the same sentence, nothing measurable is left on the table.

This topic defines those metrics. The first lesson starts with the pair most often confused: “median tracking response” and “request rate reaching the store,” both of which appeared among the previous topic’s thresholds, name two separate metrics with no necessary link between them. The lesson measures this independence in a setup, then shows where it ends and why.

Three Flows and the Class of a Number

The same system is used throughout the course: a shipment tracking and billing service. It has three flows. The read flow is the tracking query: the user supplies a tracking number and receives the shipment’s latest state. The write flow is the state event coming from the carrier: small records stream in continuously. The batch flow is the seller’s end-of-day billing: once a day, a very large job.

This topic does not select new assumptions. The twelve-row assumption table from the Back-of-the-Envelope Estimation lesson, and the computed values it produces, are used here as input. The numbers needed for the three flows are: peak request rate at the edge 513.89 req/s — 416.67 of it reads, 97.22 writes; peak rate reaching the store behind the cache 138.89 req/s; the end-of-day job scans 12 million records spanning thirty days in a four-hour window, that is 833.33 records/s and 1.20 ms per record. All of these belong to the computed-value class, and none of them was taken from outside.

The classification from the What Is System Design lesson applies to every number in this topic: a number is either a chosen assumption, a computed value derived from assumptions, or a measurement taken from a setup. This lesson’s contribution is the third class — the computed values above state the required capacity but not what a single process can carry. That number gets measured.

Two Metrics Are Two Separate Axes

Latency is the time elapsed from the start of a piece of work to its end; its unit is time. The Network Models and Protocols course split it into four components: transmission, propagation, queueing, and processing. Throughput is the number of items of work completed per unit time; its unit is work divided by time. The two metrics have different units, so one cannot stand in for the other.

The three flows place these two axes in separate corners. The tracking query wants low latency; the user waits for the response, and a single response’s duration is directly visible to them. End-of-day billing wants high throughput; 12 million records must be scanned in four hours, and when any single record finishes does not matter. In one flow of the same system the unit of scale is the millisecond, in the other it is the hour.

The critical consequence: increasing a flow’s throughput can worsen its latency. The end-of-day job can multiply its throughput by grouping records, but every record in a group then waits for the group to finish. The reverse also happens: shortening latency can lower throughput, because starting every job immediately means giving up the gain grouping provides.

There is one more asymmetry between the two metrics. Throughput is a single number: items of work completed per unit time are counted, and that is the end of it. Latency is not a single number; it is a distribution — under the same load some requests are answered quickly, others slowly. This is why latency is reported in percentiles: the median is the duration below which half of requests fall, p99 is the duration the top one percent exceeds. Reporting an average latency carries the least information, because a single very slow request skews the average and hides the distribution’s shape.

What Happens as Concurrency Increases

The setup below sets up the tracking endpoint as a local server. Every request spends two things: a wait standing in for a record lookup, and a CPU share. Both are parameters of the setup, not measurements.

// tracking/server.mjs — tracking query endpoint: each request spends a wait and a CPU share
import { createServer } from "node:http";

const WAIT = 3;                    // wait standing in for a record lookup (ms), parameter of the setup
const CPU = 0.20;                  // CPU share per request (ms), parameter of the setup

function burnCpu(ms) {            // keeps the CPU busy for ms
  const end = performance.now() + ms;
  let t = 0;
  while (performance.now() < end) t += 1;
  return t;
}

const port = Number(process.argv[2]);
if (Number.isInteger(port) === false) console.log("usage: node tracking/server.mjs <port>");
else createServer(async (req, res) => {
  const no = new URL(req.url, "http://local").searchParams.get("no") ?? "-";
  await new Promise((c) => setTimeout(c, WAIT));
  burnCpu(CPU);
  const body = JSON.stringify({ no, state: "at transit hub", zone: "35", step: 4 });
  res.writeHead(200, { "content-type": "application/json" });
  res.end(body);
}).listen(port);

The client runs at seven concurrency levels. At each level, the stated number of requests is kept in flight at all times; the number of requests sent over two seconds gives the throughput, and the individually measured durations give the latency percentiles.

// tracking/load.mjs — sends requests at increasing concurrency; each level prints throughput and latency
import { Agent, request } from "node:http";

const port = Number(process.argv[2]);
const LEVEL = [1, 2, 4, 8, 16, 32, 64];
const DURATION = 2000;             // measurement duration per level (ms)

const agent = new Agent({ keepAlive: true, maxSockets: 256 });

function oneRequest(no) {
  return new Promise((resolve, reject) => {
    const start = performance.now();
    const r = request({ port, path: `/tracking?no=${no}`, agent }, (res) => {
      res.resume();
      res.on("end", () => resolve(performance.now() - start));
    });
    r.on("error", reject);
    r.end();
  });
}

function percentile(arr, p) {
  const s = [...arr].sort((a, b) => a - b);
  return s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))];
}

async function level(n) {
  const latency = [];
  const end = performance.now() + DURATION;
  const start = performance.now();
  await Promise.all(Array.from({ length: n }, async (_, k) => {
    let i = 0;
    while (performance.now() < end) latency.push(await oneRequest(`G${k}-${i++}`));
  }));
  const elapsed = (performance.now() - start) / 1000;
  return { n, requests: latency.length, throughput: latency.length / elapsed,
    median: percentile(latency, 50), p99: percentile(latency, 99) };
}

if (Number.isInteger(port) === false) console.log("usage: node tracking/load.mjs <port>");
else {
  await oneRequest("warmup");
  console.log("concurr  requests  throughput(req/s)  median(ms)  p99(ms)  throughput*median");
  for (const n of LEVEL) {
    const s = await level(n);
    const little = (s.throughput * s.median) / 1000;
    console.log(`${String(s.n).padStart(6)}  ${String(s.requests).padStart(6)}  ` +
      `${s.throughput.toFixed(0).padStart(13)}  ${s.median.toFixed(2).padStart(10)}  ` +
      `${s.p99.toFixed(2).padStart(7)}  ${little.toFixed(2).padStart(22)}`);
  }
  agent.destroy();
}

The driver script starts the server, runs the measurement, and stops the server. The port number depends on the environment; it should be changed if it is already in use on the machine.

# measure.sh — starts the server, measures seven concurrency levels, stops the server
node tracking/server.mjs 8811 & SERVER=$!
sleep 1
node tracking/load.mjs 8811
kill $SERVER

concurr  requests  throughput(req/s)  median(ms)  p99(ms)  throughput*median
     1     463            231        4.33     4.78                    1.00
     2     936            468        4.46     5.05                    2.09
     4    1764            880        4.80     5.38                    4.23
     8    3038           1515        5.67     6.14                    8.59
    16    4849           2417        6.65     7.57                   16.06
    32    8089           4031        8.21     9.56                   33.10
    64    8593           4268       15.15    16.46                   64.64

These numbers are measurements, taken on this machine. Their absolute values will not repeat on another machine; what repeats is the table’s shape.

Two Regions

The table splits into two distinct regions, and the lesson’s claim rests on this split.

In the first region, as concurrency rises from 1 to 4, throughput climbs from 231 to 880, that is 3.81x; median latency climbs from 4.33 ms to 4.80 ms, that is 1.11x. Throughput grew nearly in step with concurrency while latency stood still. In this region the two metrics are independent: throughput is rising and says nothing about latency.

In the second region the table reverses. As concurrency rises from 32 to 64, throughput climbs from 4031 to 4268, only 1.06x; median latency climbs from 8.21 ms to 15.15 ms, 1.85x. Doubling concurrency buys 6% more throughput and double the latency. In this region independence ends: throughput has hit an upper bound, and every added request only adds to the wait.

Where the upper bound comes from is written into the setup. Every request spends 0.20 ms of CPU share, and the server runs on a single thread; the CPU share cannot be divided. Waits can overlap; a CPU share cannot. This is the distinction from the Server-Side Fundamentals course: waiting I/O is cheap, computation that occupies the CPU locks the queue.

The Relation

The last column ties the two metrics together. In every row, the product of throughput and median latency comes out equal to the concurrency level: 1.00, 2.09, 4.23, 8.59, 16.06, 33.10, 64.64. This is not a coincidence; it is queueing theory’s fundamental relation. The Caching, Queues and Asynchronous Processing course set it up as $L = \lambda W$; its counterpart here is:

$$N = X \times R$$

$N$ is the number of items of work in flight, $X$ is throughput, $R$ is latency. The design consequence of the relation is direct: if two of the three quantities are chosen, the third cannot be chosen freely. If the number of items in flight is held fixed and throughput cannot rise, latency must grow — there is no other way out. The behavior in the second region is this necessity made visible.

The same relation also gives the rule for reading a measurement. A report that states only throughput says nothing about latency unless it also states the concurrency at which it was taken; a report that states only latency cannot be read either unless it states the load. Reporting a single metric on its own is an incomplete report.

The Cost of Approaching Saturation

A model shows what happens at the entrance to the second region. In a single-server queue, with service rate $\mu$ and arrival rate $\lambda$, utilization is defined as $\rho = \lambda / \mu$, and the average time spent in the system is $1 / (\mu - \lambda)$. The computation below feeds this model with the measured upper bound. This is a model, not a measurement; it carries assumptions about the distribution of arrivals and service times and does not predict a real server’s latency. What it shows is the direction of the ratios.

// calc/queue.mjs — single-server queue MODEL: the latency cost of approaching saturation
const MU = 4268;                   // measured upper bound: this setup's saturation throughput for a process (req/s)

console.log(`service rate mu = ${MU} req/s (measured upper bound)`);
console.log("utilization   arrival rate(req/s)   modeled average time(ms)   ratio to 30% case");
const base = 1000 / (MU - 0.30 * MU);
for (const rho of [0.30, 0.50, 0.70, 0.80, 0.90, 0.95, 0.99]) {
  const T = 1000 / (MU - rho * MU);
  console.log(`${(rho * 100).toFixed(0).padStart(6)}%  ${(rho * MU).toFixed(0).padStart(19)}  ` +
    `${T.toFixed(2).padStart(27)}  ${(T / base).toFixed(1).padStart(18)}`);
}

const peak = [416.67, 833.33];   // Back-of-the-Envelope Estimation: peak read rate, and if the query assumption doubles
for (const t of peak) {
  console.log(`peak ${t} req/s -> utilization ${((t / MU) * 100).toFixed(1)}%, ` +
    `upper bound is ${(MU / t).toFixed(1)}x peak, ` +
    `modeled average time ${(1000 / (MU - t)).toFixed(2)} ms`);
}

service rate mu = 4268 req/s (measured upper bound)
utilization   arrival rate(req/s)   modeled average time(ms)   ratio to 30% case
    30%                 1280                         0.33                 1.0
    50%                 2134                         0.47                 1.4
    70%                 2988                         0.78                 2.3
    80%                 3414                         1.17                 3.5
    90%                 3841                         2.34                 7.0
    95%                 4055                         4.69                14.0
    99%                 4225                        23.43                70.0
peak 416.67 req/s -> utilization 9.8%, upper bound is 10.2x peak, modeled average time 0.26 ms
peak 833.33 req/s -> utilization 19.5%, upper bound is 5.1x peak, modeled average time 0.29 ms

The model says one thing: as utilization rises, throughput grows linearly while time does not. Going from 30% to 99% multiplies the arrival rate by 3.3 and the time by 70. The last slice of throughput is bought with unbounded growth in latency. Keeping utilization high looks like an efficiency win, and it is a latency decision.

The last two lines join the computation to the measurement. Peak read rate is 416.67 req/s, the upper bound measured in the setup is 4268 req/s; the ratio is 10.2 and utilization stays at 9.8%. When queries per user double, the ratio becomes 5.1. What matters here is not the number itself but the method: the peak rate is computed from assumptions, the upper bound is measured from the setup, and their ratio gives the margin. A real tracking endpoint does more than this setup, so its upper bound will be lower; the ratio is still read the same way. When it falls below 1, the design no longer works; between 2 and 3, a single failure is narrow enough to erase the entire margin.

Summary

• Latency is the duration of a piece of work, throughput is the number of items of work completed per unit time; their units differ and one cannot substitute for the other.
• In the measurement, as concurrency rose from 1 to 4, throughput climbed to 3.81x while median latency stayed at 1.11x: in this region the two metrics are independent.
• As concurrency rose from 32 to 64, throughput climbed to 1.06x, latency climbed to 1.85x: once throughput hits its upper bound, every added request only adds to the wait.
• $N = X \times R$ held in every measured row; if two of the three quantities are chosen the third cannot be chosen freely, so throughput or latency reported alone cannot be read.
• The queueing model showed that time does not grow linearly with utilization: from 30% to 99%, arrival rate rises 3.3x while time rises 70x; keeping utilization high is a latency decision.
• The computed peak read rate is 416.67 req/s, the measured upper bound is 4268 req/s; the ratio is 10.2 and falls to 5.1 when queries per user double.

Next Step

This lesson held a single process fixed and varied the load. The two regions that emerged show why “this system is fast” is insufficient: a system’s speed depends on the load at which it was measured, and a number taken at low load says nothing about behavior at high load. Two separate questions follow. First, how good is the system at a given load. Second, how long does that goodness hold up as load grows. The two questions are often answered with a single word, and should not be: one is named performance, the other scalability. The next lesson separates the two concepts: it measures two implementations of the end-of-day job at four separate loads, shows that both meet today’s threshold, and reveals that the difference between them can only be counted once the load is scaled up.