---
title: 'Latency and Throughput'
source: 'https://academia.sh/en/courses/introduction-to-system-design/latency-and-throughput'
course: 'Introduction to System Design'
language: en
updated: '2026-08-23T14:25:23+00:00'
license: 'CC BY-SA 4.0'
---

# Latency and Throughput

Measuring that the two metrics are two independent axes: throughput saturating and latency worsening as concurrency rises, verifying by measurement the relation among concurrency, throughput, and latency, showing the latency cost of approaching saturation with a queueing model, and classifying every number as an assumption, a computed value, or a measurement.

The previous topic covered how a design is built and communicated: separating components,
splitting the requirement into functional and non-functional halves, roughly computing the
resource need, and presenting every decision with a rationale. What the rationale rests on,
however, went unnamed. "This design is fast," "this design scales," and "this design stays up"
contain no metric; all three start an argument rather than end one, because once the other side
reverses the same sentence, nothing measurable is left on the table.

This topic defines those metrics. The first lesson starts with the pair most often confused:
"median tracking response" and "request rate reaching the store," both of which appeared among
the previous topic's thresholds, name two separate metrics with no necessary link between them.
The lesson measures this independence in a setup, then shows where it ends and why.

## Three Flows and the Class of a Number

The same system is used throughout the course: a **shipment tracking and billing service**. It
has three flows. The **read** flow is the tracking query: the user supplies a tracking number
and receives the shipment's latest state. The **write** flow is the state event coming from the
carrier: small records stream in continuously. The **batch** flow is the seller's end-of-day
billing: once a day, a very large job.

This topic does not select new assumptions. The twelve-row assumption table from the
Back-of-the-Envelope Estimation lesson, and the computed values it produces, are used here as
input. The numbers needed for the three flows are: peak request rate at the edge **513.89
req/s** — 416.67 of it reads, 97.22 writes; peak rate reaching the store behind the cache
**138.89 req/s**; the end-of-day job scans **12 million records spanning thirty days in a
four-hour window**, that is **833.33 records/s** and 1.20 ms per record. All of these belong to
the computed-value class, and none of them was taken from outside.

The classification from the What Is System Design lesson applies to every number in this topic:
a number is either a chosen **assumption**, a **computed value** derived from assumptions, or a
**measurement** taken from a setup. This lesson's contribution is the third class — the computed
values above state the required capacity but not what a single process can carry. That number
gets measured.

## Two Metrics Are Two Separate Axes

**Latency** is the time elapsed from the start of a piece of work to its end; its unit is time.
The Network Models and Protocols course split it into four components: transmission,
propagation, queueing, and processing. **Throughput** is the number of items of work completed
per unit time; its unit is work divided by time. The two metrics have different units, so one
cannot stand in for the other.

The three flows place these two axes in separate corners. The tracking query wants low latency;
the user waits for the response, and a single response's duration is directly visible to them.
End-of-day billing wants high throughput; 12 million records must be scanned in four hours, and
when any single record finishes does not matter. In one flow of the same system the unit of
scale is the millisecond, in the other it is the hour.

The critical consequence: increasing a flow's throughput can worsen its latency. The end-of-day
job can multiply its throughput by grouping records, but every record in a group then waits for
the group to finish. The reverse also happens: shortening latency can lower throughput, because
starting every job immediately means giving up the gain grouping provides.

There is one more asymmetry between the two metrics. Throughput is a single number: items of
work completed per unit time are counted, and that is the end of it. Latency is not a single
number; it is a **distribution** — under the same load some requests are answered quickly,
others slowly. This is why latency is reported in **percentiles**: the median is the duration
below which half of requests fall, p99 is the duration the top one percent exceeds. Reporting an
average latency carries the least information, because a single very slow request skews the
average and hides the distribution's shape.

## What Happens as Concurrency Increases

The setup below sets up the tracking endpoint as a local server. Every request spends two
things: a wait standing in for a record lookup, and a CPU share. Both are parameters of the
setup, not measurements.

```js
// tracking/server.mjs — tracking query endpoint: each request spends a wait and a CPU share
import { createServer } from "node:http";

const WAIT = 3;                    // wait standing in for a record lookup (ms), parameter of the setup
const CPU = 0.20;                  // CPU share per request (ms), parameter of the setup

function burnCpu(ms) {            // keeps the CPU busy for ms
  const end = performance.now() + ms;
  let t = 0;
  while (performance.now() < end) t += 1;
  return t;
}

const port = Number(process.argv[2]);
if (Number.isInteger(port) === false) console.log("usage: node tracking/server.mjs <port>");
else createServer(async (req, res) => {
  const no = new URL(req.url, "http://local").searchParams.get("no") ?? "-";
  await new Promise((c) => setTimeout(c, WAIT));
  burnCpu(CPU);
  const body = JSON.stringify({ no, state: "at transit hub", zone: "35", step: 4 });
  res.writeHead(200, { "content-type": "application/json" });
  res.end(body);
}).listen(port);
```

The client runs at seven **concurrency** levels. At each level, the stated number of requests is
kept in flight at all times; the number of requests sent over two seconds gives the throughput,
and the individually measured durations give the latency percentiles.

```js
// tracking/load.mjs — sends requests at increasing concurrency; each level prints throughput and latency
import { Agent, request } from "node:http";

const port = Number(process.argv[2]);
const LEVEL = [1, 2, 4, 8, 16, 32, 64];
const DURATION = 2000;             // measurement duration per level (ms)

const agent = new Agent({ keepAlive: true, maxSockets: 256 });

function oneRequest(no) {
  return new Promise((resolve, reject) => {
    const start = performance.now();
    const r = request({ port, path: `/tracking?no=${no}`, agent }, (res) => {
      res.resume();
      res.on("end", () => resolve(performance.now() - start));
    });
    r.on("error", reject);
    r.end();
  });
}

function percentile(arr, p) {
  const s = [...arr].sort((a, b) => a - b);
  return s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))];
}

async function level(n) {
  const latency = [];
  const end = performance.now() + DURATION;
  const start = performance.now();
  await Promise.all(Array.from({ length: n }, async (_, k) => {
    let i = 0;
    while (performance.now() < end) latency.push(await oneRequest(`G${k}-${i++}`));
  }));
  const elapsed = (performance.now() - start) / 1000;
  return { n, requests: latency.length, throughput: latency.length / elapsed,
    median: percentile(latency, 50), p99: percentile(latency, 99) };
}

if (Number.isInteger(port) === false) console.log("usage: node tracking/load.mjs <port>");
else {
  await oneRequest("warmup");
  console.log("concurr  requests  throughput(req/s)  median(ms)  p99(ms)  throughput*median");
  for (const n of LEVEL) {
    const s = await level(n);
    const little = (s.throughput * s.median) / 1000;
    console.log(`${String(s.n).padStart(6)}  ${String(s.requests).padStart(6)}  ` +
      `${s.throughput.toFixed(0).padStart(13)}  ${s.median.toFixed(2).padStart(10)}  ` +
      `${s.p99.toFixed(2).padStart(7)}  ${little.toFixed(2).padStart(22)}`);
  }
  agent.destroy();
}
```

The driver script starts the server, runs the measurement, and stops the server. The port number
depends on the environment; it should be changed if it is already in use on the machine.

```bash
# measure.sh — starts the server, measures seven concurrency levels, stops the server
node tracking/server.mjs 8811 & SERVER=$!
sleep 1
node tracking/load.mjs 8811
kill $SERVER
```

```
concurr  requests  throughput(req/s)  median(ms)  p99(ms)  throughput*median
     1     463            231        4.33     4.78                    1.00
     2     936            468        4.46     5.05                    2.09
     4    1764            880        4.80     5.38                    4.23
     8    3038           1515        5.67     6.14                    8.59
    16    4849           2417        6.65     7.57                   16.06
    32    8089           4031        8.21     9.56                   33.10
    64    8593           4268       15.15    16.46                   64.64
```

These numbers are measurements, taken on this machine. Their absolute values will not repeat on
another machine; what repeats is the table's **shape**.

## Two Regions

The table splits into two distinct regions, and the lesson's claim rests on this split.

In the first region, as concurrency rises from 1 to 4, throughput climbs from 231 to 880, that
is 3.81x; median latency climbs from 4.33 ms to 4.80 ms, that is 1.11x. Throughput grew nearly
in step with concurrency while latency stood still. **In this region the two metrics are
independent:** throughput is rising and says nothing about latency.

In the second region the table reverses. As concurrency rises from 32 to 64, throughput climbs
from 4031 to 4268, only 1.06x; median latency climbs from 8.21 ms to 15.15 ms, 1.85x. Doubling
concurrency buys 6% more throughput and double the latency. **In this region independence
ends:** throughput has hit an upper bound, and every added request only adds to the wait.

Where the upper bound comes from is written into the setup. Every request spends 0.20 ms of CPU
share, and the server runs on a single thread; the CPU share cannot be divided. Waits can
overlap; a CPU share cannot. This is the distinction from the Server-Side Fundamentals course:
waiting I/O is cheap, computation that occupies the CPU locks the queue.

## The Relation

The last column ties the two metrics together. In every row, the product of throughput and
median latency comes out equal to the concurrency level: 1.00, 2.09, 4.23, 8.59, 16.06, 33.10,
64.64. This is not a coincidence; it is queueing theory's fundamental relation. The Caching,
Queues and Asynchronous Processing course set it up as $L = \lambda W$; its counterpart here is:

$$
N = X \times R
$$

$N$ is the number of items of work in flight, $X$ is throughput, $R$ is latency. The design
consequence of the relation is direct: **if two of the three quantities are chosen, the third
cannot be chosen freely.** If the number of items in flight is held fixed and throughput cannot
rise, latency must grow — there is no other way out. The behavior in the second region is this
necessity made visible.

The same relation also gives the rule for reading a measurement. A report that states only
throughput says nothing about latency unless it also states the concurrency at which it was
taken; a report that states only latency cannot be read either unless it states the load.
Reporting a single metric on its own is an incomplete report.

## The Cost of Approaching Saturation

A model shows what happens at the entrance to the second region. In a single-server queue, with
service rate $\mu$ and arrival rate $\lambda$, **utilization** is defined as $\rho = \lambda /
\mu$, and the average time spent in the system is $1 / (\mu - \lambda)$. The computation below
feeds this model with the measured upper bound. **This is a model, not a measurement**; it
carries assumptions about the distribution of arrivals and service times and does not predict a
real server's latency. What it shows is the direction of the ratios.

```js
// calc/queue.mjs — single-server queue MODEL: the latency cost of approaching saturation
const MU = 4268;                   // measured upper bound: this setup's saturation throughput for a process (req/s)

console.log(`service rate mu = ${MU} req/s (measured upper bound)`);
console.log("utilization   arrival rate(req/s)   modeled average time(ms)   ratio to 30% case");
const base = 1000 / (MU - 0.30 * MU);
for (const rho of [0.30, 0.50, 0.70, 0.80, 0.90, 0.95, 0.99]) {
  const T = 1000 / (MU - rho * MU);
  console.log(`${(rho * 100).toFixed(0).padStart(6)}%  ${(rho * MU).toFixed(0).padStart(19)}  ` +
    `${T.toFixed(2).padStart(27)}  ${(T / base).toFixed(1).padStart(18)}`);
}

const peak = [416.67, 833.33];   // Back-of-the-Envelope Estimation: peak read rate, and if the query assumption doubles
for (const t of peak) {
  console.log(`peak ${t} req/s -> utilization ${((t / MU) * 100).toFixed(1)}%, ` +
    `upper bound is ${(MU / t).toFixed(1)}x peak, ` +
    `modeled average time ${(1000 / (MU - t)).toFixed(2)} ms`);
}
```

```
service rate mu = 4268 req/s (measured upper bound)
utilization   arrival rate(req/s)   modeled average time(ms)   ratio to 30% case
    30%                 1280                         0.33                 1.0
    50%                 2134                         0.47                 1.4
    70%                 2988                         0.78                 2.3
    80%                 3414                         1.17                 3.5
    90%                 3841                         2.34                 7.0
    95%                 4055                         4.69                14.0
    99%                 4225                        23.43                70.0
peak 416.67 req/s -> utilization 9.8%, upper bound is 10.2x peak, modeled average time 0.26 ms
peak 833.33 req/s -> utilization 19.5%, upper bound is 5.1x peak, modeled average time 0.29 ms
```

The model says one thing: as utilization rises, throughput grows linearly while time does not.
Going from 30% to 99% multiplies the arrival rate by 3.3 and the time by 70. The last slice of
throughput is bought with unbounded growth in latency. Keeping utilization high looks like an
efficiency win, and it is a latency decision.

The last two lines join the computation to the measurement. Peak read rate is 416.67 req/s, the
upper bound measured in the setup is 4268 req/s; the ratio is 10.2 and utilization stays at
9.8%. When queries per user double, the ratio becomes 5.1. What matters here is not the number
itself but the **method**: the peak rate is computed from assumptions, the upper bound is
measured from the setup, and their ratio gives the margin. A real tracking endpoint does more
than this setup, so its upper bound will be lower; the ratio is still read the same way. When it
falls below 1, the design no longer works; between 2 and 3, a single failure is narrow enough to
erase the entire margin.

## Summary

- Latency is the duration of a piece of work, throughput is the number of items of work
  completed per unit time; their units differ and one cannot substitute for the other.
- In the measurement, as concurrency rose from 1 to 4, throughput climbed to 3.81x while median
  latency stayed at 1.11x: in this region the two metrics are independent.
- As concurrency rose from 32 to 64, throughput climbed to 1.06x, latency climbed to 1.85x: once
  throughput hits its upper bound, every added request only adds to the wait.
- $N = X \times R$ held in every measured row; if two of the three quantities are chosen the
  third cannot be chosen freely, so throughput or latency reported alone cannot be read.
- The queueing model showed that time does not grow linearly with utilization: from 30% to 99%,
  arrival rate rises 3.3x while time rises 70x; keeping utilization high is a latency decision.
- The computed peak read rate is 416.67 req/s, the measured upper bound is 4268 req/s; the ratio
  is 10.2 and falls to 5.1 when queries per user double.

## Next Step

This lesson held a single process fixed and varied the load. The two regions that emerged show
why "this system is fast" is insufficient: a system's speed depends on the load at which it was
measured, and a number taken at low load says nothing about behavior at high load. Two separate
questions follow. First, how good is the system at a given load. Second, how long does that
goodness hold up as load grows. The two questions are often answered with a single word, and
should not be: one is named performance, the other scalability. The next lesson separates the
two concepts: it measures two implementations of the end-of-day job at four separate loads,
shows that both meet today's threshold, and reveals that the difference between them can only be
counted once the load is scaled up.