Performance and Scalability

The previous lesson held a single process fixed and varied the load, producing two regions: one where throughput grows with concurrency, and one where throughput hits an upper bound and latency grows instead. The same system showing two different behaviors at two different loads also showed how little a single measurement says about a system.

Two questions follow from this. How good is the system at a given load, and how long does that goodness hold up as load grows. This lesson answers the two questions with separate names and counts the difference on the end-of-day billing job. The only way to make the distinction visible is measuring at more than one load; a measurement at a single load does not even say which question it answers.

A Point Versus a Curve

Performance is the metric value measured at a given load: “this job takes 460 milliseconds on this machine with this many records” is a performance statement. It is a single point and says nothing outside that point.

Scalability is how the metric changes with load: how many times the duration grows when the load doubles. It is not a point but a curve. Nothing about a system’s scalability can be inferred from a single measurement, because a curve cannot be drawn through one point.

Confusing the two concepts leads to a practical mistake: a change that improves performance is treated as having improved scalability. The measurement below shows why that is wrong. In the Quality Attributes lesson’s list of quality families, performance falls under the performance-efficiency family; scalability is not in the list, and since the list is not closed, it is added here as a separate family. The two also differ in the measurement part of their quality scenarios: one states a threshold at a fixed load, the other states behavior as the load grows.

Two Implementations of the Same Job

End-of-day billing sums items into invoice lines. An invoice line corresponds to one seller’s one day; from here on this grouping unit is called a partition, and the seller number together with the day form the partition key. The two implementations produce the same result by different routes. A scans every item for each partition. B passes over the items once and uses the partition key as an index.

The number of items and the number of partitions are not independent. The assumption that gives the ratio stands as V13 in the Back-of-the-Envelope Estimation lesson’s table: 100 items are grouped into one invoice line. This produces a chain that holds with V3: 400,000 shipments a day, at this ratio, means 4000 invoice lines a day, that is 4000 sellers. The billing job scanning thirty days therefore sees 120,000 partitions: 4000 sellers times 30 days. The assumption’s sensitivity will be shown in the computation.

// batch/grouping.mjs — two implementations of end-of-day billing measured at the same four loads
const TARIFF = { tier: [[1, 3000], [5, 4800], [20, 9600]], zone: { "34": 100, "06": 115, "35": 125 }, minimum: 2500 };
const ZONES = ["34", "06", "35"];
const PER_PARTITION = 100;          // V13: items grouped into one invoice line

function items(n, partitions) {
  const d = new Array(n);
  for (let i = 0; i < n; i += 1) {
    d[i] = { partition: i % partitions, weight: (i % 20) + 1, zone: ZONES[i % 3], rate: (i % 5) / 20 };
  }
  return d;
}

function net(k) {
  const tier = TARIFF.tier.find(([s]) => k.weight <= s) ?? [0, 9600];
  const base = Math.max(TARIFF.minimum, Math.round((tier[1] * TARIFF.zone[k.zone]) / 100));
  return base - Math.round(base * Math.min(k.rate, 0.4));
}

function billingA(data, partitions) {    // every item is scanned for each partition
  const result = new Array(partitions).fill(0);
  for (let b = 0; b < partitions; b += 1) {
    for (const k of data) if (k.partition === b) result[b] += net(k);
  }
  return result;
}

function billingB(data, partitions) {    // single pass, aggregated by the partition key
  const result = new Array(partitions).fill(0);
  for (const k of data) result[k.partition] += net(k);
  return result;
}

function measure(f, data, partitions) {
  const start = performance.now();
  const r = f(data, partitions);
  return { ms: performance.now() - start, signature: r[0] + r[partitions - 1] };
}

// A ratio: how many times the duration grows when the load doubles. A order: log2(ratio) — 1
// linear, 2 quadratic growth. Raw ms depends on the machine, order is a property of the
// implementation.
console.log("items     part.      A(ms)     B(ms)     A/B  A us/item   B us/item   A ratio  order  same");
let previousA = 0;
for (const n of [20_000, 40_000, 80_000, 160_000]) {
  const partitions = n / PER_PARTITION;
  const data = items(n, partitions);
  const a = measure(billingA, data, partitions);
  const b = measure(billingB, data, partitions);
  const ratio = previousA === 0 ? null : a.ms / previousA;
  previousA = a.ms;
  console.log(`${String(n).padStart(7)}${String(partitions).padStart(10)}` +
    `${a.ms.toFixed(1).padStart(11)}${b.ms.toFixed(1).padStart(10)}${(a.ms / b.ms).toFixed(0).padStart(8)}` +
    `${((a.ms * 1000) / n).toFixed(2).padStart(12)}${((b.ms * 1000) / n).toFixed(2).padStart(12)}` +
    `${(ratio === null ? "-" : ratio.toFixed(2)).padStart(8)}${(ratio === null ? "-" : Math.log2(ratio).toFixed(2)).padStart(9)}` +
    `${String(a.signature === b.signature).padStart(6)}`);
}

These numbers are measurements, taken on this machine; another machine gives different values. The last two columns exist for this reason: A ratio and order are quantities derived from the raw durations, and they do not change from machine to machine. The same column confirms the two implementations produce the same totals; the durations of two implementations that give different results cannot be compared.

items     part.      A(ms)     B(ms)     A/B  A us/item   B us/item   A ratio  order  same
  20000       200       13.3       1.4       9        0.67        0.07       -        -  true
  40000       400       32.2       0.7      45        0.81        0.02    2.42     1.28  true
  80000       800      121.9       1.8      66        1.52        0.02    3.78     1.92  true
 160000      1600      471.7       2.9     160        2.95        0.02    3.87     1.95  true

What the Table Says

Absolute durations are not read in the fourth and fifth columns, but in the per-item columns.

B’s per-item duration is 0.07 at the first load and 0.02 microseconds three times after; the difference in the first row comes from warm-up, the rest is constant. When the load rises eightfold, per-item duration does not change: the job’s cost is directly proportional to the number of items.

A’s per-item duration rises from 0.67 to 2.95 microseconds in this run. As the load rises eightfold, per-item cost also grows, because every item is scanned for more partitions. The real reading is in the order column. A value of 1 would mean duration grows in direct proportion to load; 2 means duration grows with the square of the load. In the measurement, the last two doublings give 1.92 and 1.95, that is, when the load doubles, the duration quadruples. The first doubling stays at 1.28; the reason is that the runtime has not warmed up yet at the smallest load, and the computation below therefore takes its coefficient from the largest measurement, not the smallest. Absolute durations vary by machine, order does not — that is why the column was added.

Now the critical observation. In the first row, the ratio between A and B is around 10, and A’s absolute duration does not exceed fifteen milliseconds — no problem at all for a batch job. A performance measurement taken at this load finds A acceptable, and is right to. In the fourth row the ratio exceeds a hundred (160 in this run). Same system, same code, same machine; the only thing that changed is the load. The ratio itself is a function of the load, so the sentence “A is this many times slower than B” is meaningless when stated without a load.

Capacity

What turns the curve into a decision is a threshold. The Back-of-the-Envelope Estimation lesson computed that the end-of-day job runs in a four-hour window and scans 12 million records spanning thirty days. Capacity is the largest load that meets the threshold. The computation below takes the measured coefficients, finds each implementation’s duration at today’s volume and at two and four times that volume, then derives each one’s capacity.

// calc/capacity.mjs — capacity in a four-hour window from measured coefficients, and the effect of a constant-factor speedup
const WINDOW = 4 * 3600 * 1000;    // V10: batch job window (ms)
const DAILY = 400_000;             // V3: new shipments created per day
const DAYS = 30;                   // V11: days billing scans
const TODAY = DAILY * DAYS;        // scanned records
const PER_PARTITION = 100;         // assumption: items grouped into one invoice line

console.log(`scanned records = ${TODAY}, daily partitions = ${DAILY / PER_PARTITION} (= seller), ` +
  `partitions over ${DAYS} days = ${TODAY / PER_PARTITION}`);

const MEASURED = { n: 160_000, partitions: 1_600, aMs: 471.7, bMs: 2.9 };  // batch/grouping.mjs, last row
const aUnit = MEASURED.aMs / (MEASURED.n * MEASURED.partitions);   // ms / check
const bUnit = MEASURED.bMs / MEASURED.n;                           // ms / item
console.log(`input: a single run's measurement on this machine (A ${MEASURED.aMs} ms, B ${MEASURED.bMs} ms, ` +
  `n = ${MEASURED.n}); every duration and capacity below depends on these two numbers`);

const aTime = (n) => aUnit * n * (n / PER_PARTITION);
const bTime = (n) => bUnit * n;
const hours = (ms) => ms / 3_600_000;

console.log(`A unit = ${aUnit.toExponential(2)} ms/check, B unit = ${bUnit.toExponential(2)} ms/item`);
console.log("records        A time(hr)       B time(hr)       A fits window");
for (const mult of [1, 2, 4]) {
  const n = TODAY * mult;
  console.log(`${String(n).padStart(9)} (x${mult})  ${hours(aTime(n)).toFixed(3).padStart(13)}  ` +
    `${hours(bTime(n)).toFixed(5).padStart(15)}  ${String(aTime(n) <= WINDOW).padStart(17)}`);
}

const aCapacity = (unit) => Math.sqrt((WINDOW * PER_PARTITION) / unit);
const bCapacity = (unit) => WINDOW / unit;
console.log(`\nA capacity = ${(aCapacity(aUnit) / 1e6).toFixed(1)} million records ` +
  `(${(aCapacity(aUnit) / TODAY).toFixed(2)}x today)`);
console.log(`B capacity = ${(bCapacity(bUnit) / 1e9).toFixed(0)} billion records ` +
  `(${(bCapacity(bUnit) / TODAY).toFixed(0)}x today)`);
console.log(`if the implementation were 100x faster: A capacity x${(aCapacity(aUnit / 100) / aCapacity(aUnit)).toFixed(0)}, ` +
  `B capacity x${(bCapacity(bUnit / 100) / bCapacity(bUnit)).toFixed(0)}`);

const half = aUnit * TODAY * (TODAY / (PER_PARTITION / 2));
console.log(`if the assumption were 50 instead of 100: A time ${hours(half).toFixed(3)} hr ` +
  `(x${(half / aTime(TODAY)).toFixed(0)}), B time unchanged`);

scanned records = 12000000, daily partitions = 4000 (= seller), partitions over 30 days = 120000
input: a single run's measurement on this machine (A 471.7 ms, B 2.9 ms, n = 160000); every duration and capacity below depends on these two numbers
A unit = 1.84e-6 ms/check, B unit = 1.81e-5 ms/item
records        A time(hr)       B time(hr)       A fits window
 12000000 (x1)          0.737          0.00006               true
 24000000 (x2)          2.948          0.00012               true
 48000000 (x4)         11.793          0.00024              false

A capacity = 28.0 million records (2.33x today)
B capacity = 794 billion records (66207x today)
if the implementation were 100x faster: A capacity x10, B capacity x100
if the assumption were 50 instead of 100: A time 1.474 hr (x2), B time unchanged

The first line shows the partition count holds with V3: 4000 partitions a day, that is 4000 sellers, 120,000 partitions over thirty days. The partition count should not be confused with the seller count — one is a daily figure, the other a period figure, and A’s cost depends on the period figure.

The second line states what the entire computation rests on: the input is the measurement of a single run. Every hour and capacity value below changes together with those two numbers; what stays fixed is the ratios between them and the shape of the curves.

The third line is the lesson’s most important result: A meets today’s threshold. It scans 12 million records in 0.737 hours, that is forty-four minutes; well within the four-hour window. An acceptance test passes this implementation, and a performance report shows it as flawless.

When volume doubles, A rises to 2.948 hours — still within the window, but its margin has shrunk fourfold. At four times volume it is 11.793 hours, and the window breaks. Duration quadrupling while volume doubles is the definition of a scalability problem; if it were a performance problem, duration would have doubled too.

The capacity lines reduce the same thing to a single number. By this measurement, A carries 28.0 million records, 2.33 times today; the number changes on another machine, but being a small multiple of today does not. B carries 794 billion — this number is an arithmetic extrapolation, and the memory limit arrives long before that volume; its meaning is “in practice this job no longer runs into a time limit.” The decision between the two implementations rests on these margins, not on the measured durations.

A Constant Factor Does Not Beat the Order

The last two lines separate two distinct kinds of improvement.

If the implementation is sped up 100x — a cheaper comparison, a tighter loop, a better data layout — A’s capacity rises only 10x. Because duration grows with the square of the load, capacity grows with the square root of the speedup. The same 100x gain raises B’s capacity 100x. A constant-factor gain does not pay back in kind on a solution with a poor order of growth.

The last line gives the assumption’s sensitivity. If 50 items were grouped into an invoice line instead of 100, the partition count would double; A’s duration rises to 1.474 hours, that is doubles, B’s duration does not change at all. The assumption concerns only one implementation. When an assumption being wrong breaks one design without affecting the other, that is the real difference between the two designs.

Summary

• Performance is a single value measured at a given load; scalability is how the metric changes with load and cannot be inferred from a single measurement.
• In the measurement, B’s per-item duration stayed constant across an eightfold load increase (0.02 microseconds), while A’s duration grew; the derived order column gave 1.92 and 1.95 in the last two doublings, that is, duration grows with the square of the item count. Raw durations depend on the machine, order does not.
• The ratio between A and B is a function of the load: around 10 at the smallest load, exceeding a hundred at the largest. A ratio reported without a load is meaningless.
• A meets today’s threshold: by this measurement it scans 12 million records in 0.737 hours. At four times volume it rises to 11.793 hours and breaks the four-hour window.
• By this measurement, capacity for A is 28.0 million records, 2.33 times today; for B the time limit is not binding in practice and the memory limit takes its place.
• If the implementation speeds up 100x, A’s capacity rises 10x, B’s capacity rises 100x; a constant-factor gain does not beat the order of growth.

Next Step

This lesson varied the load and held the resource fixed: every measurement was taken with a single process, a single worker. The capacity computation is valid under this same assumption — so was the Back-of-the-Envelope Estimation lesson’s 1.20-millisecond-per-record share. A resource is a quantity that can be changed, and there are two ways to change it: grow the same unit, or increase the number of units. The two hit different limits, require different conditions, and their costs grow along different curves. The next lesson measures the end-of-day job’s speedup as the worker count rises from 1 to 16, finds where the speedup stops, and compares the cost of the two paths under the same threshold.