Collecting Data from Interfaces

Neither the file nor the database gave the data in a single piece. When the file was read, how many rows it had was known; when the query ran, the set was in hand. When collecting data from an interface, this assurance is missing: records arrive piece by piece, the calling side keeps track of how each piece connects to the next, the source puts a limit on call frequency, and the token determines which records can be seen.

The lesson’s measure is this: how many of the rows in the source entered the collected set. Interface design, pagination, and rate limiting are prerequisites from the point of view of the side that provides them and are not repeated here; the only things measured are how many records the consuming side received, how many it received twice, and how many it never asked for. Before the technical limit there is one more limit: collecting data from a source without the source’s terms of use and an explicit access permission is a legal and professional problem even if it is technically possible; here a permitted collection is assumed.

• DC10. The interface is fiction and is modeled with node. No network call is made: a request is a function call, a response is an object. It is a model.
• DC11. The set in the source is the same as the reading table in the previous lesson’s store — the same seed, 1,877 rows, 38,690.7 m³.
• DC12. The interface offers two orderings. The reading ordering is descending by reading value, and the internal order of equal values is rebuilt on every request; the id ordering is ascending by subscriber-period key and unique. The page size is at most 200, and anything beyond that is silently truncated.
• DC13. The rate limit quota is 12 requests. When the quota is exceeded, the response carries no records and its status is limit. Time is not measured with a real clock; the source’s window count is counted as a wait unit.
• DC14. The threshold for counting two totals as the same is 0.05 m³, because readings are reported with one decimal and a difference below that can only come from the order of summation.

The Page the Interface Gives

// api.mjs — the fictional reading interface of the regional measurement network. IT IS A
// MODEL: no network call is made, a "request" is a function call and a "response" is an
// object. The records are the same as the 1877 reading rows in the previous lesson's
// store: the same seed, the same numbers.
export const SEED = 4711, PAGE_SIZE = 50, MAX_SIZE = 200, QUOTA = 12;
const rng = (t) => { let s = t >>> 0;
  return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; }; };

// The reading is kept as an integer in deci-m3; since the sort key is an integer,
// floating-point values never enter an equality comparison.
const r = rng(SEED);
export const RECORDS = [];
for (let i = 0; i < 1400; i++)
  RECORDS.push({ subscriber_no: 10000 + i, period: "2024-01", deci: 50 + Math.floor(r() * 321) });
for (let i = 0; i < 477; i++)
  RECORDS.push({ subscriber_no: 10000 + i, period: "2024-02", deci: 50 + Math.floor(r() * 321) });
export const REGION = ["north", "east", "central", "south", "west"];
export const recordId = (k) => `${k.subscriber_no}-${k.period}`;
export const m3 = (k) => k.deci / 10;
export const regionOf = (k) => REGION[(k.subscriber_no - 10000) % 5];

// "reading": descending by reading, the internal order of equal values is rebuilt on
// every request. "id": ascending by subscriber-period key and unique.
function sorted(order, requestNo) {
  if (order === "id") return [...RECORDS].sort((a, b) => (recordId(a) < recordId(b) ? -1 : 1));
  const g = rng(SEED + requestNo * 7919);
  for (let i = 0; i < 5; i++) g();
  return RECORDS.map((k) => ({ k, p: g() })).sort((a, b) => b.k.deci - a.k.deci || a.p - b.p)
    .map((x) => x.k);
}

// The quota is the number of requests accepted in one window; when it is exceeded, the
// response carries no records. If the scope does not include "history", the source
// returns only the open period.
export function server({ quota = QUOTA } = {}) {
  let requests = 0, inWindow = 0, waits = 0;
  return {
    counts: () => ({ requests, waits }),
    wait() { inWindow = 0; waits++; },
    request({ scope = ["reading"], order = "id", offset = 0, cursor = null, size = PAGE_SIZE }) {
      requests++; inWindow++;
      if (inWindow > quota) return { status: "limit", records: [], next: cursor };
      const b = Math.min(size, MAX_SIZE);
      let set = sorted(order, requests);
      if (!scope.includes("history")) set = set.filter((k) => k.period === "2024-01");
      const start = cursor === null ? offset : set.findIndex((k) => recordId(k) > cursor);
      const slice = start < 0 ? [] : set.slice(start, start + b);
      return { status: "ok", total: set.length,
        records: slice.map((k) => ({ subscriber_no: k.subscriber_no, period: k.period,
          reading_m3: m3(k), region: regionOf(k) })),
        next: slice.length < b ? null : recordId(slice[slice.length - 1]) };
    },
  };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const s = server({ quota: Infinity }), FULL = ["reading", "history"];
  const sum = (y) => `status ${y.status}, total ${y.total}, ${y.records.length} ` +
    `records on the page, next cursor ${y.next}`;
  console.log(`seed ${SEED}; ${RECORDS.length} reading rows, ` +
    `${RECORDS.reduce((t, k) => t + m3(k), 0).toFixed(1)} m3; size ${PAGE_SIZE}, upper limit ` +
    `${MAX_SIZE}, quota ${QUOTA}`);
  console.log(`full scope: ${sum(s.request({ scope: FULL }))}`);
  console.log(`narrow scope: ${sum(s.request({ scope: ["reading"] }))}`);
  const u = () => s.request({ scope: FULL, order: "reading", offset: 49, size: 3 })
    .records.map((x) => `${x.subscriber_no}/${x.reading_m3}`).join("  ");
  console.log(`three records from the 50th record in the "reading" order, in two separate requests:\n  ${u()}\n  ${u()}`);
}

seed 4711; 1877 reading rows, 38690.7 m3; size 50, upper limit 200, quota 12
full scope: status ok, total 1877, 50 records on the page, next cursor 10024-2024-02
narrow scope: status ok, total 1400, 50 records on the page, next cursor 10049-2024-01
three records from the 50th record in the "reading" order, in two separate requests:
  11319/36  10452/36  10828/36
  11319/36  10018/36  10230/36

The difference between the two tokens is on the third line. The narrow-scope token returns the ok status, the page comes back full, the cursor advances, and there is no authorization warning anywhere. The only sign is that the total field says 1,400 instead of 1,877, and whether this number is correct cannot be understood from inside the response. Scope is not a constraint that produces an error; it shrinks the set the source gives and presents the rest as if it never existed.

The last two lines show the second problem: three records were requested twice from the same offset, and the records that came back are different. The sort key is the reading value, and that value is not unique — the 1,877 records are spread over at most 321 distinct reading values, that is, each value is shared by about six records. The order of equal values among themselves is left to the source’s decision. The only thing that looks fixed is the position in the list, but which record stands at that position is not fixed.

Same Interface, Three Collection Decisions

// collect.mjs — three collection decisions against the same interface produce three
// different sets. No decision gets an error; all responses return "ok". What is
// measured is not the number of records returned but how many of the 1877 rows in the
// source entered the collected set.
import { RECORDS, recordId, m3, regionOf, REGION, server, PAGE_SIZE } from "./api.mjs";
const FULL = ["reading", "history"];

// Cursor-based pagination: each page asks for what comes after the last key, so the
// boundary does not shift. T1 does this with a narrow token, T3 with a full-scope token.
function withCursor(scope) {
  const s = server({ quota: Infinity }), collected = []; let cursor = null;
  do { const y = s.request({ scope, order: "id", cursor, size: PAGE_SIZE });
    collected.push(...y.records); cursor = y.next; } while (cursor !== null);
  return { collected, requests: s.counts().requests };
}
// T2. Offset-based pagination, in the "reading" order: since the key is not unique,
// the page boundary falls on a different record on every request.
function withOffset() {
  const s = server({ quota: Infinity }), collected = []; let offset = 0, total = Infinity;
  while (offset < total) {
    const y = s.request({ scope: FULL, order: "reading", offset, size: PAGE_SIZE });
    total = y.total; collected.push(...y.records); offset += PAGE_SIZE;
  }
  return { collected, requests: s.counts().requests };
}

// Threshold 0.05 m3: readings are kept with one decimal, and a difference below that
// can only come from the order of summation.
const THRESHOLD = 0.05, TARGET = RECORDS.reduce((t, k) => t + m3(k), 0);
const REGION_TARGET = new Map(REGION.map((b) => [b,
  RECORDS.filter((k) => regionOf(k) === b).reduce((t, k) => t + m3(k), 0)]));
const dev = (set) => {                  // the largest deviation of the region breakdown
  const t = new Map(REGION.map((b) => [b, 0]));
  for (const x of set) t.set(x.region, t.get(x.region) + x.reading_m3);
  return Math.max(...REGION.map((b) => Math.abs(t.get(b) - REGION_TARGET.get(b))));
};
const y1 = (h) => h.map((v, j) => (j ? String(v).padStart([0, 9, 10, 9, 9, 9, 12, 11, 12][j])
  : String(v).padEnd(20))).join("");
console.log(y1(["collection decision", "rows in", "requests", "rows out", "distinct", "skipped",
  "duplicated", "total m3", "max dev m3"]));
const result = {};
for (const [name, f] of [["T1 narrow scope", () => withCursor(["reading"])],
  ["T2 offset-based", withOffset], ["T3 cursor-based", () => withCursor(FULL)]]) {
  const r = result[name] = f(), distinct = new Set(r.collected.map(recordId)).size;
  r.distinct = distinct; r.total = r.collected.reduce((t, x) => t + x.reading_m3, 0);
  console.log(y1([name, RECORDS.length, r.requests, r.collected.length, distinct,
    RECORDS.length - distinct, r.collected.length - distinct, r.total.toFixed(1),
    dev(r.collected).toFixed(1)]));
}
const t2 = result["T2 offset-based"];
console.log(`\nT2 matched the ${t2.collected.length} rows the source reported, but ` +
  `${RECORDS.length - t2.distinct} records never arrived and ${t2.collected.length - t2.distinct} records ` +
  `arrived twice; because the skipped and duplicated records carried the same reading value, the overall total ` +
  `differs from the target by ${Math.abs(t2.total - TARGET).toFixed(3)} m3 (threshold ${THRESHOLD}), while in the region ` +
  `breakdown it deviates by ${Math.round(dev(t2.collected) / THRESHOLD)} times the threshold`);

collection decision   rows in  requests rows out distinct  skipped  duplicated   total m3  max dev m3
T1 narrow scope          1877        29     1400     1400      477           0    28790.8      2016.2
T2 offset-based          1877        38     1877     1842       35          35    38690.7        70.5
T3 cursor-based          1877        38     1877     1877        0           0    38690.7         0.0

T2 matched the 1877 rows the source reported, but 35 records never arrived and 35 records arrived twice; because the skipped and duplicated records carried the same reading value, the overall total differs from the target by 0.000 m3 (threshold 0.05), while in the region breakdown it deviates by 1410 times the threshold

The last column is the largest deviation in the region breakdown.

T1 consumes the pagination correctly and still misses a quarter of the set. All twenty-nine requests came back successful, the cursor advanced properly, and the loop stopped in the right place. The loss comes entirely from the scope: the 477 rows of the past period were never requested, and the total came out at 28,790.8 m³, that is, 0.74 times the truth. A collector finishing successfully does not say which set it was authorized to see; the scope question is asked before collecting.

T2 is the lesson’s main finding. The number of rows collected is 1,877, exactly the same as the number the source reported; a check that looks at “did as many records arrive as expected” passes here. Yet when the record ids are counted, 35 records never arrived and 35 records arrived twice. Because the internal order of equal readings is rebuilt on every request, the page boundary fell on a different record each time: one moved to the end of the previous page and was skipped, another fell at the start of the next page and arrived again.

Why the error is not visible in the measure is the cleanest example of silent loss. Because the skipped and duplicated records came from the same cluster of equal values, they carry the same reading value: the total stays at 38,690.7 m³ and the difference is below the 0.05 m³ threshold. The damage appears only in a breakdown that depends on the id — in the region breakdown the largest deviation is 70.5 m³, 1,410 times the threshold.

T3 collects the same 1,877 records once in 38 requests; the only thing that makes the difference is that the cursor advances over a unique and stable key. Offset-based pagination assumes the list stays fixed between requests, and this assumption is wrong both with a non-unique sort key and with a source that gains records while collection is going on.

The Queue That Obeys the Rate Limit

// queue.mjs — the collection queue under a rate limit. There is no throttling technique:
// what is measured is how many wait units the queue that obeys the limit spends and how
// many records the queue that does not obey it leaves out. Time is not measured with a
// real clock; the source's window count is counted as a wait unit.
import { RECORDS, recordId, server, PAGE_SIZE, MAX_SIZE, QUOTA } from "./api.mjs";
const FULL = ["reading", "history"];

// H1: the queue that counts the limit response as an empty page. When no records
// arrive, it says "the set is finished".
function h1() {
  const s = server(), collected = []; let cursor = null, y;
  do { y = s.request({ scope: FULL, cursor, size: PAGE_SIZE });
    collected.push(...y.records); cursor = y.next;
  } while (y.records.length > 0 && cursor !== null);
  return { collected, ...s.counts() };
}
// H2: the queue that obeys the limit. It counts the rate limit quota itself and waits when it is full.
function h2(size) {
  const s = server(), collected = []; let cursor = null, inWindow = 0;
  do { if (inWindow === QUOTA) { s.wait(); inWindow = 0; }
    const y = s.request({ scope: FULL, cursor, size });
    inWindow++; collected.push(...y.records); cursor = y.next;
  } while (cursor !== null);
  return { collected, ...s.counts() };
}
const y1 = (h) => h.map((v, j) => (j ? String(v).padStart([0, 9, 10, 12, 10, 9][j])
  : String(v).padEnd(26))).join("");
console.log(y1(["queue", "rows in", "requests", "wait units", "rows out", "skipped"]));
for (const [name, f] of [["H1 treats limit as empty", h1], [`H2 compliant, size ${PAGE_SIZE}`,
  () => h2(PAGE_SIZE)], ["H2 compliant, size 20", () => h2(20)],
  ["H2 compliant, size 100", () => h2(100)],
  [`H2 compliant, size ${MAX_SIZE}`, () => h2(MAX_SIZE)]]) {
  const r = f(), distinct = new Set(r.collected.map(recordId)).size;
  console.log(y1([name, RECORDS.length, r.requests, r.waits, r.collected.length,
    RECORDS.length - distinct]));
}

queue                       rows in  requests  wait units  rows out  skipped
H1 treats limit as empty       1877        13           0       600     1277
H2 compliant, size 50          1877        38           3      1877        0
H2 compliant, size 20          1877        94           7      1877        0
H2 compliant, size 100         1877        19           1      1877        0
H2 compliant, size 200         1877        10           0      1877        0

H1 is the costliest misunderstanding of a rate limit. Because the thirteenth request exceeded the quota, it came back carrying no records, the collector read this as “the set is finished”, and the process ended with a zero exit code: 600 records were collected, 1,277 records were left out. An empty page and a rejected request look alike; if they are not told apart, the difference shows up only in the row count.

H2 collects the same set completely, and its cost is 3 wait units. The last three rows show where this cost comes from: the wait unit comes from dividing the number of requests by the rate limit quota. With a page size of 20, 94 requests and 7 wait units are needed; with a size of 200, 10 requests are enough and the collection ends in a single window. The collected records are 1,877 in all four rows. Page size is not a detail, it is directly a collection time decision, and the source’s upper limit determines the lower limit of this time. According to DC12, when an attempt is made to exceed this upper limit, no error comes either: a queue that asks for a 400-record page and advances its position by 400 silently skips 200 records on every turn, because the source gives 200. The number of records on the page that arrives is counted; the requested size is not assumed.

Summary

• In a set collected from an interface, the number to verify is not the number of records returned but how many of the rows in the source entered the set once.
• A narrow-scope token leaves out 477 rows without producing any warning, and the total drops to 0.74 times the truth; the scope question is asked before collecting.
• Offset-based pagination on a non-unique key skips 35 records and collects 35 records twice; the row count and the overall total look correct, and the damage shows up in the region breakdown as a 70.5 m³ deviation.
• A cursor advancing over a unique and stable key collects the same set as 1,877 records, zero skipped, and zero duplicated.
• The queue that counts the rate limit response as an empty page stops at 600 records and loses 1,277 records; the queue that obeys the limit spends 38 requests and 3 wait units, and when the page size is raised to 200 the wait drops to zero.

Next Step

The source of this lesson’s data was an interface designed for the party that receives its data: the field names were known, the page limit was declared, and the number of records was written inside the response. Part of the data, however, does not stand behind such an interface; it stands inside the structure of a page arranged for human eyes. There are no field names there; there are only position and appearance markers, and these are not a contract. The next lesson parses a fictional publication page and measures a single thing: when the structure changes, how many selectors break, how many rows drop, and how many rows survive having been read from the wrong field.