Partial Response and Field Selection

The previous two lessons settled which records a collection returns, in what order, and how many. How much of each record comes back stayed open.

The library interface has two screens. The loan list screen shows only a book’s title, a member’s name, and the issue date on every row. The loan detail screen wants every field of the record. Both draw from the same collection. If the server returns a single representation, that representation is either too narrow for the list or too wide for the detail. Both cases carry a measurable cost.

Over-Fetching and Under-Fetching

Over-fetching is sending fields the client does not use. If the loan list screen shows three fields while the server sends ten, the difference travels the network, gets parsed on the client, and is thrown away.

Under-fetching is a single request failing to feed the screen. If a loan record carries only an ISBN and a member code, getting the book’s title and the member’s name needs two more requests per row. A forty-row list means eighty extra requests.

The two look like opposites but come from the same root: the server alone decides the representation’s level of detail. The fix is to hand part of that decision to the client — but as flexibility whose boundaries the server draws.

// field-selection-server.mjs — loan collection supporting field selection and expansion
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const db = new DatabaseSync("library.db");
const respond = (response, status, data) => {
  response.writeHead(status, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(data));
};

// Allowlist: a field the client names is translated to a column expression here.
const BASE = { id: "o.id", member: "o.member", isbn: "o.isbn", issuedAt: "o.issuedAt", return: "o.return" };
const EXPANDED = { bookTitle: "k.title", bookAuthor: "k.author", memberName: "u.name", memberBranch: "u.branch" };

const server = createServer((request, response) => {
  const url = new URL(request.url, "http://127.0.0.1");
  const s = url.searchParams;

  if (/^\/books\/[^/]+$/.test(url.pathname)) {
    const isbn = url.pathname.split("/")[2];
    const b = db.prepare("SELECT * FROM book WHERE isbn = ?").get(isbn);
    return b ? respond(response, 200, b) : respond(response, 404, { error: "book_not_found" });
  }
  if (/^\/members\/[^/]+$/.test(url.pathname)) {
    const code = url.pathname.split("/")[2];
    const m = db.prepare("SELECT * FROM member WHERE code = ?").get(code);
    return m ? respond(response, 200, m) : respond(response, 404, { error: "member_not_found" });
  }
  if (url.pathname !== "/loans") return respond(response, 404, { error: "path_not_found" });

  // Expansion: fields of related resources are folded into the response.
  const expand = (s.get("expand") ?? "").split(",").filter(Boolean);
  const offered = { ...BASE };
  if (expand.includes("book")) Object.assign(offered, { bookTitle: EXPANDED.bookTitle, bookAuthor: EXPANDED.bookAuthor });
  if (expand.includes("member")) Object.assign(offered, { memberName: EXPANDED.memberName, memberBranch: EXPANDED.memberBranch });

  // Field selection: requested names are intersected with what's offered, unknown names are rejected.
  const requested = (s.get("fields") ?? "").split(",").filter(Boolean);
  const unknown = requested.filter((a) => !(a in offered));
  if (unknown.length)
    return respond(response, 422, { error: "validation", field: "fields", unknown, offered: Object.keys(offered) });
  const selected = requested.length ? requested : Object.keys(offered);

  const selection = selected.map((a) => `${offered[a]} AS ${a}`).join(", ");
  const rows = db.prepare(
    `SELECT ${selection} FROM loan o
       LEFT JOIN book k ON k.isbn = o.isbn
       LEFT JOIN member u ON u.code = o.member
      ORDER BY o.id LIMIT ?`
  ).all(Number(s.get("size") ?? 40));

  respond(response, 200, { data: rows, pagination: { size: rows.length } });
});

server.listen(8482, "127.0.0.1", () => console.log("field selection server 127.0.0.1:8482"));

Column names come from an allowlist again; the previous lesson’s distinction applies here too. The client sends the name bookTitle, the server translates it to the expression k.title. A name not on the list can never reach the query text.

Measurement

Three different strategies feed the same screen. The measurement counts the request count and the transferred body bytes.

// measurement.mjs — measures the request count and transferred bytes of three strategies
const T = "http://127.0.0.1:8482";
let requests = 0, bytes = 0;
const call = async (path) => {
  requests++;
  const text = await (await fetch(T + path)).text();
  bytes += Buffer.byteLength(text);
  return JSON.parse(text);
};
const report = (name) => { console.log(`${name.padEnd(34)} requests: ${String(requests).padStart(3)}  bytes: ${bytes}`); requests = 0; bytes = 0; };

// 1) Separately: the loan list, then the book and member for every record.
const list = await call("/loans");
for (const l of list.data) { await call(`/books/${l.isbn}`); await call(`/members/${l.member}`); }
report("separate requests (under-fetching)");

// 2) Expansion: a single request, but every field comes along.
await call("/loans?expand=book,member");
report("expansion (over-fetching)");

// 3) Expansion + field selection: a single request, only the fields the screen uses.
await call("/loans?expand=book,member&fields=id,issuedAt,bookTitle,memberName");
report("expansion + field selection");

# Forty loan records; the same screen is fed by three different strategies.
rm -f library.db && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql
sqlite3 library.db <<'SQL'
DELETE FROM loan;
WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n < 40)
INSERT INTO loan (member, isbn, issuedAt, return)
SELECT CASE WHEN n % 2 = 0 THEN 'U-1001' ELSE 'U-1002' END,
       'K-0' || (1 + n % 9), date('2026-01-01', '+' || n || ' day'), NULL
FROM s;
SQL
node field-selection-server.mjs & server=$!
sleep 0.4

node measurement.mjs
echo "--- sample response row ---"
curl -sS "http://127.0.0.1:8482/loans?expand=book,member&fields=id,issuedAt,bookTitle,memberName&size=2"; echo
curl -sS "http://127.0.0.1:8482/loans?fields=id,bookTitle&size=2"; echo

kill $server

field selection server 127.0.0.1:8482
separate requests (under-fetching) requests:  81  bytes: 8968
expansion (over-fetching)          requests:   1  bytes: 7368
expansion + field selection        requests:   1  bytes: 3741
--- sample response row ---
{"data":[{"id":1,"issuedAt":"2026-01-02","bookTitle":"Data Structures and Algorithms","memberName":"Ben Turner"},{"id":2,"issuedAt":"2026-01-03","bookTitle":"Compilers","memberName":"Alina Drake"}],"pagination":{"size":2}}
{"error":"validation","field":"fields","unknown":["bookTitle"],"offered":["id","member","isbn","issuedAt","return"]}

The first line shows under-fetching’s cost: eighty-one requests for thirty-nine records. The byte difference is not the real issue here; the request count is. Every request means a network round trip, and when trips wait on each other, latency multiplies directly. There is also repetition, since the same book and the same member get requested over and over.

The second line drops to a single request but carries 7368 bytes. Six fields the screen does not use — ISBN, member code, return date, book author, member branch — appear on every row.

The third line solves both together: a single request, 3741 bytes. The same screen is fed with roughly half the data. The ratio depends on the data; here the savings come from how large a share field names and unused values take up in the total body.

The last line is the contract defending itself. The request fields=id,bookTitle arrived without expand=book; bookTitle is not among the fields offered in that request. The server rejects the request with 422 and reports both the unknown name and the fields offered in that request. Silently dropping the field would have led the client to expect a field that was never there and render the screen with an undefined value.

The Contract’s Cost for This Flexibility

Field selection is not free, and its cost is paid on the contract surface.

First, cache keys multiply. The address is a cache key, and because the field list enters the address, fields=id,title and fields=title,id produce two separate keys. To avoid storing the same data twice, a canonical form for the field list must be settled: names are sorted and duplicates dropped. This canonical-form decision is the same kind as the one in the URI Design lesson.

Second, the combinations to test multiply. At an endpoint with five base fields and four expansion fields, the number of field sets a client could request is too large to test by hand. This is why the allowlist stays narrow: not every column is offered, only the fields screens actually want.

Third, expansion depth must be bounded. This server offers one level of expansion: from loan to book and member. If two levels were allowed — from book to branch, from branch to the responsible member — a single request would turn into a query tree branching on the server side. The single request the client sees hides the cost the server pays.

Field Selection or a Named View

The alternative to field selection is the server defining a handful of ready-made representations: ?view=list and ?view=detail, say. Flexibility drops under this approach, but three things are gained. The number of cache keys falls to two. No combinations are left to test. The server can write one query per view and optimize it.

The selection criterion is the number of clients. If a single interface uses the server, a named view is enough; the view’s definition changes as the screen does. If there are many independent clients — a mobile app, an admin panel, another service — defining a separate view for each bloats the contract; field selection needs less upkeep in that case.

The two approaches are also used together: named views for common cases, field selection for the rest. This is the resource-oriented style’s counterpart to a problem query-based styles solve from the start; the trade-off discussed in the API Styles topic turns into concrete numbers here.

Summary

• Over-fetching is carrying unused fields; under-fetching is a single request failing to feed the screen. Both come from the server alone deciding the representation’s level of detail.
• In the measurement, the same screen consumed 81 requests and 8968 bytes with separate requests, 1 request and 7368 bytes with expansion, and 1 request and 3741 bytes with expansion and field selection.
• Field names and expansion names pass through an allowlist; only column expressions the server wrote enter the query text.
• A request for an undefined field is not silently dropped, it is rejected with 422, and the fields offered in that request are reported in the response.
• Field selection multiplies cache keys and increases the combinations to test; the field list’s canonical form must be settled and the allowlist kept narrow.
• In services with few clients, named views need less upkeep than field selection; the two approaches can also be used together.

Next Step

The four lessons up to here fixed the read side: which records, in what order, how much, in how many requests. On the write side, a gap remains from the third lesson. POST is not idempotent; when the network drops or the response gets lost on the way back, the client retries the request and a second loan record is born. The client cannot know whether the request arrived, and the server cannot see that two requests came from the same intent. The next lesson closes this gap with a key the client generates, and shows by counting that two requests sent with the same key produce a single effect.