--- title: 'Web API Design' source: 'https://academia.sh/en/courses/api-design' language: en updated: '2026-08-19T05:19:26+00:00' license: 'CC BY-SA 4.0' lessons: 34 outcomes: - 'Builds a consistent REST interface through resource modeling and URI design' - 'Justifies API versioning and error contract decisions' - 'Documents and tests the contract in a machine-readable form' --- # Web API Design Establishing the contract through which programs connect to one another: the trade-offs between API styles, resource modeling and URI design, status code and body decisions, the error contract, versioning, and documenting and testing the contract in a machine-readable form. ## API Styles Different interface styles that meet the same need: the notion of a contract and of compatibility, a comparison of resource-based, remote procedure call, and query-based approaches measured by request count and bytes transferred, REST principles, binary protocols, and communication outside the request-response pattern. ### What Is an API Source: https://academia.sh/en/courses/api-design/what-is-an-api What makes an interface a contract: the consumer's unwritten assumptions, defining compatible and breaking change by consumer behavior, the different response a tolerant reader and a strict reader give to the same change, and who owns the contract. The previous course established the cases where the server hands out a file or a document to the outside: content a human will read. The library lending service's actual consumers, however, are programs. The shelf display at the branch queries a book's status, the report job that runs at night pulls a listing by author, and the audit script checks the shape of the records. All three call the same endpoint, and all three expect something different from it. The concept that emerges at this point is the **application programming interface**: the callable surface a program opens to other programs. This lesson establishes why that surface counts as a **contract** and when the contract breaks. #### The Contract Is More Than the Body Looking at the JSON body an endpoint returns is not the same as seeing the whole contract. The contract covers everything the two sides can expect from each other: - **Surface:** which addresses exist, which methods they accept, which parameters they recognize. - **Format:** the names, types, requiredness, and value sets of the fields in the body. - **Meaning:** what a field represents. Does the `status` field's `onShelf` value report the book's physical location, or its availability for loan? - **Behavior:** the result of making the same request twice, which status code reports errors, whether ordering is guaranteed. Only some of this is written down. The unwritten part is still part of the contract, because the consumer relies on it. A consumer who notices that a list always comes back in the same order writes code that assumes that order; when the order changes, the consumer breaks even though no "documented" part of the interface has changed. The contract's boundary is not drawn by the scope of the documentation but by everything the consumer has come to depend on. #### Compatibility Is a Measurement Question Whether a change is **breaking** is defined not by the size of the change but by whether existing consumers keep working after it. This definition is measurable: it is enough to run the same consumers against different published contract versions. The library service's book endpoint is written so it can publish three separate versions. ```js // server.mjs — the endpoint that publishes a book record. The CONTRACT variable selects the published version. import { createServer } from "node:http"; const CONTRACT = process.env.CONTRACT ?? "v1"; const RECORD = { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", status: "onShelf" }; const buildBody = () => { if (CONTRACT === "v1") return RECORD; if (CONTRACT === "v2-addition") return { ...RECORD, pages: 1312 }; // field added return { isbn: RECORD.isbn, title: RECORD.title, // field name and type changed author: { name: RECORD.author }, loanStatus: RECORD.status, pages: 1312 }; }; createServer((req, res) => { res.sendDate = false; const body = JSON.stringify(buildBody()); res.setHeader("Content-Type", "application/json; charset=utf-8"); res.setHeader("Content-Length", Buffer.byteLength(body)); res.writeHead(200).end(body); }).listen(8441, "127.0.0.1"); ``` Three consumers represent three separate reading habits. The shelf display reads only the two fields it needs. The report job also reads few fields, but it depends on one field's type. The audit script checks the body as a whole and counts an unrecognized field as an error. ```js // consumer.mjs — three separate consumers. Usage: node consumer.mjs const role = process.argv[2]; const record = await (await fetch("http://127.0.0.1:8441/books/978-0262033848")).json(); const ROLES = { // Tolerant reader: reads only the fields it needs, ignores fields it does not recognize. shelfDisplay: (r) => { if (typeof r.status !== "string") throw new Error("status field missing"); return `shelf display: ${r.title} -> ${r.status}`; }, // Tolerant, but depends on a field's type. report: (r) => { if (typeof r.author !== "string") throw new Error("author field is not a string"); return `report line: ${r.author.toUpperCase()} / ${r.title}`; }, // Strict reader: rejects every field not listed in the contract. auditor: (r) => { const EXPECTED = ["isbn", "title", "author", "status"]; const extra = Object.keys(r).filter((f) => !EXPECTED.includes(f)); if (extra.length) throw new Error(`unexpected field: ${extra.join(", ")}`); return `check: ${r.isbn} valid`; }, }; try { console.log(` ${role.padEnd(13)} OK ${ROLES[role](record)}`); } catch (e) { console.log(` ${role.padEnd(13)} BROKEN ${e.message}`); } ``` ```bash #!/usr/bin/env bash # Tests three contract versions against three consumers. A=http://127.0.0.1:8441/books/978-0262033848 for c in v1 v2-addition v2-change; do CONTRACT=$c node server.mjs & p=$! echo "== published contract: $c ==" printf ' body: ' curl -s --retry 20 --retry-all-errors --retry-delay 0 --retry-connrefused "$A"; echo for role in shelfDisplay report auditor; do node consumer.mjs $role; done kill $p 2>/dev/null; wait $p 2>/dev/null || true done ``` ``` == published contract: v1 == body: {"isbn":"978-0262033848","title":"Introduction to Algorithms","author":"Cormen","status":"onShelf"} shelfDisplay OK shelf display: Introduction to Algorithms -> onShelf report OK report line: CORMEN / Introduction to Algorithms auditor OK check: 978-0262033848 valid == published contract: v2-addition == body: {"isbn":"978-0262033848","title":"Introduction to Algorithms","author":"Cormen","status":"onShelf","pages":1312} shelfDisplay OK shelf display: Introduction to Algorithms -> onShelf report OK report line: CORMEN / Introduction to Algorithms auditor BROKEN unexpected field: pages == published contract: v2-change == body: {"isbn":"978-0262033848","title":"Introduction to Algorithms","author":{"name":"Cormen"},"loanStatus":"onShelf","pages":1312} shelfDisplay BROKEN status field missing report BROKEN author field is not a string auditor BROKEN unexpected field: loanStatus, pages ``` Nine results give three rules. **Adding a field is not inherently compatible.** Adding the `pages` field left two consumers unaffected and broke the audit script. The provider's "I only added something" claim showed up as an error on the consumer side. For an addition to count as compatible depends on consumers ignoring fields they do not recognize. **Renaming a field is the same as deleting it.** When the `status` field was renamed to `loanStatus`, that field ceased to exist for the shelf display. Even though the provider's intent was "fixing the name," what the consumer sees is a deleted field. **Changing a type is breaking even when the name is preserved.** The `author` field is still in place, but it is now an object instead of a string. A check that only looks at field names does not catch this change; the report job breaks at run time. #### The Two Directions of Compatibility The measurement tested only one direction: the server new, the consumers old. This direction is called **backward compatibility**, and it is the criterion that determines the provider's release decision, because the server is updated before the consumers. The reverse direction also exists, and it is easy to miss: the consumer new, the server old. **Forward compatibility** is a consumer written against a contract not yet published being able to work with the old server. In the library service, this situation arises when the terminals are updated before the central branch: the new terminal tries to read the `pages` field, and the old server never sends that field. Whether the terminal crashes or shows a blank when the field is missing is the consumer's decision, not the contract's. The two directions together give a release rule: **the server that adds the field is published first, then the consumer that reads the field; the order reverses when a field is removed.** A contract change is not a single event but a sequential transition between two sides. #### The Tolerant Reader The audit script in the measurement caused its own break by its own rule: it raised an error when it saw a field not listed in the body. The opposite of this behavior is the **tolerant reader**: a consumer that reads only the fields it needs and ignores everything it does not recognize. Tolerant reading is a consumer habit, but its result concerns the provider: an interface stays extensible to the degree that its consumers read tolerantly. In a system with strict-reading consumers, every addition requires a version bump; in a system with tolerant-reading consumers, additions roll out silently. This has a limit in the reverse direction. Tolerance does not apply to input validation. When the server ignores fields it does not recognize in an incoming body, a consumer's typo is silently swallowed: a client that writes `dueDat` instead of `dueDate` gets no error, runs with the default value, and only learns of the mistake from the result. The rule is therefore different in the two directions: **be strict in what you send, be tolerant in what you receive.** #### The Consumer Owns the Contract The owner of an interface is the party that publishes it, but the party that determines the contract's validity is the consumer. This distinction has two practical consequences. First, the cost of a breaking change grows with the number of consumers. In an internal interface with a single consumer, renaming the `status` field means publishing both sides together. In an interface distributed to terminals across branches, the same change means every terminal that has not been updated breaks. The same technical change is two separate decisions in two contexts. Second, the wider the written part of the contract, the smaller the share of unwritten assumptions the consumer has to rely on. When the types, requiredness, and value sets of fields are explicitly stated, the consumer binds by "reading" rather than by "observing." If it is written that ordering is not guaranteed, a consumer that relies on ordering is at fault. This is where the value of a machine-readable definition comes from, and it is covered later in this course. #### Summary - An API's contract is wider than its body format: surface, format, meaning, and behavior together make up the contract, and the unwritten assumptions the consumer relies on fall within this scope. - Whether a change is breaking is defined by whether existing consumers keep working after it, and it is measured by running the consumers against the new version. - In the measurement, adding a field broke the strict-reading consumer, renaming a field had the same effect as deleting it, and changing a type produced a run-time break even though the name was preserved. - The tolerant reader ignores fields it does not recognize and keeps the interface extensible through additions; the same tolerance does not apply to validating incoming requests. - The contract's cost grows with the number of consumers; every undocumented detail is a surface the consumer will bind to by observation. #### Next Step This lesson established what a contract is but left the contract's shape open: why is a book record read from an address, why is it not requested with a procedure call, why does the client not say which fields it wants? The same job can be met with three separate approaches — resource-based, remote-procedure-call style, and query-based — and the difference among these three approaches is not a matter of taste. The next lesson writes all three for the same scenario and shows the difference among them with three measures: how many requests went out, how many bytes were carried, and how much of the carried data was used? ### Comparison of API Styles Source: https://academia.sh/en/courses/api-design/comparison-of-api-styles Three styles that meet the same job: resource-based, remote-procedure-call style, and query-based approaches are written for the same scenario and compared by measuring request count, bytes on the wire, and the used share of carried fields; the lesson shows which side has to change when the requirement changes. The previous lesson established what makes an interface a contract but left the contract's shape open. The following job requested from the library service can be met by three separate approaches: **list a member's borrowed books, along with their titles and due dates.** The three approaches are as follows. In the **resource-based** style, every concept has its own address, and the client collects the necessary pieces one by one. In the **remote-procedure-call** style, there is a single address; the client sends a method name in the body, and the server returns the response prepared for that job. In the **query-based** style, there is again a single address, but the client writes which fields it wants in the body. The difference among them is not a matter of taste: who determines the response shape changes, and this has measurable consequences. #### Shared Data and the Third Style's Parser All three styles work on the same data. ```js // data.mjs — the shared library data all three styles run on export const MEMBERS = [{ id: 1, name: "Alice Kane", email: "alice@example.test", phone: "0000000000", address: "Central Branch, Floor 2", registered: "2024-01-05", fine: 0, membershipType: "full" }]; export const BOOKS = { "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", pages: 1312, publisher: "MIT", year: 2009, shelf: "R-12", summary: "Algorithm design and analysis." }, "978-0201896831": { isbn: "978-0201896831", title: "The Art of Computer Programming", author: "Knuth", pages: 650, publisher: "Addison", year: 1997, shelf: "R-03", summary: "The fundamental algorithms volume." }, "978-0131103627": { isbn: "978-0131103627", title: "The C Programming Language", author: "Kernighan", pages: 272, publisher: "Prentice", year: 1988, shelf: "R-07", summary: "The language's definition and examples." }, }; export const LOANS = [ { id: 11, memberId: 1, isbn: "978-0262033848", borrowedAt: "2024-05-01", dueDate: "2024-05-15" }, { id: 12, memberId: 1, isbn: "978-0201896831", borrowedAt: "2024-05-03", dueDate: "2024-05-17" }, { id: 13, memberId: 1, isbn: "978-0131103627", borrowedAt: "2024-05-06", dueDate: "2024-05-20" }, ]; ``` The query-based style requires a parser: the selection text the client writes is converted into a tree, and this tree is then executed against the data. The following thirty lines do this job. ```js // query.mjs — a small selection-set parser. Input: "member(1) { name loans { dueDate book { title } } }" export const parse = (text) => { const tokens = text.match(/[A-Za-z_][A-Za-z0-9_]*|\d+|[{}()]/g) ?? []; let i = 0; const selectionSet = () => { // { field field(arg) { ... } ... } const fields = []; i++; // opening curly brace while (tokens[i] !== "}") { const name = tokens[i++]; let arg = null; if (tokens[i] === "(") { arg = tokens[i + 1]; i += 3; } const sub = tokens[i] === "{" ? selectionSet() : null; fields.push({ name, arg, sub }); } i++; // closing curly brace return fields; }; tokens.unshift("{"); tokens.push("}"); return selectionSet(); }; // Executes the selection tree against the data: only the fields the client asked for come back. export const execute = (fields, root, resolver) => { const output = {}; for (const field of fields) { const value = resolver(root, field.name, field.arg); if (!field.sub) { output[field.name] = value; continue; } output[field.name] = Array.isArray(value) ? value.map((d) => execute(field.sub, d, resolver)) : execute(field.sub, value, resolver); } return output; }; ``` All three styles sit side by side in a single server; this way the comparison happens on the same data, the same process, and the same connection. ```js // server.mjs — a server that meets the same job with three styles: resource-based, remote procedure, query-based import { createServer } from "node:http"; import { MEMBERS, BOOKS, LOANS } from "./data.mjs"; import { parse, execute } from "./query.mjs"; const readBody = (req) => new Promise((resolve) => { let m = ""; req.on("data", (p) => (m += p)); req.on("end", () => resolve(m)); }); const json = (res, code, body) => { const text = JSON.stringify(body); res.setHeader("Content-Type", "application/json; charset=utf-8"); res.setHeader("Content-Length", Buffer.byteLength(text)); res.writeHead(code).end(text); }; // Resolver for the query-based style: each field name corresponds to a data access. const resolver = (root, name, arg) => { if (name === "member") return MEMBERS.find((m) => m.id === Number(arg)); if (name === "loans") return LOANS.filter((l) => l.memberId === root.id); if (name === "book") return BOOKS[root.isbn]; return root[name]; }; createServer(async (req, res) => { res.sendDate = false; const path = new URL(req.url, "http://local").pathname; // 1) Resource-based: each concept has a separate address, each address returns the full record. let e; if ((e = /^\/members\/(\d+)$/.exec(path))) return json(res, 200, MEMBERS.find((m) => m.id === +e[1])); if ((e = /^\/members\/(\d+)\/loans$/.exec(path))) return json(res, 200, LOANS.filter((l) => l.memberId === +e[1])); if ((e = /^\/books\/(.+)$/.exec(path))) return json(res, 200, BOOKS[e[1]]); // 2) Remote procedure call: single address, method name in the body; the server decides the response shape. if (path === "/rpc") { const { method, params } = JSON.parse(await readBody(req)); if (method !== "memberLoanSummary") return json(res, 200, { error: "unknown method" }); const member = MEMBERS.find((m) => m.id === params.memberId); return json(res, 200, { result: { name: member.name, loans: LOANS .filter((l) => l.memberId === member.id) .map((l) => ({ dueDate: l.dueDate, bookTitle: BOOKS[l.isbn].title })) } }); } // 3) Query-based: single address, selection set in the body; the client decides the response shape. if (path === "/query") { const { query } = JSON.parse(await readBody(req)); return json(res, 200, { data: execute(parse(query), null, resolver) }); } return json(res, 404, { error: "not found" }); }).listen(8451, "127.0.0.1", () => console.log("three styles 127.0.0.1:8451")); ``` #### Three Measures The comparison rests on three numbers. **Request count** gives the network round trips. **Bytes on the wire**, headers included, gives the data actually carried; to measure this, the client uses a layer that wraps the connection. **Carried field count** is the sum of the scalar values in the responses; how many of these are used quantifies **over-fetching**. ```js // measure.mjs — measures the three styles in the same scenario: request count, bytes on the wire, used-field ratio import { Agent, request } from "node:http"; import { connect } from "node:net"; let sent = 0, received = 0, requestCount = 0; const agent = new Agent({ keepAlive: true, maxSockets: 1 }); agent.createConnection = (option) => { // connection that counts bytes on the wire const s = connect(option); s.on("data", (p) => { received += p.length; }); const write = s.write.bind(s); s.write = (chunk, ...k) => { sent += Buffer.byteLength(chunk); return write(chunk, ...k); }; return s; }; const sendRequest = (path, body) => new Promise((resolve, reject) => { requestCount++; const r = request({ agent, host: "127.0.0.1", port: 8451, path, method: body ? "POST" : "GET", headers: body ? { "Content-Type": "application/json" } : {} }, (y) => { let m = ""; y.on("data", (p) => (m += p)); y.on("end", () => resolve(JSON.parse(m))); }); r.on("error", reject); r.end(body ? JSON.stringify(body) : undefined); }); // Sum of scalar fields carried in the responses const countLeaves = (d) => (typeof d !== "object" || d === null ? 1 : Object.values(d).reduce((t, v) => t + countLeaves(v), 0)); const reset = () => { sent = received = requestCount = 0; }; const report = (name, carried, used, note) => console.log( `${name.padEnd(17)} requests=${String(requestCount).padStart(2)} ` + `sent=${String(sent).padStart(4)} B received=${String(received).padStart(4)} B ` + `carried fields=${String(carried).padStart(2)} used=` + (carried ? `%${Math.round((100 * used) / carried)}`.padStart(5) : " -") + (note ? `\n${" ".repeat(18)}${note}` : "")); const resourceBased = async () => { reset(); let carried = 0; const member = await sendRequest("/members/1"); carried += countLeaves(member); const loans = await sendRequest("/members/1/loans"); carried += countLeaves(loans); for (const l of loans) carried += countLeaves(await sendRequest(`/books/${l.isbn}`)); return carried; }; // --- Scenario 1: member name + book title and due date for each loan (7 fields) --- console.log("-- scenario 1: member name, book title, due date --"); report("resource-based", await resourceBased(), 7); reset(); const rpc = await sendRequest("/rpc", { method: "memberLoanSummary", params: { memberId: 1 } }); report("remote procedure", countLeaves(rpc), 7, JSON.stringify(rpc.result).slice(0, 78) + " ..."); reset(); const result = await sendRequest("/query", { query: "member(1) { name loans { dueDate book { title } } }" }); report("query-based", countLeaves(result), 7, JSON.stringify(result.data).slice(0, 78) + " ..."); // --- Scenario 2: the client's requirement changed, the author's name is now wanted too (10 fields) --- console.log("\n-- scenario 2: author name added to the response --"); report("resource-based", await resourceBased(), 10, "server did not change: author was already carried"); reset(); const rpc2 = await sendRequest("/rpc", { method: "memberLoanSummaryWithAuthors", params: { memberId: 1 } }); report("remote procedure", 0, 10, `server response: ${JSON.stringify(rpc2)} -> new method needed`); reset(); const result2 = await sendRequest("/query", { query: "member(1) { name loans { dueDate book { title author } } }" }); report("query-based", countLeaves(result2), 10, "server did not change: author field added to the query"); agent.destroy(); ``` ```bash node server.mjs & p=$! curl -s --retry 20 --retry-connrefused --retry-delay 0 -o /dev/null http://127.0.0.1:8451/members/1 node measure.mjs kill $p ``` ``` three styles 127.0.0.1:8451 -- scenario 1: member name, book title, due date -- resource-based requests= 5 sent= 404 B received=1702 B carried fields=47 used= %15 remote procedure requests= 1 sent= 174 B received= 379 B carried fields= 7 used= %100 {"name":"Alice Kane","loans":[{"dueDate":"2024-05-15","bookTitle":"Introductio ... query-based requests= 1 sent= 185 B received= 403 B carried fields= 7 used= %100 {"member":{"name":"Alice Kane","loans":[{"dueDate":"2024-05-15","book":{"title ... -- scenario 2: author name added to the response -- resource-based requests= 5 sent= 404 B received=1702 B carried fields=47 used= %21 server did not change: author was already carried remote procedure requests= 1 sent= 185 B received= 159 B carried fields= 0 used= - server response: {"error":"unknown method"} -> new method needed query-based requests= 1 sent= 192 B received= 459 B carried fields=10 used= %100 server did not change: author field added to the query ``` Byte counts also include headers, so they depend on the runtime's default header set; field counts and request counts, however, depend on the data and are independent of the machine. #### What the Measurement Says **Under-fetching** shows up in the resource-based style as request count. The member record is one request, the loan list is one request, then one book request per loan: five in total. This number is not fixed; it depends on the number of loans. The pattern of fetching a record list and making a separate request for each row in that list is known as the **N+1 problem**, and the round-trip count grows as the data grows. **Over-fetching** shows up in the same style as field count. Forty-seven fields were carried, and seven were used. The member's address, phone, fine amount; the book's summary, publisher, shelf number — none of these had been requested. In the resource-based style, returning the whole record is not a defect but the definition of that style: an address corresponds to a **record**, not a **view**. The other two styles solve both problems in a single request: seven fields are carried, seven are used. The difference between them shows up in the second scenario. #### Who Determines the Response Shape In the second scenario, the requirement changed: the author's name is now wanted too. Each of the three styles had to change from a different place. **In the resource-based style, nothing changed.** The author's name was already carried; the used share went from 15% to 21%, and that was all. This is the unexpected benefit of over-fetching: a field the client needs later is, more often than not, already in its hands. **In the remote-procedure style, the server had to change.** The method the client called does not exist, and the server reports this. To get the new field, either the existing method's response is extended or a new method is added; either way, the client cannot proceed until the server is republished. In this style, the response shape is the server's decision, so a shape change is also the server's job. **In the query-based style, only the client changed.** `author` was added to the selection set, the server stayed the same, the carried fields went from seven to ten, and the used share stayed at 100%. Because the client determines the response shape, a shape change is also the client's job. This is the real trade-off among the styles, and its cost lands on the server side. In a query-based interface, the server must be able to serve every field combination the client might request; how much data access an incoming query will trigger is not known in advance. In the remote-procedure style, by contrast, the server knows exactly what each response will bring, because it fixed the shape itself. #### What Style Choice Depends On The measurement does not show that one style is better than another; it shows that every measure has an owner. If the number of consumers is small and their requirements resemble each other, the server fixing the shape produces the fewest surprises. If there are many consumers and each wants a different field combination, leaving the shape to the client spares the server from changing for every new screen. If the consumer side relies on caching, every record having its own address turns into an advantage: the same book requested a second time never goes out to the network. This last point is the subject of the next lesson. There is also one item that does not enter the measurement: **understandability**. In a resource-based interface, addresses are concept names; someone unfamiliar with the system can learn the domain by looking at the address list. In single-address styles, this information lives inside the body and can only be seen by reading the contract. #### Summary - All three styles meet the same job, but a different side determines the response shape: the record itself in the resource-based style, the server in the remote-procedure style, the client in the query-based style. - In the resource-based style, the measurement gave five requests and forty-seven carried fields; seven fields were used. Under-fetching shows up in request count, over-fetching in the used share. - A request count that grows with the number of loans is the N+1 pattern; it appears in any design that fetches a list and makes a separate request for each row. - When the requirement changed, the server had to change in the remote-procedure style and the client in the query-based style; in the resource-based style, the excess carried data made the change unnecessary. - The cost of the query-based style is on the server side: the data access an incoming query will trigger is not known in advance, whereas the cost of a fixed-shape response is known. #### Next Step The resource-based style was the option that carried the most bytes and made the most requests in the measurement, yet it is the most widely adopted style. This contradiction comes from properties the measurement does not capture: every record having its own address makes it possible for the same record, requested a second time, to never go out to the network at all; every request being self-contained makes it unimportant which server copy the request lands on. The next lesson names these properties as a set of constraints and tests all three by running them: the same request giving the same response in two separate processes, the concrete counterpart of a uniform interface, and the effect of cacheability on bytes carried. ### REST Principles Source: https://academia.sh/en/courses/api-design/rest-principles The set of constraints behind the resource-based style: demonstrating statelessness by testing the same request in two separate processes, measuring the uniform interface with a client that does not know the resource name, the effect of cacheability with an entity tag on bytes carried, and how a layered system follows from these constraints. The previous lesson measured the resource-based style and found it the most expensive option: five requests, forty-seven carried fields, 15% used. Yet most lending services are written in this style. The contradiction comes from properties the measurement does not capture. Those properties are the consequences of a set of constraints. **REST** (representational state transfer) is not a protocol or a library; it is the name of the constraints an interface can conform to. This lesson tests three of these constraints — statelessness, uniform interface, and cacheability — by running them. #### Constraints and Resource Orientation REST's fundamental unit is the **resource**: every concept that can be referred to by a name. In the library service, a book, a member, a loan, and a shelf are each a resource. What the client sees is not the resource itself but a **representation**: a copy of that resource in a specific format. The same resource can have a JSON representation as well as an HTML representation; both describe the same thing. The required constraints build on top of this unit: - **Client–server separation:** the two sides evolve separately; the only bond between them is the interface. - **Statelessness:** every request is self-contained; the server does not keep a context belonging to the client between requests. - **Cacheability:** the response itself reports whether it can be stored. - **Uniform interface:** resources are named the same way and processed with the same methods. - **Layered system:** the client does not have to know whether the party on the other side is the final server or an intermediate layer. There is a sixth constraint — **code on demand**: the server may send executable code to the client. It is the only optional constraint among them, and this lesson does not measure it. The following server carries two designs side by side, one that applies these constraints and one that violates them; because it takes its port from outside, two copies of it can be run together. ```js // server.mjs — the port is supplied from outside so the same code can run as two copies: // node server.mjs 8461 import { createServer } from "node:http"; import { createHash, randomUUID } from "node:crypto"; const PORT = Number(process.argv[2] ?? 8461); const RESOURCES = { books: [{ id: "978-0262033848", title: "Introduction to Algorithms" }, { id: "978-0201896831", title: "The Art of Computer Programming" }, { id: "978-0131103627", title: "The C Programming Language" }, { id: "978-0596007126", title: "Head First" }], members: [{ id: "u1", name: "Alice Kane" }, { id: "u2", name: "Ben Ortiz" }], loans: [{ id: "o11", member: "u1", book: "978-0262033848" }], shelves: [{ id: "R-12", floor: 2 }, { id: "R-03", floor: 1 }], // resource added later }; const cursors = new Map(); // state kept on the server side: exists only in this copy const json = (res, code, body, tag) => { const text = JSON.stringify(body); res.setHeader("Content-Type", "application/json; charset=utf-8"); res.setHeader("X-Process", process.pid); if (tag) res.setHeader("ETag", tag); res.setHeader("Content-Length", Buffer.byteLength(text)); res.writeHead(code).end(text); }; const tagOf = (body) => `"${createHash("sha256").update(JSON.stringify(body)).digest("hex").slice(0, 16)}"`; createServer((req, res) => { res.sendDate = false; const url = new URL(req.url, "http://local"); const [, first, second] = url.pathname.split("/"); if (first === "") return json(res, 200, { resources: Object.keys(RESOURCES) }); // Stateful pagination: the cursor is kept on the server, the request carries only the cursor id. if (first === "cursor" && req.method === "POST") { const id = randomUUID().slice(0, 8); cursors.set(id, 0); return json(res, 201, { cursor: id }); } if (first === "cursor" && req.method === "GET") { if (!cursors.has(second)) return json(res, 404, { error: "unknown_cursor" }); const offset = cursors.get(second); cursors.set(second, offset + 2); return json(res, 200, { offset, page: RESOURCES.books.slice(offset, offset + 2) }); } // Uniform interface: every resource is named the same way, read with the same methods. const collection = RESOURCES[first]; if (!collection) return json(res, 404, { error: "resource_not_found" }); if (second) { const entry = collection.find((k) => k.id === second); if (!entry) return json(res, 404, { error: "not found" }); const tag = tagOf(entry); if (req.headers["if-none-match"] === tag) { // conditional request: no body is sent res.setHeader("ETag", tag); res.setHeader("X-Process", process.pid); return res.writeHead(304).end(); } return json(res, 200, entry, tag); } // Stateless pagination: the page information is inside the request. const offset = Number(url.searchParams.get("offset") ?? 0); const quantity = Number(url.searchParams.get("quantity") ?? collection.length); return json(res, 200, { offset, quantity, entries: collection.slice(offset, offset + quantity) }); }).listen(PORT, "127.0.0.1", () => console.log(`copy :${PORT} pid=${process.pid}`)); ``` The measurement client consists of three parts and distributes requests alternately between the two copies. ```js // measure.mjs — tests three principles: statelessness, uniform interface, cacheability import { Agent, request } from "node:http"; import { connect } from "node:net"; let received = 0; const agent = new Agent({ keepAlive: true }); agent.createConnection = (s) => { const b = connect(s); b.on("data", (p) => (received += p.length)); return b; }; const sendRequest = (port, path, option = {}) => new Promise((resolve, reject) => { const r = request({ agent, host: "127.0.0.1", port, path, method: option.method ?? "GET", headers: option.headers ?? {} }, (y) => { let m = ""; y.on("data", (p) => (m += p)); y.on("end", () => resolve({ status: y.statusCode, tag: y.headers.etag, proc: y.headers["x-process"], body: m ? JSON.parse(m) : null })); }); r.on("error", reject); r.end(); }); // 1) STATELESSNESS — does the same request give the same response in two separate processes? console.log("-- 1) statelessness: requests are distributed alternately between the two copies --"); const { body: created } = await sendRequest(8461, "/cursor", { method: "POST" }); for (const port of [8461, 8462, 8461]) { const y = await sendRequest(port, `/cursor/${created.cursor}`); console.log(` stateful :${port} pid=${y.proc} -> ${y.status} ${JSON.stringify(y.body).slice(0, 62)}`); } for (const port of [8461, 8462, 8461]) { const y = await sendRequest(port, "/books?offset=2&quantity=2"); console.log(` stateless :${port} pid=${y.proc} -> ${y.status} ${JSON.stringify(y.body).slice(0, 62)}`); } // 2) UNIFORM INTERFACE — does the same client code read every resource without knowing its name? console.log("\n-- 2) uniform interface: resource names are learned from the server --"); const { body: root } = await sendRequest(8461, "/"); for (const name of root.resources) { const list = await sendRequest(8461, `/${name}`); const first = list.body.entries[0]; const single = await sendRequest(8461, `/${name}/${first.id}`); console.log(` ${name.padEnd(9)} list=${list.status} quantity=${list.body.entries.length}` + ` single=${single.status} tag=${single.tag}`); } const missing = await sendRequest(8461, "/branches"); console.log(` branches list=${missing.status} ${JSON.stringify(missing.body)}`); // 3) CACHEABILITY — how much does a conditional request cut the bytes carried? console.log("\n-- 3) cacheability: the same record requested ten times --"); const PATH = "/books/978-0262033848"; received = 0; for (let i = 0; i < 10; i++) await sendRequest(8461, PATH); const unconditional = received; received = 0; const first = await sendRequest(8461, PATH); for (let i = 0; i < 9; i++) await sendRequest(8461, PATH, { headers: { "If-None-Match": first.tag } }); const conditional = received; const final = await sendRequest(8461, PATH, { headers: { "If-None-Match": first.tag } }); console.log(` unconditional 10 requests -> ${unconditional} B`); console.log(` conditional 10 requests -> ${conditional} B (final response ${final.status}, gain %${ Math.round((100 * (unconditional - conditional)) / unconditional)})`); agent.destroy(); ``` ```bash node server.mjs 8461 & a=$! node server.mjs 8462 & b=$! for k in 8461 8462; do curl -s --retry 20 --retry-connrefused --retry-delay 0 -o /dev/null "http://127.0.0.1:$k/" done node measure.mjs kill $a $b ``` ``` copy :8461 pid=24387 copy :8462 pid=24388 -- 1) statelessness: requests are distributed alternately between the two copies -- stateful :8461 pid=24387 -> 200 {"offset":0,"page":[{"id":"978-0262033848","title":"Introducti stateful :8462 pid=24388 -> 404 {"error":"unknown_cursor"} stateful :8461 pid=24387 -> 200 {"offset":2,"page":[{"id":"978-0131103627","title":"The C Prog stateless :8461 pid=24387 -> 200 {"offset":2,"quantity":2,"entries":[{"id":"978-0131103627","ti stateless :8462 pid=24388 -> 200 {"offset":2,"quantity":2,"entries":[{"id":"978-0131103627","ti stateless :8461 pid=24387 -> 200 {"offset":2,"quantity":2,"entries":[{"id":"978-0131103627","ti -- 2) uniform interface: resource names are learned from the server -- books list=200 quantity=4 single=200 tag="c80ac47429ab852d" members list=200 quantity=2 single=200 tag="79aa82aae4e5fc1c" loans list=200 quantity=1 single=200 tag="6f2d536c9fff5239" shelves list=200 quantity=2 single=200 tag="bb5d3afab6485c63" branches list=404 {"error":"resource_not_found"} -- 3) cacheability: the same record requested ten times -- unconditional 10 requests -> 2370 B conditional 10 requests -> 1317 B (final response 304, gain %44) ``` Process ids change on every run. Because the id is carried in the `X-Process` header, its digit count shifts the total bytes by a few, but the ratio does not change. Byte counts also depend on the header set the runtime writes. #### Statelessness: The Request Being Self-Contained The first part tests two pagination designs under the same condition: requests are distributed alternately between the two server copies. This is a small example of the multi-copy setup behind the reverse proxy built in the previous course. **In the stateful design,** the cursor sits in the server's memory. The first request lands on 8461 and runs. When the second request lands on 8462, it gets a `404` because the same cursor is not there. When the third request lands on 8461 again, the second page comes back — meaning the error the second copy gave has also broken the cursor's state: the client has skipped a page without ever seeing it. **In the stateless design,** the page information is inside the request. The same request went three times, to two separate processes, and gave the same body all three times. The server's identity has no effect on the response whatsoever. The cost of statelessness is resending the context on every request: identity information, pagination position, and filter criteria are carried each time. What is gained in return is that server copies become interchangeable. When a copy goes down, the request can be routed to another copy, the number of copies can be scaled up under load, and none of them has to carry the client's session. #### Uniform Interface: The Same Client Code, Four Resources The client in the second part does not know any resource name. It reads the resource list from the root address, then makes the same two requests for each resource: fetch the collection, fetch the single record by the first entry's id. All four were read with the same code; not a single line changed in the client for the `shelves` resource that was added later. These are the three concrete counterparts of the uniform interface in this measurement: **resources are named by addresses**, **the same methods mean the same thing on every resource**, and **responses are self-descriptive** — how to read the body is understood from the content type header, and the record's version from the entity tag. A resource that does not exist is met the same way too, with a `404`. The cost of this is a loss of flexibility. When an operation does not sit on a single resource — something like "lend these three books at once and recompute the overdue fine" — the uniform interface does not accommodate that operation naturally. Such operations are either turned into resources themselves, or the interface moves closer to a remote procedure call at that point. #### Cacheability: Requesting the Same Record Ten Times The third part requests the same book ten times. In this run, the unconditional requests carry 2370 bytes. In the second round, the client saves the entity tag from the first response and attaches it to the next nine requests with the `If-None-Match` header; the server sees the record has not changed and returns a bodyless `304`. The bytes carried drop to 1317 — a 44% reduction. The entity tag was introduced in the Static File Serving lesson; the difference here is that the same mechanism is applied to a computed record. The source of the gain is the body not being sent; the headers are still carried, so the reduction cannot reach 100%. The ratio rises as the record grows: the record in this measurement carries only two fields. #### Layered System All three measurements together make a fourth constraint possible too. If the response depends only on the request and reports its own cacheability, a layer can be placed between the client and the server: a reverse proxy, a cache, or a router. The client does not know whether the party on the other side is the final server or an intermediate layer, and it does not need to. The reverse proxy built in the Web Server Concept lesson of the previous course is exactly this. That lesson showed that the proxy also brings a loss of information; the additional observation here is that the proxy's ability to step in depends on the interface being stateless and cacheable. In a design that keeps sessions on the server, the proxy is forced to send the request to the specific copy that holds the session, not to just any copy. #### Summary - REST is not a protocol but a set of constraints a resource-oriented interface conforms to; the fundamental unit is the resource, and what gets carried is the resource's representation. - Statelessness was seen directly in the measurement: the cursor kept on the server could not be found on the second copy and a page was skipped; page information carried inside the request gave the same response on both copies. - The uniform interface means a single client code that does not know the resource name can read all four resources; the resource added later required no change in the client. - A conditional request cut the bytes carried by about 44% when the same record was fetched ten times; the gain comes from the body not being sent, and it cannot be complete because the headers keep being carried. - A layered system depends on these two constraints: a proxy or a cache can be placed in between only when the response depends solely on the request and is storable. #### Next Step REST's constraints lean the interface on the rules of the network: addresses, methods, status codes, headers. In return, the contract stays textual and loose — a field's type or a method's signature can only be known by reading the documentation. There is an approach that does the opposite: writing the contract first in a separate schema file, generating both the client and server side from that schema, and using the wire in binary form instead of text. The next lesson builds this approach: it writes a schema file, generates a validator from it, writes its own binary framing, and compares the byte count of the same record in binary and text form. ### Remote Procedure Call and gRPC Source: https://academia.sh/en/courses/api-design/remote-procedure-call-and-grpc The approach that writes the contract first: a validator generated from a schema file, the field number being the wire's identity, a byte comparison between a custom binary framing and the text format, and how gRPC brings this design together with stream types. The previous lesson tested REST's constraints. What those constraints have in common is that they lean the contract on the rules of the network: addresses, methods, status codes. The contract's detail — a field's type, whether a field is required — stays outside the wire, in the documentation. There is an approach that does the opposite. **Remote procedure call** presents an operation over the network as though it were a local function call: the client calls a function named `GetBook`, the call goes out over the network, runs on the other side, and returns a result. For this presentation to work, the two sides must agree on the call's signature, and this agreement lives not in documentation but in a machine-readable **schema**. #### The Limit of the Procedure Call Analogy The analogy is useful but not exact, and knowing where it breaks down determines the interface's design. A local call is too fast to measure; a remote call takes at least one network round trip in the best case. A local call either runs or throws an exception; a remote call can produce a third outcome — **partial failure**: it is not known whether the request went out, or whether the work was done. In a local call, arguments are objects in memory; in a remote call, they have to be converted into a byte sequence, and the rules of that conversion are the contract itself. These three differences show up in the design of interfaces that use the remote-procedure-call style: calls are kept **coarse-grained** (a small number of full calls instead of a large number of small ones), retry behavior is defined explicitly, and the data format is tightly specified. #### The Contract Is Written First The schema file is the interface's single source of truth. The following file defines a message type and a service; every field has a **number**, a type, a name, and a requiredness. ``` message Book { 1 string isbn required 2 string title required 3 int pages optional 4 bool onLoan required } service Catalog { GetBook(GetBookRequest) -> Book SearchBooks(SearchRequest) -> stream Book } ``` Two things are generated from this file: a **validator** that checks the presence and type of fields, and an **encoder** that converts an object into a byte sequence. The encoder's format never puts field names on the wire at all; every field is referred to by its own number, and numbers are written at variable length. ```js // generate.mjs — generates a validator and a binary encoder from the schema file (the contract comes first). import { readFileSync } from "node:fs"; export const readSchema = (file) => { const messages = {}, services = {}; let open = null; for (const raw of readFileSync(file, "utf8").split("\n")) { const line = raw.trim(); if (!line) continue; let e; if ((e = /^message (\w+) \{$/.exec(line))) { open = messages[e[1]] = []; continue; } if ((e = /^service (\w+) \{$/.exec(line))) { open = services[e[1]] = []; continue; } if (line === "}") { open = null; continue; } if ((e = /^(\d+) (\w+)\s+(\w+)\s+(\S+)$/.exec(line))) open.push({ number: +e[1], type: e[2], name: e[3], required: e[4] === "required" }); else if ((e = /^(\w+)\((\w+)\) -> (stream )?(\w+)$/.exec(line))) open.push({ method: e[1], input: e[2], stream: Boolean(e[3]), output: e[4] }); } return { messages, services }; }; const TYPES = { string: "string", int: "number", bool: "boolean" }; // Generated validator: checks fields for presence and type against the schema. export const makeValidator = (fields) => (obj) => { const defects = []; for (const f of fields) { const v = obj[f.name]; if (v === undefined) { if (f.required) defects.push(`${f.name}: required field missing`); continue; } if (typeof v !== TYPES[f.type]) defects.push(`${f.name}: expected ${f.type}, got ${typeof v}`); } for (const name of Object.keys(obj)) if (!fields.some((f) => f.name === name)) defects.push(`${name}: not in schema`); return defects; }; // Variable-length integer: small numbers take fewer bytes. const writeVarint = (n) => { const b = []; do { let p = n & 0x7f; n >>>= 7; b.push(n ? p | 0x80 : p); } while (n); return Buffer.from(b); }; const readVarint = (b, i) => { let n = 0, k = 0; while (b[i] & 0x80) { n |= (b[i++] & 0x7f) << k; k += 7; } return [n | (b[i++] << k), i]; }; // Binary framing: each field is written with a (number<<3 | wireType) key; the field name never reaches the wire. export const makeEncoder = (fields) => ({ encode(obj) { const parts = []; for (const f of fields) { const v = obj[f.name]; if (v === undefined) continue; const wireType = f.type === "string" ? 2 : 0; parts.push(writeVarint((f.number << 3) | wireType)); if (wireType === 0) parts.push(writeVarint(f.type === "bool" ? (v ? 1 : 0) : v)); else { const buf = Buffer.from(v, "utf8"); parts.push(writeVarint(buf.length), buf); } } return Buffer.concat(parts); }, decode(bytes) { const obj = {}; let i = 0; while (i < bytes.length) { let key; [key, i] = readVarint(bytes, i); const field = fields.find((f) => f.number === key >> 3); if ((key & 7) === 2) { let length; [length, i] = readVarint(bytes, i); obj[field?.name ?? `unknown_${key >> 3}`] = bytes.toString("utf8", i, i + length); i += length; } else { let value; [value, i] = readVarint(bytes, i); obj[field?.name ?? `unknown_${key >> 3}`] = field?.type === "bool" ? Boolean(value) : value; } } return obj; }, }); ``` #### Measurement The measurement answers four questions: what surface is generated from the schema, which records the validator rejects, how many bytes the binary format takes, and what happens to the bytes on the wire when a field name changes? ```js // measure.mjs — tests the schema-first contract: validation, binary/text byte comparison, field renaming import { writeFileSync } from "node:fs"; import { readSchema, makeValidator, makeEncoder } from "./generate.mjs"; const { messages, services } = readSchema("library.schema"); const fields = messages.Book; const validate = makeValidator(fields); const { encode, decode } = makeEncoder(fields); console.log("-- 1) service surface generated from the schema --"); for (const y of services.Catalog) console.log(` ${y.method}(${y.input}) -> ${y.stream ? "stream " : ""}${y.output}`); console.log("\n-- 2) generated validator --"); const SAMPLES = [ { isbn: "978-0262033848", title: "Introduction to Algorithms", pages: 1312, onLoan: false }, { isbn: "978-0201896831", title: "The Art of Computer Programming", onLoan: true }, // no optional field { isbn: "978-0131103627", onLoan: true }, // required field missing { isbn: "978-0596007126", title: "Head First", pages: "272", onLoan: true }, // wrong type { isbn: "978-0000000000", title: "Test", onLoan: true, shelf: "R-12" }, // field not in schema ]; for (const o of SAMPLES) { const defects = validate(o); console.log(` ${JSON.stringify(o).slice(0, 52).padEnd(54)} ${defects.length ? "FAIL " + defects[0] : "PASS"}`); } console.log("\n-- 3) byte count of binary framing vs. text format --"); const valid = SAMPLES.filter((o) => validate(o).length === 0); for (const o of valid) { const binary = encode(o), text = Buffer.from(JSON.stringify(o), "utf8"); console.log(` ${o.isbn} binary=${String(binary.length).padStart(3)} B ` + `text=${String(text.length).padStart(3)} B gain=%${Math.round(100 * (1 - binary.length / text.length))}`); } const totalBinary = valid.reduce((t, o) => t + encode(o).length, 0); const totalText = Buffer.byteLength(JSON.stringify(valid)); console.log(` total binary=${totalBinary} B text=${totalText} B ` + `gain=%${Math.round(100 * (1 - totalBinary / totalText))}`); console.log("\n-- 4) no field name on the wire: the schema changes the field's name --"); const encoded = encode(valid[0]); console.log(` decoded with old schema : ${JSON.stringify(decode(encoded))}`); writeFileSync("new.schema", `message Book { 1 string isbn required 2 string heading required 3 int pages optional 4 bool onLoan required } `); const renamed = makeEncoder(readSchema("new.schema").messages.Book); console.log(` decoded with new schema : ${JSON.stringify(renamed.decode(encoded))}`); console.log(` same bytes, ${encoded.length} B; the only thing that changed is the name attached to the field number.`); ``` ``` -- 1) service surface generated from the schema -- GetBook(GetBookRequest) -> Book SearchBooks(SearchRequest) -> stream Book -- 2) generated validator -- {"isbn":"978-0262033848","title":"Introduction to Al PASS {"isbn":"978-0201896831","title":"The Art of Compute PASS {"isbn":"978-0131103627","onLoan":true} FAIL title: required field missing {"isbn":"978-0596007126","title":"Head First","pages FAIL pages: expected int, got string {"isbn":"978-0000000000","title":"Test","onLoan":tru FAIL shelf: not in schema -- 3) byte count of binary framing vs. text format -- 978-0262033848 binary= 49 B text= 90 B gain=%46 978-0201896831 binary= 51 B text= 81 B gain=%37 total binary=100 B text=174 B gain=%43 -- 4) no field name on the wire: the schema changes the field's name -- decoded with old schema : {"isbn":"978-0262033848","title":"Introduction to Algorithms","pages":1312,"onLoan":false} decoded with new schema : {"isbn":"978-0262033848","heading":"Introduction to Algorithms","pages":1312,"onLoan":false} same bytes, 49 B; the only thing that changed is the name attached to the field number. ``` #### What the Measurement Says **Validation is generated from the contract, not hand-written.** Three of the five records were rejected, and the three rejections are of three separate kinds: a missing required field, a wrong type, a field not in the schema. None of these checks were written in the application code; they came out of the lines in the schema file. When the schema changes, validation changes with it, and the two sides' rules cannot drift apart. **The binary format reduces the same record to roughly half the bytes.** The source of the gain is not compression but the unnecessary never being written at all: field names, quotation marks, colons, and commas never reach the wire. Numbers, too, are written not as text but as a variable-length integer — `1312` takes two bytes instead of four characters. **The field number is the identity, not the field name.** The fourth part repeats a change considered breaking in the first lesson: the `title` field is renamed to `heading`. In the text format, this change had broken the consumer. In the binary format, the same 49-byte sequence decodes without any trouble under the new schema; only the key in the output changes. The identity on the wire is the number `2`. The mirror image of this is also true, and more dangerous: **changing the number breaks silently.** If two fields' numbers are swapped, the validator says nothing, because the bytes are valid; only the values land in the wrong fields. This is where schema-first design's invariant rule comes from: once a field number has been assigned, it is never given to another field, and a retired number is never reused. #### What gRPC Brings The schema, validator, and framing in this lesson are a small example of this design. **gRPC** offers the same structure as a ready-made whole: messages are defined with a **Protocol Buffers** schema, both the client and server sides are generated from that schema, and **HTTP/2** is used as the transport layer. Choosing HTTP/2 is not a detail; it is what makes four call types possible. The `stream` marker in the schema file shows this: a call can be single-request–single-response, but it can also be server-side streaming (one request, many responses), client-side streaming (many requests, one response), or bidirectional streaming. In the library service, jobs like "send the shelf count as a stream" or "give catalog matches as they come in" fit these types. Its cost is just as clear. The content on the wire cannot be read by eye; inspecting a request requires a tool that knows the schema. A browser cannot speak this protocol directly; a translation layer has to sit in between. And using the interface requires access to the other side's schema — where REST is content with sending a request to an address and reading the response, here not a single byte can be understood without the contract. These costs also explain where gRPC is used the most: among a system's own components, in places where both sides are managed by the same team and call volume is high. #### Summary - Remote procedure call presents a network operation as though it were a local call; the analogy breaks down at latency, partial failure, and byte conversion, and these points shape the design. - In schema-first design, the contract is a machine-readable file; the validator and encoder are generated from it, so the two sides' rules cannot drift apart. - In the measurement, binary framing reduced the same records to roughly half the bytes of the text format; the gain comes from field names and delimiters never reaching the wire at all. - The identity on the wire is the field number: when the field name changes, the same bytes decode without trouble, but when the number changes, the error shows up silently as a wrong value. - gRPC combines this design with a Protocol Buffers schema and HTTP/2 transport; it makes four call types possible, and in return readability and direct browser access are lost. #### Next Step Schema-first design fixes the response shape in the contract, and this returns to the problem measured in the second lesson: when the client's requirement changes, the server has to change. There is also an approach that does the exact opposite — one where the client determines the shape, but does so not with a loose selection text but inside a type system the server publishes. The next lesson builds this approach: a single endpoint, a client-driven query bounded by a type system, per-field resolution, and the notion of a partial success response. ### GraphQL Source: https://academia.sh/en/courses/api-design/graphql Client-driven query over a single endpoint: the type system the server publishes, validating the query against the schema before it runs, per-field resolution, and the partial success response that results when one field errors; reusing the same document with variables. The previous lesson built the approach that fixes the contract in a schema, and left one gap: because the response shape is the server's decision, the server has to change when the client's requirement changes. This was the problem measured in the second lesson. **GraphQL** does the opposite. The client determines the response shape, but it does so not with free-form text but inside a **type system** the server publishes. There is a single endpoint; what is requested is written not in the address but in the body's **query document**. This lesson builds that mechanism by writing a small engine. #### The Type System Is the Surface the Server Publishes The type system states which fields exist, what type each field is, and which fields require a subselection. The function that knows how to find a field's data is called a **resolver**; if a field has no resolver, its value is read from the source object. ```js // schema.mjs — the type system and resolvers. The server's whole published surface is here. export const TYPES = { Query: { member: { type: "Member", arg: "id" }, book: { type: "Book", arg: "isbn" } }, Member: { id: "ID", name: "String", email: "String", fine: "Int", loans: { type: "Loan", list: true } }, Loan: { id: "ID", dueDate: "String", overdueDays: "Int", book: { type: "Book" } }, Book: { isbn: "ID", title: "String", author: "String", pages: "Int" }, }; const MEMBERS = [{ id: "u1", name: "Alice Kane", email: "alice@example.test", fine: 0 }]; const BOOKS = { "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", pages: 1312 }, "978-0201896831": { isbn: "978-0201896831", title: "The Art of Computer Programming", author: "Knuth", pages: 650 }, }; const LOANS = [{ id: "o11", member: "u1", isbn: "978-0262033848", dueDate: "2024-05-15" }, { id: "o12", member: "u1", isbn: "978-0201896831", dueDate: "2024-05-17" }]; // Resolver: each field fetches its own data. One field's error does not stop the others. export const RESOLVERS = { "Query.member": (_, arg) => MEMBERS.find((m) => m.id === arg), "Query.book": (_, arg) => BOOKS[arg], "Member.loans": (member) => LOANS.filter((l) => l.member === member.id), "Loan.book": (loan) => BOOKS[loan.isbn], "Loan.overdueDays": () => { throw new Error("overdue service is not responding"); }, }; ``` The `Loan.overdueDays` resolver throwing an error is deliberate: in a real setup, the overdue amount comes from a separate service, and that service might not respond. This field exists to demonstrate the partial success response. #### The Engine's Three Steps The query document passes through three steps: **parsing** (the text turns into a selection tree), **validation** (the tree is checked against the type system), and **execution** (each field is computed by its own resolver). Validation coming before execution is decisive: for an invalid query, no resolver runs, and therefore no data access happens at all. ```js // execute.mjs — a small query engine: parsing, validation against the type system, per-field execution import { TYPES, RESOLVERS } from "./schema.mjs"; export const parse = (text) => { // converts a query document into a selection tree const s = text.match(/[A-Za-z_][A-Za-z0-9_]*|[{}():]|"[^"]*"|\d+/g) ?? []; let i = 0; const selectionSet = () => { const fields = []; i++; while (s[i] !== "}") { const name = s[i++]; let arg = null; if (s[i] === "(") { arg = s[i + 3].replace(/^"|"$/g, ""); i += 5; } // name ( key : value ) fields.push({ name, arg, sub: s[i] === "{" ? selectionSet() : null }); } i++; return fields; }; s.unshift("{"); s.push("}"); return selectionSet(); }; export const validate = (fields, typeName = "Query", path = []) => { // checks against the type system const defects = []; for (const field of fields) { const def = TYPES[typeName]?.[field.name]; const at = [...path, field.name]; if (!def) { defects.push(`${at.join(".")}: no such field on type ${typeName}`); continue; } const isObject = typeof def === "object"; if (isObject && !field.sub) defects.push(`${at.join(".")}: ${def.type} is an object, a subfield must be selected`); else if (!isObject && field.sub) defects.push(`${at.join(".")}: ${def} is a scalar, a subfield cannot be selected`); else if (isObject) defects.push(...validate(field.sub, def.type, at)); } return defects; }; export const execute = (fields, root) => { const errors = []; const step = (fields, source, typeName, path) => { const output = {}; for (const field of fields) { const def = TYPES[typeName][field.name]; const at = [...path, field.name]; const resolver = RESOLVERS[`${typeName}.${field.name}`] ?? ((k) => k[field.name]); let value; try { value = resolver(source, field.arg); } catch (e) { errors.push({ path: at.join("."), message: e.message }); output[field.name] = null; continue; } if (typeof def !== "object") { output[field.name] = value ?? null; continue; } output[field.name] = def.list ? value.map((d, k) => step(field.sub, d, def.type, [...at, k])) : step(field.sub, value, def.type, at); } return output; }; return { data: step(fields, root, "Query", []), errors }; }; export const substituteVariables = (document, variables) => document.replace(/\$(\w+)/g, (_, name) => JSON.stringify(variables[name])); ``` ```js // server.mjs — a single endpoint. The path is the same for every query; what is requested is written in the body. import { createServer } from "node:http"; import { parse, validate, execute, substituteVariables } from "./execute.mjs"; createServer(async (req, res) => { res.sendDate = false; let raw = ""; for await (const chunk of req) raw += chunk; const { document, variables = {} } = JSON.parse(raw); const fields = parse(substituteVariables(document, variables)); const defects = validate(fields); // Validation happens before execution: for an invalid query, no resolver runs at all. const body = defects.length ? { errors: defects } : execute(fields, null); const text = JSON.stringify(body); res.setHeader("Content-Type", "application/json; charset=utf-8"); res.setHeader("Content-Length", Buffer.byteLength(text)); res.writeHead(defects.length ? 400 : 200).end(text); }).listen(8471, "127.0.0.1", () => console.log("single endpoint 127.0.0.1:8471/graph")); ``` #### Measurement ```js // measure.mjs — measures the single endpoint's four properties const query = async (document, variables) => { const body = JSON.stringify({ document, variables }); const y = await fetch("http://127.0.0.1:8471/graph", { method: "POST", body }); const text = await y.text(); return { status: y.status, bytes: Buffer.byteLength(text), sent: Buffer.byteLength(body), body: JSON.parse(text) }; }; const print = (label, y) => console.log(` ${label.padEnd(22)} HTTP ${y.status} ` + `sent=${String(y.sent).padStart(3)} B received=${String(y.bytes).padStart(3)} B\n` + ` ${JSON.stringify(y.body)}`); console.log("-- 1) same endpoint, two different client requirements --"); print("shelf display", await query('member(id: "u1") { name }')); print("overdue report", await query('member(id: "u1") { name loans { dueDate book { title author } } }')); console.log("\n-- 2) type system: query is validated before execution --"); print("nonexistent field", await query('member(id: "u1") { name phone }')); print("subselection on scalar", await query('member(id: "u1") { name { length } }')); print("no selection on object", await query('member(id: "u1") { name loans }')); console.log("\n-- 3) partial success: one field's resolver errors --"); print("overdue field", await query('member(id: "u1") { name loans { dueDate overdueDays } }')); console.log("\n-- 4) variables: same document, two different values --"); const DOCUMENT = "book(isbn: $id) { title author pages }"; for (const id of ["978-0262033848", "978-0201896831"]) print(`id=${id.slice(-4)}`, await query(DOCUMENT, { id })); ``` ```bash node server.mjs & p=$! curl -s --retry 20 --retry-connrefused --retry-delay 0 -o /dev/null \ -X POST -d '{"document":"book(isbn: \"978-0262033848\") { title }"}' http://127.0.0.1:8471/graph node measure.mjs kill $p ``` ``` single endpoint 127.0.0.1:8471/graph -- 1) same endpoint, two different client requirements -- shelf display HTTP 200 sent= 42 B received= 53 B {"data":{"member":{"name":"Alice Kane"}},"errors":[]} overdue report HTTP 200 sent= 82 B received=245 B {"data":{"member":{"name":"Alice Kane","loans":[{"dueDate":"2024-05-15","book":{"title":"Introduction to Algorithms","author":"Cormen"}},{"dueDate":"2024-05-17","book":{"title":"The Art of Computer Programming","author":"Knuth"}}]}},"errors":[]} -- 2) type system: query is validated before execution -- nonexistent field HTTP 400 sent= 48 B received= 57 B {"errors":["member.phone: no such field on type Member"]} subselection on scalar HTTP 400 sent= 53 B received= 77 B {"errors":["member.name: String is a scalar, a subfield cannot be selected"]} no selection on object HTTP 400 sent= 48 B received= 75 B {"errors":["member.loans: Loan is an object, a subfield must be selected"]} -- 3) partial success: one field's resolver errors -- overdue field HTTP 200 sent= 72 B received=318 B {"data":{"member":{"name":"Alice Kane","loans":[{"dueDate":"2024-05-15","overdueDays":null},{"dueDate":"2024-05-17","overdueDays":null}]}},"errors":[{"path":"member.loans.0.overdueDays","message":"overdue service is not responding"},{"path":"member.loans.1.overdueDays","message":"overdue service is not responding"}]} -- 4) variables: same document, two different values -- id=3848 HTTP 200 sent= 89 B received= 99 B {"data":{"book":{"title":"Introduction to Algorithms","author":"Cormen","pages":1312}},"errors":[]} id=6831 HTTP 200 sent= 89 B received=102 B {"data":{"book":{"title":"The Art of Computer Programming","author":"Knuth","pages":650}},"errors":[]} ``` #### What the Measurement Says **A single endpoint, two different response shapes.** The shelf display received 53 bytes, the overdue report 245 bytes. Both went to the same address, and nothing changed on the server. This is the property the query-based style showed in the second lesson; the difference here is that the selection is made not with free-form text but inside a published type system. **The type system catches three separate errors before execution.** A field that does not exist, a subselection made underneath a scalar, and an object requested without a subselection — all three are rejected with `400`, and no resolver runs. This shows that a query-based interface is not loose: the client can choose the field combination, but it cannot step outside the schema. **A partial success response is not an HTTP error.** The overdue field's resolver threw an error, but the response came back `200`. Most of the requested fields in the data portion are filled in, the field that errored is `null`, and the error list reports every failure along with its **path**: `member.loans.0.overdueDays`. This behavior is the direct consequence of per-field resolution — because each field fetches its own data, one field's failure does not affect its neighbor. Its counterpart on the consumer side is that the HTTP status code alone is not a sufficient signal. A client that receives `200` still has to inspect the body. The notion of partial success was described from the client side in the Application Architecture course; here, the same behavior's source on the server becomes visible. **Variables make the query document reusable.** The same document worked for two different books. Sending variables separately has a second benefit too: because the document stays fixed, it can be recognized, cached, and cost-estimated in advance on the server side. #### Costs This flexibility has four costs, and three of them show up in the measurement. **Caching becomes harder.** Every query goes to a single address with the `POST` method. The entity tag mechanism from the previous lesson does not work directly here, because the address does not name a record. Caching is set up either on the client side, keyed by the query document, or at the field level. **The response's cost is not known in advance.** The server cannot compute how much data access an incoming query will trigger before the query arrives. A query with growing depth, or a nested selection inside a long list, can force a lot of work with a single request. This requires protections like query cost and depth limits. **Per-field resolution multiplies data access.** The `Loan.book` resolver ran twice for the two loans in the measurement; this number grows as the loan count grows. The N+1 pattern seen in the resource-based style in the second lesson has moved inside the server here. Its solution through batch loading is covered in this course's GraphQL in Detail topic. **The error contract splits in two.** A single request carries both a transport-level status code and an error list inside the body. If it is not explicitly decided when to use which, consumers are forced to look for errors in two separate places. #### Summary - GraphQL works over a single endpoint; what is requested is written not in the address but in the query document in the body, and the client's selection set determines the response shape. - The type system the server publishes bounds the selection: in the measurement, a nonexistent field, a subselection on a scalar, and an object without a subselection were all rejected with `400` before execution. - Per-field resolution lets the others keep working when one field's resolver errors; the result is a partial success response with a `200` status that carries both the data and the error list. - Variables keep the document fixed; the same document is reused with different values and becomes recognizable on the server side. - The costs are reflected in the measurement: a single address and `POST` make caching harder, the work a response will trigger is not known in advance, per-field resolution multiplies data access, and the error contract splits into two places. #### Next Step Working over a single endpoint, with an operation name carried in the body, is not a new idea. There is a family of protocols that built the same mechanism and left a wide legacy behind it: a family where every message is wrapped in an envelope, the operation name and the error are also carried inside that envelope, and the contract is published in a separate definition file. The next lesson builds this envelope-based approach with its own small parser, measures the envelope's byte cost, and shows which of today's interface decisions were inherited from there. ### Legacy Protocols Source: https://academia.sh/en/courses/api-design/legacy-protocols The design and legacy of envelope-based protocols: separating header from body, carrying the operation name inside the envelope, reporting errors inside the envelope, how the envelope's cost changes with record count, and which of today's interface decisions were inherited from here. The previous lesson built an interface that works over a single endpoint, with a selection document carried in the body. This mechanism is not new. There is a family of protocols that built the same idea — single address, single method, naming the operation in the body — much earlier, and part of today's interface decisions are inherited directly from there. This family's common building block is the **envelope**: a container that wraps the entire message and holds a **header** and a **body** inside it. **SOAP** is the best-known example of this structure and is built on XML; its `Envelope`, `Header`, `Body`, and `Fault` elements are exactly what this lesson's `Envelope`, `Header`, `Body`, and `Fault` elements are built to match. #### The Idea of the Envelope The problem the envelope solves is this: carrying a message's own metadata alongside it, independent of the protocol carrying the message. HTTP headers already do this job, but the envelope design is not tied to HTTP — the same envelope can travel over a message queue or a mail message too. Its cost and its gain sit together. The envelope has three structural decisions: - **The header carries cross-cutting concerns:** the trace id, credentials, the operation name. Intermediate layers can read this part and decide without ever parsing the body. - **The body carries the payload:** data specific to the operation. The payload's shape changes from operation to operation. - **The error also goes into the body:** failure is reported by a fault element inside the body. Even when the transport carries a status code — SOAP's HTTP binding sends the fault with a `500` — the error's contract lives inside the envelope; on transports like a queue or mail it is the only place. #### A Small Implementation The following file contains the functions that build and read the envelope. The XML parser converts nested elements into an object; it does not support attributes or namespaces, because what the lesson needs is the structure itself. ```js // envelope.mjs — a small XML parser and envelope builders. // Envelope/Header/Body/Fault elements are this lesson's counterpart to SOAP's Envelope/Header/Body/Fault. const escape = (s) => String(s).replace(/&/g, "&").replace(//g, ">"); const unescape = (s) => s.replace(/</g, "<").replace(/>/g, ">").replace(/&/g, "&"); export const writeXml = (name, content) => typeof content === "object" && content !== null ? `<${name}>${Object.entries(content).map(([a, d]) => writeXml(a, d)).join("")}` : `<${name}>${escape(content)}`; // Parser: turns nested elements into an object. Siblings with the same name are collected into an array. export const readXml = (text) => { const element = /<(\w+)>([\s\S]*?)<\/\1>/g; const output = {}; let e, found = false; while ((e = element.exec(text))) { found = true; const value = /<\w+>/.test(e[2]) ? readXml(e[2]) : unescape(e[2]); if (e[1] in output) output[e[1]] = [].concat(output[e[1]], value); else output[e[1]] = value; } return found ? output : unescape(text); }; export const wrap = (operation, trace, body) => `${writeXml("Header", { Operation: operation, Trace: trace })}` + `${Object.entries(body).map(([a, d]) => writeXml(a, d)).join("")}`; export const faultEnvelope = (code, reason, detail) => `
Fault
` + writeXml("Fault", { Code: code, Reason: reason, Detail: detail }) + ``; // The one place the intermediary reads: the header can be taken without parsing the body at all. export const extractHeader = (envelope) => readXml(/
[\s\S]*?<\/Header>/.exec(envelope)[0]).Header; ``` ```js // server.mjs — single endpoint, single method: the operation name is in the envelope's header. import { createServer } from "node:http"; import { readXml, wrap, faultEnvelope, extractHeader } from "./envelope.mjs"; const BOOKS = { "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", pages: 1312 }, "978-0201896831": { isbn: "978-0201896831", title: "The Art of Computer Programming", author: "Knuth", pages: 650 }, }; createServer(async (req, res) => { res.sendDate = false; let raw = ""; for await (const p of req) raw += p; const header = extractHeader(raw); // routing: only the header is read const body = readXml(raw).Envelope.Body; let output; if (header.Operation === "GetBook") { const book = BOOKS[body.BookRequest.isbn]; output = book ? wrap("GetBookResponse", header.Trace, { Book: book }) : faultEnvelope("Client", "book not found", `isbn: ${body.BookRequest.isbn}`); } else output = faultEnvelope("Client", "unknown operation", header.Operation); res.setHeader("Content-Type", "text/xml; charset=utf-8"); res.setHeader("Content-Length", Buffer.byteLength(output)); // Both faults and successful requests return with the same status code: the error is in the envelope, not the transport layer. res.writeHead(200).end(output); }).listen(8481, "127.0.0.1", () => console.log("envelope endpoint 127.0.0.1:8481/service")); ``` #### Measurement ```js // measure.mjs — measures three properties of the envelope-based protocol import { wrap, readXml, extractHeader } from "./envelope.mjs"; const call = async (operation, trace, body) => { const request = wrap(operation, trace, body); const y = await fetch("http://127.0.0.1:8481/service", { method: "POST", headers: { "Content-Type": "text/xml" }, body: request }); const text = await y.text(); return { status: y.status, sent: Buffer.byteLength(request), received: Buffer.byteLength(text), text, parsed: readXml(text).Envelope.Body }; }; console.log("-- 1) single endpoint, operation name in the envelope --"); const good = await call("GetBook", "trace-1", { BookRequest: { isbn: "978-0262033848" } }); console.log(` request : ${wrap("GetBook", "trace-1", { BookRequest: { isbn: "978-0262033848" } })}`); console.log(` response : ${good.text}`); console.log(` HTTP ${good.status} sent=${good.sent} B received=${good.received} B`); console.log("\n-- 2) envelope cost: the same data in JSON and envelope form --"); const RECORD = { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", pages: 1312 }; for (const quantity of [1, 5, 20]) { const list = Array.from({ length: quantity }, () => RECORD); const json = Buffer.byteLength(JSON.stringify({ Book: list })); const envelope = Buffer.byteLength(wrap("GetBookResponse", "trace-1", { Book: list })); console.log(` ${String(quantity).padStart(2)} record json=${String(json).padStart(4)} B ` + `envelope=${String(envelope).padStart(4)} B difference=%${Math.round(100 * (envelope / json - 1))}`); } console.log("\n-- 3) the error is in the envelope, not the transport layer --"); const bad = await call("GetBook", "trace-2", { BookRequest: { isbn: "978-0000000000" } }); console.log(` response : ${bad.text}`); console.log(` a client that looks at the status code -> HTTP ${bad.status} : "request successful"`); console.log(` a client that parses the body -> ${JSON.stringify(bad.parsed.Fault)}`); console.log("\n-- 4) an intermediary reads the header without parsing the body --"); for (const msg of [good.text, bad.text]) { const bodyLength = /[\s\S]*<\/Body>/.exec(msg)[0].length; console.log(` header=${JSON.stringify(extractHeader(msg))} (body ${bodyLength} B, not parsed)`); } ``` ```bash node server.mjs & p=$! curl -s --retry 20 --retry-connrefused --retry-delay 0 -o /dev/null -X POST \ -d '
x
' http://127.0.0.1:8481/service node measure.mjs kill $p ``` ``` envelope endpoint 127.0.0.1:8481/service -- 1) single endpoint, operation name in the envelope -- request :
GetBooktrace-1
978-0262033848 response :
GetBookResponsetrace-1
978-0262033848Introduction to AlgorithmsCormen1312 HTTP 200 sent=157 B received=234 B -- 2) envelope cost: the same data in JSON and envelope form -- 1 record json= 104 B envelope= 241 B difference=%132 5 record json= 480 B envelope= 709 B difference=%48 20 record json=1890 B envelope=2484 B difference=%31 -- 3) the error is in the envelope, not the transport layer -- response :
Fault
Clientbook not foundisbn: 978-0000000000 a client that looks at the status code -> HTTP 200 : "request successful" a client that parses the body -> {"Code":"Client","Reason":"book not found","Detail":"isbn: 978-0000000000"} -- 4) an intermediary reads the header without parsing the body -- header={"Operation":"GetBookResponse","Trace":"trace-1"} (body 136 B, not parsed) header={"Operation":"Fault"} (body 115 B, not parsed) ``` #### What the Measurement Says **The envelope's cost is fixed, the payload's cost is variable.** For a single record, the envelope is 132% larger than the same data's JSON form; at twenty records, this difference drops to 31%. The envelope itself — the header section, the wrapping elements — is a fixed cost independent of record count; the per-record overhead comes from the closing tags and repeats with every record. The envelope is expensive for small, frequent calls; the ratio drops for large payloads. **The cost of reporting the error in the envelope lands on the consumer.** When a nonexistent book was requested, the server returned `200`. A client that looks at the status code counts the request as successful; seeing the error requires parsing the body. This is the source of the error contract splitting in two, mentioned in the fourth lesson, and it comes back in exactly the same shape in GraphQL's partial success response. **The header can be read independently of the body.** The last part of the measurement shows that a body over a hundred bytes can have its header taken without being parsed at all. This means an intermediate layer — a router, an authorization layer, a tracing layer — can do its job without understanding the payload. The **trace id** the header carries links the same request's records across different components together. #### The Legacy Left Behind The aspects of envelope-based design carried forward to today are a direct continuation of the properties seen in the measurement. **A machine-readable contract.** This family's interfaces publish which operations exist and each operation's input–output types in a separate definition file. Two habits remain from this: the contract being an executable file separate from documentation, and the client side being generated from that file. The same idea shows up as the schema file in the fourth lesson, and later in this course as OpenAPI- and JSON-Schema-based definitions. **A structured error.** The triple inside the fault element — code, reason, detail — is the ancestor of today's standard error bodies: a code the machine will read, an explanation the human will read, and a context-specific extra. This structure is detailed in the Error Response Format lesson. **Separating cross-cutting concerns.** Keeping identity, trace, and routing information apart from the payload has been preserved; in HTTP-based interfaces, this separation lives in HTTP headers instead of the envelope's header. **Carrying the operation name in the body.** The pattern of a single endpoint with the operation name in the body persists in both remote-procedure-call-style HTTP interfaces and in GraphQL. What was not inherited is just as clear. The desire to be transport-independent renders HTTP's own semantics — methods, status codes, conditional requests, cache directives — unused. The cacheability gain measured in the third lesson cannot be obtained in a design where every request is `POST`. Together with the text format's overhead of detail, this is the real reason that opened the way for designs that do the same job with fewer bytes and fewer layers. #### Summary - Envelope-based protocols work with a container that wraps the message as a header and a body; the operation name is carried not in the address but in the envelope's header, and the transport layer is treated as replaceable. - In the measurement, the envelope's byte cost came out to 132% for a single record and 31% for twenty records: the envelope itself is a fixed cost, and the closing tags are a cost that repeats per record. - Because the server in the measurement reports the error only inside the envelope, it returns `200` even on a failed operation; a client that looks at the status code cannot see the error and is forced to parse the body. - Because the header can be read independently of the body, intermediate layers can do routing, authorization, and tracing without ever parsing the payload. - The inherited legacy is four items: the machine-readable contract file, the structured error shape, separating cross-cutting concerns from the payload, and a named operation over a single endpoint. #### Next Step The shared assumption of the five styles so far is the same: the client initiates communication, the server responds, the exchange ends. Some of the library service's jobs do not fit this pattern. A book returning to the shelf is an event that happens on the server, and nobody asks the waiting client about it. A count job runs for minutes, and keeping the request open is not an option. The next lesson builds communication forms outside the request–response pattern and compares two of them by measurement: how many connections and how much delay does an event stream versus long polling take to deliver the same notification? ### Synchronous and Asynchronous APIs Source: https://academia.sh/en/courses/api-design/synchronous-and-asynchronous-apis Communication forms outside the request–response pattern: event stream, long polling, and short polling are compared on the same event sequence by connection count, bytes carried, and delivery delay; a long-running job is modeled with an accepted request and a separate status resource. The five styles so far shared a common assumption: the client initiates communication, the server responds, the exchange ends. Some of the library service's jobs do not fit this pattern. Three situations break this pattern. First, **the event occurs on the server**: nobody notifies a client waiting for a book to return to the shelf, and the client does not know when to ask either. Second, **the work takes a long time**: a branch count job runs for minutes, and keeping the request open for that whole time holds up both the connection and the client. Third, **the same notification goes out to many consumers**: each consumer asking separately repeats the same work. This lesson writes and measures the first situation in three forms, then builds the pattern for the second situation. #### Publishing the Same Event in Three Forms The server publishes six events — books returning to the shelf — and serves the same event sequence from three separate endpoints. **Event stream** keeps the connection open and writes as an event occurs. **Long polling** holds the request without a response until a new event comes up. **Short polling** answers every request immediately; it can come back empty, and the client asks again at fixed intervals. In all three, the client sends the id of the last event it has seen. This is required in the polling forms: events that occur in the gap between two requests would otherwise be lost. ```js // server.mjs — publishes the same event stream in three forms: event stream, long polling, short polling import { createServer } from "node:http"; const EVENTS = []; // published events; the id is the array index const waiters = []; // requests waiting for a response in long polling const streams = new Set(); // open event-stream connections const jobs = new Map(); // state of long-running jobs let jobCounter = 0; const publish = (event) => { EVENTS.push(event); for (const y of streams) y.write(`id: ${EVENTS.length}\ndata: ${JSON.stringify(event)}\n\n`); while (waiters.length) waiters.pop()(); }; const json = (res, body) => { const text = JSON.stringify(body); res.setHeader("Content-Type", "application/json; charset=utf-8"); res.setHeader("Content-Length", Buffer.byteLength(text)); res.writeHead(200).end(text); }; const after = (n) => EVENTS.slice(n).map((o, i) => ({ id: n + i + 1, ...o })); createServer((req, res) => { res.sendDate = false; const url = new URL(req.url, "http://local"); const since = Number(url.searchParams.get("after") ?? 0); // 1) Event stream: the connection stays open, the server writes as events occur. if (url.pathname === "/stream") { res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-store", Connection: "keep-alive" }); for (const o of after(since)) res.write(`id: ${o.id}\ndata: ${JSON.stringify(o)}\n\n`); streams.add(res); req.on("close", () => streams.delete(res)); return; } // 2) Long polling: if there is no new event, the response is held; it is written once the event arrives. if (url.pathname === "/long") { if (EVENTS.length > since) return json(res, { events: after(since) }); const timeout = setTimeout(() => { const k = waiters.indexOf(resume); if (k >= 0) waiters.splice(k, 1); json(res, { events: [] }); }, 2000); const resume = () => { clearTimeout(timeout); json(res, { events: after(since) }); }; waiters.push(resume); return; } // 3) Short polling: every request is answered immediately, the client asks again at fixed intervals. if (url.pathname === "/short") return json(res, { events: after(since) }); // 4) Long-running job: the request is accepted, the result is tracked from a separate resource. if (url.pathname === "/counts" && req.method === "POST") { const id = `job-${++jobCounter}`; jobs.set(id, { status: "running", result: null }); setTimeout(() => jobs.set(id, { status: "done", result: { missing: 3 } }), 600); res.setHeader("Location", `/jobs/${id}`); res.setHeader("Content-Type", "application/json; charset=utf-8"); return res.writeHead(202).end(JSON.stringify({ job: id })); } if (url.pathname.startsWith("/jobs/")) return json(res, jobs.get(url.pathname.slice(6))); // The publication is triggered from outside so the measurements run under the same condition. if (url.pathname === "/start") { start(); return json(res, { started: true }); } return json(res, { error: "not found" }); }).listen(8491, "127.0.0.1", () => console.log("event endpoint 127.0.0.1:8491")); // Six events, at 250 ms intervals: a book returning to the shelf is an event that occurs on the server. const start = () => { let n = 0; const t = setInterval(() => { publish({ type: "book_back_on_shelf", isbn: `978-000000000${n}`, at: Date.now() }); if (++n === 6) clearInterval(t); }, 250); }; ``` For the measurement to be valid, all three clients have to watch the same event sequence at the same time. So the publication does not start on its own: it is triggered from outside once all three clients are connected. ```js // measure.mjs — measures three forms on the same event stream, at the same time: connections, bytes, delivery delay import { Agent, request } from "node:http"; import { connect } from "node:net"; const createClient = () => { const state = { connections: 0, requests: 0, bytes: 0, delays: [] }; const agent = new Agent({ keepAlive: false }); // every request opens a new connection agent.createConnection = (s) => { state.connections++; const b = connect(s); b.on("data", (p) => (state.bytes += p.length)); return b; }; return { state, agent }; }; const record = (state, events) => { for (const o of events) state.delays.push(Date.now() - o.at); return events.length ? events[events.length - 1].id : null; }; const get = (client, path) => new Promise((resolve) => { client.state.requests++; request({ agent: client.agent, host: "127.0.0.1", port: 8491, path }, (y) => { let m = ""; y.on("data", (p) => (m += p)); y.on("end", () => resolve(JSON.parse(m))); }).end(); }); // 1) EVENT STREAM — a single connection stays open, the client reads as the server writes const stream = createClient(); const streamTask = new Promise((done) => { stream.state.requests++; request({ agent: stream.agent, host: "127.0.0.1", port: 8491, path: "/stream?after=0" }, (y) => { y.on("data", (p) => { for (const block of String(p).split("\n\n")) { const e = /data: (.+)/.exec(block); if (e) record(stream.state, [JSON.parse(e[1])]); } if (stream.state.delays.length === 6) { y.destroy(); done(); } }); }).end(); }); // 2) LONG POLLING — the server holds the response until an event arrives; the client reconnects const long = createClient(); const longTask = (async () => { let last = 0; while (long.state.delays.length < 6) last = record(long.state, (await get(long, `/long?after=${last}`)).events) ?? last; })(); // 3) SHORT POLLING — asking again at a fixed interval const short = createClient(); const shortTask = (async () => { let last = 0; while (short.state.delays.length < 6) { last = record(short.state, (await get(short, `/short?after=${last}`)).events) ?? last; if (short.state.delays.length < 6) await new Promise((c) => setTimeout(c, 500)); } })(); await new Promise((c) => setTimeout(c, 200)); // publication starts once all three clients are connected await fetch("http://127.0.0.1:8491/start"); await Promise.all([streamTask, longTask, shortTask]); const summarize = (label, state) => { const d = state.delays; console.log(` ${label.padEnd(14)} connections=${String(state.connections).padStart(2)} ` + `bytes=${String(state.bytes).padStart(4)} events=${d.length} ` + `average delay=${String(Math.round(d.reduce((t, v) => t + v, 0) / d.length)).padStart(3)} ms ` + `max=${String(Math.max(...d)).padStart(3)} ms`); }; console.log("-- six events, published at 250 ms intervals --"); summarize("event stream", stream.state); summarize("long polling", long.state); summarize("short polling", short.state); // 4) LONG-RUNNING JOB — the request is accepted, the result is tracked from a separate resource console.log("\n-- long-running job: accepted request --"); const t0 = Date.now(); const accepted = await fetch("http://127.0.0.1:8491/counts", { method: "POST" }); const location = accepted.headers.get("location"); console.log(` POST /counts -> HTTP ${accepted.status} Location: ${location}`); for (let i = 0; i < 5; i++) { const d = await (await fetch(`http://127.0.0.1:8491${location}`)).json(); console.log(` GET ${location} (+${Date.now() - t0} ms) -> ${JSON.stringify(d)}`); if (d.status === "done") break; await new Promise((c) => setTimeout(c, 200)); } ``` ```bash node server.mjs > /dev/null & p=$! curl -s --retry 20 --retry-connrefused --retry-delay 0 -o /dev/null "http://127.0.0.1:8491/short?after=0" node measure.mjs kill $p ``` ``` -- six events, published at 250 ms intervals -- event stream connections= 1 bytes= 681 events=6 average delay= 3 ms max= 4 ms long polling connections= 6 bytes=1182 events=6 average delay= 2 ms max= 2 ms short polling connections= 5 bytes=1068 events=6 average delay=168 ms max=294 ms -- long-running job: accepted request -- POST /counts -> HTTP 202 Location: /jobs/job-1 GET /jobs/job-1 (+8 ms) -> {"status":"running","result":null} GET /jobs/job-1 (+212 ms) -> {"status":"running","result":null} GET /jobs/job-1 (+417 ms) -> {"status":"running","result":null} GET /jobs/job-1 (+624 ms) -> {"status":"done","result":{"missing":3}} ``` Delay values and byte counts depend on the machine and the headers the runtime writes; the count of six events, the connection counts, and the ordering by size do not. #### What the Measurement Says **Event stream carries six events over a single connection.** The client connects once and reads as the server writes; the delay is on the order of milliseconds. Bytes carried are also the lowest, because a single set of headers was sent for all six events. **Long polling achieves the same delay with six connections.** The response closes after every event, and the client reconnects; every reconnection means a new request line and a new set of headers. The byte count is roughly double that of the event stream. The delay is low because the server held the request until the event arrived — the cost of waiting is not in bytes but in the number of open requests: every waiting client holds a response object and a connection on the server. **Short polling ties the delay to the polling interval.** The average delay is 168 milliseconds, the largest 294 milliseconds; the polling interval is half a second, and the expected average delay is about half the interval. Connection count here depends not on the number of events but on the time elapsed: even with no events at all, the client keeps asking, and every empty response adds to the bytes carried. The choice among the three is not made by a single measure. Short polling is the simplest and is enough where delay does not matter; to intermediate layers, it is just an ordinary request. Long polling lowers the delay but raises the number of requests held open on the server. Event stream wins on both counts, and in exchange it requires a long-lived connection: intermediate layers must not close the connection, the server must be able to allocate a connection per client, and the stream must be able to reconnect if it breaks. This last point corresponds to the `after` parameter in the measurement. **Server-Sent Events** sends back the id of the last event received with the `Last-Event-ID` header when reconnecting a dropped connection, and the server picks up where it left off. Without this mechanism, no asynchronous publication is reliable: events that occur at the moment of disconnection are silently lost. #### One-Way and Two-Way Event stream is one-way: server to client. If the client also needs to send data continuously — like a handheld terminal reporting every tag it reads during a shelf count instantly — a two-way connection is needed, and its counterpart is the **WebSocket** protocol. Its cost is stepping outside HTTP's semantics: the notions of method, status code, cache directive, and conditional request no longer apply once the connection is established. None of the gains measured in the third lesson hold over an open socket. The selection criterion is simple: if data comes only from the server, an event stream is enough; if both sides send data continuously, a two-way connection is needed. #### The Accepted Request The last part of the measurement builds the second situation. When a branch count is requested, the server does not try to finish the work; it **accepts** the request, returns `202`, and reports the resource the job will be tracked from with the `Location` header. The client learns the status by asking that resource; once the job finishes, the same resource carries the result. This pattern has three benefits. Request duration is separated from the job's duration, so the timeout problem disappears. Because the job's status is a resource, it stays inside the uniform interface; it can be listed, canceled, stored. And what happens when the same job is requested twice becomes a separate decision — this is the subject of this course's Idempotency Keys lesson. It is not even required for the client to ask in order to learn that the job is done: when the status resource is combined with an event stream, the notification reaches the client on its own once the job finishes. The two patterns are used together, not as substitutes for each other. #### Summary - The request–response pattern falls short in three situations: an event occurring on the server, work taking a long time, and the same notification going out to many consumers. - In the measurement, event stream carried six events over one connection and 681 bytes, long polling over six connections and 1182 bytes; both stayed at millisecond-level delay. - In short polling, the delay depends on the polling interval: at a half-second interval, an average of 168 and a maximum of 294 milliseconds were measured, and the connection count tracked the time elapsed rather than the event count. - Every polling form has to carry the id of the last event seen; events that occur during a disconnection or in the gap between requests would otherwise be lost. - A long-running job is accepted with `202` and tracked with a separate status resource; this separates request duration from job duration, and the job turns into a resource inside the uniform interface. #### Next Step This topic compared the styles and measured what each one makes easier and what it makes harder. What comes next is the detail of a choice: actually designing the resource-based style. In the measurements, resources were assumed ready-made — book, member, loan, job — yet their being resources was a decision. Which domain concepts turn into a resource, and which stay as another resource's field? Which resource does an action like "lend the book" sit on? The next topic's first lesson starts with this question and derives the resource model from the library domain's concepts. ## Resource and Contract Design Turning domain concepts into resources, URI layout, mapping HTTP methods and status codes correctly, body format, pagination, filtering, field selection, idempotency keys, and link-oriented responses. ### Resource Modeling Source: https://academia.sh/en/courses/api-design/resource-modeling Turning domain concepts into addressable resources: an endpoint list that grows with action names, modeling an operation as a resource's state change, computed resources with no table of their own, and the cost of choosing an identity. The API Styles topic finished the work of choosing: it established which constraint favors resource-oriented, remote procedure call, and query-based approaches over one another, and where synchronous and asynchronous communication diverge. When the decision for the library loan service falls on the resource-oriented style, the work does not end — the real design starts there. The style says "address resources and speak through a uniform interface," but it does not say what counts as a resource. This lesson fills that gap. There is a domain at hand — book, member, loan, branch, overdue fine, loan period — and these concepts are not all of the same kind. Some are addressed, some live only as a field inside another resource, and some become resources without a row in any table. The criterion that draws the line is identity. #### What a Resource Is, and Is Not A **resource** is something the client will want to refer to individually and that has an identity that stays the same over time. Both halves of this definition are functional. Something without an identity cannot be addressed; addressing something no one refers to only inflates the surface. Apply the criterion to the loan service. A **book** is a resource: it is searched for on its own in the catalog, its detail is shown, it is updated. A **member** is a resource. A **loan** is a resource — though it looks like an action at first glance, it is something with its own identity, a start, an end, and a history. A **branch** is a resource. The **loan period**, by contrast, is not a resource: the fourteen-day period is a rule, and it lives inside the loan record or in the service's configuration. The **overdue fine** is not a resource on its own either; it is a component of a member's debt. The resource itself is abstract. The JSON body sent to the client is its **representation** at that moment. The same resource can have more than one representation: a summary list row and a detail page describe the same loan record at different levels of detail. Separating the resource from its representation is what makes field selection and partial response possible in later lessons. #### The Cost of Turning Actions into Address Names When a service is written without resource thinking, addresses turn into action names: `/issueLoan`, `/returnLoan`, `/memberLoans`. Every new client need opens a new route. The two servers below answer the same questions over the same data; what differs is how their address sets grow. ```js // action-server.mjs — a design that opens a new route for every client need import { createServer } from "node:http"; const BOOKS = [ { isbn: "978-0201896831", title: "The Art of Computer Programming", author: "Knuth", branch: "S-01" }, { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", branch: "S-02" }, ]; const LOANS = [ { id: 1, member: "U-1001", isbn: "978-0262033848", return: null }, { id: 2, member: "U-1002", isbn: "978-0201896831", return: "2026-03-04" }, ]; const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // Each line is one client need; the line count grows as needs grow. const ROUTES = { "/bookList": () => BOOKS, "/bookDetail": (s) => BOOKS.find((b) => b.isbn === s.get("isbn")) ?? null, "/addBook": () => ({ result: "added" }), "/removeBook": () => ({ result: "removed" }), "/loanList": () => LOANS, "/loanDetail": (s) => LOANS.find((l) => l.id === Number(s.get("id"))) ?? null, "/issueLoan": () => ({ result: "issued" }), "/returnLoan": () => ({ result: "return received" }), "/memberLoans": (s) => LOANS.filter((l) => l.member === s.get("member")), "/branchBooks": (s) => BOOKS.filter((b) => b.branch === s.get("branch")), "/openLoans": () => LOANS.filter((l) => l.return === null), }; const server = createServer((request, response) => { const url = new URL(request.url, "http://127.0.0.1"); if (url.pathname === "/routes") return respond(response, 200, { count: Object.keys(ROUTES).length }); const handler = ROUTES[url.pathname]; if (!handler) return respond(response, 404, { error: "path_not_found" }); respond(response, 200, handler(url.searchParams)); }); server.listen(8471, "127.0.0.1", () => console.log("action server 127.0.0.1:8471")); ``` ```js // resource-server.mjs — a design that meets the same needs with resource addresses import { createServer } from "node:http"; const BOOKS = [ { isbn: "978-0201896831", title: "The Art of Computer Programming", author: "Knuth", branch: "S-01" }, { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen", branch: "S-02" }, ]; const LOANS = [ { id: 1, member: "U-1001", isbn: "978-0262033848", return: null }, { id: 2, member: "U-1002", isbn: "978-0201896831", return: "2026-03-04" }, ]; const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // Three address patterns; filter criteria travel in the query part. const PATTERNS = [ [/^\/books$/, (e, s) => { const branch = s.get("branch"); return branch ? BOOKS.filter((b) => b.branch === branch) : BOOKS; }], [/^\/books\/([\w-]+)$/, (e) => BOOKS.find((b) => b.isbn === e[1]) ?? null], [/^\/loans$/, (e, s) => { let result = LOANS; if (s.get("member")) result = result.filter((l) => l.member === s.get("member")); if (s.get("status") === "open") result = result.filter((l) => l.return === null); return result; }], [/^\/loans\/(\d+)$/, (e) => LOANS.find((l) => l.id === Number(e[1])) ?? null], ]; const server = createServer((request, response) => { const url = new URL(request.url, "http://127.0.0.1"); if (url.pathname === "/patterns") return respond(response, 200, { count: PATTERNS.length }); for (const [pattern, handler] of PATTERNS) { const match = pattern.exec(url.pathname); if (match) return respond(response, 200, handler(match, url.searchParams)); } respond(response, 404, { error: "path_not_found" }); }); server.listen(8472, "127.0.0.1", () => console.log("resource server 127.0.0.1:8472")); ``` Both servers are long-running; the script below starts both, asks the same questions, and stops them at the end. ```bash # Starts both servers in turn, asks each the same four questions, stops them. node action-server.mjs & action=$! sleep 0.4 node resource-server.mjs & resource=$! sleep 0.4 ask() { printf '%-22s %s\n' "$1" "$(curl -sS "$2")"; } echo "--- action based ---" ask "S-01 branch books" "http://127.0.0.1:8471/branchBooks?branch=S-01" ask "U-1001 loans" "http://127.0.0.1:8471/memberLoans?member=U-1001" ask "open loans" "http://127.0.0.1:8471/openLoans" ask "route count" "http://127.0.0.1:8471/routes" echo "--- resource based ---" ask "S-01 branch books" "http://127.0.0.1:8472/books?branch=S-01" ask "U-1001 loans" "http://127.0.0.1:8472/loans?member=U-1001" ask "open loans" "http://127.0.0.1:8472/loans?status=open" ask "address pattern count" "http://127.0.0.1:8472/patterns" kill $action $resource ``` ``` action server 127.0.0.1:8471 resource server 127.0.0.1:8472 --- action based --- S-01 branch books [{"isbn":"978-0201896831","title":"The Art of Computer Programming","author":"Knuth","branch":"S-01"}] U-1001 loans [{"id":1,"member":"U-1001","isbn":"978-0262033848","return":null}] open loans [{"id":1,"member":"U-1001","isbn":"978-0262033848","return":null}] route count {"count":11} --- resource based --- S-01 branch books [{"isbn":"978-0201896831","title":"The Art of Computer Programming","author":"Knuth","branch":"S-01"}] U-1001 loans [{"id":1,"member":"U-1001","isbn":"978-0262033848","return":null}] open loans [{"id":1,"member":"U-1001","isbn":"978-0262033848","return":null}] address pattern count {"count":4} ``` The response bodies match line for line; the numbers do not. Eleven routes against four address patterns. The difference comes from the patterns naming the **resource, not the client need**. "Branch books" and "open loans" are not separate concepts; both are filtered views of an existing collection, and the filter criterion travels in the address's query part. The number itself is not the real problem. The real problem is that the action-based list is **not closed**. If one screen wants "overdue loans," the list grows to twelve; if another wants "overdue loans at a branch," it grows to thirteen. On the resource-based side, a new query criterion is added and the address set stays the same. Breaking this link between endpoint count and need count is resource modeling's measurable output. #### An Operation, Too, Is a Resource's State Change The most eye-catching line in the action-based list was `/returnLoan`. A return is a verb, so it does not look like a resource. But what gets returned is a loan record that already has an identity in the system; a return is that record's `return` field going from empty to filled. No new address is needed — an existing resource's state changes. The service below shows this on the library schema. The schema stays the same throughout the lesson: the branch, book, member, and loan tables. ```sql -- schema.sql — the core schema of the library loan service CREATE TABLE branch ( code TEXT PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE book ( isbn TEXT PRIMARY KEY, title TEXT NOT NULL, author TEXT NOT NULL, year INTEGER NOT NULL, branch TEXT NOT NULL REFERENCES branch(code) ); CREATE TABLE member ( code TEXT PRIMARY KEY, name TEXT NOT NULL, branch TEXT NOT NULL REFERENCES branch(code) ); CREATE TABLE loan ( id INTEGER PRIMARY KEY, member TEXT NOT NULL REFERENCES member(code), isbn TEXT NOT NULL REFERENCES book(isbn), issuedAt TEXT NOT NULL, return TEXT ); INSERT INTO branch VALUES ('S-01','Central'), ('S-02','Lakeside'); INSERT INTO book VALUES ('978-0201896831','The Art of Computer Programming','Knuth',1968,'S-01'), ('978-0262033848','Introduction to Algorithms','Cormen',1990,'S-02'), ('978-0131103627','The C Programming Language','Ritchie',1978,'S-01'); INSERT INTO member VALUES ('U-1001','Alina Drake','S-01'), ('U-1002','Ben Turner','S-02'); INSERT INTO loan (member, isbn, issuedAt, return) VALUES ('U-1001','978-0262033848','2026-03-01',NULL), ('U-1001','978-0131103627','2026-02-10','2026-02-24'), ('U-1002','978-0201896831','2026-03-04',NULL); ``` ```js // resource-service.mjs — a loan is a resource; a return is its state change import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const readBody = (request) => new Promise((resolve) => { let data = ""; request.on("data", (p) => (data += p)); request.on("end", () => resolve(data ? JSON.parse(data) : {})); }); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; const server = createServer(async (request, response) => { const path = new URL(request.url, "http://127.0.0.1").pathname; const loan = /^\/loans\/(\d+)$/.exec(path); const status = /^\/members\/([\w-]+)\/status$/.exec(path); // A single loan record: corresponds to one table row. if (loan && request.method === "GET") { const row = db.prepare("SELECT * FROM loan WHERE id = ?").get(Number(loan[1])); return row ? respond(response, 200, row) : respond(response, 404, { error: "loan_not_found" }); } // Return: not a new address, a state change on an existing resource. if (loan && request.method === "PATCH") { const body = await readBody(request); db.prepare("UPDATE loan SET return = ? WHERE id = ? AND return IS NULL") .run(body.return, Number(loan[1])); return respond(response, 200, db.prepare("SELECT * FROM loan WHERE id = ?").get(Number(loan[1]))); } // Member status: has no row in any table, computed from two tables. if (status && request.method === "GET") { const counts = db.prepare(` SELECT COUNT(*) AS total, SUM(return IS NULL) AS open FROM loan WHERE member = ?`).get(status[1]); const member = db.prepare("SELECT name, branch FROM member WHERE code = ?").get(status[1]); if (!member) return respond(response, 404, { error: "member_not_found" }); return respond(response, 200, { member: status[1], name: member.name, branch: member.branch, openLoan: counts.open, totalLoans: counts.total, canBorrowMore: counts.open < 2, }); } respond(response, 404, { error: "path_not_found" }); }); server.listen(8473, "127.0.0.1", () => console.log("resource service 127.0.0.1:8473")); ``` ```bash # Builds the schema, starts the server, reads a loan record, returns it, asks for status. rm -f library.db && sqlite3 library.db < schema.sql node resource-service.mjs & server=$! sleep 0.4 curl -sS http://127.0.0.1:8473/loans/1; echo curl -sS -X PATCH -H 'content-type: application/json' \ -d '{"return":"2026-03-12"}' http://127.0.0.1:8473/loans/1; echo curl -sS http://127.0.0.1:8473/members/U-1001/status; echo sqlite3 library.db "SELECT COUNT(*) || ' rows, open: ' || SUM(return IS NULL) FROM loan;" kill $server ``` ``` resource service 127.0.0.1:8473 {"id":1,"member":"U-1001","isbn":"978-0262033848","issuedAt":"2026-03-01","return":null} {"id":1,"member":"U-1001","isbn":"978-0262033848","issuedAt":"2026-03-01","return":"2026-03-12"} {"member":"U-1001","name":"Alina Drake","branch":"S-01","openLoan":0,"totalLoans":2,"canBorrowMore":true} 3 rows, open: 1 ``` The last line matters: the return did not produce a new row, it filled a field on an existing row. The table's record count is three, and it stays three. Opening a separate address for a return creates the illusion that something separate was created in the system, when in fact nothing was. The reverse is also true, and it should not be overstated. If a transition has its own rules, its own authorization, and its own record — a loan's transfer to another member, say — opening that transition as a separate address tied to the resource is defensible. The criterion is again identity: if the transition itself leaves a queryable record, it earns the right to be a resource; if it does not, it is a field change. #### A Table and a Resource Are Not the Same Thing The output's third line belongs to the `/members/U-1001/status` address, and this resource has no row in any table. Open loan count, total loan count, and eligibility for another loan are computed from two tables. It is still a resource: its identity is fixed, the client refers to it directly, and the result is a single representation. The clerk's screen calls this one address; it does not run three separate queries and apply the rule on the client. The correspondence is not one-directional either. A single table can feed more than one resource: the `loan` table feeds both the loan record collection and a member's loan history. The resource model is not the database schema projected outward; the schema looks for the best shape for storage, and the resource model looks for the concepts the client will want to refer to. That the two often overlap does not mean they are the same thing — and if this distinction is not kept, every reorganization of the database breaks the outside contract. #### The Cost of Choosing an Identity What to use as a resource's identity is a design decision. There are two options: the **natural key** defined in the Data Modeling and Relational Theory course — the ISBN for a book — and the **surrogate key** — a generated number for a loan record. A natural key is readable and meaningful; a surrogate key is meaningless but stable. The difference shows up when data gets corrected. ```bash # What happens to references when a resource's identity changes? rm -f identity.db sqlite3 identity.db <<'SQL' CREATE TABLE book (isbn TEXT PRIMARY KEY, title TEXT NOT NULL); CREATE TABLE loan (id INTEGER PRIMARY KEY, isbn TEXT NOT NULL REFERENCES book(isbn)); INSERT INTO book VALUES ('978-0262033848', 'Introduction to Algorithms'); INSERT INTO loan (isbn) VALUES ('978-0262033848'); -- The ISBN in the catalog record was entered wrong; it is being fixed. UPDATE book SET isbn = '978-0262046305' WHERE isbn = '978-0262033848'; SELECT 'book table : ' || isbn FROM book; SELECT 'loan table : ' || isbn FROM loan; SELECT 'orphan record : ' || COUNT(*) FROM loan l LEFT JOIN book b ON b.isbn = l.isbn WHERE b.isbn IS NULL; SQL ``` ``` book table : 978-0262046305 loan table : 978-0262033848 orphan record : 1 ``` A single correction produced one orphan record. In this engine, foreign key checking is off by default and is turned on with `PRAGMA foreign_keys = ON`; if it were on, the update would be rejected and the correction could not be made. Either way there is a cost: either the reference breaks or the correction is blocked. The counterpart on the API side is heavier, because references live outside the system. The address `/books/978-0262033848` may be written into a log, put in an email, embedded in another service's configuration. Once the identity changes, that address can no longer be found. This is why the criterion for a resource's identity is not "is it meaningful," but **"are we sure it will not change."** Natural keys that can change stay as a field on the resource and are offered as a search criterion; a surrogate key is used for the identity. #### Summary - A resource is something whose identity stays fixed over time and that the client refers to directly; not every domain concept is a resource — some are a resource's field or the service's rule. - The resource itself is abstract; the body sent to the client is its representation, and the same resource can have more than one representation. - The number of endpoints opened with action names grows together with client needs; when the same needs are met with four address patterns, this link breaks. - Transitions like a return do not create a new resource, they change an existing resource's state; transitions that leave their own record, however, earn the right to be a separate resource. - The resource model is not a reflection of the database schema: a resource may correspond to no table at all, and a table may feed more than one resource. - The criterion for identity is not meaningfulness but stability; natural keys that can change are carried as a field instead of an identity. #### Next Step Resources are identified, but how their addresses are written has not been decided yet. This lesson used the forms `/books`, `/loans/1`, and `/members/U-1001/status` without questioning them, yet each was a choice. Is a collection name singular or plural, is a loan record referenced through `/loans/1` or through `/members/U-1001/loans/1`, and if both paths are open, which one counts as correct? The next lesson takes on path layout by building a matcher table and measures where nested addresses produce ambiguity. ### URI Design Source: https://academia.sh/en/courses/api-design/uri-design The path layout of resource addresses: the division of labor between path and query part, naming collections and singular resources, the clash between static and parameter segments, the parent segment of nested paths, and the canonical-address decision. The previous lesson identified the resources: book, member, loan record, branch, and a member status with no table of its own. Its examples used addresses like `/books`, `/loans/1`, and `/members/U-1001/status` without questioning them. Yet each one was a decision, and every decision had an alternative. This lesson opens those decisions up: which part of an address carries identity and which sets the view, how a collection name is written, which path counts as valid when a loan record has two, and what a trailing slash changes. All of it produces measurable results once a matcher table runs. #### The Division of Labor Between Path and Query The How the Internet Works course introduced the parts of an address: scheme, host, path, query, and fragment. In API design there is a sharp division of labor among these parts. **The path carries a resource's identity.** `/books/978-0131103627` points to a specific book; that book is not found at any other path. **The query part sets the resource's view.** `/loans?status=open` is not a new resource, it is a filtered view of the loan collection. The same division holds for sorting, pagination, and field selection; all of these are the subject of later lessons and all of them live in the query part. The practical consequence of this division is this: **the path carries no verb.** Addresses like `/returnLoan`, `/searchBook`, `/removeMember` name the operation, not the resource. What says what to do is the method; the path only says what to do it to. The next lesson takes up in detail how methods take on that job. Collection names are written in the plural: `/books`, `/members`, `/loans`. The reasoning is not aesthetic. When `/book/978-0131103627` is used alongside `/books`, every address-building spot on the client has to remember separately whether it is singular or plural. One rule — the collection is plural, its member sits under it — removes that burden. Which rule is chosen matters less than that it never changes. #### The Clash Between a Static Segment and a Parameter Segment Address patterns form a table, and the table is scanned in order. When the order is wrong, the error is silent. The matcher below reports which pattern matched inside the response, so the decision becomes visible. ```js // path-matcher.mjs — matcher that tries address patterns in order; reports the matched pattern import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // Order matters: the list is scanned top to bottom, first match wins. const PATTERNS = [ ["/books/{isbn}", /^\/books\/([^/]+)$/, (e) => db.prepare("SELECT * FROM book WHERE isbn = ?").get(e[1]) ?? null], ["/books/search", /^\/books\/search$/, () => db.prepare("SELECT isbn, title FROM book WHERE author LIKE 'K%'").all()], ["/members/{code}/loans/{id}", /^\/members\/([^/]+)\/loans\/(\d+)$/, (e) => db.prepare("SELECT * FROM loan WHERE id = ?").get(Number(e[2])) ?? null], ]; const server = createServer((request, response) => { const path = new URL(request.url, "http://127.0.0.1").pathname; for (const [name, pattern, handler] of PATTERNS) { const match = pattern.exec(path); if (match) return respond(response, 200, { matched: name, result: handler(match) }); } respond(response, 404, { error: "path_not_found" }); }); server.listen(8474, "127.0.0.1", () => console.log("path matcher 127.0.0.1:8474")); ``` The schema is the `schema.sql` file set up in the previous lesson; the script rebuilds the database on every run. ```bash rm -f library.db && sqlite3 library.db < schema.sql node path-matcher.mjs & server=$! sleep 0.4 request() { printf '%-34s %s\n' "$1" "$(curl -sS "http://127.0.0.1:8474$1")"; } request "/books/978-0131103627" request "/books/search" request "/members/U-1002/loans/1" request "/books/978-0131103627/" request "/Books/978-0131103627" kill $server ``` ``` path matcher 127.0.0.1:8474 /books/978-0131103627 {"matched":"/books/{isbn}","result":{"isbn":"978-0131103627","title":"The C Programming Language","author":"Ritchie","year":1978,"branch":"S-01"}} /books/search {"matched":"/books/{isbn}","result":null} /members/U-1002/loans/1 {"matched":"/members/{code}/loans/{id}","result":{"id":1,"member":"U-1001","isbn":"978-0262033848","issuedAt":"2026-03-01","return":null}} /books/978-0131103627/ {"error":"path_not_found"} /Books/978-0131103627 {"error":"path_not_found"} ``` Five lines show four separate problems. The second line is the clash. `/books/search` never reached the search pattern; the parameter segment `{isbn}` caught it first and treated `search` as an ISBN, looking it up in the database. The result came back `null` — the client concludes not that the search endpoint is broken, but that the book does not exist. The **specificity** criterion defined in the Application Architecture course applies here too: static-segment patterns should be tried before parameter-segment ones. If you do not want to rely on order, there is an alternative — move the search criterion into the collection address's query part: `/books?author=Knuth`. In this form no segment is left to clash. The third line is more serious. `/members/U-1002/loans/1` returned member U-1001's loan record. The pattern captured the parent segment but the handler never used it, querying with only `{id}`. The nested path falls into being decoration: the address carries a claim of ownership, and the server never checks that claim. The fourth and fifth lines are a canonical-form problem. The trailing slash and the capital letter each caused the path to fail to match. Both came back as "not found," even though the resource is right there. #### Validating the Parent Segment and Canonical Form The matcher below applies all three fixes together: the static segment moves to the front, the parent segment joins the query, and a trailing slash redirects to the canonical address. ```js // path-matcher-2.mjs — matcher that tries specificity first and validates the parent segment import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // Static-segment patterns first; the parameter segment stays last. const PATTERNS = [ ["/books/search", /^\/books\/search$/, () => [200, db.prepare("SELECT isbn, title FROM book WHERE author LIKE 'K%'").all()]], ["/books/{isbn}", /^\/books\/([^/]+)$/, (e) => { const b = db.prepare("SELECT * FROM book WHERE isbn = ?").get(e[1]); return b ? [200, b] : [404, { error: "book_not_found" }]; }], ["/members/{code}/loans/{id}", /^\/members\/([^/]+)\/loans\/(\d+)$/, (e) => { // Saying the parent segment is decoration is wrong: no path if the record is not that member's. const l = db.prepare("SELECT * FROM loan WHERE id = ? AND member = ?").get(Number(e[2]), e[1]); return l ? [200, l] : [404, { error: "loan_not_found" }]; }], ]; const server = createServer((request, response) => { let path = new URL(request.url, "http://127.0.0.1").pathname; // Canonical form: a trailing slash collapses to a single address. if (path.length > 1 && path.endsWith("/")) { response.writeHead(308, { location: path.slice(0, -1) }); return response.end(); } for (const [name, pattern, handler] of PATTERNS) { const match = pattern.exec(path); if (match) { const [status, result] = handler(match); return respond(response, status, { matched: name, result }); } } respond(response, 404, { error: "path_not_found" }); }); server.listen(8475, "127.0.0.1", () => console.log("fixed matcher 127.0.0.1:8475")); ``` ```bash rm -f library.db && sqlite3 library.db < schema.sql node path-matcher-2.mjs & server=$! sleep 0.4 request() { printf '%-30s %3s %s\n' "$1" \ "$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:8475$1")" \ "$(curl -sS "http://127.0.0.1:8475$1")"; } request "/books/search" request "/members/U-1002/loans/1" request "/members/U-1001/loans/1" request "/books/978-0131103627/" # Same address, following the redirect: printf '%-30s %3s %s\n' "/books/978-0131103627/ (-L)" \ "$(curl -sSL -o /dev/null -w '%{http_code}' http://127.0.0.1:8475/books/978-0131103627/)" \ "$(curl -sSL http://127.0.0.1:8475/books/978-0131103627/)" kill $server ``` ``` fixed matcher 127.0.0.1:8475 /books/search 200 {"matched":"/books/search","result":[{"isbn":"978-0201896831","title":"The Art of Computer Programming"}]} /members/U-1002/loans/1 404 {"matched":"/members/{code}/loans/{id}","result":{"error":"loan_not_found"}} /members/U-1001/loans/1 200 {"matched":"/members/{code}/loans/{id}","result":{"id":1,"member":"U-1001","isbn":"978-0262033848","issuedAt":"2026-03-01","return":null}} /books/978-0131103627/ 308 /books/978-0131103627/ (-L) 200 {"matched":"/books/{isbn}","result":{"isbn":"978-0131103627","title":"The C Programming Language","author":"Ritchie","year":1978,"branch":"S-01"}} ``` Read the second and third lines together: the same loan record gives two different results with two different parent segments. The parent segment is now a condition, not a claim. This is a correctness decision, not a security one; authorization is a separate topic, covered in the Authentication and Authorization course. The gain here is that whatever the address claims, the server verifies it. The fourth line shows the redirect: the trailing-slash address turns into the canonical one with a 308, body empty. The fifth line makes the same request following the redirect and reaches the resource with a 200. This kind of redirect preserves the method and the body — a distinction that becomes decisive whenever a write request needs redirecting. The capital-letter problem was deliberately left unfixed. A path is case-sensitive and should stay that way; accepting `/Books` as well would mean producing an unbounded number of addresses for the same resource. The right decision is to document one spelling and reject the rest. #### Separators Inside an Identity The character that separates path segments is the slash; if an identity value contains that character, the address breaks. ```bash # What happens when a path separator appears inside an identity? node -e ' const code = "S-01/Central"; console.log("raw :", "/branches/" + code); console.log("encoded :", "/branches/" + encodeURIComponent(code)); const url = new URL("http://127.0.0.1/branches/" + encodeURIComponent(code)); console.log("decoded :", decodeURIComponent(url.pathname.split("/")[2])); ' ``` ``` raw : /branches/S-01/Central encoded : /branches/S-01%2FCentral decoded : S-01/Central ``` In the raw form, a single identity gets split into two segments and the matcher mistakes it for a sub-resource. In the encoded form, the segment keeps its integrity. This is the client's responsibility and it should be written into the contract. The more robust path is restricting the identity's format from the start: identities that contain no separator, no space, and no capital letter never raise this problem at all. #### The Canonical-Address Decision A resource can be reached by more than one path: `/loans/1` and `/members/U-1001/loans/1` return the same record. Both can be valid, but one must be chosen as the **canonical address**. The selection criterion is this: if the resource has an independent identity, the canonical address is the shorter one. A loan record has its own identity, so `/loans/1` is canonical; the nested form is a shortcut used while browsing a member's records. Settling on a canonical address does work in three places. When a response links to a resource, which form to write is settled. When a new record is created, the address to put in the `Location` header is settled. When the address is used as a cache key, the same resource is not stored under two different keys. Depth is bounded by this same criterion. A four-level path like `/branches/S-01/members/U-1001/loans/1` needs four separate checks, and all four have to run on every request. As a rule, one level is enough: the resource that owns the collection, then the collection, then its member. Needing more depth than that is a sign that the relationship is better written as a query criterion instead. #### Summary - The path carries a resource's identity, the query part sets its view; the path carries no verb, the method says what to do. - Collections are named in the plural and this decision does not change throughout the course; consistency matters more than which rule was chosen. - Static-segment patterns must be tried before parameter-segment ones; otherwise the address `/books/search` is mistaken for an ISBN and the error looks like "not found." - A nested path's parent segment is a condition, not decoration: if the record does not belong to that member, the response must be a 404. - Every resource gets one settled canonical address; variants like a trailing slash are redirected to it with a 308, and capitalization variants are never accepted at all. - If identity values contain the path separator, the address must be encoded; the more robust fix is restricting the identity's format from the start. #### Next Step Now that addresses are settled, the question of where the verb comes from remains. This lesson said the path layout carries no verb and moved on with "the method says what to do." But choosing a method is not choosing a name: every method has properties that the client and the components in between rely on. The assumption that a read does not change data, and that sending the same request twice produces a single effect, both come from here. The next lesson defines these properties and tests each one by counting rows in the database. ### Correct Use of HTTP Methods Source: https://academia.sh/en/courses/api-design/correct-use-of-http-methods The promises a method carries: a safe method not changing data, and an idempotent method producing a single effect when repeated; testing each property by counting rows in the database, PATCH's increment and assignment faces, and the identity choice between PUT and POST. The previous lesson set the path aside for a resource's identity and moved on with "the method says what to do." That sentence holds more than a division of labor. An HTTP method is not just a command sent to the server; it is a promise that the client, caches, and every component in between rely on when they look at the request. This lesson defines those promises and measures each one. The measurement is uniform: the same request is sent twice in a row, and the row count and field values in the database are counted at every step. Whether a method keeps its promise is settled by numbers, not by argument. #### A Method Is Not a Name, It Is a Promise The How the Internet Works course introduced two properties. A **safe** method leaves no observable change on the server; it reads, it does not write. An **idempotent** method makes no difference to the server's final state whether the same request is sent once or five times in a row. The two are independent: every safe method is idempotent, but not every idempotent method is safe. | Method | Safe | Idempotent | Carries a body | |---|---|---|---| | GET | yes | yes | no | | HEAD | yes | yes | no | | OPTIONS | yes | yes | no | | POST | no | no | yes | | PUT | no | yes | yes | | PATCH | no | depends on design | yes | | DELETE | no | yes | usually no | These properties are not the server's internal business; they are commitments made outward. A cache can store a GET response and answer the next request without going to the server — trusting that GET has no side effect. A client can resend a PUT request after a network error — trusting that repeating it is harmless. The request layer in the Application Architecture course encoded this trust as **retryability**. If the commitment is not kept, what breaks is not the server, but the parties that trust it. #### Safety: A Read Does Not Change Data The service below carries a deliberate flaw for the measurement: a loan return can be made both the right way and through a read-shaped address. ```js // method-server.mjs — a service set up to measure method properties import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const readBody = (request) => new Promise((resolve) => { let data = ""; request.on("data", (p) => (data += p)); request.on("end", () => resolve(data ? JSON.parse(data) : {})); }); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(data === null ? "" : JSON.stringify(data)); }; const server = createServer(async (request, response) => { const path = new URL(request.url, "http://127.0.0.1").pathname; const method = request.method; const book = /^\/books\/([^/]+)$/.exec(path); const loan = /^\/loans\/(\d+)$/.exec(path); const returnPath = /^\/loans\/(\d+)\/return$/.exec(path); if (method === "GET" && path === "/loans") return respond(response, 200, db.prepare("SELECT * FROM loan").all()); // WRONG: a read-shaped address changes data. if (method === "GET" && returnPath) { db.prepare("UPDATE loan SET return = '2026-03-12' WHERE id = ? AND return IS NULL") .run(Number(returnPath[1])); return respond(response, 200, { result: "return received" }); } if (method === "POST" && path === "/loans") { const g = await readBody(request); const s = db.prepare("INSERT INTO loan (member, isbn, issuedAt, return) VALUES (?,?,?,NULL)") .run(g.member, g.isbn, g.issuedAt); return respond(response, 201, { id: Number(s.lastInsertRowid) }); } if (method === "PUT" && book) { const g = await readBody(request); // Full replacement: the sent representation determines the whole record. const exists = db.prepare("SELECT 1 FROM book WHERE isbn = ?").get(book[1]); db.prepare(`INSERT INTO book (isbn, title, author, year, branch, copies) VALUES (?,?,?,?,?,?) ON CONFLICT(isbn) DO UPDATE SET title=excluded.title, author=excluded.author, year=excluded.year, branch=excluded.branch, copies=excluded.copies`) .run(book[1], g.title, g.author, g.year, g.branch, g.copies); return respond(response, exists ? 200 : 201, db.prepare("SELECT * FROM book WHERE isbn = ?").get(book[1])); } if (method === "PATCH" && book) { const g = await readBody(request); if ("addCopies" in g) // increment: depends on the previous value db.prepare("UPDATE book SET copies = copies + ? WHERE isbn = ?").run(g.addCopies, book[1]); if ("copies" in g) // assignment: independent of the previous value db.prepare("UPDATE book SET copies = ? WHERE isbn = ?").run(g.copies, book[1]); return respond(response, 200, db.prepare("SELECT isbn, copies FROM book WHERE isbn = ?").get(book[1])); } if (method === "DELETE" && loan) { db.prepare("DELETE FROM loan WHERE id = ?").run(Number(loan[1])); return respond(response, 204, null); } respond(response, 404, { error: "path_not_found" }); }); server.listen(8476, "127.0.0.1", () => console.log("method server 127.0.0.1:8476")); ``` Now suppose a client that follows the links on a page in order arrives at this service. Such a client only walks GET; it does not know what is behind a link. ```bash # What happens when a link crawler follows a read-shaped address? rm -f library.db && sqlite3 library.db < schema.sql sqlite3 library.db "ALTER TABLE book ADD COLUMN copies INTEGER NOT NULL DEFAULT 1;" node method-server.mjs & server=$! sleep 0.4 open() { sqlite3 library.db "SELECT COUNT(*) FROM loan WHERE return IS NULL;"; } printf 'open loans at start : %s\n' "$(open)" # A client that follows every link on a page: it only walks GET. for path in /loans /loans/1/return /loans/3/return; do printf 'GET %-22s -> %s\n' "$path" "$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:8476$path")" done printf 'open loans after GET: %s\n' "$(open)" kill $server ``` ``` method server 127.0.0.1:8476 open loans at start : 2 GET /loans -> 200 GET /loans/1/return -> 200 GET /loans/3/return -> 200 open loans after GET: 0 ``` Two open loan records closed without any user performing a return. The client broke no rule: for a link-following program, GET is a question it is free to ask. The server is the one that broke the rule. The scale of this flaw goes beyond a design mistake. A page-preview tool, a link-checking script, a browser that prefetches an address bar entry — all operate on the same assumption. In an example schema with two tables, two rows are lost; in a real catalog, the loss is as large as the number of links followed. In a correct design, an address like `/loans/{id}/return` is never opened; a return is made with a `PATCH /loans/{id}` request, as shown in the previous lesson. If the address is read-shaped, the method must be a read too, and the reverse holds as well. #### Idempotency: Sending the Same Request Twice Idempotency measurement is direct. Every method is called twice in a row; if the result of two calls differs from that of one, the method is not idempotent. ```bash # Every method is called twice in a row; the effect on the database is counted after each call. rm -f library.db && sqlite3 library.db < schema.sql sqlite3 library.db "ALTER TABLE book ADD COLUMN copies INTEGER NOT NULL DEFAULT 1;" node method-server.mjs & server=$! sleep 0.4 T=http://127.0.0.1:8476 say() { printf '%-26s loans=%s copies=%s\n' "$1" \ "$(sqlite3 library.db 'SELECT COUNT(*) FROM loan;')" \ "$(sqlite3 library.db "SELECT copies FROM book WHERE isbn='978-0131103627';")"; } say "start" curl -sS -o /dev/null $T/loans; curl -sS -o /dev/null $T/loans say "GET x2" for i in 1 2; do curl -sS -o /dev/null -X POST -H 'content-type: application/json' \ -d '{"member":"U-1002","isbn":"978-0131103627","issuedAt":"2026-03-10"}' $T/loans; done say "POST x2" for i in 1 2; do curl -sS -o /dev/null -X PUT -H 'content-type: application/json' \ -d '{"title":"The C Programming Language","author":"Ritchie","year":1988,"branch":"S-01","copies":4}' \ $T/books/978-0131103627; done say "PUT x2" for i in 1 2; do curl -sS -o /dev/null -X PATCH -H 'content-type: application/json' \ -d '{"addCopies":2}' $T/books/978-0131103627; done say "PATCH increment x2" for i in 1 2; do curl -sS -o /dev/null -X PATCH -H 'content-type: application/json' \ -d '{"copies":6}' $T/books/978-0131103627; done say "PATCH assignment x2" printf '%-26s %s %s\n' "DELETE x2 (status code)" \ "$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE $T/loans/2)" \ "$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE $T/loans/2)" say "DELETE x2" kill $server ``` ``` method server 127.0.0.1:8476 start loans=3 copies=1 GET x2 loans=3 copies=1 POST x2 loans=5 copies=1 PUT x2 loans=5 copies=4 PATCH increment x2 loans=5 copies=8 PATCH assignment x2 loans=5 copies=6 DELETE x2 (status code) 204 204 DELETE x2 loans=4 copies=6 ``` Read the lines one at a time. **GET x2** changed no number: the method is safe, so it is idempotent. **POST x2** raised the loan count from three to five: two requests, two records. This is POST's counterpart in this design — a new member is added to the collection and the server determines its identity. HTTP does not bind POST to one operation; it asks that the body be processed according to the resource's own semantics, and appending to a collection is the most common form of that processing. A POST resent because the network dropped produces a second loan record. This is not a design flaw, it is POST's nature; its fix is the ninth lesson's subject. **PUT x2** set the copy count to 4 and left it there. Because the sent body determines the whole record, the second request produces the same final state as the first. How many times it is sent does not change the result. **PATCH increment x2** raised the copy count from 4 to 8. The body carries an `addCopies` field, so the new value depends on the old one. This is not a flaw in PATCH; it is proof that PATCH's idempotency depends on the body. **PATCH assignment x2** used the same method to produce the value 6, and the second request did not change it. The design rule this yields is clear: a partial-update body built on assignment rather than increment makes the method idempotent. If an increment is genuinely needed — a counter, a balance, stock — idempotency cannot be achieved through the body; the key mechanism from the ninth lesson is needed. **DELETE x2** returned 204 to both requests, and the loan count dropped from five to four, not dropping further on the second delete. This is idempotency's definition: the final state is the same. Returning 404 to the second request is also a defensible choice and does not break idempotency, because idempotency is a property of server state, not of the response code. Which one to choose is the next lesson's subject. #### The Choice Between PUT and POST Both write; what draws the line is who determines the identity. **POST** is used when the server determines the identity. The client sends a body to the `/loans` collection, the server generates the new record's identity and reports it in the response. The client cannot know the target address before sending the request. **PUT** is used when the client knows the identity. A PUT to `/books/978-0131103627` creates the resource at that address if it does not exist, or replaces it with the sent representation if it does. The server above reflects this distinction in the status code too: 201 if the record did not exist, 200 if it did. PUT's second property is often overlooked: **fields left out are emptied.** PUT is a full replacement, not a partial update. If the `author` field is absent from the body, the record's author counts as cleared. If this behavior is not wanted, the right method is PATCH. A service that runs PUT like a partial update does not break the client's idempotency assumption, but it makes the contract ambiguous: the same body produces two different results across two services. #### Method Mismatch If an address exists but does not support the requested method — say, a DELETE arrives at the `/members/U-1001/status` resource — the response should not be "not found." The resource is right there; the method just cannot be applied. HTTP defines a separate status code for this and expects the response to report which methods are accepted. The server above does not make this distinction; it returns 404 to anything that does not match. The next lesson measures this gap and closes it. #### Summary - A method is a commitment the server makes outward: a safe method leaves no observable change, an idempotent one does not change the final state when repeated. - A read-shaped address changing data turns every link-following program into something destructive; in the measurement, three GET requests closed two open loan records. - POST is not idempotent and does not try to be; because the server generates the identity, every request produces a new record. - PUT is a full replacement and is idempotent; fields left out are emptied, so it is not suited to partial updates. - PATCH's idempotency depends on the body: an assignment-shaped body is idempotent, an increment-shaped one is not — in the measurement, the copy count rose from 4 to 8. - DELETE is idempotent; a second delete request returning a different status code does not break this property, because idempotency is a property of server state. #### Next Step This lesson largely assumed the response status codes: 201 on create, 204 on delete, 404 for everything unmatched. Yet each of these is a choice, and when the choice is made wrong, the client does not notice the error. Does a resource's absence return the same code as lacking permission to access it? How is a request rejected by a business rule told apart from one with a malformed body? The next lesson asks the same set of scenarios of two different servers, sets the codes they produce side by side, and shows what a wrong mapping produces on the client. ### Status Code Selection Source: https://academia.sh/en/courses/api-design/status-code-selection Answering the same set of scenarios with two different code mappings and measuring the difference on the client: distinguishing close codes, the meanings of the 2xx family, the silent success a wrong mapping produces, and retries spent for nothing. The previous lesson measured the promises a method carries, and ended on a gap: a server that returns 404 to anything unmatched collapses a missing resource and an inapplicable method into the same number. Throughout that lesson, the codes 201, 204, and 409 were used without justification too. This lesson builds that justification. The status code is the one part of a response the client can read **without interpretation**. The body's `error` field is written to be read by a human and differs across services; the status code, by contrast, is standard, and components like the request layer, the cache, and the retry mechanism make their decisions by looking at it. Choosing the wrong code means these components decide wrong — and that is a measurable difference. #### The Same Scenarios, Two Mappings The server below is a single file that runs in two modes. In `strict` mode, every outcome is reported with its own code. In `loose` mode, a common shortcut is applied: 200 with an `error` field in the body for anything not found, and 500 for every other failure. ```js // code-server.mjs — answers the same scenarios in two different status-code mapping modes // Usage: node code-server.mjs import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const [PORT, MODE] = [Number(process.argv[2]), process.argv[3]]; const MEMBER_LIMIT = 2; const db = new DatabaseSync("library.db"); const readBody = (request) => new Promise((resolve) => { let data = ""; request.on("data", (p) => (data += p)); request.on("end", () => resolve(data)); }); const send = (response, status, data, extraHeaders = {}) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8", ...extraHeaders }); response.end(data === null ? "" : JSON.stringify(data)); }; // Loose mode: reports every outcome as 200 or 500, leaving the distinction to the body. const report = (response, status, data, extraHeaders = {}) => { if (MODE === "strict") return send(response, status, data, extraHeaders); if (status >= 400 && status < 500) return send(response, status === 404 ? 200 : 500, data ?? {}); return send(response, 200, data ?? { result: "ok" }); }; const server = createServer(async (request, response) => { const path = new URL(request.url, "http://127.0.0.1").pathname; const book = /^\/books\/([^/]+)$/.exec(path); const loan = /^\/loans\/(\d+)$/.exec(path); if (request.method === "GET" && book) { const b = db.prepare("SELECT * FROM book WHERE isbn = ?").get(book[1]); return b ? report(response, 200, b) : report(response, 404, { error: "book_not_found" }); } if (request.method === "POST" && path === "/loans") { const type = request.headers["content-type"] ?? ""; if (!type.startsWith("application/json")) return report(response, 415, { error: "unsupported_type", expected: "application/json" }); const raw = await readBody(request); let g; try { g = JSON.parse(raw); } catch { return report(response, 400, { error: "body_not_parsed" }); } if (!db.prepare("SELECT 1 FROM book WHERE isbn = ?").get(g.isbn)) return report(response, 422, { error: "validation", field: "isbn", reason: "not_in_catalog" }); const open = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member = ? AND return IS NULL").get(g.member).n; if (open >= MEMBER_LIMIT) return report(response, 409, { error: "member_limit", limit: MEMBER_LIMIT, open }); const s = db.prepare("INSERT INTO loan (member, isbn, issuedAt, return) VALUES (?,?,?,NULL)") .run(g.member, g.isbn, g.issuedAt); const id = Number(s.lastInsertRowid); return report(response, 201, { id }, { location: `/loans/${id}` }); } if (request.method === "DELETE" && loan) { db.prepare("DELETE FROM loan WHERE id = ?").run(Number(loan[1])); return report(response, 204, null); } // Resource exists but the method cannot be applied: not "not found." if (/^\/members\/[^/]+\/status$/.test(path)) return report(response, 405, { error: "method_not_allowed" }, { allow: "GET" }); report(response, 404, { error: "path_not_found" }); }); server.listen(PORT, "127.0.0.1", () => console.log(`${MODE} mode 127.0.0.1:${PORT}`)); ``` On the other side stands a simplified version of the request layer from the Application Architecture course. It has one rule: the 5xx class counts as retryable, the 4xx class does not. ```js // client.mjs — client that decides based on the status code; counts retries // Usage: node client.mjs const BASE = process.argv[2]; const JSON_HEADER = { "content-type": "application/json" }; const SCENARIOS = [ ["read", "GET", "/books/978-0131103627", null, null], ["missing", "GET", "/books/000-0000000000", null, null], ["create", "POST", "/loans", JSON_HEADER, '{"member":"U-1002","isbn":"978-0131103627","issuedAt":"2026-03-10"}'], ["limit", "POST", "/loans", JSON_HEADER, '{"member":"U-1002","isbn":"978-0131103627","issuedAt":"2026-03-11"}'], ["isbn", "POST", "/loans", JSON_HEADER, '{"member":"U-1001","isbn":"000-0000000000","issuedAt":"2026-03-10"}'], ["malformed", "POST", "/loans", JSON_HEADER, "{broken"], ["type", "POST", "/loans", { "content-type": "text/plain" }, "hello"], ["delete", "DELETE", "/loans/2", null, null], ["method", "DELETE", "/members/U-1001/status", null, null], ]; let totalRequests = 0, silentSuccess = 0; for (const [name, method, path, headers, body] of SCENARIOS) { let attempt = 0, response; do { attempt++; totalRequests++; response = await fetch(BASE + path, { method, headers: headers ?? {}, body }); } while (response.status >= 500 && attempt < 4); const text = await response.text(); const ok = response.status < 400; const bodyHasError = text.includes('"error"'); if (ok && bodyHasError) silentSuccess++; const location = response.headers.get("location"); console.log( `${name.padEnd(9)} ${String(response.status).padEnd(4)} ` + `${(ok ? "success" : "error").padEnd(7)} attempt: ${attempt}` + (location ? ` location: ${location}` : ""), ); } console.log(`total requests: ${totalRequests}, success-counted errors: ${silentSuccess}`); ``` ```bash # The same client is run against both modes; the request counts are compared. setup() { rm -f library.db && sqlite3 library.db < schema.sql; } setup; node code-server.mjs 8477 strict & a=$!; sleep 0.4 echo "--- strict mode ---"; node client.mjs http://127.0.0.1:8477 kill $a setup; node code-server.mjs 8478 loose & b=$!; sleep 0.4 echo "--- loose mode ---"; node client.mjs http://127.0.0.1:8478 kill $b ``` ``` strict mode 127.0.0.1:8477 --- strict mode --- read 200 success attempt: 1 missing 404 error attempt: 1 create 201 success attempt: 1 location: /loans/4 limit 409 error attempt: 1 isbn 422 error attempt: 1 malformed 400 error attempt: 1 type 415 error attempt: 1 delete 204 success attempt: 1 method 405 error attempt: 1 total requests: 9, success-counted errors: 0 loose mode 127.0.0.1:8478 --- loose mode --- read 200 success attempt: 1 missing 200 success attempt: 1 create 200 success attempt: 1 limit 500 error attempt: 4 isbn 500 error attempt: 4 malformed 500 error attempt: 4 type 500 error attempt: 4 delete 200 success attempt: 1 method 500 error attempt: 4 total requests: 24, success-counted errors: 1 ``` #### Reading the Measurement Nine scenarios finished in nine requests under the strict mapping; the loose mapping needed twenty-four. The difference comes from four retried scenarios that share one trait: **none of them get fixed by repeating.** The member limit is full, the ISBN is not in the catalog, the body is malformed, the content type is wrong. Reporting these with 500 tells the client "I have a temporary problem, try again." The client keeps that promise and tries three more times — the request count triples, the result does not change. The second number is sneakier. Under the loose mapping, one failure **counted as success**: a client asking for a missing book got a 200 and, not reading the body's `error` field, assumed it had one. An empty book card, an error never logged, a bug never found — all from this one line. The third difference is not in the lines but in what is missing. In the strict mapping, the `create` scenario carries a `Location` header alongside its 201; in the loose mapping, that header is gone. It is the only place that reports the new resource's address. Without it, the client has to either read the identity from the body and build the address itself, or refetch the collection, just to reach the loan record it created. #### Distinguishing Close Codes The question that drives code selection is this: **which part of the request was not accepted?** - **400** — The body could not be read. The request is broken at the format level; the JSON did not parse, a required header is missing. The server cannot make sense of the request, so it cannot say anything about the content. - **415** — The body might be readable, but the content type is not supported. The problem is not the content, it is the format it was sent in. - **422** — The body parsed, the fields were read, but the values were not accepted. An ISBN not in the catalog is like this: the syntax is right, the sent value is not valid. The practical value of this distinction is that the client can trace the error back to the form — it can say which field was rejected. Field-level error reporting is a separate lesson's subject. - **409** — Both the body and the values are valid, but the system's current state does not allow the request to be fulfilled. A full member limit is exactly this. The difference between 422 and 409 is time: 422 is always invalid for the same body, 409 can become valid once the state changes. - **404** — The addressed resource does not exist. The problem is in the path, not the body. - **405** — The resource exists, the method cannot be applied. The response reports which methods are accepted through the `Allow` header; the server above sends `Allow: GET`. Confusing this with 404 pushes the client to look in the wrong place: it tries to fix the address, when it needs to fix the method instead. Identity and permission codes — 401 and 403 — were deliberately left off this list. The distinction between them rests on whether the identity is known, and it is covered in the Authentication and Authorization course. The only rule here is that neither collapses into 404. #### The 2xx Family Success, too, is not a single code. **200** is general success and carries a body. **201** says a new resource was created; when the new resource's address cannot be read off the request address — as with a POST to a collection — it must carry a `Location` header alongside it, because the client cannot build that address itself. **204** reports success with no body; it is used for deletions and whenever there is no new representation to return. **202** says the request was accepted but not yet completed; it needs a separate resource to track the work's outcome, and it falls under asynchronous processing. The selection criterion, again, is the client's decision. A client that gets 204 does not try to read the body; one that gets 200 does. Returning 200 with an empty body for a delete forces the client to parse a body that is not there, and produces a body error. #### Collapsing the Unknown into 500 The loose mode's real flaw is not using the 500 code, it is using it **for everything.** 500 is the code the server uses to report its own fault: an unexpected exception, an unreachable database, a broken assumption. Using it for the client's error breaks two things at once: the client does not fix a problem it could fix, since it was told the problem is not on its side; and on the server, real faults mix with client errors and become unmeasurable. The right behavior is to map every known failure to its own code and reserve 500 for the case that cannot be classified. The body of an unclassified case gives no detail either; writing the internal error message outward leaks the server's internal structure into the contract. #### Summary - The status code is the part of a response the client reads without interpretation; the cache, the request layer, and the retry mechanism decide by looking at it. - The same nine scenarios finished in nine requests under the strict mapping and produced twenty-four under the loose one; the difference comes from errors that will not be fixed being reported with 5xx and retried. - Under the loose mapping, one failure counted as success: a "not found" returned with 200 turned into an error the client never noticed. - Error code selection looks at which part of the request was not accepted: 400 for format, 415 for content type, 422 for field values, 409 for the system's current state, 404 for the address, 405 for the method. - A 201 response is incomplete without a `Location` header; 204 reports success with no body and does not force the client to parse one. - 500 is reserved only for a fault the server cannot classify; used for client errors, it misleads the client and makes real faults unmeasurable. #### Next Step The codes are settled, but what goes inside the response bodies is still open. This lesson wrote bodies haphazardly: in one place just an identity field, in another an error name and a limit value. What casing will field names use, how will dates be written, does a missing field count the same as an empty value, will a collection response be a bare array? The next lesson takes on body format and shows by running it where wrong choices silently corrupt data — in parsing large numbers and in date notation. ### Request and Response Bodies Source: https://academia.sh/en/courses/api-design/request-and-response-bodies The contract value of body format: one spelling for field naming, the silent loss of large integers and decimal amounts, writing dates with zone information, separating a missing field from an empty value, and the envelope decision for collection responses. The previous lesson settled the response's machine-read part — the status code. What is left is the body. In the measurement servers, bodies were written haphazardly: in one place just an identity field, elsewhere an error name and a limit value, in another the table's column names as they stood. Body format is as much a contract as the status code. If a field name changes, the client breaks; if how a number is written is chosen wrong, the client does not break — something worse happens, it silently reads the wrong value. This lesson lays out the body decisions, and two of them are shown by measurement: parsing numbers and dates, and separating a missing field from an empty value. #### The Body Is a Contract The first decision is field naming, and what matters is consistency, not the choice itself. If one response has `issuedAt`, another `issuedDate`, and a third `issued_date`, the client writes a separate parser for every endpoint. One casing is chosen for the whole course and it does not change. The second decision is that the body is not a copy of the database schema. In the previous lesson the loan record was returned directly as a `SELECT *` result; this leaks column names into the contract. When a column is renamed or added, the client is affected. The right behavior is to build the representation explicitly: which fields go out is written down, the rest stays inside. The third decision concerns enumerated values. A loan record's status is reported with fixed codes like `open`, `returned`, `overdue` — not display text like "Open" or "Returned." Text depends on language and wording, while the code is part of the contract and can be compared. What appears on screen is the client's decision. #### Numbers: A Silent Loss In the library catalog, every copy has an accession number that comes from another system. An overdue fine is also a decimal amount. What happens when both are written as numbers in JSON is worth examining. ```bash # Two silent losses in a JSON body: a large integer and an unformatted date. node -e ' const body = `{"accession": 9007199254740993, "fine": 0.1, "extra": 0.2}`; const c = JSON.parse(body); console.log("sent accession :", body.match(/\d{16}/)[0]); console.log("parsed accession :", String(c.accession)); console.log("equal :", String(c.accession) === body.match(/\d{16}/)[0]); console.log("fine + extra :", c.fine + c.extra); ' echo "---" TZ=Europe/Istanbul node -e ' for (const d of ["12/03/2026", "2026-03-12", "2026-03-12T00:00:00", "2026-03-12T00:00:00+03:00"]) console.log(d.padEnd(28), new Date(d).toISOString()); ' ``` ``` sent accession : 9007199254740993 parsed accession : 9007199254740992 equal : false fine + extra : 0.30000000000000004 --- 12/03/2026 2026-12-02T21:00:00.000Z 2026-03-12 2026-03-12T00:00:00.000Z 2026-03-12T00:00:00 2026-03-11T21:00:00.000Z 2026-03-12T00:00:00+03:00 2026-03-11T21:00:00.000Z ``` The first three lines show an identity breaking during parsing. JSON's number type's precision depends on the parser's floating-point representation; once the double-precision representation defined in the How Computers Work course exceeds its integer limit, a value rounds to the nearest representable one. `...993` is sent to the server, `...992` is read on the client. No error is raised; the loss stays invisible until a query made with that identity comes back empty. The fourth line is a second problem from the same root: the result of `0.1 + 0.2` is not exactly `0.3`. When monetary amounts travel as decimal numbers, addition produces accumulating errors. Both problems have the same fix: **numbers that will not be computed on travel as strings.** Identities are strings — no arithmetic is done on them anyway. Monetary amounts travel either as an integer in the smallest unit (cents) or as a string, and comparison is done in that form. An identity looking like a number does not make it one. #### Dates: No Such Thing as a Format-Free String The output's second section shows date fields, and all four lines hit a different trap. This block was run with `TZ=Europe/Istanbul`; the third line's result changes when the local time zone changes — and that is exactly the problem. `12/03/2026` was parsed not as March 12 but as December 2. Because the order of day and month cannot be read from the notation itself, the parser applied its own assumption. If a field like this exists in a contract, half its clients read eight months into the future. `2026-03-12` is a zone-free day and was taken as midnight UTC. `2026-03-12T00:00:00`, carrying no zone information, was counted as **local** time and shifted back a day once converted to UTC. The same string corresponds to two different instants on two different machines. Only the last line — the form that writes the zone offset explicitly — points to the same instant everywhere. The rule is this: **if an instant is being reported, the zone offset is written.** Fields like the moment a loan is issued travel in the full form. If only a calendar day is being reported — a due date, say — the field's name should say so and its notation should carry no time of day at all. Mixing the two kinds in the same field produces one-day shifts across daylight-saving transitions. #### Absent and Empty Are Not the Same Thing In a partial update, the client can say three things: set this field to this value, clear this field, do not touch this field. In a JSON body, the first two are conveyed with a value, the third with **the field being absent altogether.** A form that ignores this distinction can never carry out a "clear" request. ```js // field-server.mjs — the "absent" vs. "empty" distinction in a partial update import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const readBody = (request) => new Promise((resolve) => { let data = ""; request.on("data", (p) => (data += p)); request.on("end", () => resolve(data ? JSON.parse(data) : {})); }); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; const server = createServer(async (request, response) => { const path = new URL(request.url, "http://127.0.0.1").pathname; const loose = /^\/loose\/loans\/(\d+)$/.exec(path); const strict = /^\/strict\/loans\/(\d+)$/.exec(path); if (request.method !== "PATCH" || !(loose || strict)) return respond(response, 404, { error: "path_not_found" }); const id = Number((loose ?? strict)[1]); const g = await readBody(request); const existing = db.prepare("SELECT * FROM loan WHERE id = ?").get(id); if (!existing) return respond(response, 404, { error: "loan_not_found" }); if (loose) { // A form that does not separate an empty value from an absent field: null can never be written. const next = g.return ?? existing.return; db.prepare("UPDATE loan SET return = ? WHERE id = ?").run(next, id); } else { // Checks whether the field is present in the body; its value is evaluated separately. if ("return" in g) db.prepare("UPDATE loan SET return = ? WHERE id = ?").run(g.return, id); } respond(response, 200, db.prepare("SELECT id, return FROM loan WHERE id = ?").get(id)); }); server.listen(8479, "127.0.0.1", () => console.log("field server 127.0.0.1:8479")); ``` ```bash # The same three bodies are sent to both forms; the return field's final value is compared. rm -f library.db && sqlite3 library.db < schema.sql node field-server.mjs & server=$! sleep 0.4 send() { curl -sS -X PATCH -H 'content-type: application/json' -d "$2" "http://127.0.0.1:8479$1"; echo; } for mode in loose strict; do echo "--- $mode ---" send "/$mode/loans/2" '{"return":"2026-03-20"}' send "/$mode/loans/2" '{"member":"U-1001"}' send "/$mode/loans/2" '{"return":null}' done kill $server ``` ``` field server 127.0.0.1:8479 --- loose --- {"id":2,"return":"2026-03-20"} {"id":2,"return":"2026-03-20"} {"id":2,"return":"2026-03-20"} --- strict --- {"id":2,"return":"2026-03-20"} {"id":2,"return":"2026-03-20"} {"id":2,"return":null} ``` The first two lines of both modes match: a value is assigned, the field is kept if absent from the body. The third line diverges. In the loose form, the `return` field could not be cleared, because the `??` operator treats `null` as "not given" and restores the old value. There is no way to fix a loan record closed by mistake; the client sends the request, gets a 200, and nothing changes. In the strict form, the distinction is read from the body's structure: if the field is present, its value — `null` included — is applied; if not, it is left untouched. This behavior must be written into the contract, because the client needs to knowingly use the difference between "not sending the field" and "sending it empty." The same distinction holds on the response side. Sending `null` for a field with no value says something different from omitting it: the first says "this field exists but is empty," the second says "this field is not in this representation." One rule is chosen for the whole course; sometimes `null`, sometimes an absent field in a response forces the client to handle both possibilities. #### Collection Response: Bare Array or Envelope A collection response's shortest form is a bare array. Being short is its only advantage. No extra information can be attached to an array: how many records there are, where the next page starts, what criteria the query ran with. This information can travel in headers, but headers may not be cached alongside the body, and reading them takes separate work on the client. The **envelope** form wraps the body in an object: records sit in one field, pagination information in another. Its cost is one extra level of nesting, its gain is that the response can describe itself. The next two lessons — pagination and connection-based responses — will use that field, so the envelope form is chosen for the whole course. Singular resource responses, however, are not wrapped. The body returned from `/loans/2` is directly the loan record's representation; wrapping it in a field adds no information and just forces every client to descend one more level. #### Summary - One casing is chosen for field naming and does not change through the course; the body is an explicitly built representation, not a copy of the database schema. - Numbers that will not be computed on — identities — travel as strings; in the measurement, `9007199254740993` became `9007199254740992` when parsed, with no error raised. - Monetary amounts do not travel as decimal numbers; an integer in the smallest unit or a string is used instead. - Fields that report an instant write the zone offset explicitly; zone-free notations correspond to different instants on different machines, and notations with an ambiguous day/month order can drift by months. - In a partial update, whether a field is present in the body and whether its value is empty are separate pieces of information; a form that ignores this distinction makes clearing a field impossible. - Collection responses are wrapped in an envelope, because pagination and navigation information will travel inside the body; singular resource responses are not wrapped. #### Next Step The envelope's empty field is the next lesson's subject. While the loan collection was limited to three rows, sending every record in a single response was not a problem; at tens of thousands of records, that response overwhelms both server and client. Records need to be handed out in pieces, and there is more than one way to do it. The next lesson builds offset-based pagination, actually produces a reader seeing a record twice or not at all once a record is inserted in between, and then shows a method that does not drift under the same scenario. ### Pagination Source: https://academia.sh/en/courses/api-design/pagination Serving a collection in pieces: the repeats and skips offset-based pagination produces when a record is inserted in between, cursor-based pagination not drifting under the same scenario, and deep pagination's cost in query plan and duration. The previous lesson settled on wrapping collection responses in an envelope and left the envelope's `pagination` field empty. This lesson fills that field. The problem starts with numbers. With twenty records in the loan collection, sending all of them in a single response is not a problem; at two hundred thousand records, the same response strains the server's memory, the network, and the client's parser together. The collection has to be handed out in pieces. There are two ways to say where a piece starts, and the difference between them shows up once the collection changes between requests. #### Two Methods In **offset-based** pagination, the client says which page number it wants; the server skips that many rows from the start of the sorted result and returns the next ones. The `LIMIT` and `OFFSET` clauses from the SQL Fundamentals course map to this directly. The client writes `?page=3`, the server applies `OFFSET 10`. In **cursor-based** pagination, the client reports the position of the last record it saw; the server returns what comes after it. There is no page number; every response carries the starting point for the next request. The criterion is the sort key, so the method only works under **deterministic ordering**: if the sort key can produce ties, a unique field is added alongside it. The server below serves both: the same table, the same ordering, two addresses. ```js // page-server.mjs — serves the same collection with two pagination methods import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; const server = createServer((request, response) => { const url = new URL(request.url, "http://127.0.0.1"); const s = url.searchParams; const size = Math.min(Number(s.get("size") ?? 5), 50); if (url.pathname === "/offset/loans") { const page = Number(s.get("page") ?? 1); const rows = db.prepare( "SELECT id, issuedAt FROM loan ORDER BY id DESC LIMIT ? OFFSET ?" ).all(size, (page - 1) * size); return respond(response, 200, { data: rows.map((r) => r.id), pagination: { page, size }, }); } if (url.pathname === "/cursor/loans") { // No cursor: start from the top; a cursor asks for identities below it. const cursor = s.get("cursor"); const rows = cursor === null ? db.prepare("SELECT id FROM loan ORDER BY id DESC LIMIT ?").all(size) : db.prepare("SELECT id FROM loan WHERE id < ? ORDER BY id DESC LIMIT ?") .all(Number(cursor), size); const last = rows.at(-1); return respond(response, 200, { data: rows.map((r) => r.id), pagination: { nextCursor: last ? String(last.id) : null }, }); } respond(response, 404, { error: "path_not_found" }); }); server.listen(8480, "127.0.0.1", () => console.log("page server 127.0.0.1:8480")); ``` The page size is taken from the client but capped at an upper bound. This is an inseparable part of the contract: an unbounded `size` parameter leaves the door open to pulling the entire collection in a single request. #### Drift Measurement The measurement sets up this scenario: the client fetches the first page, the collection changes in the meantime, then it asks for the second page. Two kinds of change are tried separately — inserting a record in between and deleting one. Data is reseeded at the start of every round, so all four scenarios run from the same starting point. ```bash # A twenty-record collection; the first page is fetched, the collection changes, the second page is fetched. rm -f library.db && sqlite3 library.db < schema.sql seed() { sqlite3 library.db <<'SQL' DELETE FROM loan; WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n < 20) INSERT INTO loan (member, isbn, issuedAt, return) SELECT 'U-1001', '978-0262033848', date('2026-01-01', '+' || n || ' day'), NULL FROM s; SQL } seed node page-server.mjs & server=$! sleep 0.4 insert() { sqlite3 library.db "INSERT INTO loan (member, isbn, issuedAt, return) VALUES ('U-1002','978-0201896831','2026-02-01',NULL), ('U-1002','978-0201896831','2026-02-02',NULL);"; } delete() { sqlite3 library.db "DELETE FROM loan WHERE id IN (20, 19);"; } get() { curl -sS "http://127.0.0.1:8480$1"; echo; } for effect in insert delete; do seed; printf 'offset / %s page 1 : ' "$effect"; get "/offset/loans?page=1&size=5" $effect; printf 'offset / %s page 2 : ' "$effect"; get "/offset/loans?page=2&size=5" seed; printf 'cursor / %s page 1 : ' "$effect"; get "/cursor/loans?size=5" $effect; printf 'cursor / %s page 2 : ' "$effect"; get "/cursor/loans?cursor=16&size=5" done kill $server ``` ``` page server 127.0.0.1:8480 offset / insert page 1 : {"data":[20,19,18,17,16],"pagination":{"page":1,"size":5}} offset / insert page 2 : {"data":[17,16,15,14,13],"pagination":{"page":2,"size":5}} cursor / insert page 1 : {"data":[20,19,18,17,16],"pagination":{"nextCursor":"16"}} cursor / insert page 2 : {"data":[15,14,13,12,11],"pagination":{"nextCursor":"11"}} offset / delete page 1 : {"data":[20,19,18,17,16],"pagination":{"page":1,"size":5}} offset / delete page 2 : {"data":[13,12,11,10,9],"pagination":{"page":2,"size":5}} cursor / delete page 1 : {"data":[20,19,18,17,16],"pagination":{"nextCursor":"16"}} cursor / delete page 2 : {"data":[15,14,13,12,11],"pagination":{"nextCursor":"11"}} ``` Two of the four scenarios are flawed. **Offset produced repeats on insertion.** The first page ran from 20 down to 16. Once two new records entered in between, the start of the ordering shifted by two positions; the second page now starts at 17. The client saw records 17 and 16 **twice**. On a screen that accumulates pages into a list, the same loan record shows up as two rows. **Offset produced skips on deletion.** Once two records were deleted, the ordering pulled up by two positions and the second page started at 13. Records 15 and 14 never appeared on any page. This is more dangerous than a repeat: what is missing goes unnoticed. **Cursor gave the same result in both cases:** 15 down to 11. The reason is clear. A cursor carries not which position a page is in, but **where the reader left off.** Inserting a record at the front of the collection, or deleting one from the front, does not affect the criterion "identities smaller than 16." Newly inserted records are seen once they come to the front on a later round; that is a result of freshness, not drift. #### The Cost of Deep Pagination There is a second difference independent of drift, and it shows up as the page number grows. Skipped rows are not skipped for free: the engine still has to find them. ```bash # Deep pagination over two million rows: comparing plan and duration. rm -f deep.db sqlite3 deep.db <<'SQL' CREATE TABLE loan (id INTEGER PRIMARY KEY, member TEXT NOT NULL, issuedAt TEXT NOT NULL); WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n < 2000000) INSERT INTO loan (member, issuedAt) SELECT 'U-' || (1000 + n % 500), date('2020-01-01', '+' || (n % 2000) || ' day') FROM s; CREATE INDEX loan_issued_at ON loan (issuedAt DESC, id DESC); SQL sqlite3 deep.db <<'SQL' .echo on EXPLAIN QUERY PLAN SELECT id FROM loan ORDER BY issuedAt DESC, id DESC LIMIT 20 OFFSET 1999980; EXPLAIN QUERY PLAN SELECT id FROM loan WHERE (issuedAt, id) < ('2020-01-02', 100) ORDER BY issuedAt DESC, id DESC LIMIT 20; SQL echo "--- timings ---" sqlite3 deep.db <<'SQL' .timer on SELECT count(*) FROM (SELECT id FROM loan ORDER BY issuedAt DESC, id DESC LIMIT 20 OFFSET 1999980); SELECT count(*) FROM (SELECT id FROM loan WHERE (issuedAt, id) < ('2020-01-02', 100) ORDER BY issuedAt DESC, id DESC LIMIT 20); SQL ``` ``` EXPLAIN QUERY PLAN SELECT id FROM loan ORDER BY issuedAt DESC, id DESC LIMIT 20 OFFSET 1999980; QUERY PLAN `--SCAN loan USING COVERING INDEX loan_issued_at EXPLAIN QUERY PLAN SELECT id FROM loan WHERE (issuedAt, id) < ('2020-01-02', 100) ORDER BY issuedAt DESC, id DESC LIMIT 20; QUERY PLAN `--SEARCH loan USING COVERING INDEX loan_issued_at (issuedAt { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // Allowlists: a name coming from the client is translated to a column name here. const FILTERS = { author: "author = ?", yearMin: "year >= ?", branch: "branch = ?", search: "title LIKE ?", }; const SORTS = { year: "year", title: "title", author: "author" }; const server = createServer((request, response) => { const url = new URL(request.url, "http://127.0.0.1"); if (url.pathname !== "/books") return respond(response, 404, { error: "path_not_found" }); const s = url.searchParams; const size = Math.min(Number(s.get("size") ?? 3), 50); // Filters: only names in the allowlist turn into a condition. const conditions = [], values = []; for (const [name, clause] of Object.entries(FILTERS)) { const value = s.get(name); if (value === null) continue; conditions.push(clause); values.push(name === "search" ? `%${value}%` : value); // value never enters the query text } // Sort: rejected if not in the allowlist, never falls back to a default. const sortName = s.get("sort") ?? "year"; const column = SORTS[sortName]; if (!column) return respond(response, 422, { error: "validation", field: "sort", allowed: Object.keys(SORTS), }); // Cursor: loose mode carries only the sort column, strict mode also carries the tiebreaker. const cursor = s.get("cursor"); const strict = s.get("mode") !== "loose"; if (cursor !== null) { if (strict) { const [d, k] = cursor.split("|"); conditions.push(`(${column}, isbn) > (?, ?)`); values.push(d, k); } else { conditions.push(`${column} > ?`); values.push(cursor); } } const where = conditions.length ? `WHERE ${conditions.join(" AND ")}` : ""; const order = strict ? `${column}, isbn` : column; const rows = db.prepare( `SELECT isbn, title, author, year FROM book ${where} ORDER BY ${order} LIMIT ?` ).all(...values, size); const last = rows.at(-1); respond(response, 200, { data: rows.map((r) => `${r.year} ${r.isbn}`), pagination: { nextCursor: last ? (strict ? `${last[column]}|${last.isbn}` : String(last[column])) : null, }, }); }); server.listen(8481, "127.0.0.1", () => console.log("filter server 127.0.0.1:8481")); ``` Notice how the query text gets built. Every piece entering the `conditions` array is a fixed string the server itself wrote; no character coming from the client reaches it. Everything the client sends lives in the `values` array and is handed to the query as a bound variable. The catalog is expanded so sort ties become visible: ```sql -- catalog.sql — an expanded catalog containing sort ties DELETE FROM book; INSERT INTO book (isbn, title, author, year, branch) VALUES ('K-01','Computer Networks','Tanenbaum',1978,'S-01'), ('K-02','Data Structures and Algorithms','Aho',1983,'S-01'), ('K-03','Compilers','Aho',1983,'S-02'), ('K-04','Operating Systems','Tanenbaum',1983,'S-02'), ('K-05','Shell Scripting',"O'Reilly",1986,'S-01'), ('K-06','Graph Algorithms','Sedgewick',1990,'S-01'), ('K-07','Numerical Methods','Press',1990,'S-02'), ('K-08','Network Programming','Stevens',1990,'S-01'), ('K-09','Formal Languages','Hopcroft',1995,'S-02'); ``` The client that walks the pages is a separate file too; it does not build the cursor itself, it takes it from the response. ```js // walk.mjs — client that walks every page by following the cursor // Usage: node walk.mjs const MODE = process.argv[2]; let cursor = null, page = 0; const collected = []; while (page < 10) { const url = new URL("http://127.0.0.1:8481/books"); url.searchParams.set("mode", MODE); url.searchParams.set("sort", "year"); if (cursor !== null) url.searchParams.set("cursor", cursor); const data = await (await fetch(url)).json(); if (data.data.length === 0) break; page++; collected.push(...data.data); console.log(`${MODE.padEnd(7)} page ${page}: ${data.data.join(" ")}`); cursor = data.pagination.nextCursor; } console.log(`${MODE.padEnd(7)} total : ${collected.length} rows\n`); ``` ```bash # The effect of the allowlist, bound variables, and sort ties on pagination. rm -f library.db && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql node filter-server.mjs & server=$! sleep 0.4 get() { curl -sS -G "http://127.0.0.1:8481/books" "$@"; echo; } echo "--- filter and allowlist ---" printf 'author=Aho : '; get --data-urlencode "author=Aho" printf "author=O'Reilly: "; get --data-urlencode "author=O'Reilly" printf 'search=Network : '; get --data-urlencode "search=Network" printf 'sort=branch : '; get --data-urlencode "sort=branch" echo "--- pagination on a column with sort ties (catalog has 9 books) ---" node walk.mjs loose node walk.mjs strict kill $server ``` ``` filter server 127.0.0.1:8481 --- filter and allowlist --- author=Aho : {"data":["1983 K-02","1983 K-03"],"pagination":{"nextCursor":"1983|K-03"}} author=O'Reilly: {"data":["1986 K-05"],"pagination":{"nextCursor":"1986|K-05"}} search=Network : {"data":["1978 K-01","1990 K-08"],"pagination":{"nextCursor":"1990|K-08"}} sort=branch : {"error":"validation","field":"sort","allowed":["year","title","author"]} --- pagination on a column with sort ties (catalog has 9 books) --- loose page 1: 1978 K-01 1983 K-02 1983 K-03 loose page 2: 1986 K-05 1990 K-06 1990 K-07 loose page 3: 1995 K-09 loose total : 7 rows strict page 1: 1978 K-01 1983 K-02 1983 K-03 strict page 2: 1983 K-04 1986 K-05 1990 K-06 strict page 3: 1990 K-07 1990 K-08 1995 K-09 strict total : 9 rows ``` #### Keeping the Value Out of the Query Text The second line is the most concrete proof of this distinction. The value `O'Reilly` contains an apostrophe, and that is exactly the character that ends an SQL string literal. If the value had been written into the query text, the query would split at that point and the rest would be read as SQL. Carried as a bound variable, none of that happens: the value is only a value, it finds the record and returns it. This is not a problem of "escaping" apostrophe-carrying names. An escape character looks like a solution too, but it needs a separate rule for every data type and dialect, and gets forgotten somewhere. With a bound variable there is nothing to forget; the query text stays fixed regardless of the client. The `search=Network` filter in the third line carries the same principle one step further. Wildcard characters are added on the server — the client-supplied value `Network` is turned into `%Network%` — so the client cannot write a pattern, it can only supply the text to search for. #### Rejecting What Is Outside the Allowlist The fourth line returned a 422 to the `sort=branch` request and reported the allowed names in the response. Both decisions here are deliberate. First, the request was **rejected**; it did not silently fall back to a default sort. Falling back to a default means the client never learns it sent a wrong name; the screen shows the wrong order and no one goes looking for why. The criterion from the Status Code Selection lesson applies here too: an unaccepted field value is reported with 422. Second, the response lists the allowed names. This is a self-describing part of the contract; it steers the client toward fixing the request without going to the documentation. The allowlist also keeps sorting's cost under control. If sorting were allowed by any column, a request to sort by a column with no index would force a full table read. Only columns that have been decided to be supported — and indexed if needed — go on the list. #### A Sort Tie Loses Records The output's last section loops back to the Pagination lesson. There are nine books in the catalog; loose mode showed seven of them. K-04 and K-08 never appeared on any page. The reason is measurably plain. In loose mode, the cursor carries only the year. The first page ends at 1983 and the cursor becomes `1983`; the second page arrives with the condition `year > 1983`, and the K-04 record sharing that same year falls outside it. The same thing repeats at 1990, and K-08 is lost. As long as the sort key is not unique, the criterion "after this value" always leaves out part of the ties. In strict mode, the cursor carries two fields: the sort column and a unique identity. Both the condition and the sort now work over two fields. Because a definite order is established among records sharing the same year, "after this point" is well defined, and all nine of the nine records appear. Two rules follow from this. Whatever the sort criterion, a unique field is appended to the end of it — this is what **deterministic ordering** requires. And the cursor carries the entire sort criterion: a single-field cursor can only describe a single-field sort. #### Summary - A query parameter carries two kinds of information: name and value. A name is translated to a column through an allowlist, a value travels as a bound variable; the two do not enter SQL by the same path. - Only fixed pieces the server wrote enter the query text; no character from the client reaches the query's structure. An apostrophe-carrying author name is found without incident because of this. - Search patterns are built on the server; the client supplies the text to search for, not the wildcard. - A sort request outside the allowlist is rejected with 422 and the allowed names are reported in the response; silently falling back to a default makes the error invisible. - The allowlist also bounds sorting's cost: only columns decided to be supported go on the list. - If the sort key is not unique, cursor-based pagination skips part of the ties; in the measurement, seven of nine records appeared. Appending a unique field to the sort and the cursor removes this loss. #### Next Step Which records come back, and in what order, is settled. What is left is **how much** of each record comes back. While the catalog list screen shows only a book's title and author, the server sends every field on every row; the loan history screen, meanwhile, fires off separate requests for information it cannot find in a single response. Both problems come from the same place: the server alone decides the representation's level of detail. The next lesson builds the mechanism that lets the client choose fields and measures the difference in transferred bytes. ### Partial Response and Field Selection Source: https://academia.sh/en/courses/api-design/partial-response-and-field-selection Letting the client take part in a representation's level of detail: measuring over- and under-fetching in request count and bytes, building field selection and expansion through an allowlist, and the cost this flexibility adds to the contract. The previous two lessons settled which records a collection returns, in what order, and how many. How much **of each record** comes back stayed open. The library interface has two screens. The loan list screen shows only a book's title, a member's name, and the issue date on every row. The loan detail screen wants every field of the record. Both draw from the same collection. If the server returns a single representation, that representation is either too narrow for the list or too wide for the detail. Both cases carry a measurable cost. #### Over-Fetching and Under-Fetching **Over-fetching** is sending fields the client does not use. If the loan list screen shows three fields while the server sends ten, the difference travels the network, gets parsed on the client, and is thrown away. **Under-fetching** is a single request failing to feed the screen. If a loan record carries only an ISBN and a member code, getting the book's title and the member's name needs two more requests per row. A forty-row list means eighty extra requests. The two look like opposites but come from the same root: the server alone decides the representation's level of detail. The fix is to hand part of that decision to the client — but as flexibility whose boundaries the server draws. ```js // field-selection-server.mjs — loan collection supporting field selection and expansion import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // Allowlist: a field the client names is translated to a column expression here. const BASE = { id: "o.id", member: "o.member", isbn: "o.isbn", issuedAt: "o.issuedAt", return: "o.return" }; const EXPANDED = { bookTitle: "k.title", bookAuthor: "k.author", memberName: "u.name", memberBranch: "u.branch" }; const server = createServer((request, response) => { const url = new URL(request.url, "http://127.0.0.1"); const s = url.searchParams; if (/^\/books\/[^/]+$/.test(url.pathname)) { const isbn = url.pathname.split("/")[2]; const b = db.prepare("SELECT * FROM book WHERE isbn = ?").get(isbn); return b ? respond(response, 200, b) : respond(response, 404, { error: "book_not_found" }); } if (/^\/members\/[^/]+$/.test(url.pathname)) { const code = url.pathname.split("/")[2]; const m = db.prepare("SELECT * FROM member WHERE code = ?").get(code); return m ? respond(response, 200, m) : respond(response, 404, { error: "member_not_found" }); } if (url.pathname !== "/loans") return respond(response, 404, { error: "path_not_found" }); // Expansion: fields of related resources are folded into the response. const expand = (s.get("expand") ?? "").split(",").filter(Boolean); const offered = { ...BASE }; if (expand.includes("book")) Object.assign(offered, { bookTitle: EXPANDED.bookTitle, bookAuthor: EXPANDED.bookAuthor }); if (expand.includes("member")) Object.assign(offered, { memberName: EXPANDED.memberName, memberBranch: EXPANDED.memberBranch }); // Field selection: requested names are intersected with what's offered, unknown names are rejected. const requested = (s.get("fields") ?? "").split(",").filter(Boolean); const unknown = requested.filter((a) => !(a in offered)); if (unknown.length) return respond(response, 422, { error: "validation", field: "fields", unknown, offered: Object.keys(offered) }); const selected = requested.length ? requested : Object.keys(offered); const selection = selected.map((a) => `${offered[a]} AS ${a}`).join(", "); const rows = db.prepare( `SELECT ${selection} FROM loan o LEFT JOIN book k ON k.isbn = o.isbn LEFT JOIN member u ON u.code = o.member ORDER BY o.id LIMIT ?` ).all(Number(s.get("size") ?? 40)); respond(response, 200, { data: rows, pagination: { size: rows.length } }); }); server.listen(8482, "127.0.0.1", () => console.log("field selection server 127.0.0.1:8482")); ``` Column names come from an allowlist again; the previous lesson's distinction applies here too. The client sends the name `bookTitle`, the server translates it to the expression `k.title`. A name not on the list can never reach the query text. #### Measurement Three different strategies feed the same screen. The measurement counts the request count and the transferred body bytes. ```js // measurement.mjs — measures the request count and transferred bytes of three strategies const T = "http://127.0.0.1:8482"; let requests = 0, bytes = 0; const call = async (path) => { requests++; const text = await (await fetch(T + path)).text(); bytes += Buffer.byteLength(text); return JSON.parse(text); }; const report = (name) => { console.log(`${name.padEnd(34)} requests: ${String(requests).padStart(3)} bytes: ${bytes}`); requests = 0; bytes = 0; }; // 1) Separately: the loan list, then the book and member for every record. const list = await call("/loans"); for (const l of list.data) { await call(`/books/${l.isbn}`); await call(`/members/${l.member}`); } report("separate requests (under-fetching)"); // 2) Expansion: a single request, but every field comes along. await call("/loans?expand=book,member"); report("expansion (over-fetching)"); // 3) Expansion + field selection: a single request, only the fields the screen uses. await call("/loans?expand=book,member&fields=id,issuedAt,bookTitle,memberName"); report("expansion + field selection"); ``` ```bash # Forty loan records; the same screen is fed by three different strategies. rm -f library.db && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql sqlite3 library.db <<'SQL' DELETE FROM loan; WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n < 40) INSERT INTO loan (member, isbn, issuedAt, return) SELECT CASE WHEN n % 2 = 0 THEN 'U-1001' ELSE 'U-1002' END, 'K-0' || (1 + n % 9), date('2026-01-01', '+' || n || ' day'), NULL FROM s; SQL node field-selection-server.mjs & server=$! sleep 0.4 node measurement.mjs echo "--- sample response row ---" curl -sS "http://127.0.0.1:8482/loans?expand=book,member&fields=id,issuedAt,bookTitle,memberName&size=2"; echo curl -sS "http://127.0.0.1:8482/loans?fields=id,bookTitle&size=2"; echo kill $server ``` ``` field selection server 127.0.0.1:8482 separate requests (under-fetching) requests: 81 bytes: 8968 expansion (over-fetching) requests: 1 bytes: 7368 expansion + field selection requests: 1 bytes: 3741 --- sample response row --- {"data":[{"id":1,"issuedAt":"2026-01-02","bookTitle":"Data Structures and Algorithms","memberName":"Ben Turner"},{"id":2,"issuedAt":"2026-01-03","bookTitle":"Compilers","memberName":"Alina Drake"}],"pagination":{"size":2}} {"error":"validation","field":"fields","unknown":["bookTitle"],"offered":["id","member","isbn","issuedAt","return"]} ``` The first line shows under-fetching's cost: eighty-one requests for thirty-nine records. The byte difference is not the real issue here; the request count is. Every request means a network round trip, and when trips wait on each other, latency multiplies directly. There is also repetition, since the same book and the same member get requested over and over. The second line drops to a single request but carries 7368 bytes. Five fields the screen does not use — ISBN, member code, return date, book author, member branch — appear on every row. The third line solves both together: a single request, 3741 bytes. The same screen is fed with roughly half the data. The ratio depends on the data; here the savings come from how large a share field names and unused values take up in the total body. The last line is the contract defending itself. The request `fields=id,bookTitle` arrived without `expand=book`; `bookTitle` is not among the fields offered in that request. The server rejects the request with 422 and reports both the unknown name and the fields offered in that request. Silently dropping the field would have led the client to expect a field that was never there and render the screen with an `undefined` value. #### The Contract's Cost for This Flexibility Field selection is not free, and its cost is paid on the contract surface. First, **cache keys multiply**. The address is a cache key, and because the field list enters the address, `fields=id,title` and `fields=title,id` produce two separate keys. To avoid storing the same data twice, a canonical form for the field list must be settled: names are sorted and duplicates dropped. This canonical-form decision is the same kind as the one in the URI Design lesson. Second, **the combinations to test multiply**. At an endpoint with five base fields and four expansion fields, the number of field sets a client could request is too large to test by hand. This is why the allowlist stays narrow: not every column is offered, only the fields screens actually want. Third, **expansion depth must be bounded**. This server offers one level of expansion: from loan to book and member. If two levels were allowed — from book to branch, from branch to the responsible member — a single request would turn into a query tree branching on the server side. The single request the client sees hides the cost the server pays. #### Field Selection or a Named View The alternative to field selection is the server defining a handful of ready-made representations: `?view=list` and `?view=detail`, say. Flexibility drops under this approach, but three things are gained. The number of cache keys falls to two. No combinations are left to test. The server can write one query per view and optimize it. The selection criterion is the number of clients. If a single interface uses the server, a named view is enough; the view's definition changes as the screen does. If there are many independent clients — a mobile app, an admin panel, another service — defining a separate view for each bloats the contract; field selection needs less upkeep in that case. The two approaches are also used together: named views for common cases, field selection for the rest. This is the resource-oriented style's counterpart to a problem query-based styles solve from the start; the trade-off discussed in the API Styles topic turns into concrete numbers here. #### Summary - Over-fetching is carrying unused fields; under-fetching is a single request failing to feed the screen. Both come from the server alone deciding the representation's level of detail. - In the measurement, the same screen consumed 81 requests and 8968 bytes with separate requests, 1 request and 7368 bytes with expansion, and 1 request and 3741 bytes with expansion and field selection. - Field names and expansion names pass through an allowlist; only column expressions the server wrote enter the query text. - A request for an undefined field is not silently dropped, it is rejected with 422, and the fields offered in that request are reported in the response. - Field selection multiplies cache keys and increases the combinations to test; the field list's canonical form must be settled and the allowlist kept narrow. - In services with few clients, named views need less upkeep than field selection; the two approaches can also be used together. #### Next Step The four lessons up to here fixed the read side: which records, in what order, how much, in how many requests. On the write side, a gap remains from the third lesson. POST is not idempotent; when the network drops or the response gets lost on the way back, the client retries the request and a second loan record is born. The client cannot know whether the request arrived, and the server cannot see that two requests came from the same intent. The next lesson closes this gap with a key the client generates, and shows by counting that two requests sent with the same key produce a single effect. ### Idempotency Keys Source: https://academia.sh/en/courses/api-design/idempotency-keys The idempotency key that prevents a repeated effect on retry: matching the key to its scope and a body digest, a second request with the same key replaying the first response, rejecting a conflict from a different body, and the key store's lifetime. The third lesson measured that POST is not idempotent: sending the same request twice produced two loan records. Back then, that was noted as POST's nature and left alone. Once the eighth lesson closed the read side, this gap remained. The gap comes from a real situation. The client sends a request to create a loan record, and the network drops. The network-error class of the error contract from the Application Architecture course described exactly this: **the request may have reached the server, or it may not have.** If the response was lost on the way back, the record was created but the client does not know it. If the client retries, there is a risk of a second record; if it does not, the member may end up without the book. Neither option is acceptable. This lesson builds a third way: attaching a client-generated identity to the request. #### What the Key Does An **idempotency key** is the unique name a client gives to an operation's intent. It travels in the `Idempotency-Key` header and has three rules. The key is generated **per intent, not per request.** A key is generated when the user presses the "issue loan" button; three attempts made because the network dropped all carry that same key. If the user wants to check out a second book, a new key is generated. The key **is not enough on its own**. The server must also store which operation the key is tied to: which address, with which body. Otherwise, if the client mistakenly reuses the same key on a different request, that request gets silently swallowed. The key **is stored together with the response**. When the second request arrives, the operation is not redone; the response generated the first time is returned as is. The client learns the result even if it never saw the first response. ```js // key-server.mjs — loan creation guarded by an idempotency key import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; import { createHash } from "node:crypto"; const db = new DatabaseSync("library.db"); db.exec(`CREATE TABLE IF NOT EXISTS idempotency ( key TEXT PRIMARY KEY, path TEXT NOT NULL, digest TEXT NOT NULL, status INTEGER, response TEXT, createdAt TEXT NOT NULL )`); const readBody = (request) => new Promise((resolve) => { let data = ""; request.on("data", (p) => (data += p)); request.on("end", () => resolve(data)); }); const send = (response, status, data, extra = {}) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8", ...extra }); response.end(JSON.stringify(data)); }; const server = createServer(async (request, response) => { const path = new URL(request.url, "http://127.0.0.1").pathname; if (request.method !== "POST" || path !== "/loans") return send(response, 404, { error: "path_not_found" }); const raw = await readBody(request); const key = request.headers["idempotency-key"]; const digest = createHash("sha256").update(raw).digest("hex").slice(0, 16); const body = JSON.parse(raw); // No key: every call produces a new record. if (!key) return send(response, ...create(body)); const record = db.prepare("SELECT * FROM idempotency WHERE key = ?").get(key); if (record) { // Same key, different body: the client reused the key. if (record.path !== path || record.digest !== digest) return send(response, 409, { error: "key_conflict", key }); return send(response, record.status, JSON.parse(record.response), { "idempotency-replayed": "true" }); } // Reserving the key and doing the work happen in one block, with no wait in between. db.prepare("INSERT INTO idempotency (key, path, digest, createdAt) VALUES (?,?,?,datetime('now'))") .run(key, path, digest); const [status, data] = create(body); db.prepare("UPDATE idempotency SET status = ?, response = ? WHERE key = ?") .run(status, JSON.stringify(data), key); send(response, status, data); }); function create(body) { if (!db.prepare("SELECT 1 FROM book WHERE isbn = ?").get(body.isbn)) return [422, { error: "validation", field: "isbn" }]; const s = db.prepare("INSERT INTO loan (member, isbn, issuedAt, return) VALUES (?,?,?,NULL)") .run(body.member, body.isbn, body.issuedAt); return [201, { id: Number(s.lastInsertRowid), member: body.member, isbn: body.isbn, issuedAt: body.issuedAt }]; } server.listen(8483, "127.0.0.1", () => console.log("key server 127.0.0.1:8483")); ``` #### Measurement The same body is sent twice: first without a key, then with one. The catalog is set up with the `catalog.sql` file from the seventh lesson. ```bash # The same request is sent twice: first without a key, then with an idempotency key. rm -f library.db && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql sqlite3 library.db "DELETE FROM loan;" node key-server.mjs & server=$! sleep 0.4 BODY='{"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"}' say() { printf '%-30s loan rows: %s\n' "$1" "$(sqlite3 library.db 'SELECT COUNT(*) FROM loan;')"; } send() { curl -sS -D - -o /tmp/body.$$ -X POST -H 'content-type: application/json' "$@" \ -d "$BODY" http://127.0.0.1:8483/loans \ | grep -iE '^(HTTP/|idempotency-replayed)' | tr -d '\r'; cat /tmp/body.$$; echo; rm -f /tmp/body.$$; } echo "--- no key, twice ---" send; send; say "no key x2" sqlite3 library.db "DELETE FROM loan; DELETE FROM idempotency;" echo "--- same key, twice ---" send -H 'Idempotency-Key: OD-7f21' send -H 'Idempotency-Key: OD-7f21' say "with key x2" echo "--- same key, different body ---" curl -sS -o /dev/null -w 'status: %{http_code}\n' -X POST -H 'content-type: application/json' \ -H 'Idempotency-Key: OD-7f21' -d '{"member":"U-1002","isbn":"K-03","issuedAt":"2026-03-11"}' \ http://127.0.0.1:8483/loans say "after conflict" kill $server ``` ``` key server 127.0.0.1:8483 --- no key, twice --- HTTP/1.1 201 Created {"id":1,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"} HTTP/1.1 201 Created {"id":2,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"} no key x2 loan rows: 2 --- same key, twice --- HTTP/1.1 201 Created {"id":1,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"} HTTP/1.1 201 Created idempotency-replayed: true {"id":1,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"} with key x2 loan rows: 1 --- same key, different body --- status: 409 after conflict loan rows: 1 ``` The two requests without a key produced two records and returned two different identities. The two requests with a key produced a single record and returned **the same body**: the second response's `id` field is also 1. The second response also carries the `Idempotency-Replayed` header; this header is not mandatory, but it lets the client and the logs distinguish the situation. The last section tests reusing the key. The same `OD-7f21` key arrived with a different body, and the request was rejected with 409; the loan record count stayed at one. Had the body digest not been stored, this request would have been silently swallowed and the client would have assumed a second loan was created. Rejecting it is the right behavior: only the client can fix a key used incorrectly. #### Concurrent Requests Two requests arriving one after another is the easy case. The hard case is both arriving at the same time: while the client times out and retries, the first request may still be being processed. ```bash # Five requests with the same key are sent concurrently; the resulting records are counted. rm -f library.db response-*.json && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql sqlite3 library.db "DELETE FROM loan;" node key-server.mjs & server=$! sleep 0.4 pids="" for i in 1 2 3 4 5; do curl -sS -o "response-$i.json" -X POST -H 'content-type: application/json' \ -H 'Idempotency-Key: OD-9c04' \ -d '{"member":"U-1002","isbn":"K-05","issuedAt":"2026-03-12"}' \ http://127.0.0.1:8483/loans & pids="$pids $!" done wait $pids for f in response-*.json; do cat "$f"; echo; done | sort -u > variants.txt echo "distinct response bodies : $(wc -l < variants.txt | tr -d ' ')" echo "body : $(cat variants.txt)" echo "loan rows : $(sqlite3 library.db 'SELECT COUNT(*) FROM loan;')" echo "--- idempotency store age ---" sqlite3 library.db " INSERT INTO idempotency (key, path, digest, status, response, createdAt) VALUES ('OD-old','/loans','0000',201,'{}',datetime('now','-30 day')); SELECT key || ' age: ' || CAST(julianday('now') - julianday(createdAt) AS INT) || ' days' FROM idempotency ORDER BY createdAt; DELETE FROM idempotency WHERE createdAt < datetime('now','-7 day'); SELECT 'remaining after cleanup: ' || COUNT(*) FROM idempotency;" kill $server ``` ``` key server 127.0.0.1:8483 distinct response bodies : 1 body : {"id":1,"member":"U-1002","isbn":"K-05","issuedAt":"2026-03-12"} loan rows : 1 --- idempotency store age --- OD-old age: 30 days OD-9c04 age: 0 days remaining after cleanup: 1 ``` Five concurrent requests produced a single record, and all five got the same body. What makes this work is the code's structure: there is no wait point between recording the key and doing the work. If a wait were introduced in between, both requests could see "no key" and both could create a record. This arrangement being sufficient on a single-process server does not mean it stays sufficient in a multi-process deployment. The general fix is **writing the key as a primary key**: the second write attempt is rejected by the database, and that request is routed to wait and reread, or to an "operation in progress" response. How races of this kind are isolated at the transaction level is the subject of the Transactions topic in the Advanced SQL course. #### The Key Store's Lifetime The output's last section shows the key store aging. A key written thirty days ago was cleaned up under a seven-day retention window. The retention decision has two bounds. The lower bound is set by the client's **longest retry window**: if a client is still retrying twenty-four hours later, the key must be kept that long. The retry budget defined in the Retry and Backoff lesson of the Application Architecture course gives this window. The upper bound is set by storage cost and the size of the response bodies written into the table. Once the window closes, a request arriving with the same key counts as a new operation and produces a new record. This is the design's accepted limit; the duration is chosen with this in mind. One last boundary is scope. A key must be unique together with the authenticated client, not on its own: two different clients producing the same key string must not lead one's request to receive the other's response. In the table above, the key alone is the primary key; in a real deployment, the client's identity also becomes part of the key. #### Summary - Under a network error, whether the request reached the server is unknown; retrying risks a repeated effect, not retrying risks a missing one. - An idempotency key is generated per intent, not per request; every attempt of the same operation carries the same key. - The server stores the key together with the path and a body digest; in the measurement, two requests without a key produced two records, two requests with one produced one record, and the second response matched the first exactly. - The same key arriving with a different body is rejected with 409; silently swallowing it would show the client an operation that never happened as if it had. - No wait must fall between recording the key and doing the work; five concurrent requests produced a single record under this structure. - The key store's retention period cannot be shorter than the client's longest retry window; once the window closes, the same key counts as a new operation. #### Next Step Across nine lessons, the client was assumed to build every address itself: the collection address, the page cursor, the field list, even the address of a newly created record. To do this, the client embeds the contract's shape into itself — and when the contract changes, every client gets updated separately. The next lesson loosens this dependency: the response carries within itself the addresses the client will use in its next request. Which operations a loan record is open to is also read from the same place; a returned record no longer offers a return link. ### Connection-Based Responses Source: https://academia.sh/en/courses/api-design/connection-based-responses The response carrying navigation and action links within itself: walking pages by following only the next link, action links that open and close based on a resource's state, and measuring the client's address-building burden. Across nine lessons, the client was assumed to build addresses itself. It knew the collection address, the page cursor, the field list, even the address of a newly created record. This has a cost: the contract's shape gets embedded into the client. If the cursor parameter's name changes, if the method used for a return changes, or if an operation becomes conditional, every client has to be updated separately. This lesson loosens that dependency. The response carries within itself the addresses the client will use in its next request; the client knows only the entry address and reads the rest. The second, more important consequence is this: which operations a resource **is open to** is also read from the response. A loan record that has been returned no longer offers a return link. #### The Structure of a Link A link must carry two pieces of information: which address to go to and with which method. A link that carries only an address makes read and write indistinguishable. Action links need a third piece: which fields are expected in the body. Which links a response carries depends on the resource's kind. A collection response carries navigation links: itself, and the next page if there is one. A singular resource response carries its own address, the addresses of related resources, and the operations applicable at that moment. ```js // link-server.mjs — a service whose responses carry navigation and action links inside them import { createServer } from "node:http"; import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("library.db"); const readBody = (request) => new Promise((resolve) => { let data = ""; request.on("data", (p) => (data += p)); request.on("end", () => resolve(data ? JSON.parse(data) : {})); }); const respond = (response, status, data) => { response.writeHead(status, { "content-type": "application/json; charset=utf-8" }); response.end(JSON.stringify(data)); }; // A record's current state determines which operations are open. const recordView = (o) => ({ id: o.id, member: o.member, isbn: o.isbn, issuedAt: o.issuedAt, return: o.return, links: { self: { path: `/loans/${o.id}`, method: "GET" }, member: { path: `/members/${o.member}`, method: "GET" }, ...(o.return === null ? { return: { path: `/loans/${o.id}`, method: "PATCH", fields: ["return"] } } : {}), }, }); const server = createServer(async (request, response) => { const url = new URL(request.url, "http://127.0.0.1"); const single = /^\/loans\/(\d+)$/.exec(url.pathname); const member = /^\/members\/([\w-]+)$/.exec(url.pathname); if (request.method === "GET" && member) { const m = db.prepare("SELECT * FROM member WHERE code = ?").get(member[1]); return m ? respond(response, 200, m) : respond(response, 404, { error: "member_not_found" }); } if (request.method === "GET" && url.pathname === "/loans") { const size = Math.min(Number(url.searchParams.get("size") ?? 3), 50); const cursor = url.searchParams.get("cursor"); const rows = cursor === null ? db.prepare("SELECT * FROM loan ORDER BY id LIMIT ?").all(size) : db.prepare("SELECT * FROM loan WHERE id > ? ORDER BY id LIMIT ?").all(Number(cursor), size); const last = rows.at(-1); // The next page is reported only if it actually exists. const remaining = last ? db.prepare("SELECT COUNT(*) AS n FROM loan WHERE id > ?").get(last.id).n : 0; return respond(response, 200, { data: rows.map(recordView), links: { self: { path: `/loans?size=${size}`, method: "GET" }, ...(remaining > 0 ? { next: { path: `/loans?size=${size}&cursor=${last.id}`, method: "GET" } } : {}), }, }); } if (request.method === "GET" && single) { const o = db.prepare("SELECT * FROM loan WHERE id = ?").get(Number(single[1])); return o ? respond(response, 200, recordView(o)) : respond(response, 404, { error: "loan_not_found" }); } if (request.method === "PATCH" && single) { const g = await readBody(request); const id = Number(single[1]); if ("return" in g) db.prepare("UPDATE loan SET return = ? WHERE id = ?").run(g.return, id); return respond(response, 200, recordView(db.prepare("SELECT * FROM loan WHERE id = ?").get(id))); } respond(response, 404, { error: "path_not_found" }); }); server.listen(8484, "127.0.0.1", () => console.log("link server 127.0.0.1:8484")); ``` Two details matter in particular. The `next` link is generated only if a record genuinely remains, so the client never needs to write a rule like "stop if an empty page arrives." The `return` link is generated only while the record's `return` field is empty; the business rule ends up embedded inside the representation. #### A Client That Knows Only the Entry Address The client on the other side writes a single address and builds no other. ```js // navigator.mjs — client that knows only the entry address, reads the rest from responses const BASE = "http://127.0.0.1:8484"; const ENTRY = "/loans?size=3"; const addressesBuilt = 1; // only the entry address is written on the client const call = async (link, body) => { const options = { method: link.method }; if (body) { options.headers = { "content-type": "application/json" }; options.body = JSON.stringify(body); } return (await fetch(BASE + link.path, options)).json(); }; // 1) Pages are walked only by following the "next" link. let page = await (await fetch(BASE + ENTRY)).json(); let pageCount = 1; const records = []; while (true) { records.push(...page.data); console.log(`page ${pageCount}: ${page.data.map((r) => r.id).join(", ")}` + ` next: ${page.links.next?.path ?? "none"}`); if (!page.links.next) break; page = await call(page.links.next); pageCount++; } console.log(`total ${records.length} records, ${pageCount} pages`); // 2) The first record with an open action link is found and that link is followed. const open = records.find((r) => r.links.return); console.log(`\nbefore action id=${open.id} links: ${Object.keys(open.links).join(", ")}`); const updated = await call(open.links.return, { return: "2026-03-15" }); console.log(`after action id=${updated.id} links: ${Object.keys(updated.links).join(", ")}`); // 3) The record's own address is also read from the response. const reread = await call(updated.links.self); console.log(`reread id=${reread.id} return=${reread.return} ` + `links: ${Object.keys(reread.links).join(", ")}`); console.log(`\naddresses built on the client: ${addressesBuilt}`); ``` ```bash # Seven loan records; the client runs the entire flow knowing only the entry address. rm -f library.db && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql sqlite3 library.db <<'SQL' DELETE FROM loan; INSERT INTO loan (id, member, isbn, issuedAt, return) VALUES (1,'U-1001','K-01','2026-02-01','2026-02-14'), (2,'U-1002','K-02','2026-02-05','2026-02-19'), (3,'U-1001','K-03','2026-03-01',NULL), (4,'U-1002','K-04','2026-03-02',NULL), (5,'U-1001','K-05','2026-03-03',NULL), (6,'U-1002','K-06','2026-03-04',NULL), (7,'U-1001','K-07','2026-03-05',NULL); SQL node link-server.mjs & server=$! sleep 0.4 node navigator.mjs echo "--- raw view of a single record ---" curl -sS http://127.0.0.1:8484/loans/4; echo kill $server ``` ``` link server 127.0.0.1:8484 page 1: 1, 2, 3 next: /loans?size=3&cursor=3 page 2: 4, 5, 6 next: /loans?size=3&cursor=6 page 3: 7 next: none total 7 records, 3 pages before action id=3 links: self, member, return after action id=3 links: self, member reread id=3 return=2026-03-15 links: self, member addresses built on the client: 1 --- raw view of a single record --- {"id":4,"member":"U-1002","isbn":"K-04","issuedAt":"2026-03-02","return":null,"links":{"self":{"path":"/loans/4","method":"GET"},"member":{"path":"/members/U-1002","method":"GET"},"return":{"path":"/loans/4","method":"PATCH","fields":["return"]}}} ``` Three pages, seven records, one written address. The word `cursor` never appears in the client's code; how the cursor is encoded, what parameter name carries it, even whether pagination is cursor-based or offset-based, is none of the client's concern. When the server changes that decision, the client does not change. The second section is more interesting. **Before** the return operation, `return` was among the record's links; after the operation, it was gone. The client does not answer "can this record be returned" by looking at a field's value and applying its own rule; it looks at whether the link exists. The rule lives on the server, and it lives in exactly one place. Its counterpart on the interface side is direct: the return button's visibility is tied to whether the `return` link exists. When the business rule changes — say, members with a fine now have to return books at a branch — the server stops generating the link and the button disappears on its own. #### The Raw Representation and the Contract's Boundary The output's last line gives a single record's raw representation. Links are gathered into a separate field of the body, not mixed in with the record's own fields. This separation must be kept, or the name `return` would be both a date field and a link name. Connection-based design has a cost too, and it should not be overstated. The body grows. Every record in a seven-record list carrying three links dedicates a significant portion of the body to links. The criterion from the Partial Response lesson applies here too: the link block should be closeable through field selection. The client's dependency does not disappear, it relocates. The client is now dependent not on the address format but on **link names.** The name `next` changing is just as breaking as the cursor parameter's name changing. The gain is that link names change less often than address formats do. Generating links costs the server work. The collection endpoint above runs an extra count query just to find out whether a next page exists. This count is expensive on large collections; the common fix is fetching one more record than the page size and deciding a next page exists if that extra one comes back. Finally, not all links carry the same weight of necessity. Without navigation links, the client has to know the pagination format; without action links, it has to reimplement the business rules. The first pays for itself in almost every service; the second is valuable in services where business rules are complex and change often. #### Summary - A link carries at least two pieces of information: address and method; action links also report the fields expected in the body. - A collection response carries navigation links, a singular resource response carries related-resource and action links; links are gathered into a separate field of the body. - In the measurement, the client walked three pages and collected seven records by writing a single address; the cursor parameter's name never appeared in the client's code. - Action links are generated from a resource's current state: a returned record carries no `return` link, so the business rule lives in exactly one place. - The next-page link is generated only while a record remains; the client never needs to write a rule to stop on an empty page. - Its cost is the body's growth, the server's burden of generating links, and the dependency shifting from address format to link names. #### Next Step Resource and contract design is complete: resources were identified, their addresses settled, methods and status codes mapped, body format decided, collections paginated and filtered, write requests guarded against a repeated effect, responses came to carry their own navigation information. All of this holds when things go right. In the error case, though, every endpoint still makes up its own body: an `error` field in one place, `field` and `reason` in another, just a code somewhere else. The next topic starts from here, and its first lesson settles error responses into a single structure: a standard problem-details format, which fields it carries, and how the client uses it. ## Errors, Versions and Documentation Binding error responses to a single shape, reporting field-level validation, path-, header- and content-based versioning, detecting breaking changes from a schema diff, measuring the deprecation window, and testing the contract with machine-readable documentation, contract tests and mock servers. ### Error Response Format Source: https://academia.sh/en/courses/api-design/error-response-format The cost to the client of improvising the error body per endpoint, the five core fields of the standard problem details format, and verifying with a check that a single-layer problem catalog produces the body. The Connection-Based Responses lesson put navigation information inside the successful response: when the client received a loan record, it could read from the response itself how to reach the return action and the member. Resource modeling, address layout, status code selection, body naming, and links — everything designed so far described the case where **the request succeeded**. Failed requests fell outside that design. In the loan service's real traffic there are requests where the book cannot be found, no copy remains, a limit is exceeded, or the body is malformed. This lesson answers a single question: what should the error response's body look like so the client does not have to relearn it for every endpoint? #### Two Error Formats, One Service The server below serves the loan service's familiar core — looking up a book, lending it, returning it — and can produce the error body in two separate formats. The routing code is identical in both; only the body that the `sendError()` call produces changes. The `SCATTERED` table is the shape a codebase settles into over time when the error body is never written into the contract: each condition is reported with whatever names occurred to the person who wrote that line. The `PROBLEM` table instead maps every condition to a problem type catalog. ```js // problem.mjs — the problem details layer: the single place that produces the body export const BASE_TYPE = "https://example.library/problems/"; // Catalog: each problem type is defined once. Type -> permanent id, title, status code. export const CATALOG = { resource_not_found: { title: "Resource not found", status: 404 }, malformed_body: { title: "Request body could not be parsed", status: 400 }, no_copy_available: { title: "No copy available to loan", status: 409 }, member_limit: { title: "Member reached the loan limit", status: 409 }, validation: { title: "Request body failed validation", status: 422 }, }; let counter = 0; const occurrenceId = () => `oc-${String(++counter).padStart(4, "0")}`; // problem(code, detail, extra) -> { body, status } export function problem(code, detail, extra = {}) { const record = CATALOG[code]; if (!record) throw new Error(`unknown problem type in catalog: ${code}`); return { status: record.status, body: { type: BASE_TYPE + code.replaceAll("_", "-"), title: record.title, status: record.status, detail, instance: occurrenceId(), ...extra, }, }; } ``` ```js // server.mjs — loan service; same routes, two separate error formats // Usage: node server.mjs scattered (body improvised per endpoint) // node server.mjs problem (problem details format) import { createServer } from "node:http"; import { problem } from "./problem.mjs"; const STYLE = process.argv[2] ?? "scattered"; // Scattered format: whatever body was written for each error condition on the day. const SCATTERED = { book_not_found: (b) => [404, { error: "not_found", resource: "book", id: b.id }], loan_not_found: (b) => [404, { message: `loan not found: ${b.id}` }], member_not_found: () => [404, { error: { code: 4041, description: "member not found" } }], malformed_body: () => [400, { ok: false, reason: "invalid JSON" }], no_copy_available: () => [409, { conflict: "no_copy_available", remaining: 0 }], member_limit: (b) => [409, { errorMessage: "member limit exceeded", limit: b.limit }], path_not_found: (b) => [404, { status: 404, path: b.path }], }; // Problem format: every condition maps to a problem type in the catalog. const PROBLEM = { book_not_found: (b) => problem("resource_not_found", `ISBN ${b.id} is not in the catalog.`, { resource: "book" }), loan_not_found: (b) => problem("resource_not_found", `Loan record ${b.id} does not exist.`, { resource: "loan" }), member_not_found: (b) => problem("resource_not_found", `Member ${b.id} does not exist.`, { resource: "member" }), malformed_body: () => problem("malformed_body", "Body is not valid JSON.", { resource: "body" }), no_copy_available: (b) => problem("no_copy_available", `Copies of ${b.title} are all on loan.`, { isbn: b.id, remaining: 0 }), member_limit: (b) => problem("member_limit", `Member holds ${b.open} books, limit ${b.limit}.`, { limit: b.limit, open: b.open }), path_not_found: (b) => problem("resource_not_found", `Path ${b.path} is not defined.`, { resource: "path" }), }; const sendError = (res, code, context = {}) => { if (STYLE === "problem") { const { status, body } = PROBLEM[code](context); res.writeHead(status, { "content-type": "application/problem+json; charset=utf-8" }); return res.end(JSON.stringify(body)); } const [status, body] = SCATTERED[code](context); res.writeHead(status, { "content-type": "application/json; charset=utf-8" }); res.end(JSON.stringify(body)); }; const BOOKS = new Map([ ["978-0201896831", { title: "The Art of Computer Programming", copies: 2 }], ["978-0262033848", { title: "Introduction to Algorithms", copies: 1 }], ]); const MEMBERS = new Set(["U-1001", "U-1002"]); const MEMBER_LIMIT = 2; const loans = [{ id: "O-1", member: "U-1001", isbn: "978-0262033848" }]; const readBody = (req) => new Promise((resolve) => { let v = ""; req.on("data", (p) => (v += p)); req.on("end", () => resolve(v)); }); const respond = (res, code, obj) => { res.writeHead(code, { "content-type": "application/json; charset=utf-8" }); res.end(JSON.stringify(obj)); }; createServer(async (req, res) => { res.sendDate = false; const path = req.url.split("?")[0]; if (req.method === "GET" && path.startsWith("/books/")) { const isbn = path.slice("/books/".length); const book = BOOKS.get(isbn); if (!book) return sendError(res, "book_not_found", { id: isbn }); return respond(res, 200, { isbn, ...book }); } if (req.method === "GET" && path.startsWith("/loans/")) { const id = path.slice("/loans/".length); const record = loans.find((o) => o.id === id); if (!record) return sendError(res, "loan_not_found", { id }); return respond(res, 200, record); } if (req.method === "POST" && path === "/loans") { let body; try { body = JSON.parse((await readBody(req)) || "{}"); } catch { return sendError(res, "malformed_body"); } if (!MEMBERS.has(body.member)) return sendError(res, "member_not_found", { id: body.member }); const book = BOOKS.get(body.isbn); if (!book) return sendError(res, "book_not_found", { id: body.isbn }); const held = loans.filter((o) => o.isbn === body.isbn).length; if (held >= book.copies) return sendError(res, "no_copy_available", { id: body.isbn, title: book.title }); const open = loans.filter((o) => o.member === body.member).length; if (open >= MEMBER_LIMIT) return sendError(res, "member_limit", { limit: MEMBER_LIMIT, open }); const id = `O-${loans.length + 1}`; loans.push({ id, member: body.member, isbn: body.isbn }); return respond(res, 201, { id, member: body.member, isbn: body.isbn }); } if (req.method === "POST" && path === "/returns") { const body = JSON.parse((await readBody(req)) || "{}"); const index = loans.findIndex((o) => o.id === body.loanId); if (index < 0) return sendError(res, "loan_not_found", { id: body.loanId }); loans.splice(index, 1); return respond(res, 200, { status: "returned" }); } sendError(res, "path_not_found", { path }); }).listen(8431, "127.0.0.1", () => console.log(`server 127.0.0.1:8431 style=${STYLE}`)); ``` #### The Problem Details Format The shape the `PROBLEM` table produces is not improvised: there is a standardized format for the error body. **Problem details** is defined by RFC 9457, is carried with the `application/problem+json` content type, and has five core fields. - **`type`** — the problem type's permanent identifier, a URI. This is the field the client branches on. It is an identifier, not text; it stays the same even if the title changes. - **`title`** — the problem type's short, human-readable name. Apart from localization it stays the same for the same `type`. - **`status`** — the HTTP status code's copy inside the body. - **`detail`** — an explanation specific to this **instance**: which ISBN, which member, which limit. - **`instance`** — this single occurrence's identifier; it lets the log record be found. The distinction between `type` and `detail` is the format's core. `type` names the **class**, `detail` describes the **instance**. The client decides by looking at the `type` field ("show the reservation button if there is no copy"), and only shows the `detail` field to the user or writes it to the log. If this distinction is not maintained, the client is forced into text comparison, and a spelling fix on the server breaks the client's logic. Repeating the `status` field in the body looks unnecessary at first glance — it is already in the status line. It becomes necessary in two situations: when the response is captured as a body into a log or a queue, the status line is lost; and when an intermediary changes the status code, the mismatch between the body's value and the status line makes that change visible. The standard allows adding **extension members** outside the core. The number of remaining copies, the limit value, which resource was looked up and not found — these are extension members, and they depend on the problem type: the same `type` always comes with the same extension members. #### Measurement The script below produces the server's eight error conditions in turn, collects the bodies, and counts three things: how many distinct content types come back, how many distinct key sets result, and how many distinct combinations the five core fields appear in. ```js // check.mjs — measures the format consistency of a server's error responses // Usage: node check.mjs const BASE = process.argv[2]; const CORE = ["type", "title", "status", "detail", "instance"]; // Each row produces one error condition: [name, method, path, body] const CASES = [ ["book not found (GET)", "GET", "/books/978-0000000000", null], ["loan not found (GET)", "GET", "/loans/O-99", null], ["malformed body", "POST", "/loans", "{this is not json"], ["member not found", "POST", "/loans", { member: "U-9999", isbn: "978-0201896831" }], ["book not found (POST)", "POST", "/loans", { member: "U-1001", isbn: "978-0000000000" }], ["no copy available", "POST", "/loans", { member: "U-1002", isbn: "978-0262033848" }], ["loan not found (return)", "POST", "/returns", { loanId: "O-99" }], ["path not found", "GET", "/shelves", null], ]; const keysOf = (n, prefix = "") => Object.entries(n).flatMap(([a, d]) => d && typeof d === "object" && !Array.isArray(d) ? keysOf(d, `${prefix}${a}.`) : [`${prefix}${a}`]); const coreShapes = new Set(); const fullShapes = new Set(); const types = new Set(); console.log("case content type core extension members"); for (const [name, method, path, body] of CASES) { const response = await fetch(BASE + path, { method, headers: body == null ? {} : { "content-type": "application/json" }, body: body == null ? undefined : typeof body === "string" ? body : JSON.stringify(body), }); const text = await response.text(); const type = (response.headers.get("content-type") ?? "").split(";")[0]; types.add(type); let list; try { list = keysOf(JSON.parse(text)); } catch { list = [""]; } const core = CORE.filter((a) => list.includes(a)); const extra = list.filter((a) => !CORE.includes(a)).sort(); coreShapes.add(core.join(",")); fullShapes.add([...core, ...extra].join(",")); const status = core.length === CORE.length ? "full" : `partial(${core.length}/5)`; console.log(`${name.padEnd(24)} ${type.padEnd(24)} ${status.padEnd(14)} ${extra.join(",") || "-"}`); } console.log(`\nerror conditions tested: ${CASES.length}`); console.log(`distinct content types: ${types.size} (${[...types].join(", ")})`); console.log(`distinct full key sets: ${fullShapes.size}`); console.log(`distinct core shapes: ${coreShapes.size}`); ``` ```bash #!/usr/bin/env bash # Starts server.mjs in each format in turn and runs the same check. for style in scattered problem; do node server.mjs "$style" & s=$! sleep 0.5 node check.mjs http://127.0.0.1:8431 kill "$s"; wait "$s" 2>/dev/null echo done ``` ``` server 127.0.0.1:8431 style=scattered case content type core extension members book not found (GET) application/json partial(0/5) error,id,resource loan not found (GET) application/json partial(0/5) message malformed body application/json partial(0/5) ok,reason member not found application/json partial(0/5) error.code,error.description book not found (POST) application/json partial(0/5) error,id,resource no copy available application/json partial(0/5) conflict,remaining loan not found (return) application/json partial(0/5) message path not found application/json partial(1/5) path error conditions tested: 8 distinct content types: 1 (application/json) distinct full key sets: 6 distinct core shapes: 2 server 127.0.0.1:8431 style=problem case content type core extension members book not found (GET) application/problem+json full resource loan not found (GET) application/problem+json full resource malformed body application/problem+json full resource member not found application/problem+json full resource book not found (POST) application/problem+json full resource no copy available application/problem+json full isbn,remaining loan not found (return) application/problem+json full resource path not found application/problem+json full resource error conditions tested: 8 distinct content types: 1 (application/problem+json) distinct full key sets: 2 distinct core shapes: 1 ``` In the scattered format, eight error conditions produce six distinct key sets, and none of them share a common core. `error`, `message`, `reason`, `errorMessage`, `error.description` — all of them carry the same thing, the "what happened" information, and each one uses a different name. The request layer built in the Application Architecture course has to map the response to an **error contract**, and no mapping can be written against this table. The client either writes six separate parsers or looks at a single field and ignores the rest; the second option is common, and it turns into the sentence "an error occurred" on the user's screen. In the problem format, the number of distinct core shapes drops to one, and all five fields are present on every row. The full key set count is two, because the extension members differ, and they should differ: the `no_copy_available` problem has a `remaining` field, the `resource_not_found` problem does not. But this difference is tied to the **problem type**, not the endpoint. When the client reads the `type` field it knows which extension members will come; in the scattered format it had to know which endpoint it had called. The content type row shows a second gain. In the scattered format, the error response and the success response carry the same content type; the intermediaries and logging tools in between cannot tell whether this is an error without parsing the body. `application/problem+json` gives that distinction without looking at the body. #### The Same Problem Returned From Different Endpoints In the table, the `resource_not_found` problem appears on six rows. For the client to be able to rely on this, those rows' `type`, `title`, and `status` fields must be identical. The script below produces the same problem from two different endpoints, then requests a type that is not written into the catalog. ```bash #!/usr/bin/env bash # Produces the same problem type from two different endpoints; tries a type outside the catalog. node server.mjs problem & server=$! sleep 0.5 for path in /books/978-0000000000 /loans/O-99; do echo "--- $path ---" curl -sS -D - -o /tmp/body "http://127.0.0.1:8431$path" | grep -i '^HTTP\|^content-type' cat /tmp/body; echo done kill "$server"; wait "$server" 2>/dev/null echo "--- requesting a problem type outside the catalog ---" node -e 'import("./problem.mjs").then(({ problem }) => { try { problem("shelf_is_dusty", "A made-up error."); } catch (h) { console.log(h.message); } });' ``` ``` server 127.0.0.1:8431 style=problem --- /books/978-0000000000 --- HTTP/1.1 404 Not Found content-type: application/problem+json; charset=utf-8 {"type":"https://example.library/problems/resource-not-found","title":"Resource not found","status":404,"detail":"ISBN 978-0000000000 is not in the catalog.","instance":"oc-0001","resource":"book"} --- /loans/O-99 --- HTTP/1.1 404 Not Found content-type: application/problem+json; charset=utf-8 {"type":"https://example.library/problems/resource-not-found","title":"Resource not found","status":404,"detail":"Loan record O-99 does not exist.","instance":"oc-0002","resource":"loan"} --- requesting a problem type outside the catalog --- unknown problem type in catalog: shelf_is_dusty ``` The first three fields are identical; `detail` and `instance` differ. The client's status code mapping looks at the `type` field and writes a single branch for both endpoints; the text to show the user comes from the `detail` field; `instance` is written into the support record. The last line shows the catalog's second function. The catalog does not only hold the mapping between `type`, `title`, and `status`; it errors when a type it does not have is requested. A problem not written into the catalog cannot silently produce a new shape. What prevents the format from scattering again over time is not that the format is designed correctly, but that misuse is visible at runtime. #### What Does Not Belong in the Body Producing the error body from a single place makes what stays out controllable, just as much as what goes in. Stack traces, database query text, file paths, and internal identifiers do not go to the client; they are kept in the server log, matched by the `instance` field. When the client reports a problem, it gives the occurrence identifier, and the full context is found from the log. The criterion rests on who can do what: information the client can **change its behavior** on goes into the body (no copies left, limit is two, ISBN not found); information the client can do nothing about goes into the log. This criterion gives the same result for both security and usability reasons. #### Summary - When the error body is written per endpoint, eight error conditions produce six distinct key sets and none of them share a common core; the client cannot build a single error contract on this table. - The problem details format's five core fields each serve a separate purpose: `type` names the class, `title` is the class's fixed name, `status` repeats the status code in the body, `detail` describes the instance, `instance` matches the single occurrence to the log. - The client branches on the `type` field; branching on the `detail` text turns a spelling fix on the server into a breaking change. - Producing the body from a single layer built on the problem type catalog brings the distinct core shape count down to one; extension members can still differ, because they depend on the problem type, not the endpoint. - The catalog makes the format's re-scattering visible at runtime by erroring when a type it does not have is requested. - Information the client can change its behavior on is written into the body; information it cannot is written into the log record matched by `instance`. #### Next Step The `validation` type in the catalog was never used in this lesson. The reason is that a validation error does not fit into a single sentence: a loan request with four wrong fields at once cannot be sufficiently described by the title "request body failed validation." The client needs to show each wrong field next to its own input box, and that requires the server to report, field by field, which field was rejected and why. The next lesson adds a field-level error list to problem details, decides the rule by which field names match the client's fields, and measures the case where the match breaks down. ### Validation Errors Source: https://academia.sh/en/courses/api-design/validation-errors Adding a field-level error list to problem details, the request-round difference between stopping at the first error and collecting all of them, and the body path rule that makes field names match the client's input boxes. The previous lesson bound the error body to a single shape but left the `validation` type in the problem catalog unused. The reason is that this type is structurally different from the others. "No copy available" is a single fact; a sentence in the `detail` field describes it completely. "Request body failed validation" is not a fact but a **set of facts**: which fields of the body were rejected, by which rules? The loan service's lending request is no longer a single book; it carries a member, multiple items, and a return date. This lesson builds field-level error reporting and settles two decisions by measurement: how many errors get reported at once, and by what name the fields are referred to. #### How Many Errors at Once Validation can stop at the first field that violates a rule, or it can scan the whole body and collect the errors. The difference between them is how many times the user has to send a request to fix the form. The validator below takes the schema as data. The schema keys are **body paths** and are written in JSON Pointer form: `/member`, `/items/1/isbn`. The `*` marker expands over array elements, so the `/items/*/isbn` pattern applies to every item in the body. ```js // validate.mjs — validates the body against a schema, produces a field-level error list // Field name is JSON Pointer (RFC 6901): "/member", "/items/1/isbn" export const RULES = { required: (d) => (d === undefined || d === null || d === "" ? { code: "required" } : null), pattern: (d, p) => (new RegExp(p).test(String(d)) ? null : { code: "format", param: p }), enum: (d, p) => (p.includes(d) ? null : { code: "not_in_enum", param: p }), array: (d, p) => !Array.isArray(d) ? { code: "not_an_array" } : d.length < p.min ? { code: "too_short", param: p } : d.length > p.max ? { code: "too_long", param: p } : null, date: (d) => (/^\d{4}-\d{2}-\d{2}$/.test(String(d)) ? null : { code: "not_a_date" }), maxDays: (d, p, today) => { const diff = Math.round((Date.parse(d) - Date.parse(today)) / 86400000); if (Number.isNaN(diff)) return null; // date rule already reported it if (diff < 0) return { code: "in_past" }; return diff > p ? { code: "too_far_ahead", param: p } : null; }, }; // The * inside a template is expanded to real array indices: /items/*/isbn -> /items/0/isbn const read = (body, path) => path.split("/").slice(1).reduce((d, p) => (d == null ? undefined : d[p]), body); function expandPaths(body, template) { const star = template.indexOf("/*"); if (star < 0) return [template]; const arr = read(body, template.slice(0, star)); if (!Array.isArray(arr)) return []; return arr.flatMap((_, i) => expandPaths(body, template.slice(0, star) + `/${i}` + template.slice(star + 2))); } // validate(schema, body, today) -> [{ path, code, param }] export function validate(schema, body, today) { const errors = []; for (const [template, rules] of Object.entries(schema)) { for (const path of expandPaths(body, template)) { const value = read(body, path); for (const [name, param] of rules) { if (name !== "required" && (value === undefined || value === null || value === "")) break; const result = RULES[name](value, param, today); if (result) { errors.push({ path, ...result }); break; } // one error per field } } } return errors; } ``` To compare the two collection styles, a body with four fields wrong at once is taken, and the user is assumed to fix only the errors **reported to them**. ```js // turns.mjs — compares stop-at-first-error validation with collect-all validation import { validate } from "./validate.mjs"; const TODAY = "2026-03-01"; const SCHEMA = { "/member": [["required"], ["pattern", "^U-\\d{4}$"]], "/items": [["required"], ["array", { min: 1, max: 3 }]], "/items/*/isbn": [["required"], ["pattern", "^97[89]-\\d{10}$"]], "/items/*/branch": [["required"], ["enum", ["central", "shore", "hill"]]], "/returnDate": [["required"], ["date"], ["maxDays", 30]], }; // The body the user first submits: four fields wrong at once. const FIRST = { member: "1001", items: [ { isbn: "978-0262033848", branch: "central" }, { isbn: "0262033848", branch: "sea" }, ], returnDate: "2026-06-01", }; // The user's fix order: whichever error is reported gets fixed. const FIXES = { "/member": (b) => (b.member = "U-1001"), "/items/1/isbn": (b) => (b.items[1].isbn = "978-0201896831"), "/items/1/branch": (b) => (b.items[1].branch = "shore"), "/returnDate": (b) => (b.returnDate = "2026-03-20"), }; function roundCount(collectAll) { const body = structuredClone(FIRST); let round = 0; while (round < 10) { round++; const errors = validate(SCHEMA, body, TODAY); if (errors.length === 0) return round; const reported = collectAll ? errors : [errors[0]]; console.log(` round ${round}: ${reported.map((h) => `${h.path}=${h.code}`).join(" ")}`); for (const h of reported) FIXES[h.path](body); } return round; } console.log("validation that stops at the first error:"); const a = roundCount(false); console.log(` request rounds until success: ${a}\n`); console.log("validation that collects all errors:"); const b = roundCount(true); console.log(` request rounds until success: ${b}`); ``` ``` validation that stops at the first error: round 1: /member=format round 2: /items/1/isbn=format round 3: /items/1/branch=not_in_enum round 4: /returnDate=too_far_ahead request rounds until success: 5 validation that collects all errors: round 1: /member=format /items/1/isbn=format /items/1/branch=not_in_enum /returnDate=too_far_ahead request rounds until success: 2 ``` Five rounds against two. The number itself depends on how many fields in the body are wrong — $n$ wrong fields mean $n+1$ rounds in stop-at-first-error validation — but the behavioral difference is independent of the count: stopping at the first error **hides** the existence of the next error from the user. The user thinks the job is done after every fix and gets rejected again. One detail is preserved: the validator stops at the first error **per field**. The `/member` field must be both required and pattern-matching; when it is left blank, there is no benefit in reporting both the "required" and "format" errors, since the second is a consequence of the first. The rule is: scanning continues across fields, but the first error within a field is enough. #### The Three Fields of an Error Record Every error record carries three things. **`path`** is the field's address in the body. **`code`** is the error's stable identifier — the same distinction as the validation schemas built in the Application Architecture course in the Frontend Development curriculum: the code is for branching, the message is for display. **`param`** is the number the code requires: which pattern, which options, how many days. The code standing in for the message makes it possible to produce the text shown to the user **on the client side**. The server sends `too_far_ahead` and `30`; the client turns this into a sentence in its own language and its own tone. If the server sent text, translation, pluralization rules, and tone decisions would move onto the server; none of that is the server's job. This triple is added to problem details as an **extension member**. The five core fields stay in place; the `errors` array joins them. ```js // server.mjs — service that validates a loan request and returns 422 problem details import { createServer } from "node:http"; import { validate } from "./validate.mjs"; const TODAY = "2026-03-01"; // fixed so the example is reproducible const SCHEMA = { "/member": [["required"], ["pattern", "^U-\\d{4}$"]], "/items": [["required"], ["array", { min: 1, max: 3 }]], "/items/*/isbn": [["required"], ["pattern", "^97[89]-\\d{10}$"]], "/items/*/branch": [["required"], ["enum", ["central", "shore", "hill"]]], "/returnDate": [["required"], ["date"], ["maxDays", 30]], }; // The one entry from the previous lesson's problem catalog that this lesson needs. let counter = 0; const validationProblem = (errors) => ({ type: "https://example.library/problems/validation", title: "Request body failed validation", status: 422, detail: `${errors.length} fields were not accepted.`, instance: `oc-${String(++counter).padStart(4, "0")}`, errors, }); const readBody = (req) => new Promise((resolve) => { let v = ""; req.on("data", (p) => (v += p)); req.on("end", () => resolve(v)); }); // Core fields stacked, each error on its own line: keep the body readable. const format = (s) => [ "{", ...["type", "title", "status", "detail", "instance"].map((a) => ` "${a}": ${JSON.stringify(s[a])},`), ' "errors": [', ...s.errors.map((h, i) => ` ${JSON.stringify(h)}${i < s.errors.length - 1 ? "," : ""}`), " ]", "}", ].join("\n"); createServer(async (req, res) => { res.sendDate = false; const body = JSON.parse((await readBody(req)) || "{}"); const errors = validate(SCHEMA, body, TODAY); if (errors.length) { res.writeHead(422, { "content-type": "application/problem+json; charset=utf-8" }); return res.end(format(validationProblem(errors))); } res.writeHead(201, { "content-type": "application/json; charset=utf-8" }); res.end(JSON.stringify({ id: "O-2", member: body.member, items: body.items.length })); }).listen(8433, "127.0.0.1", () => console.log("validation server 127.0.0.1:8433")); ``` ```bash #!/usr/bin/env bash # Sends a loan request with four wrong fields; shows the 422 body. node server.mjs & server=$! sleep 0.5 curl -sS -w '\n[%{http_code}] %{content_type}\n' -X POST -H 'content-type: application/json' \ -d '{"member":"1001","items":[{"isbn":"978-0262033848","branch":"central"},{"isbn":"0262033848","branch":"sea"}],"returnDate":"2026-06-01"}' \ http://127.0.0.1:8433/loans echo "--- fixed body ---" curl -sS -w '\n[%{http_code}] %{content_type}\n' -X POST -H 'content-type: application/json' \ -d '{"member":"U-1001","items":[{"isbn":"978-0262033848","branch":"central"},{"isbn":"978-0201896831","branch":"shore"}],"returnDate":"2026-03-20"}' \ http://127.0.0.1:8433/loans kill "$server"; wait "$server" 2>/dev/null ``` ``` validation server 127.0.0.1:8433 { "type": "https://example.library/problems/validation", "title": "Request body failed validation", "status": 422, "detail": "4 fields were not accepted.", "instance": "oc-0001", "errors": [ {"path":"/member","code":"format","param":"^U-\\d{4}$"}, {"path":"/items/1/isbn","code":"format","param":"^97[89]-\\d{10}$"}, {"path":"/items/1/branch","code":"not_in_enum","param":["central","shore","hill"]}, {"path":"/returnDate","code":"too_far_ahead","param":30} ] } [422] application/problem+json; charset=utf-8 --- fixed body --- {"id":"O-2","member":"U-1001","items":2} [201] application/json; charset=utf-8 ``` The status code being **422**, not 400, is a deliberate choice. 400 reports that the request could not be **understood** by the server: malformed JSON, a missing header, an unrecognized content type. The body here has been understood completely — parsed, its fields read, run through the rules — and its **meaning** was rejected. The distinction is useful in the log too: a rise in the 400 rate shows that the client built the request wrong, a rise in the 422 rate shows that the user entered wrong data; the two have different causes and different fixes. #### The Field Name's Counterpart on the Client For the error list to be useful, the client has to know **next to which input box** to show each error. The error presentation in the Accessible Component Patterns course assumes this match: each error is bound to its own field's accessible description, and the error summary moves focus to that field. If the match cannot be established, all that is left is a single generic message. The measurement below reports the same four errors under three separate server contracts and counts how many of them the client can match to their box. ```js // match.mjs — measures the client-side match rate of three field naming conventions // The client's input boxes are named by the contract's rule: // name = JSON Pointer with the leading slash dropped and "/" replaced by "." const BOXES = ["member", "items.0.isbn", "items.0.branch", "items.1.isbn", "items.1.branch", "returnDate"]; const toBoxName = (path) => path.replace(/^\//, "").replaceAll("/", "."); // The same four errors, reported under three separate server contracts. const CONTRACTS = { "no field name": { errors: [{ message: "The information you submitted has four errors." }], toBox: () => null, }, "server's internal names": { errors: [ { field: "member_id", code: "format" }, { field: "items[1].isbn", code: "format" }, { field: "items[1].branch", code: "not_in_enum" }, { field: "due_date", code: "too_far_ahead" }, ], toBox: (h) => h.field, }, "body path (JSON Pointer)": { errors: [ { path: "/member", code: "format" }, { path: "/items/1/isbn", code: "format" }, { path: "/items/1/branch", code: "not_in_enum" }, { path: "/returnDate", code: "too_far_ahead" }, ], toBox: (h) => toBoxName(h.path), }, }; console.log("contract reported matched can be shown next to its box"); for (const [name, { errors, toBox }] of Object.entries(CONTRACTS)) { const matched = errors.filter((h) => BOXES.includes(toBox(h))); console.log( `${name.padEnd(27)} ${String(errors.length).padStart(9)} ${String(matched.length).padStart(7)} ` + `${matched.length ? matched.map((h) => toBox(h)).join(", ") : "none"}` ); } ``` ``` contract reported matched can be shown next to its box no field name 1 0 none server's internal names 4 0 none body path (JSON Pointer) 4 4 member, items.1.isbn, items.1.branch, returnDate ``` The second row is worse than the first, because it is deceptive: the server appears to report four separate errors, and there are field names too, but those names come from the server's own internal data model. `member_id`, `items[1].branch` — there is no such field in the client's body. The client either keeps a hand-written translation table between the server's internal names and its own field names, or shows the errors unmatched. If a translation table is kept, every internal rename on the server breaks the client; in other words, something outside the contract starts behaving like part of the contract. The rule is: **the error path is the address of the field in the body the client sent.** Not the server's internal model names, but the structure of the request body. This rule guarantees two properties. It is total — every field in the body has a path, array elements included. And it is derivable in one direction: because the client builds the body itself, it can write the rule that converts a path into a box name, without needing to keep a table in the reverse direction. The only remaining decision is which format the body path is written in. JSON Pointer is a defined standard, expresses array indices naturally, and has well-defined escaping rules. A dot separator could be chosen instead; what matters is not the format itself but that it is **documented and defined for every field**. #### Summary - When validation stops at the first error, $n$ wrong fields require $n+1$ request rounds; when errors are reported in bulk, two rounds are enough and the existence of the next error is not hidden from the user. - Scanning continues across fields, but the first error within a field is enough: successive rule violations on the same field are consequences of each other. - An error record carries three fields — path, code, parameter; the text shown to the user is produced on the client from the code and the parameter, not from the server. - A request that could not be understood is reported with 400, a request that was understood but whose meaning was rejected is reported with 422; the distinction also separates the cause behind a rise in the log. - The error path is the address in the body the client sent, not the server's internal model name; when internal names are reported, the match rate drops to zero out of four. - The field naming convention must be total and derivable in one direction; what matters is not which format is chosen but that it is documented and defined for every field. #### Next Step The validation schema was taken as fixed in this lesson: `/items` accepts at most three elements, the return date can be at most thirty days ahead. What happens when the library decides to change these limits? Raising the item limit to five breaks no client; lowering it to three breaks every client that sends a request with four items. Renaming the same body field, however, breaks every client at once. Change to the contract is inevitable; the next lesson takes up the mechanism that makes this change manageable, and by publishing two versions of the same resource on a single server, compares the cost to client code of path-based, header-based, and content-based versioning. ### Versioning Strategies Source: https://academia.sh/en/courses/api-design/versioning-strategies Publishing two versions of the same resource on a single server, the measured equal cost to client code of path-based, header-based, and content-based versioning, and the real difference that comes from whether intermediaries can tell versions apart. The validation schema was taken as fixed in the previous lesson: at most three items, the return date at most thirty days ahead. These numbers can change; more importantly, the body's **structure** can change. The loan record's `member` field turns from a plain identifier string into an object when a name is also needed, the single `isbn` field turns into an array when loans with multiple items are required, and the `returnDate` field is renamed `dueDate`. What these changes have in common is that they break clients that are already working. Versioning does not prevent breakage; it keeps the **old contract in effect**. This lesson runs three versioning styles side by side on the same server and measures two things: how many lines of client code change, and whether intermediaries can tell the versions apart. #### Two Versions of the Same Resource The server below serves a single loan record in two versions. The versions are derived from the same data; only the shape of the exported body differs. All three styles run in the same process. ```js // server.mjs — publishes the same loan record in two versions, three separate versioning styles import { createServer } from "node:http"; const RECORD = { id: "O-1", member: "U-1001", name: "Alice Carter", isbn: "978-0262033848", dueDate: "2026-03-20" }; // v1: flat shape, single book. v2: member object, item array, field renamed. const VERSIONS = { 1: (k) => ({ id: k.id, member: k.member, isbn: k.isbn, returnDate: k.dueDate }), 2: (k) => ({ id: k.id, member: { id: k.member, name: k.name }, items: [{ isbn: k.isbn }], dueDate: k.dueDate }), }; const MEDIA_TYPE = "application/vnd.library.loan+json"; // Extracts the version parameter from the Accept header: ...+json;version=2 const acceptVersion = (accept) => { const m = /version=(\d+)/.exec(accept ?? ""); return m ? Number(m[1]) : null; }; const write = (res, version, extraHeaders) => { res.writeHead(200, { "content-type": `${MEDIA_TYPE}; version=${version}`, ...extraHeaders }); res.end(JSON.stringify(VERSIONS[version](RECORD))); }; createServer((req, res) => { res.sendDate = false; const path = req.url.split("?")[0]; // 1) Path-based: version is part of the address. Two versions are two separate resource addresses. const m = /^\/v(\d+)\/loans\/(.+)$/.exec(path); if (m && VERSIONS[m[1]]) return write(res, Number(m[1]), {}); if (path === "/loans/O-1") { // 2) Content-based: version is a parameter of the media type. Requires Vary: Accept. const accepted = acceptVersion(req.headers.accept); if (accepted && VERSIONS[accepted]) return write(res, accepted, { vary: "Accept" }); // 3) Header-based: version is in a separate header. Vary: that header. const header = Number(req.headers["api-version"] ?? 1); if (VERSIONS[header]) return write(res, header, { vary: "Api-Version" }); } res.writeHead(404, { "content-type": "application/problem+json; charset=utf-8" }); res.end(JSON.stringify({ type: "https://example.library/problems/resource-not-found", title: "Resource not found", status: 404, detail: `${path} does not exist.`, instance: "oc-0001" })); }).listen(8434, "127.0.0.1", () => console.log("server 127.0.0.1:8434")); ``` ```bash #!/usr/bin/env bash # Requests the same record's two versions with three separate versioning styles. node server.mjs & server=$! sleep 0.5 echo "--- path-based ---" curl -sS -w ' [%{content_type}]\n' http://127.0.0.1:8434/v1/loans/O-1 curl -sS -w ' [%{content_type}]\n' http://127.0.0.1:8434/v2/loans/O-1 echo "--- header-based ---" curl -sS -H 'Api-Version: 1' -w ' [%{content_type}]\n' http://127.0.0.1:8434/loans/O-1 curl -sS -H 'Api-Version: 2' -w ' [%{content_type}]\n' http://127.0.0.1:8434/loans/O-1 echo "--- content-based ---" curl -sS -H 'Accept: application/vnd.library.loan+json;version=1' -w ' [%{content_type}]\n' http://127.0.0.1:8434/loans/O-1 curl -sS -H 'Accept: application/vnd.library.loan+json;version=2' -w ' [%{content_type}]\n' http://127.0.0.1:8434/loans/O-1 kill "$server"; wait "$server" 2>/dev/null ``` ``` server 127.0.0.1:8434 --- path-based --- {"id":"O-1","member":"U-1001","isbn":"978-0262033848","returnDate":"2026-03-20"} [application/vnd.library.loan+json; version=1] {"id":"O-1","member":{"id":"U-1001","name":"Alice Carter"},"items":[{"isbn":"978-0262033848"}],"dueDate":"2026-03-20"} [application/vnd.library.loan+json; version=2] --- header-based --- {"id":"O-1","member":"U-1001","isbn":"978-0262033848","returnDate":"2026-03-20"} [application/vnd.library.loan+json; version=1] {"id":"O-1","member":{"id":"U-1001","name":"Alice Carter"},"items":[{"isbn":"978-0262033848"}],"dueDate":"2026-03-20"} [application/vnd.library.loan+json; version=2] --- content-based --- {"id":"O-1","member":"U-1001","isbn":"978-0262033848","returnDate":"2026-03-20"} [application/vnd.library.loan+json; version=1] {"id":"O-1","member":{"id":"U-1001","name":"Alice Carter"},"items":[{"isbn":"978-0262033848"}],"dueDate":"2026-03-20"} [application/vnd.library.loan+json; version=2] ``` All three styles produce the same two bodies. The difference is where the version is written: to the address in the path-based style, to a separate request-specific header in the header-based style, to a parameter of the media type in the `Accept` header in the content-based style. The content-based style uses HTTP's own content negotiation mechanism. The `Content-Type` the server returns also carries the version, so the response reports its own version. In the header-based style, this information is in a separate header, and it is not a concept HTTP knows about; for it to work, the client and the server must agree on the same header name. #### The Cost to Client Code The first justification for choosing a style is usually presented as "the client changes less code." This is a measurable claim. The program below generates a client's source with three call sites for each of the three styles, and counts the number of changed lines in two scenarios: all call sites migrating to v2, and only one migrating. ```js // client-cost.mjs — changed line count in client code for three versioning styles // Three call sites, three scenarios: [1,1,1] all v1 | [2,2,2] full migration | [2,1,1] partial migration // Constant names stay stable: the majority version keeps the main constant, the minority gets an added constant. const MEDIA_TYPE = "application/vnd.library.loan+json"; const CALL_SITES = [["getLoan ", "(id)", "/loans/${id}"], ["listLoans", "() ", "/loans"], ["getMember", "(id)", "/members/${id}"]]; const majority = (s) => (s.filter((v) => v === 1).length >= 2 ? 1 : 2); const call = (k, address, extra) => `export const ${k[0]} = ${k[1]} => request(\`${address}\`${extra});`; const STYLES = { "path-based": (s) => { const c = majority(s), name = (v) => (v === c ? "BASE" : `BASE_V${v}`); return [ `const BASE = "http://127.0.0.1:8434/v${c}";`, ...[...new Set(s)].filter((v) => v !== c).map((v) => `const BASE_V${v} = "http://127.0.0.1:8434/v${v}";`), ...CALL_SITES.map((k, i) => call(k, "${" + name(s[i]) + "}" + k[2], "")), ]; }, "header-based": (s) => { const c = majority(s), name = (v) => (v === c ? "VERSION" : `VERSION_V${v}`); return [ 'const BASE = "http://127.0.0.1:8434";', `const VERSION = { "api-version": "${c}" };`, ...[...new Set(s)].filter((v) => v !== c).map((v) => `const VERSION_V${v} = { "api-version": "${v}" };`), ...CALL_SITES.map((k, i) => call(k, "${BASE}" + k[2], `, ${name(s[i])}`)), ]; }, "content-based": (s) => { const c = majority(s), name = (v) => (v === c ? "VERSION" : `VERSION_V${v}`); return [ 'const BASE = "http://127.0.0.1:8434";', `const VERSION = { accept: "${MEDIA_TYPE};version=${c}" };`, ...[...new Set(s)].filter((v) => v !== c).map((v) => `const VERSION_V${v} = { accept: "${MEDIA_TYPE};version=${v}" };`), ...CALL_SITES.map((k, i) => call(k, "${BASE}" + k[2], `, ${name(s[i])}`)), ]; }, }; const changed = (before, after) => after.filter((x) => !before.includes(x)).length; console.log("style lines full migration partial migration"); for (const [name, f] of Object.entries(STYLES)) { const v1 = f([1, 1, 1]); console.log(`${name.padEnd(15)} ${String(v1.length).padStart(5)} ` + `${String(changed(v1, f([2, 2, 2]))).padStart(15)} ${String(changed(v1, f([2, 1, 1]))).padStart(18)}`); } for (const name of ["path-based", "header-based"]) { console.log(`\n— ${name} client during partial migration —`); for (const s of STYLES[name]([2, 1, 1])) console.log(s); } ``` ``` style lines full migration partial migration path-based 4 1 2 header-based 5 1 2 content-based 5 1 2 — path-based client during partial migration — const BASE = "http://127.0.0.1:8434/v1"; const BASE_V2 = "http://127.0.0.1:8434/v2"; export const getLoan = (id) => request(`${BASE_V2}/loans/${id}`); export const listLoans = () => request(`${BASE}/loans`); export const getMember = (id) => request(`${BASE}/members/${id}`); — header-based client during partial migration — const BASE = "http://127.0.0.1:8434"; const VERSION = { "api-version": "1" }; const VERSION_V2 = { "api-version": "2" }; export const getLoan = (id) => request(`${BASE}/loans/${id}`, VERSION_V2); export const listLoans = () => request(`${BASE}/loans`, VERSION); export const getMember = (id) => request(`${BASE}/members/${id}`, VERSION); ``` All three styles give the same numbers: one line in a full migration, two in a partial migration. This was not the expected result; it is a common belief that path-based versioning is more expensive. The belief's source is the assumption that the address is scattered across call sites. When the address is collected into a single constant, the version is collected into that constant too, and the cost equalizes with the header-based style. What the measurement actually shows is this: **the cost is however many places the version information sits in the client, not the style.** If it can sit in a single place in all three styles, all three are the same price. In that case, the choice has to rest on a justification outside client code. #### What Intermediaries See That justification lies in the intermediaries the request passes through on the path between client and server. The program below is a small cache and runs in two modes: a mode that produces the key from the address alone, and a mode that also folds the request headers declared in the response's `Vary` header into the key. ```js // cache.mjs — a small cache that sits in front of the server // Usage: node cache.mjs address (key is the request address only) // node cache.mjs vary (key is address + the headers named in the response's Vary) import { createServer } from "node:http"; const MODE = process.argv[2] ?? "address"; const UPSTREAM = "http://127.0.0.1:8434"; const entries = new Map(); // key -> { body, type } const varyInfo = new Map(); // address -> header names declared in the response's Vary createServer(async (req, res) => { res.sendDate = false; const requestHeaders = { accept: req.headers.accept ?? "", "api-version": req.headers["api-version"] ?? "" }; const keyFor = (names) => req.url + names.map((a) => `|${a}=${requestHeaders[a]}`).join(""); const names = MODE === "vary" ? (varyInfo.get(req.url) ?? []) : []; const found = entries.get(keyFor(names)); if (found) { res.writeHead(200, { "content-type": found.type, "x-cache": "hit" }); return res.end(found.body); } const response = await fetch(UPSTREAM + req.url, { headers: requestHeaders }); const body = await response.text(); const type = response.headers.get("content-type"); const vary = (response.headers.get("vary") ?? "").split(",").map((s) => s.trim().toLowerCase()).filter(Boolean); varyInfo.set(req.url, vary); entries.set(keyFor(MODE === "vary" ? vary : []), { body, type }); res.writeHead(200, { "content-type": type, "x-cache": "miss" }); res.end(body); }).listen(8435, "127.0.0.1", () => console.log(`cache 127.0.0.1:8435 mode=${MODE}`)); ``` ```bash #!/usr/bin/env bash # Behind the cache: first the client requesting v2, then the client requesting v1. node server.mjs & s=$! sleep 0.4 ask() { # ask