Input Validation and Encoding

The Observability and Reliability course made the loan system visible and made it withstand failure. Across that course every incoming request was valid: the load generator sent bodies matching the contract, and the chaos trials broke the infrastructure, not the request. How far a request that does not conform to the contract can travel inside the system was never measured.

This lesson’s subject is that boundary. The split loan system has three external endpoints, and every body arriving at them is untrusted data until proven otherwise. The vulnerabilities are not covered — the client side belongs to Frontend Quality, the scanning side to Non-Functional Testing — here, the implementation of the defense is measured.

The Untrusted-Data Boundary

The contract is written first: which endpoint accepts which field, in which shape. Whether each request should be accepted or rejected comes from the loan system’s rules and is labeled by hand.

AS1 (assumption): the labels are complete and correct; the counts are relative to this label set and say nothing about a request shape outside it.

// requests.mjs — the loan system's three external endpoints, field contract, and hand-labeled request set
const O = "POST /loans", K = "GET /catalog", N = "POST /member/note";

export const SCHEMA = {
  [O]: { memberId: { t: "str", pattern: /^U-\d{5}$/, required: 1 },
         isbn:     { t: "str", pattern: /^\d{13}$/, required: 1 },
         days:     { t: "num", min: 1, max: 30 } },
  [K]: { q:        { t: "str", max: 64, required: 1 },
         sort:     { t: "str", options: ["title", "author", "year"] },
         page:     { t: "num", min: 1, max: 500 } },
  [N]: { memberId: { t: "str", pattern: /^U-\d{5}$/, required: 1 },
         note:     { t: "str", max: 120, required: 1 } },
};

// [endpoint, body, legitimate?] — the legitimate value is a hand-assigned decision, not measured.
export const REQUESTS = [
  [O, { memberId: "U-10432", isbn: "9789750718533", days: 21 }, 1],
  [O, { memberId: "U-10432", isbn: "9789750718533" }, 1],
  [K, { q: "lost time" }, 1],
  [K, { q: "O'Neill", sort: "author" }, 1],
  [K, { q: "pencil", page: 3 }, 1],
  [N, { memberId: "U-10432", note: "cover worn" }, 1],
  [N, { memberId: "U-77120", note: "5 < 7 pages missing on return" }, 1],
  [O, { memberId: "U-00001", isbn: "9786050000000", days: 1 }, 1],
  [K, { q: "history -- 20th century" }, 1],
  [N, { memberId: "U-31005", note: "book; left with staff" }, 1],
  [O, { memberId: "U-10432", isbn: "978-975-07-1853-3" }, 1],   // hyphenated legitimate form
  [O, { memberId: "u-10432", isbn: "9789750718533" }, 0],       // lowercase
  [O, { memberId: "U-10432", isbn: "978975071853" }, 0],        // twelve digits
  [O, { memberId: "U-10432", isbn: "9789750718533", days: 400 }, 0],
  [O, { memberId: "U-10432", isbn: "9789750718533", days: "21" }, 0],
  [O, { memberId: ["U-10432"], isbn: "9789750718533" }, 0],
  [O, { memberId: "U-10432", isbn: "9789750718533", penalty: 0 }, 0],
  [K, { q: "k".repeat(200) }, 0],
  [K, { q: "pencil", sort: "book.internal_note" }, 0],
  [K, { q: "pencil", page: -2 }, 0],
  [N, { memberId: "U-10432" }, 0],
  [N, { memberId: "U-10432", note: "ok", role: "admin" }, 0],
].map(([endpoint, body, m], i) => ({ no: i + 1, endpoint, body, expected: m ? "accept" : "reject" }));

export const countBy = (e) => REQUESTS.filter((i) => i.expected === e).length;

The out-of-contract requests carry no malicious payload; they carry the wrong shape, the wrong type, and extra fields.

Two Decision Shapes

The allowlist passes nothing except what the contract writes down; the blocklist rejects named bad shapes and passes everything else. The unknownField setting produces a third row.

// validator.mjs — two decision shapes looking at the same request set
import { SCHEMA } from "./requests.mjs";

// Allowlist: nothing passes except what the contract writes down.
// unknownField = "reject" | "pass"  (the second is this lesson's silent setting)
export function allowlist(endpoint, body, { unknownField = "reject" } = {}) {
  const fields = SCHEMA[endpoint];
  if (!fields) return { decision: "reject", reason: "undefined endpoint" };
  const reject = (n) => ({ decision: "reject", reason: n });
  for (const [name, v] of Object.entries(body)) {
    const k = fields[name];
    if (!k) { if (unknownField === "reject") return reject(`unknown field ${name}`); continue; }
    if (k.t === "str") {
      if (typeof v !== "string") return reject(`${name} is not a string`);
      if (k.pattern && !k.pattern.test(v)) return reject(`${name} does not match the pattern`);
      if (k.max && v.length > k.max) return reject(`${name} is too long`);
      if (k.options && !k.options.includes(v)) return reject(`${name} is outside the options`);
    } else {
      if (!Number.isInteger(v)) return reject(`${name} is not an integer`);
      if (v < k.min || v > k.max) return reject(`${name} is out of range`);
    }
  }
  for (const [name, k] of Object.entries(fields))
    if (k.required && body[name] === undefined) return reject(`${name} is missing`);
  return { decision: "accept", reason: "" };
}

// Blocklist: named bad patterns are rejected, everything else passes.
const BANNED = ["<", ">", "'", '"', "--", ";", "..", "\\"];
export function blocklist(endpoint, body) {
  for (const [name, v] of Object.entries(body)) {
    const m = typeof v === "string" ? v : String(v ?? "");
    for (const b of BANNED) if (m.includes(b)) return { decision: "reject", reason: `${name} carries banned token ${b}` };
    if (m.length > 256) return { decision: "reject", reason: `${name} is too long` };
  }
  return { decision: "accept", reason: "" };
}

// False accept: a request that should have been rejected was accepted.
// False reject: a request that should have been accepted was rejected.
export function measure(requests, decide) {
  const r = { falseAccept: 0, falseReject: 0, escaped: [], snagged: [] };
  for (const i of requests) {
    const c = decide(i.endpoint, i.body).decision;
    if (c === i.expected) continue;
    if (i.expected === "reject") { r.falseAccept++; r.escaped.push(i.no); }
    else { r.falseReject++; r.snagged.push(i.no); }
  }
  return r;
}

// compare.mjs — the same 22 requests, three decision shapes; the third is the first two's silent setting
import { REQUESTS, SCHEMA, countBy } from "./requests.mjs";
import { allowlist, blocklist, measure } from "./validator.mjs";

const extra = (i) => Object.keys(i.body).filter((a) => !SCHEMA[i.endpoint][a]);   // field not in the contract
const fieldCount = Object.values(SCHEMA).reduce((t, a) => t + Object.keys(a).length, 0);
console.log(`${Object.keys(SCHEMA).length} endpoints, ${fieldCount} field contract, ${REQUESTS.length} requests: ` +
  `${countBy("accept")} legitimate, ${countBy("reject")} out of contract\n`);

const SHAPE = [
  ["allowlist", (endpoint, b) => allowlist(endpoint, b)],
  ["allowlist (pass)", (endpoint, b) => allowlist(endpoint, b, { unknownField: "pass" })],
  ["blocklist", blocklist],
];

console.log(`${"decision shape".padEnd(22)}${"false accept".padStart(13)}${"false reject".padStart(14)}${"extra-field record".padStart(20)}${"error".padStart(6)}`);
for (const [name, f] of SHAPE) {
  const r = measure(REQUESTS, f);
  let a = 0, err = 0;
  for (const i of REQUESTS) {
    try { if (f(i.endpoint, i.body).decision === "accept" && extra(i).length) a++; } catch { err++; }
  }
  console.log(name.padEnd(22) + String(r.falseAccept).padStart(13) + String(r.falseReject).padStart(14) +
    String(a).padStart(20) + String(err).padStart(6));
  console.log(`  passed out of contract: ${r.escaped.join(", ") || "none"}`);
  console.log(`  legitimate snagged    : ${r.snagged.map((n) => `${n} (${f(REQUESTS[n - 1].endpoint, REQUESTS[n - 1].body).reason})`).join(", ") || "none"}`);
}

3 endpoints, 8 field contract, 22 requests: 11 legitimate, 11 out of contract

decision shape         false accept  false reject  extra-field record error
allowlist                         0             1                   0     0
  passed out of contract: none
  legitimate snagged    : 11 (isbn does not match the pattern)
allowlist (pass)                  2             1                   2     0
  passed out of contract: 17, 22
  legitimate snagged    : 11 (isbn does not match the pattern)
blocklist                        11             4                   2     0
  passed out of contract: 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22
  legitimate snagged    : 4 (q carries banned token '), 7 (note carries banned token <), 9 (q carries banned token --), 10 (note carries banned token ;)

The blocklist passed all eleven out-of-contract requests: the list only recognizes tokens it knows by name, and neither the twelve-digit identifier nor the extra field carries any of them. The same list rejected four legitimate requests — an apostrophe in an author’s name, a less-than sign in a note, two dashes in a search string, a semicolon. The blocklist loses on both columns at once.

AS2 (assumption): the allowlist’s contract was written by looking at the shape of the legitimate requests in the set; the zero false accepts is partly a result of that closeness. The residue is the eleventh request: the hyphenated identifier is legitimate, and the contract does not recognize it. This is the allowlist’s cost — every unanticipated legitimate shape is a false reject.

The middle row is this lesson’s first silent setting. With the unknownField policy set to pass, the thrown-error count is still zero, but two requests are accepted and land in the store carrying two fields the contract does not know: a penalty value and a role value. No code reads these fields today.

Where the Boundary Sits

A loan request passes through three steps: recorded at the endpoint, reserved in the catalog at the service, and written to the store. The first two are side effects that must be undone.

AS3 (assumption): the side effect is measured by step count, not seconds; every step’s duration is treated as constant here.

// boundary.mjs — the same validator, five different boundaries; what is measured is where the side effect stops
import { REQUESTS, SCHEMA } from "./requests.mjs";
import { allowlist } from "./validator.mjs";

const STEPS = ["endpoint", "service", "store"];   // side effect: request record, catalog reservation, write

function run(request, placement, recorded) {
  let sideEffect = 0;
  for (const a of STEPS) {
    if (a === placement && recorded.includes(request.endpoint) &&
        allowlist(request.endpoint, request.body).decision === "reject") return { sideEffect, written: false };
    sideEffect++;
  }
  return { sideEffect, written: true };
}

const ALL = Object.keys(SCHEMA);
const MISSING = ALL.filter((u) => u !== "POST /member/note");   // one endpoint is waiting to be wired in
const invalid = REQUESTS.filter((i) => i.expected === "reject");
const legitimate = REQUESTS.filter((i) => i.expected === "accept");

console.log(`${invalid.length} out-of-contract requests, ${legitimate.length} legitimate requests, ${STEPS.length}-step pipeline`);
console.log(`\n${"placement".padEnd(20)}${"covered endpoints".padStart(18)}${"stopped".padStart(9)}${"invalid written".padStart(17)}${"wasted side effect".padStart(20)}${"legitimate written".padStart(20)}`);

for (const [name, at, recorded] of [["endpoint", "endpoint", ALL], ["endpoint (missing)", "endpoint", MISSING],
    ["service", "service", ALL], ["store", "store", ALL], ["none", "-", []]]) {
  const b = invalid.map((i) => run(i, at, recorded)), m = legitimate.map((i) => run(i, at, recorded));
  console.log(name.padEnd(20) + String(recorded.length).padStart(18) +
    String(b.filter((r) => !r.written).length).padStart(9) +
    String(b.filter((r) => r.written).length).padStart(17) +
    String(b.reduce((t, r) => t + (r.written ? 0 : r.sideEffect), 0)).padStart(20) +
    String(m.filter((r) => r.written).length).padStart(20));
}

const passed = invalid.filter((i) => run(i, "endpoint", MISSING).written).map((i) => i.no);
console.log(`\nendpoint POST /member/note not recorded -> out-of-contract records that reach the store: ${passed.length} (request ${passed.join(", ")})`);
console.log(`errors raised by the application at this placement: 0`);

11 out-of-contract requests, 11 legitimate requests, 3-step pipeline

placement            covered endpoints  stopped  invalid written  wasted side effect  legitimate written
endpoint                             3       11                0                   0                  10
endpoint (missing)                   2        9                2                   0                  10
service                              3       11                0                  11                  10
store                                3       11                0                  22                  10
none                                 0        0               11                   0                  11

endpoint POST /member/note not recorded -> out-of-contract records that reach the store: 2 (request 21, 22)
errors raised by the application at this placement: 0

All three full placements stop the same number of requests; where they diverge is how much work is wasted. Validation at the endpoint stops with zero side effect; at the service it wastes eleven request records; at the store it wastes twenty-two steps — eleven of them a catalog reservation set aside and then put back.

The second row is the second silent setting. Per-endpoint registration means three write sites, and a fourth endpoint needs a fourth write site added by hand. Skip it, and the application still runs, no test breaks, and two out-of-contract records land silently in the store. A single pipeline reduces this risk to one point.

The legitimate written column says one last thing: at every validated placement, only ten of the eleven legitimate requests reach the store. The cost of a false reject is an unrecorded loan transaction.

Output Encoding

A value that passes validation is not safe; it is contract-compliant. A member note accepts free text, because a quote, a comma, and a less-than sign legitimately appear in a reader’s note. This value is written to three places: a markup body, inside an attribute’s quotes, and a comma-separated export line. The measure is one question: does the value come back the same after being placed raw and read back?

// encoding.mjs — legitimate values that passed validation, three separate output contexts
// Measure: is a value the same after being placed raw and read back with the container's own rule?
const VALUES = ["cover worn", "5 < 7 pages missing on return", 'member "urgent" note dropped',
  "shelf 3, floor 2", "first line\nsecond line", 'donation, "used" 5 books'];

const body = (s) => s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
const attribute = (s) => body(s).replace(/"/g, "&quot;").replace(/\n/g, "&#10;");
const quoted = (s) => `"${s.replace(/"/g, '""')}"`;
const decode = (s) => s.replace(/&quot;/g, '"').replace(/&#10;/g, "\n")
  .replace(/&lt;/g, "<").replace(/&gt;/g, ">").replace(/&amp;/g, "&");

const split = (row) => {                     // quote-aware field splitter
  const fields = []; let s = "", q = false;
  for (let i = 0; i < row.length; i++) {
    const c = row[i];
    if (q && c === '"' && row[i + 1] === '"') { s += '"'; i++; }
    else if (c === '"') q = !q;
    else if (!q && (c === "," || c === "\n")) { fields.push(s); s = ""; }
    else s += c;
  }
  return [...fields, s];
};

const CONTEXTS = [
  ["body", (x) => `<note>${x}</note>`, body, (k) => {
    const end = k.indexOf("<", "<note>".length);          // field is read up to the next container marker
    return k.startsWith("<note>") && k.slice(end) === "</note>" ? decode(k.slice("<note>".length, end)) : null;
  }],
  ["attribute", (x) => `<record title="${x}"/>`, attribute,
    (k) => { const m = /title="([^"]*)"/.exec(k); return m ? decode(m[1]) : null; }],
  ["quoted", (x) => `U-10432,${x},2026-03-04`, quoted,
    (k) => { const a = split(k); return a.length === 3 ? a[1] : null; }],
];

const attempt = (encoder) => {
  const list = [];
  for (const [name, wrap, correct, read] of CONTEXTS)
    VALUES.forEach((v, i) => {
      if (read(wrap(encoder(v, correct))) !== v) list.push(`${name}/${i + 1}`);
    });
  return list;
};

console.log(`${VALUES.length} values x ${CONTEXTS.length} contexts = ${VALUES.length * CONTEXTS.length} placements\n`);
console.log(`${"encoder".padEnd(24)}${"not readable back".padStart(18)}  where`);
for (const [name, f] of [["no encoding", (v) => v], ["single encoder (body)", body],
    ["context-aware", (v, correct) => correct(v)]]) {
  const l = attempt(f);
  console.log(`${name.padEnd(24)}${String(l.length).padStart(18)}  ${l.join(" ") || "-"}`);
}

6 values x 3 contexts = 18 placements

encoder                  not readable back  where
no encoding                              7  body/2 attribute/3 attribute/6 quoted/3 quoted/4 quoted/5 quoted/6
single encoder (body)                    7  attribute/3 attribute/6 quoted/2 quoted/3 quoted/4 quoted/5 quoted/6
context-aware                            0  -

The single encoder gained nothing: seven of the eighteen placements still fail to read back, and the list changed. The body context was fixed, but the second value broke in the quoted context — the less-than sign in the note turned into an escape sequence and now sits as raw text in the export line. This is the third silent outcome: the file opens, the lines are read, and the note inside it is no longer the note that was written.

The correct implementation encodes not on input, but on write; a value encoded on input gets encoded twice when it reaches a second context.

Summary

• On the same twenty-two requests, the allowlist produced 0 false accepts and 1 false reject; the blocklist produced 11 false accepts and 4 false rejects. What it passed carried no malicious token, only the wrong shape, the wrong type, and extra fields.
• The allowlist’s cost is its single false reject: a legitimate identifier written with a hyphen was rejected, and a loan transaction went unrecorded.
• Validation stopped at the endpoint wasted 0 steps; at the service, 11; at the store, 22 — the number of requests stopped was the same in all three; only the cost differs.
• A single encoder was not enough for three contexts: 7 of 18 placements failed to read back, and as one context was fixed another broke. Encoding is done separately in every output context.
• All three wrong settings are silent, and the thrown-error total is zero: the unknownField policy sent 2 extra fields to the store, a forgotten write site let 2 out-of-contract records through, and the wrong-context encoder corrupted one line. Per-endpoint registration repeats this at 3 points; a single pipeline stops it at 1.

Next Step

The validation boundary guarantees the shape of the body. What it does not guarantee is how that value is used inside the system. A search string, a sort field, or a notification template that fully conforms to the contract stops being data and becomes part of another language the moment it is placed into a query string, a command line, or a template body by concatenation. The next lesson shows these three places share the same root and measures a single decision: how many different outcomes the same request set produces when a value is concatenated into text versus passed as a separate operand.