Output Encoding

The previous lesson counted entry points and showed that validation derived from a declaration closes fifteen points on its own. A value that passes validation now conforms to the contract: the subscriber note does not exceed two hundred characters, the tariff is in the set, the period is formatted. Validation says nothing about what happens to this value when it leaves the system.

This lesson counts exit points. The same subscriber note is written into a page body, into an attribute’s quotes, into a script variable, and into an address component. All four take the same value; all four read that value by a different rule. What is measured is whether the context is chosen right, and how many exit points have to make that choice on their own.

Four Output Contexts

A context has three parts: the container that wraps the value, the container’s own encoder, and the container’s own reader. The measure is a single question — after the value is placed in the container and read back by the container’s rule, does the same value come out?

IO5 (assumption): the read-back represents the rule of the party that parses the container. A placement counts as broken when the read-back does not give the value back exactly; how harmful the breakage is is not measured.

// context.mjs — four output contexts: the container, its own encoder, and its own reader
// Measure: after the value is placed in the container and read back by the container's rule, is it the same?
export const VALUES = ["cracked meter cover", 'subscriber "urgent" note dropped', "shelf 3 & floor 2",
  "reading < 5 m3", "first line\nsecond line", "percent 5 increase %"];

const body = (s) => s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
export const ENCODER = {
  body,
  attribute: (s) => body(s).replace(/"/g, "&quot;").replace(/\n/g, "&#10;"),
  script: (s) => JSON.stringify(s).slice(1, -1),
  address: (s) => encodeURIComponent(s),
};

const decode = (s) => s.replace(/&quot;/g, '"').replace(/&#10;/g, "\n")
  .replace(/&lt;/g, "<").replace(/&gt;/g, ">").replace(/&amp;/g, "&");

export const CONTEXT = {
  body: [(x) => `<note>${x}</note>`, (k) => {
    const i = k.indexOf("<", 6);
    return k.startsWith("<note>") && k.slice(i) === "</note>" ? decode(k.slice(6, i)) : null;
  }],
  attribute: [(x) => `<record note="${x}"/>`,
    (k) => { const m = /note="([^"]*)"/.exec(k); return m ? decode(m[1]) : null; }],
  script: [(x) => `const note = "${x}";`, (k) => {
    try { return JSON.parse(k.slice(k.indexOf('"'), k.lastIndexOf('"') + 1)); } catch { return null; }
  }],
  address: [(x) => `/report?note=${x}&format=summary`, (k) => {
    try {
      const a = k.split("?")[1].split("&").find((p) => p.startsWith("note="));
      return a === undefined ? null : decodeURIComponent(a.slice(5));
    } catch { return null; }
  }],
};

// Single placement: value is encoded, placed in the container, read back.
export const readsBack = (context, encoderName, value) => {
  const [container, read] = CONTEXT[context];
  return read(container(ENCODER[encoderName](value))) === value;
};

All six values are legitimate subscriber and field notes: a quote, an ampersand, a less-than sign, a line break, a percent sign. None of them is attack input; all of them are texts a field worker could write.

When the Context Is Chosen Wrong

Four contexts and four encoders give sixteen pairings. Four of them are the correct pairing; the remaining twelve are the context chosen wrong.

// choice.mjs — four contexts x four encoders: how many cases break when the context is chosen wrong
import { VALUES, ENCODER, CONTEXT, readsBack } from "./context.mjs";

const C = Object.keys(CONTEXT), E = Object.keys(ENCODER);
console.log(`${VALUES.length} values, ${C.length} contexts, ${E.length} encoders, ` +
  `${C.length * E.length} pairings, ${VALUES.length * C.length * E.length} placements\n`);
console.log("number of broken values (row: context, column: chosen encoder)");
console.log("context".padEnd(12) + E.map((e) => e.padStart(11)).join(""));
let right = 0, wrong = 0;
for (const c of C) {
  const row = E.map((e) => VALUES.filter((v) => !readsBack(c, e, v)).length);
  E.forEach((e, i) => (e === c ? (right += row[i]) : (wrong += row[i])));
  console.log(c.padEnd(12) + row.map((n) => String(n).padStart(11)).join(""));
}
console.log(`\nbroken in the ${C.length} pairings with the right context: ${right}`);
console.log(`broken in the ${C.length * E.length - C.length} pairings with the wrong context: ${wrong}`);

6 values, 4 contexts, 4 encoders, 16 pairings, 96 placements

number of broken values (row: context, column: chosen encoder)
context            body  attribute     script    address
body                  0          0          3          6
attribute             1          0          2          6
script                4          4          0          6
address               3          5          4          0

broken in the 4 pairings with the right context: 0
broken in the 12 pairings with the wrong context: 44

The diagonal is zero: every context returns all six values with its own encoder. Of the seventy-two placements off the diagonal, forty-four break.

The last column is the most instructive spot in this table. The address encoder breaks all six of the six values, no matter which context it is placed in. This is not under-escaping but over-escaping: spaces and punctuation turn into percent-sign sequences, and the note read on the page is no longer the note that was written. The wrong context produces error in both directions at once; the assumption that a strict encoder is harmless does not hold up under measurement.

The script row is the second point. The body encoder breaks four values in the script context, because the quote and the line break are not special under the body rule — the three characters special there are different ones. It is not enough for the encoder to be “safer”; it must be the same rule as the container’s rule.

Exit Point Inventory

The context choice is not made in one place; it is made separately at every exit point. The exit points of the regional measurement network are counted: the subscriber portal, invoicing, the field application, and the report generator.

IO6 (assumption): the real context of every exit point is known and correctly recorded in the inventory. What is measured is whether the encoder chosen at the call site matches that context.

// exit-points.mjs — exit point inventory (fictional model) and two writing forms
import { VALUES, ENCODER, CONTEXT } from "./context.mjs";

const CODEC = { ...ENCODER, raw: (s) => s };
const roundTrips = (c, e, v) => { const [container, read] = CONTEXT[c]; return read(container(CODEC[e](v))) === v; };

// [service, exit point, real context, encoder chosen at the call site]
const EXIT = [
  ["subscriber portal", "profile heading", "body", "body"],
  ["subscriber portal", "note list", "body", "body"],
  ["subscriber portal", "form preview", "attribute", "body"],
  ["subscriber portal", "search link", "address", "raw"],
  ["subscriber portal", "status script", "script", "body"],
  ["invoice", "statement line", "body", "body"],
  ["invoice", "print heading", "attribute", "attribute"],
  ["invoice", "download link", "address", "address"],
  ["field application", "work order card", "body", "body"],
  ["field application", "map label", "attribute", "raw"],
  ["field application", "server call", "address", "address"],
  ["report generator", "summary body", "body", "body"],
  ["report generator", "link generator", "address", "body"],
  ["report generator", "embedded data script", "script", "script"],
];
// New exit points: no one chose an encoder at the call site.
const NEW_EXITS = [["field application", "photo heading", "attribute", "raw"],
  ["report generator", "external link", "address", "raw"], ["subscriber portal", "notification script", "script", "raw"]];

const measure = (list, choice) => {
  let brokenPoints = 0, brokenPlacements = 0;
  for (const [, , context, byHand] of list) {
    const n = VALUES.filter((v) => !roundTrips(context, choice === "central" ? context : byHand, v)).length;
    if (n) brokenPoints++;
    brokenPlacements += n;
  }
  return [brokenPoints, brokenPlacements];
};

console.log(`${"stage".padEnd(28)}${"points".padStart(6)}${"placements".padStart(12)}` +
  `${"scattered points".padStart(18)}${"scattered placements".padStart(22)}${"central".padStart(9)}`);
for (const [name, l] of [["current inventory", EXIT], ["three new points added", [...EXIT, ...NEW_EXITS]]]) {
  const [dp, dpl] = measure(l, "scattered"), [, cpl] = measure(l, "central");
  console.log(name.padEnd(28) + String(l.length).padStart(6) + String(l.length * VALUES.length).padStart(12) +
    String(dp).padStart(18) + String(dpl).padStart(22) + String(cpl).padStart(9));
}
const count = (c) => EXIT.filter((x) => x[2] === c).length;
console.log("\ncontext distribution: " + Object.keys(CONTEXT).map((c) => `${c} ${count(c)}`).join("  "));
console.log("call-site choice: right " + EXIT.filter((x) => x[2] === x[3]).length +
  ", wrong " + EXIT.filter((x) => x[3] !== "raw" && x[2] !== x[3]).length +
  ", none at all " + EXIT.filter((x) => x[3] === "raw").length);

stage                       points  placements  scattered points  scattered placements  central
current inventory               14          84                 5                    11        0
three new points added          17         102                 8                    16        0

context distribution: body 5  attribute 3  script 2  address 4
call-site choice: right 9, wrong 3, none at all 2

Five of the fourteen exit points write broken output: three have the wrong encoder chosen, two have no encoder at all. Under the central form, breakage is zero, because the encoder is not chosen at the call site — it derives from the point’s context declaration. The difference between the two is not a difference of knowledge but a difference in where the decision is made: nine call sites chose right, five did not; under the central form there is nothing to choose.

The second row shows how coverage behaves at a new point. When three new exit points are added, the number of broken points under the scattered form rises from five to eight — every new exit point is directly a candidate for breakage, because it has no encoder unless someone writes one. Under the central form, the same three points stay at zero.

Nested Context

Not every point in the inventory writes into a single context. When a piece of script is placed inside an attribute’s quotes, the value sits inside two containers and is read by two rules.

// nested.mjs — a second context embedded inside a first: script context inside an attribute
import { VALUES, ENCODER } from "./context.mjs";

const container = (x) => `<record op="const note = &quot;${x}&quot;;"/>`;
const decode = (s) => s.replace(/&quot;/g, '"').replace(/&#10;/g, "\n")
  .replace(/&lt;/g, "<").replace(/&gt;/g, ">").replace(/&amp;/g, "&");
const read = (k) => {                    // first the outer container, then the inner container is read by its own rule
  const m = /op="([^"]*)"/.exec(k);
  if (!m) return null;
  const code = decode(m[1]);
  try { return JSON.parse(code.slice(code.indexOf('"'), code.lastIndexOf('"') + 1)); } catch { return null; }
};

console.log(`${VALUES.length} values, two nested contexts: script inside an attribute\n`);
console.log(`${"encoder choice".padEnd(32)}${"not read back".padStart(16)}`);
for (const [name, f] of [["outer context only (attribute)", ENCODER.attribute],
    ["inner context only (script)", ENCODER.script],
    ["outer then inner", (d) => ENCODER.script(ENCODER.attribute(d))],
    ["inner then outer", (d) => ENCODER.attribute(ENCODER.script(d))]])
  console.log(name.padEnd(32) + String(VALUES.filter((d) => read(container(f(d))) !== d).length).padStart(16));

6 values, two nested contexts: script inside an attribute

encoder choice                     not read back
outer context only (attribute)                 2
inner context only (script)                    1
outer then inner                               2
inner then outer                               0

A single encoder is not enough in either direction: the outer context’s breaks two values, the inner context’s breaks one value. When the two encoders are used together, order decides the result. Going from outside in, two values stay broken; going from inside out, all six are read back correctly.

The coverage consequence of this is direct. A declaration that holds a single context per exit point cannot express a nested point; those points stay uncovered by the central pattern, and the decision returns to the call site. A declaration that wants to keep coverage fixed must carry not a context but a context sequence.

The context distribution line says something separate: of the fourteen points, five are in the body context, four in address, three in attribute, two in script. All four contexts are in use, and none of them dominates. It is impossible for a codebase working with a single encoder to give the right result across this distribution; the number of off-diagonal cells comes directly from this distribution.

The Second Place the Pattern Falls Short

Even when the central pattern chooses the right encoder at every point, it rests on one assumption: the value coming from storage is raw. When that assumption is lifted, the pattern breaks for a reason outside its control.

// storage.mjs — what the central pattern does when the value enters storage already encoded
import { VALUES, ENCODER, CONTEXT } from "./context.mjs";

const C = Object.keys(CONTEXT);
const place = (c, v) => { const [container, read] = CONTEXT[c]; return read(container(ENCODER[c](v))); };
console.log(`${VALUES.length} values x ${C.length} contexts; the central pattern chooses the right encoder at every point\n`);
console.log(`${"storage decision".padEnd(28)}${"broken".padStart(9)}  per context`);
for (const [name, store] of [["raw value stored", (v) => v],
    ["encoded once on the way in", ENCODER.body]]) {
  const row = C.map((c) => VALUES.filter((v) => place(c, store(v)) !== v).length);
  console.log(name.padEnd(28) + String(row.reduce((a, n) => a + n, 0)).padStart(9) +
    "  " + C.map((c, i) => `${c} ${row[i]}`).join("  "));
}

6 values x 4 contexts; the central pattern chooses the right encoder at every point

storage decision               broken  per context
raw value stored                    0  body 0  attribute 0  script 0  address 0
encoded once on the way in          8  body 2  attribute 2  script 2  address 2

When the value is encoded once on the way in and stored that way, all four contexts each break two values. The encoder choice is right, the container is right, the reader is right; what breaks is the transformation the value goes through on the way into storage. And the two notes carrying the ampersand and the less-than sign arrive at the reader as escape sequences, because they get encoded a second time on the way out.

The coverage consequence of this breakage is heavier than the first one. A wrong choice at the call site breaks one point; a wrong storage decision breaks every point that reads that value — all fourteen of them in this inventory at once. The place for encoding is the output, and the reason is not a principle but a count: at output the decision is made per context, at input a single decision is imposed on every context.

Summary

• Four contexts and four encoders gave sixteen pairings. None of the 24 placements in the correct pairings broke; 44 of the 72 placements in the twelve pairings with the wrong context could not be read back.
• The wrong context breaks in both directions at once: the address encoder broke all six of the six values in the other three contexts — not under-escaping but over-escaping. The body encoder broke four values in the script context.
• Of the fourteen exit points, 9 had the right encoder chosen, 3 had the wrong one, and 2 had none chosen at all; under the scattered form, 5 points and 11 placements are broken. When the encoder is derived from the context declaration, breakage is 0.
• When three new exit points were added, the number of broken points under the scattered form rose from 5 to 8 and stayed at 0 under the central form. A new exit point is broken from the start unless someone writes its encoder.
• When the value is encoded once on the way in and stored that way, 8 placements broke even though the central pattern worked correctly. A wrong choice at the call site breaks a single point; a wrong storage decision breaks all 14 points that read that value at once.
• In the nested context, a single encoder was not enough: the outer context’s broke 2 values, the inner context’s broke 1. Order decides the outcome — inside out gives 0, outside in gives 2. A declaration that holds a single context per point cannot cover these points.

Next Step

Output encoding targets the container the value will be read from. Placing the value into a query text somewhere inside the system is a separate problem: there, the container is not a document but a query language, and the party reading that language has to separate the data from the command within the same text. The next lesson counts query points and measures a single question: how many query values pass through an interface that carries them as a separate operand, how many query values are appended into the text itself, and which of these two forms closes the point on its own when a new query is added.