Access Control

The previous lesson covered copying and rebuilding an index as a whole; its shared assumption was that everyone with access to the cluster is authorized to see the entire index. In a library catalog this is not true: records closed off by a donation condition, loan history, and acquisition correspondence sit in the same cluster without being open to everyone. This lesson does not measure authentication; it measures the search cluster’s authorization surface.

Authorization Surface

A role here is the product of three things: which indexes are reachable, which document filter applies, and which fields are visible. The setup has three indexes — the open catalog, loan records, the acquisition source — and a portion of the catalog records carries an internal note field only staff can see. The cluster is again modeled in process: a role is nothing more than this triple.

CO21 — the donation batch is 1500 records, comes from a single topic, and its access is restricted by the donation condition. The effect is linear: as the batch grows, the statistical drift below grows with it. CO22 — loan and source records carry the title and topic terms of the book in question. CO23 — a key expires at the sixtieth query of a hundred-query stream.

// cluster/access.mjs — index, document, and field-level access control. Generator and index are
// the same as earlier lessons. Access is modeled IN-PROCESS: there is no authentication, a role is
// only the triple (indexes, document filter, fields).

const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"];
const SPECIAL = {
  child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"],
  novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"],
  travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"],
  essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"],
};
const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"];
const TOPIC = Object.keys(SPECIAL);

function corpus({ count = 4000, seed = 20260731 } = {}) {
  let s = seed % 2147483647;
  const r = () => (s = (s * 48271) % 2147483647) / 2147483647;
  const pick = (a) => a[Math.floor(r() * a.length)], docs = [];
  for (let i = 1; i <= count; i += 1) {
    const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)];
    if (r() < 0.5) terms.push(pick(COMMON));
    for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1)
      terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE));
    docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") });
  }
  return docs;
}

function buildIndex(docs) {
  const postings = new Map(), lengths = new Map();
  for (const d of docs) {
    const t = d.text.split(" "), counts = new Map();
    for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1);
    lengths.set(d.id, t.length);
    for (const [x, n] of counts) {
      if (!postings.has(x)) postings.set(x, []);
      postings.get(x).push([d.id, n]);                       // posting: id + frequency
    }
  }
  const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1);
  return { postings, lengths, N: docs.length, avg };
}

// Scoring: term frequency, inverse document frequency, and length. The score comes from
// WHICHEVER INDEX PROVIDES THE STATISTIC; this lesson's second measurement compares exactly that choice.
function search(ix, terms, k) {
  const N = ix.N, avg = ix.avg, scores = new Map();
  let scanned = 0;
  for (const t of terms) {
    const g = ix.postings.get(t) ?? [], df = g.length;
    const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5));
    for (const [id, tf] of g) {
      scanned += 1;
      const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg);
      scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm);
    }
  }
  const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k);
  return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size };
}


const K = 10, STREAM = 100, LIFETIME = 60, RESTRICTED = 1500;
const QUERY = ["story", "lighthouse", "book", "donation"];   // a story with a donation record, its title mentions a lighthouse
const pad = (x, n) => String(x).padStart(n);

// Catalog: 4000 open records + a single-topic batch restricted by donation terms (separate
// seed, ids moved past 4000). Loans and the source are separate indexes.
const OPEN = corpus({ count: 4000 });
const CATALOG = [...OPEN, ...corpus({ count: 20000, seed: 20260801 }).filter((d) => d.topic === "story")
  .slice(0, RESTRICTED).map((d, i) => ({ ...d, id: 4001 + i, restricted: true }))];
const LOANS = corpus({ count: 900, seed: 20260901 }).map((d) => ({ ...d, id: 20000 + d.id }));
const SOURCE = corpus({ count: 400, seed: 20261001 }).map((d) => ({ ...d, id: 30000 + d.id }));

// Internal note field: acquisition correspondence, its own vocabulary and seed.
const NOTE_WORDS = ["donation", "condition", "clause", "acquisition", "purchase", "buying", "correspondence", "invoice", "testament"];
let n = 20261101;
const rr = () => (n = (n * 48271) % 2147483647) / 2147483647;
for (const d of CATALOG) if (rr() < 0.35)
  d.note = Array.from({ length: 3 + Math.floor(rr() * 4) }, () => NOTE_WORDS[Math.floor(rr() * 9)]).join(" ");

// An index for one (document set, field) pair; the "note" field has only documents with a note.
const build = (docs, field) => buildIndex(field === "note"
  ? docs.filter((d) => d.note).map((d) => ({ id: d.id, text: d.note })) : docs);

// A query is scored on every (index, field) pair and scores are summed; filter is document-level access control.
// this is the RAW match BEFORE filtering: if this is what gets written to the counter, restricted documents leak out.
function ask(indexes, terms, filter = () => true) {
  const scores = new Map();
  let scanned = 0, raw = 0;
  for (const ix of indexes) {
    const c = search(ix, terms, Infinity);
    scanned += c.scanned; raw += c.matched;
    for (const a of c.candidates) if (filter(a.id)) scores.set(a.id, (scores.get(a.id) ?? 0) + a.p);
  }
  const y = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).map(([id]) => id);
  return { all: y, top: y.slice(0, K), matched: y.length, scanned, raw };
}
const shared = (a, b) => a.filter((x) => b.includes(x)).length;

console.log(`catalog ${CATALOG.length} documents (4000 open + ${RESTRICTED} restricted by donation terms), loans ${LOANS.length}, source`);
console.log(`${SOURCE.length}, note field on ${CATALOG.filter((d) => d.note).length} documents. Seed 20260731 and 20261101. Query "story lighthouse book donation".\n`);
console.log("role / key             | indexes | docs | fields      | answered | scanned | matched | top 3 result");
console.log("------------------------|---------|------|-------------|----------|---------|---------|------------------");
for (const [name, set, allow, fields, answered] of [
  ["reader", [CATALOG], (d) => !d.restricted, ["text"], STREAM],
  ["staff", [CATALOG, LOANS], () => true, ["text", "note"], STREAM],
  ["staff, note off", [CATALOG, LOANS], () => true, ["text"], STREAM],
  ["unscoped key", [CATALOG, LOANS, SOURCE], () => true, ["text", "note"], STREAM],
  ["expired key", [], () => true, [], LIFETIME],
]) {
  const c = ask(set.flatMap((x) => fields.map((f) => build(x.filter(allow), f))), QUERY);
  console.log(`${name.padEnd(23)} | ${pad(set.length, 7)} | ${pad(set.reduce((t, x) => t + x.filter(allow).length, 0), 4)} | ` +
    `${(fields.join(", ") || "-").padEnd(11)} | ${pad(`${answered}/${STREAM}`, 8)} | ${pad(c.scanned, 7)} | ${pad(c.matched, 7)} | ${c.top.slice(0, 3).join(" ") || "-"}`);
}

const before = ask([build(OPEN, "text")], QUERY);
const after = ask([build(CATALOG, "text")], QUERY, (id) => id <= 4000);
console.log("\ndocument-level filtering (reader role, two orderings, same 4000 open documents):");
console.log("ordering        | index providing statistics | matched | reported counter | top 10 shared | documents that swapped places");
for (const [name, set, c, counter] of [["filter first", "restricted-out 4000", before, before.matched],
  ["filter after", "full catalog 5500", after, after.raw]]) {
  console.log(`${name.padEnd(15)} | ${set.padEnd(27)} | ${pad(c.matched, 7)} | ${pad(counter, 16)} | ` +
    `${pad(shared(c.top, before.top) + "/10", 12)} | ${pad(c.all.filter((x, i) => before.all[i] !== x).length, 22)}`);
}
console.log(`top 10, filter first: ${before.top.join(" ")}`);
console.log(`top 10, filter after: ${after.top.join(" ")}`);

console.log(`\nrun-independent: an unscoped key gets the same triple as the most privileged role and` +
  ` an expired key leaves ${STREAM - LIFETIME} of ${STREAM} queries unanswered.`);

catalog 5500 documents (4000 open + 1500 restricted by donation terms), loans 900, source
400, note field on 1931 documents. Seed 20260731 and 20261101. Query "story lighthouse book donation".

role / key             | indexes | docs | fields      | answered | scanned | matched | top 3 result
------------------------|---------|------|-------------|----------|---------|---------|------------------
reader                  |       1 | 4000 | text        |  100/100 |    2998 |    2463 | 1613 3589 2806
staff                   |       2 | 6400 | text, note  |  100/100 |    6882 |    4743 | 4551 4589 5265
staff, note off         |       2 | 6400 | text        |  100/100 |    6130 |    4542 | 20332 20557 20765
unscoped key            |       3 | 6800 | text, note  |  100/100 |    7175 |    4986 | 4551 4589 5265
expired key             |       0 |    0 | -           |   60/100 |       0 |       0 | -

document-level filtering (reader role, two orderings, same 4000 open documents):
ordering        | index providing statistics | matched | reported counter | top 10 shared | documents that swapped places
filter first    | restricted-out 4000         |    2463 |             2463 |        10/10 |                      0
filter after    | full catalog 5500           |    2463 |             3963 |         7/10 |                   1433
top 10, filter first: 1613 3589 2806 493 1188 1100 3214 308 476 1012
top 10, filter after: 1613 3589 2806 493 1188 3127 1100 3214 3695 3561

run-independent: an unscoped key gets the same triple as the most privileged role and an expired key leaves 40 of 100 queries unanswered.

The Same Query at Three Levels

The first table runs a single question under five authorizations. The reader reaches one index and 4000 open records, getting 2463 matches; staff reaches two indexes and 6400 records, matched rises to 4743, and all three top results change. What grows the set is not only permission but the other indexes that permission brings into the same ranking.

The third row isolates the field level: same staff role, same two indexes, same 6400 records, the only difference is closing the note field. Matched drops to 4542 — 201 records matched only through the note field — and the top three shift from the catalog to loan records. Closing a field does not just remove documents from the list; it also removes one of the scoring components for the ones that remain.

The fourth row is the default-open setup: a key with no written scope reaches three indexes and 6800 records, gaining a breadth granted to no role, and pushing scanned entries to 7175. The fifth shows lifetime: an expired key leaves forty of a hundred queries unanswered.

Filtering Leaking into Scoring

The second table is this lesson’s real measurement. The reader role can be implemented two ways: the restricted 1500 records are either kept out of the index from the start (filter first), or scored over the full catalog and then dropped from the list (filter after). Both return the identical 2463 documents — the set is exactly the same.

The order is not the same. Only seven of the top ten results are shared: 3127 appears at rank six, 3695 and 3561 at nine and ten, while 308, 476, and 1012 drop off the list; in the deep ranking, 1433 of the 2463 documents have swapped places. The reason is a single number: inverse document frequency. In the filter-after setup, the term “story” is counted across 5500 documents, and because the restricted batch comes from a single topic, that term’s distinctiveness drops. Documents the reader cannot see are deciding the order of the documents they can see.

The same setup leaks information too: the reported counter is 2463 under filter-first, 3963 under filter-after, and the 1500-record gap between them is the exact size of the restricted batch — the counter is telling the reader the count of records they cannot see. Whether filtering is applied before or after scoring is not a performance detail: it decides both the order and the leak.

Summary

• Authorization makes three separate cuts: the reader got 2463 matches with 1 index and 4000 records, staff 4743 with 2 indexes and 6400 records; all three top results changed entirely. Closing the note field dropped matched to 4542.
• A key with no written scope reached three indexes and 6800 records, pushing scanned entries from 2998 to 7175; an expired key left forty of a hundred queries unanswered.
• Document-level filtering changed the order without changing the set: across the same 2463 documents, only seven of the top ten were shared, 1433 documents swapped places, and the counter leaked the 1500 restricted records.

Course Wrap-Up

Lesson	Changed Set	Changed Order	Cost
Engine vs. Database	LIKE 68, index 12	plan order, id order	124,863 → 1,076 bytes
Inverted Index	position 23 → 10	posting list in id order	intersection 19,622 → 470
Analyzer Chain	80 / 122 / 463	all five of the top five changed	187,780 → 146,080 bytes
Mappings and Field Types	child 93 vs. 0	all ten of the top ten shifted	topic 8,203 → 11,686 bytes
Dynamic and Explicit Mapping	conflict dropped 111 documents	a dropped document is invisible in the order	metadata 1,595 → 3,022 bytes
Document Lifecycle	#10 moved from set to set	from rank 7 to rank 509	119,722 → 136,322 bytes
Query Language Structure	4,211 / 132 / 71	from rank 2,023 to the top ten	21,303 → 7,576 entries
Term and Phrase Queries	757 → 144	adjacent matches 1 → 10	216,480 → 330,578 bytes
Context Queries	must_not 512, should 0	nine of the top ten changed	entries 2,678 → 4,638
Filtering and Query Distinction	1,138 in both contexts	average 173 ranks	score computations 2,276 → 1,138
Relevance Scoring	precision 0.353 → 0.640	component absence, 499 ranks	567 results for recall
Score Tuning	boost 325, field choice 199	all ten of the top ten changed	698 → 265 entries
Aggregation Queries	208 / 168 / 1 bucket	order is bucket order	1,045 buckets, 25,080 bytes
Highlighting and Pagination	12 of 60 matches excluded	order unchanged	page 20, 800 candidates
Semantic and Vector Search	383 at a 0.5 threshold	six of the top ten from a synonym	153,600 bytes
Node Roles	delayed state rejected 800 records	the top five changed entirely	coordinating share 0.8% → 11.4%
Shards and Replicas	515 → 379 on node loss	top ten shared 10.00 → 4.75	candidates 10 → 160
Split Brain Problem	210 records lost in the minority	the two halves gave two lists	600 → 1590 messages
Bulk Indexing Optimization	600 missing with refresh off	top ten 6/10	4000 segments, 24,000 lookups
Segment Merging	unchanged, 20,637	33 of 50 ranks differed	3,111,596 → 2,498,875 bytes
Index Lifecycle	10,262 → 12,274	two of the top ten from a deleted day	entries 19,476 → 1,993
Snapshot and Restore	9,286 → 16,517	50 ranks changed	10,182,428 → 3,000,334 bytes
Access Control	reader 2463, staff 4743	top ten shared 7/10	entries 2,998 → 7,175

The table’s three columns are this course’s rule: a search decision is defended only by its effect on the set and the order. “A more relevant result” is not a decision; an analyzer, a boost, a shard count, or an authorization rule is defended only by stating which documents return, in what order, and at what cost.

This is the seventh and last course of M17. Data Modeling and Relational Theory built how data gets modeled, SQL Fundamentals how that model gets queried, Advanced SQL why the same query slows down, Relational Database Administration how a running system gets operated, Non-Relational Data Models what another path gains and gives up, In-Memory Stores and Caching Systems how a memory budget gets spent. This course closed the last gap: reaching data whose key is unknown by going through the text.

What comes after this is a habit of measurement, and its starting point is your own data’s access pattern: do questions arrive by key or through the text, how many documents return, in what order, and who decides that order. That was the only thing these twenty-three lessons did.