--- title: 'Search Engines and Text Retrieval' source: 'https://academia.sh/en/courses/search-engines' language: en updated: '2026-08-23T07:00:49+00:00' license: 'CC BY-SA 4.0' lessons: 23 outcomes: - 'Explains how the inverted index and the analyzer chain affect a search result' - 'Tunes relevance and filtering behavior through the query language' - 'Makes cluster, shard, and index lifecycle decisions' --- # Search Engines and Text Retrieval Reaching data whose key is unknown by going through the text: the inverted index's term-to-document mapping, the analyzer chain deciding the result through tokenization, normalization, and filters, mappings and field types with the text and keyword distinction, the query language's leaf and compound structure, term and phrase queries, context conditions, the filtering and query distinction, term-frequency-based relevance scoring and its tuning, aggregation queries, highlighting and deep pagination, semantic search, and node roles, shard and replica settings, segment merging, the index lifecycle, snapshots, and access control. ## Indexing and Analysis Text becoming searchable: which question separates a search engine from a relational database, the construction and cost of the inverted index mapping term to document, the analyzer chain of tokenization, normalization, and filters deciding the result, the distinction between text and keyword field types, the schema-explosion risk of dynamic mapping, and the mechanics of a document being added, updated, and deleted. ### Search Engine vs. Database Source: https://academia.sh/en/courses/search-engines/search-engine-vs-database Reaching a document through text asking for a different structure than reaching one through a known key: the distinction between key-known access and word-known access, the same catalog question answered by a relational LIKE scan and by an inverted index mapping term to document, counting the records and entries each path scans, the document set each returns, the documents each misses, and the effect of case and inflection on the set. The In-Memory Stores and Caching Systems course closed on one assumption shared by every access it covered: the key of what was being searched for **was known**. A session id, a counter name, a sorted set's name was given, and the store found that name and returned the value. The relational side worked the same way — the index was built on a key, and the query reached that key through an equality, range, or prefix condition. Questions in a library catalog do not arrive that way. A reader does not remember the book's identity, only two words that appear in its summary: "books whose summary contains deniz (sea) and çocuk (child)." This question does not reach toward a key, it reaches **into** the text. This course builds that question, and every lesson counts the same two things: how a decision changed **which documents** came back, how it changed **which order** they came back in, and what that decision **cost**. A more relevant result does not count as a decision in this course. #### Going by Word, Not by Key A relational index is a structure sorted on the **entire** value. It answers equality, range, and prefix questions through that sort order; once the part being searched for is not at the start of the value, the sort order has nothing left to say. `LIKE '%deniz%'` is exactly this case: because the match can sit anywhere in the text, the index is skipped and every row is read. Index theory was built in M17/K04; the only point that matters here is that the index cannot help with this question. The inverted index does the opposite. It does not go from document to its fields, it goes from **term to documents**: the text is split into words, and for each word the ids of the documents it appears in are stored. When the question arrives, no table is read, only the lists for two terms. The structure's name and its byte accounting are built in the next lesson; this lesson only compares what the two paths answer to **the same question**. #### The Same Question, Two Paths The measurement uses a corpus generated from a library catalog, and this corpus stays the same across all three topics in the course. **IA1:** the corpus carries 600 book records, seed 20250317; each record holds a title, summary, subject tags, author, year, language, and shelf code; the summary averages 25 words and the word pool is limited — this keeps the term count smaller than a real catalog would, without changing the ratios. **IA2:** on the relational path, the number of rows scanned is the table's row count; the query plan's `SCAN` line confirms this, and the bytes read are the summary field's total bytes. **IA3:** an index entry is counted as a 4-byte document id. **IA4:** the list of inflected forms is given by hand; the classification rests on this ten-form table. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/scan.mjs — the same question in two structures: a relational LIKE scan versus an // inverted index mapping term to document. Scanned unit, returned set, extra and missed docs are counted. import { DatabaseSync } from "node:sqlite"; import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; const db = new DatabaseSync(":memory:"); db.exec("CREATE TABLE book(id INTEGER PRIMARY KEY, title TEXT, summary TEXT)"); const insert = db.prepare("INSERT INTO book VALUES (?, ?, ?)"); for (const b of corpus) insert.run(b.id, b.title, b.summary); db.exec("CREATE INDEX book_summary ON book(summary)"); // B-tree index: on the summary field const plan = db.prepare("EXPLAIN QUERY PLAN SELECT id FROM book WHERE summary LIKE ?").all("%deniz%"); const like = (...s) => db.prepare(`SELECT id FROM book WHERE ${s.map(() => "summary LIKE ?").join(" AND ")}`) .all(...s.map((x) => `%${x}%`)).map((r) => r.id); // --- inverted index: tokenization and Turkish lowercasing; stemming is not part of this lesson --- const tokenize = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); const index = new Map(); for (const b of corpus) { for (const t of new Set(tokenize(b.summary))) { if (!index.has(t)) index.set(t, []); index.get(t).push(b.id); } } const search = (...terms) => { const lists = terms.map((t) => index.get(t) ?? []); const shortest = lists.reduce((a, l) => (l.length < a.length ? l : a)); return { results: shortest.filter((id) => lists.every((l) => l.includes(id))), postings: lists.reduce((t, l) => t + l.length, 0) }; }; const bytes = corpus.reduce((t, b) => t + Buffer.byteLength(b.summary), 0); console.log(`seed ${SEED}; corpus ${DOC_COUNT} docs, summary field ${bytes} bytes, ${index.size} terms in index`); console.log(`query plan: ${plan[0].detail} — every row is still read even with the B-tree index`); const query = ["deniz", "çocuk"]; const likeIds = like(...query), indexResult = search(...query); const INFLECTED_FORMS = new Set(["denizin", "denizler", "denize", "denizden", "denizde", "çocuğun", "çocuklar", "çocuğa", "çocuktan", "çocukta"]); // IA4: known inflection table const extra = likeIds.filter((id) => !indexResult.results.includes(id)); const inflected = extra.filter((id) => tokenize(corpus[id - 1].summary).some((t) => INFLECTED_FORMS.has(t))); console.log(`\nquery: books with "${query.join('" and "')}" in the summary`); console.log(`${"path".padEnd(16)}${"scanned unit".padStart(15)}${"bytes read".padStart(13)}` + `${"docs returned".padStart(15)}${"first four ids".padStart(20)}`); console.log("relational LIKE".padEnd(16) + `${DOC_COUNT} rows`.padStart(15) + String(bytes).padStart(13) + String(likeIds.length).padStart(15) + likeIds.slice(0, 4).join(",").padStart(20)); console.log("inverted index".padEnd(16) + `${indexResult.postings} postings`.padStart(15) + String(indexResult.postings * 4).padStart(13) + String(indexResult.results.length).padStart(15) + indexResult.results.slice(0, 4).join(",").padStart(20)); console.log(`\nLIKE brings back ${extra.length} extra docs, of two kinds:`); const example = (cond) => extra.find((id) => cond(tokenize(corpus[id - 1].summary))); for (const [label, id] of [["inflected form (the index misses it)", example((t) => t.some((x) => INFLECTED_FORMS.has(x)))], ["match inside another word", example((t) => !t.some((x) => INFLECTED_FORMS.has(x)))]]) { const matching = tokenize(corpus[id - 1].summary).filter((t) => t.includes("deniz") || t.includes("çocuk")); console.log(` ${label}: #${id} -> ${[...new Set(matching)].join(", ")}`); } console.log(` ${inflected.length} docs of the first kind, ${extra.length - inflected.length} docs of the second kind`); for (const variant of ["istanbul", "İstanbul"]) { const l = like(variant), d = search(variant.toLocaleLowerCase("tr")); console.log(`\nquery "${variant}": LIKE ${l.length} docs (${DOC_COUNT} rows scanned), ` + `inverted index ${d.results.length} docs (${d.postings} postings scanned)`); } ``` ``` seed 20250317; corpus 600 docs, summary field 124863 bytes, 170 terms in index query plan: SCAN book USING COVERING INDEX book_summary — every row is still read even with the B-tree index query: books with "deniz" and "çocuk" in the summary path scanned unit bytes read docs returned first four ids relational LIKE 600 rows 124863 68 237,333,364,490 inverted index 269 postings 1076 12 8,77,107,118 LIKE brings back 59 extra docs, of two kinds: inflected form (the index misses it): #237 -> denizler, çocuklar match inside another word: #14 -> deniz, çocukluk 43 docs of the first kind, 16 docs of the second kind query "istanbul": LIKE 0 docs (600 rows scanned), inverted index 73 docs (73 postings scanned) query "İstanbul": LIKE 73 docs (600 rows scanned), inverted index 73 docs (73 postings scanned) ``` #### Where the Set Difference Comes From The two paths answer the same question differently: LIKE returns 68 documents, the inverted index 12, and 9 are common to both. The 59 that LIKE brings back on top do not reduce to one cause — they have two separate sources, and the output counts them separately. The first source is **inflection**, and it covers 43 documents. Record `#237`'s summary carries the forms "denizler" and "çocuklar". By the reader's question this document is relevant; the inverted index misses it, because the index in this lesson stores the word as written, and "denizler" and "deniz" become two separate terms. This is not a shortcoming of the index but a **setting** — which form a word is stored in is a separate decision made in the third lesson. The second source is a **match inside the stem** and it covers 16 documents. Record `#14`'s summary contains "çocuk", but the word satisfying the second condition is "çocukluk" (childhood): `LIKE '%çocuk%'` looks at a character sequence in the middle of the text and does not recognize word boundaries. The same source also matches "denizci", "Karadeniz", and "kitapçı". These documents are brought to the question by mistake and mislead the reader. The last two lines show the other end of the set. The "İstanbul" query returns 73 documents through LIKE, and the same question written in lowercase returns **zero** documents. The reason is specific to Turkish: the lowercase of the letter `İ` is not `i`, it is a non-ASCII letter, and case folding does not recognize this pair. The inverted index returns 73 documents for both spellings, because the text is put through the same transformation both when it is indexed and when it is queried. This is exactly where a search engine departs from a relational scan: **applying the same transformation at write time and at query time**. #### Order and Cost The order column makes a third difference visible. LIKE starts with the ids `237, 333, 364, 490`; this order is not id order, it is the order of the covering index the query plan chose for itself. The inverted index starts with `8, 77, 107, 118`, that is, document id order. What the two orders share is this: **neither is a chosen order**. Both fall out of the structure's storage layout. Which document is more relevant is asked for the first time in this course only in the second topic. The cost side is not symmetric either. LIKE requires no preparation: a record is searchable the moment it is written, but every question reads 600 records and 124,863 bytes. The inverted index answers the question by reading 269 postings, touching roughly 1,076 bytes — a hundred and sixteen times smaller by bytes read. In return, the index must be **built and stored**; every new document must be processed into the index, and this structure takes up space. What this lesson measures is only the query side; the write side's accounting is worked out byte by byte in the next lesson. #### Summary - A relational index is sorted on the entire value and cannot help a `LIKE '%word%'` question; the query plan shows `SCAN` even with an index present, and all 600 records are read. - The inverted index goes from term to document: the same question is answered by reading 269 postings, and the bytes touched drop from 124,863 to 1,076. - The two paths' sets are not the same: LIKE returns 68, the inverted index 12, and 9 are common to both. Of the 59 LIKE brings back on top, 43 are inflected forms (which the index misses), 16 are matches inside a stem (which LIKE brings back by mistake). - Case makes the set flip entirely in Turkish: the "istanbul" query returns 0 documents through LIKE, "İstanbul" returns 73; the inverted index puts both spellings through the same rule at write time and query time, so it returns 73 documents for either spelling. - Neither path's returned order is a chosen order either: one is the order the query plan's index uses, the other is document id. The question of relevance is not asked at all in this topic. #### Next Step This lesson used the inverted index as a black box: a term went in, document ids came out. What is inside the structure, how many bytes it holds, and where a document's count of the same term is stored were not asked. The next lesson builds the index piece by piece: the term list, the document list under each term, the term count per document, and the word's position in the text. What gets measured is each of these four components' cost in bytes, and how many times position information makes the index grow. ### Inverted Index Source: https://academia.sh/en/courses/search-engines/inverted-index Building the structure that maps term to document and its byte accounting: the distinction between the dictionary and the posting list, pricing the document id, term frequency, and position information in a posting entry separately, measuring the index's ratio to the source text across three formats, the effect of keeping posting lists in document id order on intersection cost, and the adjacency distinction that position information buys. The previous lesson used the inverted index as a black box: a term went in, document ids came out. This lesson opens the box. The structure has two parts. The **dictionary** holds each distinct term in the corpus exactly once and places a pointer next to each term to that term's list. The **posting list** is the record of the documents the term appears in: each entry reports one document. In this course, a **posting** is an index record that reports a term's occurrence in a particular document. The same word names a message published in another curriculum; that usage is unrelated to this one. A posting entry carries at minimum the document id, and may optionally carry **how many times** the term occurs in that document (term frequency) and **where** it occurs (position). All three components carry a cost, and this lesson counts that cost. #### Dictionary, Postings, and Position A sorted index is built on the entire value and goes from a record to a value; the inverted index does the opposite, going from a term to a set of documents, which is why the same document appears in dozens of posting lists at once. That is the difference in one sentence, and it is also what determines index size: the index's size does not scale with the number of documents, it scales with **the number of distinct terms per document**. **IA1:** the corpus is the same corpus as the previous lesson — 600 book records, seed 20250317. **IA2:** the byte accounting uses the following constants — a dictionary entry costs the term's UTF-8 bytes plus 8 bytes (document frequency and a list pointer), a posting entry costs 4 bytes for the document id, 4 more bytes if term frequency is stored, and 4 bytes per position. **IA3:** the indexed text is the union of the `title` and `summary` fields; the previous lesson indexed only the `summary` field, so the term count differs there. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/index-measurement.mjs — the same corpus is built in three index formats: id only, // id+frequency, id+frequency+position. Term, posting entry, byte, scanned entry and comparisons are counted. import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; import { InvertedIndex } from "./index.mjs"; const text = (b) => `${b.title} ${b.summary}`; // IA3: indexed fields const source = corpus.reduce((t, b) => t + Buffer.byteLength(text(b)), 0); const build = (settings) => { const d = new InvertedIndex(settings); for (const b of corpus) d.add(b.id, text(b)); return d; }; const formats = [["id only", { frequency: false, position: false }], ["id + freq", { frequency: true, position: false }], ["id + freq + position", {}]]; console.log(`seed ${SEED}; corpus ${DOC_COUNT} docs, indexed text ${source} bytes`); console.log(`${"index format".padEnd(24)}${"terms".padStart(7)}${"postings".padStart(9)}${"positions".padStart(10)}` + `${"dictionary".padStart(11)}${"posting bytes".padStart(15)}${"position bytes".padStart(15)}${"total".padStart(9)}${"share of text".padStart(14)}`); let full; for (const [label, settings] of formats) { const d = build(settings), b = d.bytes(); full = d; console.log(label.padEnd(24) + String(b.terms).padStart(7) + String(b.postings).padStart(9) + String(b.positions).padStart(10) + String(b.dictionary).padStart(11) + String(b.postingBytes).padStart(15) + String(b.positionBytes).padStart(15) + String(b.total).padStart(9) + `%${(b.total * 100 / source).toFixed(1)}`.padStart(14)); } const first = full.postings("deniz").slice(0, 3).map((g) => `(#${g.id}, tf ${g.tf}, position ${g.position.join("-")})`); console.log(`\n"deniz" posting list ${full.postings("deniz").length} entries; first three: ${first.join(" ")}`); const tfDistribution = new Map(); for (const g of full.postings("deniz")) tfDistribution.set(g.tf, (tfDistribution.get(g.tf) ?? 0) + 1); console.log(`term frequency distribution: ${[...tfDistribution].sort().map(([t, n]) => `tf ${t} -> ${n} docs`).join(", ")}`); // comparison with an unsorted list: for each entry, a linear search through the other list const unsorted = (...t) => { const lists = t.map((x) => full.postings(x)), shortest = lists.reduce((a, x) => (x.length < a.length ? x : a)); let comparisons = 0; const matches = shortest.filter((g) => lists.every((x) => { for (const y of x) { comparisons += 1; if (y.id === g.id) return true; } return false; })); return { n: matches.length, comparisons }; }; console.log(`\n${"query".padEnd(20)}${"entries scanned".padStart(16)}${"id-sorted".padStart(12)}` + `${"unsorted".padStart(10)}${"docs returned".padStart(15)}${"first three ids".padStart(18)}`); for (const terms of [["deniz"], ["çocuk"], ["deniz", "çocuk"], ["deniz", "çocuk", "kitap"]]) { const result = full.search(...terms), unsortedComparisons = terms.length > 1 ? String(unsorted(...terms).comparisons) : "-"; console.log(terms.join("+").padEnd(20) + String(result.postings).padStart(16) + String(result.comparisons).padStart(12) + unsortedComparisons.padStart(10) + String(result.results.length).padStart(15) + result.results.slice(0, 3).join(",").padStart(18)); } // what position information buys: appearing in the same document versus appearing side by side const [termA, termB] = ["deniz", "günleri"]; const secondPositions = new Map(full.postings(termB).map((g) => [g.id, g.position])); const adjacent = full.postings(termA).filter((g) => secondPositions.has(g.id) && g.position.some((x) => secondPositions.get(g.id).some((y) => y - x === 1))).map((g) => g.id); console.log(`\n"${termA} ${termB}": same document ${full.search(termA, termB).results.length}, adjacent ${adjacent.length} docs ` + `(${adjacent.slice(0, 3).map((i) => `#${i} ${corpus[i - 1].title}`).join(", ")})`); ``` ``` seed 20250317; corpus 600 docs, indexed text 135824 bytes index format terms postings positions dictionary posting bytes position bytes total share of text id only 185 14609 0 2850 58436 0 61286 %45.1 id + freq 185 14609 0 2850 116872 0 119722 %88.1 id + freq + position 185 14609 16565 2850 116872 66260 185982 %136.9 "deniz" posting list 261 entries; first three: (#3, tf 1, position 0) (#4, tf 1, position 14) (#7, tf 1, position 14) term frequency distribution: tf 1 -> 206 docs, tf 2 -> 52 docs, tf 3 -> 3 docs query entries scanned id-sorted unsorted docs returned first three ids deniz 261 261 - 261 3,4,7 çocuk 84 84 - 84 5,6,8 deniz+çocuk 345 470 19622 25 8,67,77 deniz+çocuk+kitap 452 630 22135 3 158,309,491 "deniz günleri": same document 23, adjacent 10 docs (#3 Deniz Günleri, #123 Büyük Deniz Günleri, #154 Beyaz Deniz Günleri) ``` #### Three Formats, Three Prices The first table indexes the same 600 documents three times, and the only thing that changes is what the posting entry carries. The dictionary is identical across all three: 185 terms, 2,850 bytes. The index's weight is not in the dictionary, it is in the **14,609 posting entries**. When only the document id is stored, the index is 61,286 bytes, about 45.1 percent of the source text. Adding term frequency exactly doubles the posting section, and the index reaches 119,722 bytes, 88.1 percent of the text. Adding position brings it to 185,982 bytes — **larger than the text itself**. This last line is the lesson's most concrete result: an inverted index that keeps position can take up more space than the text it indexes. The reason shows up in the second and third columns — against 14,609 posting entries there are 16,565 position records, because a term can occur more than once in the same document. The frequency distribution of the term `deniz` counts this directly: once in 206 documents, twice in 52 documents, three times in 3 documents. #### Order Is a Decision Posting lists are kept in document id order, and this is not a preference, it is a decision that determines how the intersection works. The `deniz+çocuk` query reads two lists of 261 and 84 entries, 345 entries total. Because both lists are sorted, the intersection runs in a single pass, and **470 comparisons** are enough. If the same intersection ran on unsorted lists, each entry of the shorter list would require a search from the start of the longer list: **19,622 comparisons**, forty times more. For the three-term query, the ratio is 630 against 22,135. The order's second consequence shows up in the returned set. `deniz` alone returns 261 documents, `çocuk` returns 84; their intersection drops to 25 documents, and to 3 once a third term is added. Every condition narrows the set, and the cost of narrowing is a growing entry count: from 345 to 452. The returned documents arrive in the order `8, 67, 77`; this order is again document id, that is, the order documents entered the index. Which document is more relevant is still not asked. #### What Position Buys Position information is the index's most expensive component — 66,260 bytes on its own, 35.6 percent of the index. In return it buys exactly one thing: knowing whether two terms sit **next to each other** in a document. The last line counts this. The terms `deniz` and `günleri` occur together in 23 documents; only 10 of these documents have the two terms adjacent, and the titles of those 10 documents genuinely follow the pattern `Deniz Günleri`. An index that does not keep position cannot tell the two sets apart and returns all 23 documents. The decision is put this way: 66,260 bytes buys a 13-document narrowing. Whether that narrowing is worth it in a catalog search depends on the type of question — for a reader searching for a book by two words without knowing its exact title, the adjacency distinction gives the correct result directly. The query form that uses this distinction is built in the second topic; this lesson only shows that the data sits in the index and what it holds. #### Summary - The inverted index has two parts: the dictionary (185 terms, 2,850 bytes) and the posting lists (14,609 entries). The weight sits on the posting side; the dictionary is about 1.5 percent of the index. - What a posting entry carries sets the price: id only costs 61,286 bytes (45.1 percent of the text), adding term frequency costs 119,722 bytes (88.1 percent), adding position costs 185,982 bytes (136.9 percent) — the index becomes larger than the source text. - 16,565 position records fall against 14,609 posting entries; the term `deniz` occurs once in 206 documents, twice in 52, three times in 3. - Keeping posting lists in document id order reduces the intersection to a single pass: `deniz+çocuk` finishes in 470 comparisons, against 19,622 comparisons on an unsorted list. - Position information costs 66,260 bytes and buys the ability to narrow the set from 23 documents to 10 in the `deniz günleri` query. #### Next Step This lesson's index split text into words with a single rule: split on every character that is not a letter or digit, then lowercase using Turkish rules. That rule itself was never questioned. Yet `denizler` and `deniz` become two separate terms because of this rule, and the 43-document loss from the first lesson comes from exactly this. The next lesson indexes the same corpus with three separate analyzer chains — raw, lowercasing, and stemming with a stop-word filter — and runs the same query on all three: the returned set, order, and index size are measured side by side across the three configurations. ### Analyzer Chain Source: https://academia.sh/en/courses/search-engines/analyzer-chain The rule for turning text into terms deciding the outcome: building the chain of tokenization, normalization, and filters in three configurations over the same corpus, comparing raw, lowercasing, and stemming-plus-stop-word chains on index size, returned set, and the top five results, and counting the gap created by Turkish case folding, the apostrophe, and inflectional suffixes. The previous lesson's index split text into words with a single rule, and that rule was never questioned. Yet `denizler` and `deniz` being two separate terms is not a law of nature, it is the result of that rule. The sequence of rules that turns text into terms is called the **analyzer chain**, and it has three steps: **tokenization** splits the text into pieces, **normalization** collapses the pieces to a single spelling, and **filters** change or drop the remaining tokens. It is essential that this chain runs in two places at once: when a document is indexed and when a query is evaluated. If the two use different rules, the query searches for a term that does not exist in the index. The lesson builds this chain in three separate configurations and runs the same questions on all three. #### Links in the Chain Here, **tokenization** means splitting text into search units. In M06/K01, a token names the symbol entering a grammar parser, where the splitting criterion is a grammar rule; here the criterion is the search contract. **Normalization** is likewise a separate thing: M17/K01's relational normalization removes data repetition by splitting it across tables, while normalization here collapses different spellings of the same word into a single term. Three configurations are compared. The **raw** chain splits only on whitespace; punctuation stays stuck to the token, and case is kept as written. The **lowercase** chain splits on every character that is not a letter or digit and folds case using the Turkish locale — this is the chain used in the previous two lessons. The **stem + stop** chain adds two filters on top of this: terms occurring in more than 60 percent of documents are dropped, and the longest inflectional suffix is stripped once from the remaining tokens. **IA1:** the corpus is unchanged — 600 records, seed 20250317, indexed fields `title` and `summary`. **IA2:** the byte constants are the same constants as the previous lesson and are identical across all three chains. **IA3:** the stop word list is not written by hand, it is measured: the document-frequency threshold is 60 percent. **IA4:** the stemmer strips the longest suffix once and never leaves a stem shorter than three letters; it does not know consonant softening or vowel drop. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/analyzer.mjs — the same corpus is indexed with three analyzer chains: raw, lowercase, // stemming + stop word. Index size, returned set, first five results, and Turkish measures. import { corpus, DOC_COUNT, SEED, STEMS as STEM_FORMS } from "./catalog.mjs"; import { InvertedIndex } from "./index.mjs"; const text = (b) => `${b.title} ${b.summary}`; const SPLIT = (s) => s.split(/[^\p{L}\p{N}]+/u).filter(Boolean); const RAW = (s) => s.split(/\s+/).filter(Boolean); // no splitting, no transformation const LOWERCASE = (s) => SPLIT(s.toLocaleLowerCase("tr")); const SUFFIXES = ["ları", "leri", "lerin", "ların", "nin", "nın", "nun", "nün", "dan", "den", "tan", "ten", "ler", "lar", "ya", "ye", "da", "de", "ta", "te", "in", "ın", "un", "ün", "a", "e"] .sort((x, y) => y.length - x.length); const stem = (t) => { // strips the longest suffix once, stem >= 3 for (const e of SUFFIXES) if (t.length - e.length >= 3 && t.endsWith(e)) return t.slice(0, -e.length); return t; }; // stop word list is chosen by measurement: a term occurring in more than 60% of documents const pre = new InvertedIndex({ analyze: LOWERCASE, position: false }); for (const b of corpus) pre.add(b.id, text(b)); const STOP_WORDS = new Set([...pre.dictionary].filter(([, l]) => l.length > DOC_COUNT * 0.6).map(([t]) => t)); const STEMMED = (s) => LOWERCASE(s).filter((t) => !STOP_WORDS.has(t)).map(stem); const chain = [["raw", RAW], ["lowercase", LOWERCASE], ["stem + stop", STEMMED]]; const index = new Map(chain.map(([label, analyze]) => { const d = new InvertedIndex({ analyze }); for (const b of corpus) d.add(b.id, text(b)); return [label, d]; })); const source = corpus.reduce((t, b) => t + Buffer.byteLength(text(b)), 0); console.log(`seed ${SEED}; corpus ${DOC_COUNT} docs, ${source} bytes of text`); console.log(`stop words (df > 60%): ${[...STOP_WORDS].join(", ")}`); console.log(`\n${"analyzer".padEnd(14)}${"terms".padStart(7)}${"postings".padStart(9)}${"positions".padStart(10)}` + `${"index bytes".padStart(12)}${"share of text".padStart(14)}`); for (const [label, d] of index) { const b = d.bytes(); console.log(label.padEnd(14) + String(b.terms).padStart(7) + String(b.postings).padStart(9) + String(b.positions).padStart(10) + String(b.total).padStart(12) + `%${(b.total * 100 / source).toFixed(1)}`.padStart(14)); } console.log(`\n${"query".padEnd(11)}${"analyzer".padEnd(13)}${"query term".padEnd(15)}` + `${"docs returned".padStart(14)}${"first five ids".padStart(22)}${"changed in first five".padStart(23)}`); for (const query of ["Denizler", "kitap", "İstanbul"]) { const row = chain.map(([label, analyze]) => { const terms = analyze(query), r = terms.length ? index.get(label).search(...terms) : { results: [] }; return [label, terms.join("+") || "-", r.results]; }); const baseline = row.find(([label]) => label === "lowercase")[2].slice(0, 5); for (const [label, terms, results] of row) { const five = results.slice(0, 5); const changed = label === "lowercase" ? "-" : String(baseline.filter((x, i) => five[i] !== x).length + Math.max(0, five.length - baseline.length)); console.log(query.padEnd(11) + label.padEnd(13) + terms.padEnd(15) + String(results.length).padStart(14) + (five.join(",") || "-").padStart(22) + changed.padStart(23)); } } // --- Turkish measure 1: case folding (Turkish locale versus a general rule) --- const general = new InvertedIndex({ analyze: (s) => SPLIT(s.toLowerCase()), position: false }); for (const b of corpus) general.add(b.id, text(b)); const diverging = [...general.dictionary.keys()].filter((t) => !pre.dictionary.has(t)); console.log(`\ncase folding: Turkish rule ${pre.dictionary.size} terms, general rule ${general.dictionary.size} ` + `terms; diverging terms ${JSON.stringify(diverging)}`); console.log(` "istanbul" query: Turkish rule ${pre.search("istanbul").results.length} docs, ` + `general rule ${general.search("istanbul").results.length} docs`); console.log(` apostrophe: "İstanbul'un" -> ${JSON.stringify(LOWERCASE("İstanbul'un"))}`); // --- Turkish measure 2: the stemmer over 12 stems x 6 forms --- let single = 0; const split = []; for (const row of STEM_FORMS) { const k = new Set(row.map(stem)); if (k.size === 1) single += 1; else split.push(`${row[0]}: ${[...k].join("/")}`); } console.log(`\nstemming: ${STEM_FORMS.length} stems x 6 forms; ${single} stems collapsed to one term, ` + `${split.length} stems split -> ${split.join(", ")}`); const stemGroups = new Map(); for (const t of pre.dictionary.keys()) (stemGroups.get(stem(t)) ?? stemGroups.set(stem(t), []).get(stem(t))).push(t); const merged = [...stemGroups].filter(([, l]) => l.length > 1); console.log(`${pre.dictionary.size} terms collapse to ${stemGroups.size} stems; ${merged.length} stems collected more than one term`); console.log(`example: ${merged.slice(0, 3).map(([k, l]) => `${k} <- ${l.join(", ")}`).join(" | ")}`); ``` ``` seed 20250317; corpus 600 docs, 135824 bytes of text stop words (df > 60%): ve, gündelik, hayat, üzerine, notlar, sunar analyzer terms postings positions index bytes share of text raw 261 14727 16489 187780 %138.3 lowercase 185 14609 16565 185982 %136.9 stem + stop 122 11722 12627 146080 %107.6 query analyzer query term docs returned first five ids changed in first five Denizler raw Denizler 80 15,16,18,20,22 5 Denizler lowercase denizler 122 3,15,16,18,20 - Denizler stem + stop deniz 463 2,3,4,7,8 5 kitap raw kitap 54 13,44,53,80,92 5 kitap lowercase kitap 107 7,10,13,28,30 - kitap stem + stop kitap 178 2,5,6,7,10 5 İstanbul raw İstanbul 0 - 5 İstanbul lowercase istanbul 73 2,21,30,32,35 - İstanbul stem + stop istanbul 73 2,21,30,32,35 0 case folding: Turkish rule 185 terms, general rule 186 terms; diverging terms ["i","stanbul","zmir"] "istanbul" query: Turkish rule 73 docs, general rule 0 docs apostrophe: "İstanbul'un" -> ["istanbul","un"] stemming: 12 stems x 6 forms; 7 stems collapsed to one term, 5 stems split -> kitap: kitap/kitab, çocuk: çocuk/çocuğ, şehir: şehir/şehr, bahçe: bahç/bahçe, mektup: mektup/mektub 185 terms collapse to 128 stems; 18 stems collected more than one term example: gemi <- gemi, gemide, gemiye, gemiler, geminin, gemiden | öykü <- öyküleri, öyküler | mektup <- mektup, mektuptan, mektupta, mektuplar ``` #### Three Chains, Three Sets The first table shows what the chain does to the index. The raw chain produces the most terms: 261. The reason is not richness, it is the same word being counted multiple times through punctuation and case. The lowercase chain brings the term count down to 185; the stem and stop filters bring it to 122. Posting entries drop from 14,727 to 11,722, and the index from 187,780 bytes to 146,080 bytes — the filters erase about 22 percent of the index. The second table shows the decision that actually matters. The `Denizler` query returns 80 documents on the raw chain, 122 on the lowercase chain, 463 on the stem chain. Same corpus, same question, **a factor of nearly six**. The raw chain only finds the token `Denizler`, written exactly with a capital letter; the lowercase chain finds the form `denizler` regardless of case; the stem chain reduces the query to the stem `deniz` and so brings back every document containing any of the six inflected forms. This was the source of the 43-document loss counted in the first lesson; the stem chain closes the part of that loss that sits on the `deniz` stem, and in return the set quadruples. The order column counts this result separately. Taking the lowercase chain as the baseline, for `Denizler` and `kitap` **all five of the first five ids** change in both the raw and stem chains. Order here is not a relevance ranking, it is the posting list's id layout; even so, the first results a user sees change completely when the chain changes. The `İstanbul` query is the extreme case: the raw chain returns zero documents, because the token in the text takes the form `İstanbul'un`, a single piece together with the apostrophe. #### The Measure of Turkish The third section counts three concrete difficulties of Turkish. **Case folding:** the same text produces 185 terms under the Turkish locale and 186 under the general rule, and the diverging terms are `["i", "stanbul", "zmir"]`. The general rule converts the letter `İ` into `i` combined with a dot above, and the splitter does not count that dot as a letter, so the word `İstanbul` splits into `i` and `stanbul`. The result is measured: the `istanbul` query returns 73 documents under the Turkish rule and **0 documents** under the general rule. **Apostrophe:** the token `İstanbul'un` splits in two as `["istanbul", "un"]`. The split rescues the query, but it also produces a meaningless `un` term. **Stemming:** the measure is taken over 6 inflected forms each of 12 stems. Seven stems collapse to a single term, five stems split: `kitap/kitab`, `çocuk/çocuğ`, `şehir/şehr`, `bahç/bahçe`, `mektup/mektub`. Four come from consonant softening and vowel drop, one (`bahç/bahçe`) from a letter belonging to the stem itself being mistaken for a suffix and dropped. There is a cost in the opposite direction too: as 185 terms collapse to 128 stems, 18 stems collect more than one term, and not all of these are correct merges. This lesson does not **solve** Turkish stemming, it counts where the rule holds and where it does not; when the query is `kitap`, documents containing `kitabın` still stay outside the set. The stop word filter was also built by measurement: six terms were dropped because they occur in more than 60 percent of documents. The presence of non-conjunction words on the list shows that the list was determined by frequency, not by hand. #### Summary - The analyzer chain consists of tokenization, normalization, and filters, and it must work identically on both the indexing side and the query side. - The three chains produce 261, 185, and 122 terms on the same corpus; the index holds 187,780, 185,982, and 146,080 bytes — the filters erase about 22 percent of the index. - The `Denizler` query returns 80, 122, and 463 documents across the three chains; measured against the baseline, all five of the first five ids change depending on the chain. The chain choice directly determines both the set and the order. - Turkish case folding flips the decision: the `istanbul` query returns 73 documents under the Turkish locale and 0 under the general rule; the apostrophe splits the token `İstanbul'un` in two. - The stemmer collapses 7 of 12 stems to a single term and splits 5 — four from consonant softening and vowel drop, one from a stem letter dropped as if it were a suffix; as 185 terms collapse to 128 stems, 18 stems collect more than one term. #### Next Step This lesson's three chains all applied the same rule to the entire text. But not every field in the catalog is text: the shelf code `TR-879.4`, the language `Türkçe`, the subject tag `çocuk edebiyatı` (children's literature) — are these text to be split into words, or values that must stay a single piece? The next lesson indexes the same field with two types and measures the difference: which type exact matching works on, which type sorting is defined on, how many buckets aggregation produces, and how far apart the two types' index sizes fall. ### Mappings and Field Types Source: https://academia.sh/en/courses/search-engines/mappings-and-field-types A field's type deciding the indexing rule: separating the text field from the keyword field, indexing the same field with both types and comparing term count, posting entries, and index bytes, the effect of exact matching versus partial matching on the returned set, the cross-match a multi-valued field produces, whether a sort key is defined, and how aggregation buckets diverge between the two types. The previous lesson's three chains all applied the same rule to the entire text. But not every field in the catalog is text. The shelf code `TR-866.3` marks a location, and splitting it makes no sense; the language `Türkçe` is a value; the subject tag `çocuk edebiyatı` (children's literature) consists of two words but is a single category. The summary, however, really is text, and every word inside it is a separate search entry. The declaration that states which rule a field is indexed by is called a **mapping**; the decision the mapping carries is called the **field type**. This lesson compares two types. A **text field** passes the value through the analyzer chain; as a result the field spreads across multiple terms. A **keyword field** does not split or transform the value: the entire value is a single term. #### Two Types, the Same Field The measurement runs over three fields: the multi-valued `subjects`, and the single-valued `shelf` and `language`. Each field is indexed twice, and four things are counted: index size, the set an exact-value query returns, whether sorting is defined, and the number of buckets aggregation produces. **IA1:** the corpus is unchanged — 600 records, seed 20250317. **IA2:** the byte constants are the same constants as the previous lessons; position is not kept in this lesson. **IA3:** in a multi-valued field, the keyword type makes each value a separate term, the text type parses all the values as a single piece of text. **IA4:** because a text field can carry more than one term per document, a sort key must be chosen; here the "smallest term" rule is chosen. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/field-type.mjs — the same fields are indexed with two types: text field (goes through // the analyzer) and keyword field (value stays a single term). Bytes, exact match, sort, aggregation. import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; import { InvertedIndex } from "./index.mjs"; const SEPARATOR = "\u0001"; // value boundary const fieldValues = { subjects: (b) => b.subjects, shelf: (b) => [b.shelf], language: (b) => [b.language] }; const TEXT = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); const KEYWORD = (s) => s.split(SEPARATOR).filter(Boolean); // value is not split const build = (field, analyze) => { const d = new InvertedIndex({ analyze, position: false }); for (const b of corpus) d.add(b.id, fieldValues[field](b).join(SEPARATOR)); return d; }; const index = new Map(); for (const field of Object.keys(fieldValues)) for (const [type, analyze] of [["text", TEXT], ["keyword", KEYWORD]]) index.set(`${field}/${type}`, build(field, analyze)); console.log(`seed ${SEED}; corpus ${DOC_COUNT} docs; fields: subjects (multi-valued), shelf, language`); console.log(`\n${"field".padEnd(9)}${"type".padEnd(9)}${"terms".padStart(7)}${"postings".padStart(9)}` + `${"index bytes".padStart(13)}${"sample terms".padStart(28)}`); for (const [key, d] of index) { const [field, type] = key.split("/"), b = d.bytes(); console.log(field.padEnd(9) + type.padEnd(9) + String(b.terms).padStart(7) + String(b.postings).padStart(9) + String(b.total).padStart(13) + [...d.dictionary.keys()].slice(0, 2).join(" | ").padStart(28)); } const commonShelf = [...corpus.reduce((m, b) => m.set(b.shelf, (m.get(b.shelf) ?? 0) + 1), new Map())] .sort((x, y) => y[1] - x[1])[0][0]; // the shelf code repeated most often in the corpus const queries = [["subjects", "çocuk edebiyatı", "exact tag"], ["subjects", "çocuk", "single word of tag"], ["subjects", "çocuk tarihi", "tag not present"], ["shelf", commonShelf, "exact shelf code"], ["shelf", "TR", "shelf prefix"], ["language", "Türkçe", "value as written"], ["language", "türkçe", "value lowercased"]]; console.log(`\n${"field".padEnd(9)}${"query".padEnd(18)}${"description".padEnd(20)}${"text".padStart(8)}${"keyword".padStart(10)}`); for (const [field, s, note] of queries) { const m = index.get(`${field}/text`).search(...TEXT(s)).results.length; const k = index.get(`${field}/keyword`).search(s).results.length; console.log(field.padEnd(9) + s.padEnd(18) + note.padEnd(20) + String(m).padStart(8) + String(k).padStart(10)); } // --- sorting: a keyword field has one term per document, a text field can have more than one --- const termsOf = (d, id) => [...d.dictionary].filter(([, l]) => l.some((g) => g.id === id)).map(([t]) => t); const textShelf = index.get("shelf/text"), keywordShelf = index.get("shelf/keyword"); const multiTerm = corpus.filter((b) => termsOf(textShelf, b.id).length > 1).length; const smallest = (d, id) => termsOf(d, id).sort()[0]; const sortBy = (d) => corpus.map((b) => [smallest(d, b.id), b.id]).sort((x, y) => (x[0] < y[0] ? -1 : x[0] > y[0] ? 1 : x[1] - y[1])); const keywordOrder = sortBy(keywordShelf), textOrder = sortBy(textShelf); const different = keywordOrder.slice(0, 10).filter(([, id], i) => textOrder[i][1] !== id).length; console.log(`\nsort order (by shelf field): in the text type, ${multiTerm} documents have more than one term in the field; ` + `${new Set(textOrder.map(([t]) => t)).size} distinct sort keys`); console.log(`in the keyword type, ${new Set(keywordOrder.map(([t]) => t)).size} distinct keys; ` + `${different} of the first ten documents differ between the two orders`); console.log(` keyword first three: ${keywordOrder.slice(0, 3).map(([t, i]) => `${t} (#${i})`).join(", ")}`); console.log(` text first three : ${textOrder.slice(0, 3).map(([t, i]) => `${t} (#${i})`).join(", ")}`); // --- aggregation: bucket count on the subjects field --- const buckets = (d) => [...d.dictionary].map(([t, l]) => [t, l.length]).sort((x, y) => y[1] - x[1] || (x[0] < y[0] ? -1 : 1)); for (const type of ["keyword", "text"]) { const result = buckets(index.get(`subjects/${type}`)); console.log(`\nsubjects aggregation (${type}): ${result.length} buckets, ${result.reduce((t, [, n]) => t + n, 0)} total document count`); console.log(` largest three: ${result.slice(0, 3).map(([t, n]) => `${t} ${n}`).join(", ")}`); } ``` ``` seed 20250317; corpus 600 docs; fields: subjects (multi-valued), shelf, language field type terms postings index bytes sample terms subjects text 17 1430 11686 biyografi | kısa subjects keyword 12 999 8203 biyografi | kısa öykü shelf text 112 1800 15610 en | 893 shelf keyword 549 600 13584 EN-893.5 | FR-881.4 language text 4 600 4866 almanca | fransızca language keyword 4 600 4867 Almanca | Fransızca field query description text keyword subjects çocuk edebiyatı exact tag 93 93 subjects çocuk single word of tag 93 0 subjects çocuk tarihi tag not present 9 0 shelf TR-866.3 exact shelf code 3 3 shelf TR shelf prefix 154 0 language Türkçe value as written 310 310 language türkçe value lowercased 310 0 sort order (by shelf field): in the text type, 600 documents have more than one term in the field; 9 distinct sort keys in the keyword type, 549 distinct keys; 10 of the first ten documents differ between the two orders keyword first three: DE-801.1 (#460), DE-801.8 (#94), DE-802.5 (#594) text first three : 0 (#19), 0 (#31), 0 (#42) subjects aggregation (keyword): 12 buckets, 999 total document count largest three: coğrafya 95, çocuk edebiyatı 93, biyografi 88 subjects aggregation (text): 17 buckets, 1430 total document count largest three: coğrafya 95, edebiyatı 93, çocuk 93 ``` #### The Byte Difference Depends on the Field Itself The first table does not give a single rule, because which type is cheaper depends on the field. In the `subjects` field the text type produces both more terms (17 against 12) and more posting entries (1,430 against 999), and at 11,686 bytes it comes in above the keyword type's 8,203 bytes: once two-word tags are split, each document leaves two entries in the field. In the `shelf` field the direction reverses. The text type shrinks the dictionary — 112 pieces instead of 549 separate codes, because the same number and prefix repeat across hundreds of codes — but triples the posting entries: 1,800 instead of 600. In total the text type is again more expensive (15,610 against 13,584), but this time the expense comes from the posting side. The `language` field is the borderline case where the two types differ by only one byte: because the value is already a single word, the type decision does not change the size, it only preserves the spelling. #### Exact Matching, Sorting, and Aggregation The second table shows the decision's effect on the set. When the exact tag is asked for, both types return the same 93 documents; the distinction is in the other rows. When the tag's **single word** is asked for, the text field gives 93 documents, the keyword field gives 0: the text field is open to partial matching, the keyword field plainly is not. The same behavior repeats in the shelf field with the `TR` prefix: 154 documents against 0. The third row is the real warning. A tag reading `çocuk tarihi` does not **exist** in the corpus; despite this, the text field returns 9 documents. These are documents that carry the tags `çocuk edebiyatı` and `deniz tarihi` together: because the field is multi-valued, words coming from two separate values behave as a single match. The keyword field does not bring back these documents, because for it every tag is an unsplit whole. The cost in the opposite direction shows up in the language row: when asked with the spelling `türkçe`, the keyword field returns 0 documents, because the value carries a capital letter. The sorting difference is sharper. In the keyword field every document has exactly one term, and the shelf code produces 549 separate sort keys; sorting is defined. In the text field **all 600 documents** have more than one term in the field, so the question "sorted by which term" has no answer. Once the chosen rule takes the smallest term, only 9 keys remain, and all ten of the first ten documents swap places: sorting becomes meaningless in practice. On the aggregation side, too, the two types pull two different tables out of the same data. The keyword field gives 12 buckets and a document count of 999 — this is the exact count of tags in the corpus. The text field gives 17 buckets and a count of 1,430; the `çocuk edebiyatı` tag has split into two buckets, `çocuk` and `edebiyatı`, each counting 93 and landing at the top of the list. In a report that wants to see the category distribution, these two tables answer the same question differently. #### Summary - A mapping declares which rule a field is indexed by; a text field passes the value through the analyzer and splits it into terms, a keyword field keeps the value as a single, unsplit term. - The direction of the byte difference depends on the field: in `subjects`, the text type is more expensive at 11,686 against 8,203 bytes; in `shelf`, it shrinks the dictionary from 549 to 112 terms but triples the posting entries from 600 to 1,800. - Exact matching is only exact in a keyword field: the `çocuk` query returns 93 documents in the text field and 0 in the keyword field; the `TR` prefix gives 154 against 0. - In a multi-valued field, the text type produces a cross-match: the tag `çocuk tarihi`, which does not exist in the corpus, returns 9 documents because the words come from two separate tags. - Sorting is undefined in the text field: all 600 documents carry more than one term, the chosen rule leaves 9 keys, and all ten of the first ten documents swap places. Aggregation produces 17 buckets instead of 12, and `çocuk edebiyatı` splits into two separate buckets. #### Next Step The mapping in this lesson was written by hand: which field would be indexed with which type was known in advance. In a real catalog import, this information is often missing — records arrive from outside, each source carries its own fields, and the index itself makes the type decision. The next lesson measures this decision: how the field count grows as document diversity increases, what happens when the same name arrives with two different types, and how many bytes each field costs in index metadata. ### Dynamic and Explicit Mapping Source: https://academia.sh/en/courses/search-engines/dynamic-and-explicit-mapping Who decides the mapping: comparing dynamic mapping, which adds every incoming field to the index, against explicit mapping, whose field list is fixed in advance, on the same catalog import; the field count growing with document diversity; counting the mapping metadata per field in bytes; and measuring the documents rejected when the same name arrives with two types, and their fall from the query set. In the previous lesson, which field would be indexed with which type was known in advance. In a real catalog import, this information is often missing: records arrive from donation lists, periodical registries, and digital copy records, each source carries its own fields, and none of them are declared in advance. In this situation there are two routes. **Dynamic mapping** adds every new field name to the mapping the moment it sees it, and infers its type from that field's **first value**. **Explicit mapping** fixes the field list in advance; a field not on the list does not enter the index. Both take in the same records, but one chooses flexibility, the other control. This lesson measures the difference between them with four numbers: field count, metadata bytes, accepted documents, and returned set. #### The Import's Fields For the measurement, the same 600 records are enriched as if they arrived from three additional sources. Donation records bring, alongside the `donor` field, fields named after the donation year and the donor's last name: `donation_1994`, `note_kaya`. Periodical records carry `volume`, `issue`, and `periodicity`; `issue` is an integer in most records, and a text like `3-7` in combined issues. Digital copy records add `file_format`, `size_bytes`, and `page`. Every record also gets a field named after a census year: `census_2019`. **IA1:** the corpus is unchanged — 600 records, seed 20250317; the extra fields are derived from the same seed. **IA2:** mapping metadata per field costs the field name's bytes plus 24 bytes (type, analyzer pointer, and options). **IA3:** the explicit mapping consists of seven fields, and a field not on the list is not indexed. **IA4:** in dynamic mapping, a field's type is determined by its first value; a conflicting type arriving later rejects the entire document. **IA5:** terms are qualified by field name, meaning `subjects:roman` is a single term. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/mapping.mjs — the same records taken in under two mappings: dynamic (every incoming field // enters the index) and explicit (field list fixed in advance). Field count, metadata bytes, rejected documents, set difference. import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; import { InvertedIndex } from "./index.mjs"; const FIELD_OVERHEAD = 24; // IA2: mapping record per field let c = SEED; const random = () => (c = (c * 1103515245 + 12345) % 2147483648) / 2147483648; const LAST_NAMES = ["Yılmaz", "Kaya", "Demir", "Şahin", "Çelik", "Aydın", "Doğan", "Arslan", "Koç", "Ertem"]; // catalog import: each source brings its own fields (donation, periodical, digital copy) const records = corpus.map((b) => { const k = { title: b.title, summary: b.summary, subjects: b.subjects.join(" "), author: b.author, year: b.year, language: b.language, shelf: b.shelf }; const t = random(); if (t < 0.35) { const year = 1968 + Math.floor(random() * 57), s = LAST_NAMES[Math.floor(random() * 10)]; Object.assign(k, { donor: `${s} family`, [`donation_${year}`]: "accepted", [`note_${s.toLocaleLowerCase("tr")}`]: "archived" }); } else if (t < 0.6) { k.volume = 1 + Math.floor(random() * 12); k.issue = random() < 0.25 ? `${1 + Math.floor(random() * 4)}-${5 + Math.floor(random() * 4)}` : 1 + Math.floor(random() * 12); k.periodicity = "monthly"; } else if (t < 0.8) { Object.assign(k, { file_format: "scan", size_bytes: 1 << (18 + Math.floor(random() * 4)), page: 40 + Math.floor(random() * 400) }); } k[`census_${2015 + Math.floor(random() * 10)}`] = 1 + Math.floor(random() * 3); return k; }); const EXPLICIT = new Map([["title", "text"], ["summary", "text"], ["subjects", "text"], ["author", "text"], ["year", "number"], ["language", "text"], ["shelf", "text"]]); // IA3: explicit mapping is seven fields const typeOf = (v) => (typeof v === "number" ? "number" : "text"); // term qualified by field name: "subjects:roman" is a single term const termOf = (a, v) => String(v).toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u) .filter(Boolean).map((t) => `${a}:${t}`).join(" "); function take(records, explicit) { // build the mapping, accept or reject the document const field = explicit ? new Map(explicit) : new Map(); const index = new InvertedIndex({ analyze: (s) => s.split(" ").filter(Boolean), position: false }); const rejected = [], skipped = new Map(); let accepted = 0; for (const [i, k] of records.entries()) { const conflict = []; for (const [a, v] of Object.entries(k)) { const type = typeOf(v); if (explicit && !field.has(a)) { skipped.set(a, (skipped.get(a) ?? 0) + 1); continue; } if (!field.has(a)) field.set(a, type); else if (field.get(a) !== type) conflict.push(`${a} (${field.get(a)} -> ${type})`); } if (conflict.length) { rejected.push([i + 1, conflict[0]]); continue; } index.add(i + 1, Object.entries(k).filter(([a]) => field.has(a)).map(([a, v]) => termOf(a, v)).join(" ")); accepted += 1; } const metadata = [...field.keys()].reduce((t, a) => t + Buffer.byteLength(a) + FIELD_OVERHEAD, 0); return { field, index, rejected, skipped, accepted, metadata }; } const dynamic = take(records, null), explicit = take(records, EXPLICIT); console.log(`seed ${SEED}; ${DOC_COUNT} catalog records, three extra sources (donation, periodical, digital copy)`); console.log(`\nfield count growing with document count (dynamic mapping)`); console.log(`${"docs processed".padStart(15)}${"fields".padStart(8)}${"metadata bytes".padStart(16)}`); for (const n of [50, 150, 300, 600]) { const d = take(records.slice(0, n), null); console.log(String(n).padStart(15) + String(d.field.size).padStart(8) + String(d.metadata).padStart(16)); } console.log(`\n${"mapping".padEnd(9)}${"fields".padStart(7)}${"metadata bytes".padStart(16)}${"terms".padStart(7)}` + `${"index bytes".padStart(13)}${"accepted".padStart(9)}${"rejected".padStart(9)}${"unindexed fields".padStart(19)}`); for (const [label, r] of [["dynamic", dynamic], ["explicit", explicit]]) console.log(label.padEnd(9) + String(r.field.size).padStart(7) + String(r.metadata).padStart(16) + String(r.index.bytes().terms).padStart(7) + String(r.index.bytes().total).padStart(13) + String(r.accepted).padStart(9) + String(r.rejected.length).padStart(9) + String(r.skipped.size).padStart(19)); console.log(`\ntype conflict: ${dynamic.rejected.length} documents rejected; first three ` + `${dynamic.rejected.slice(0, 3).map(([i, a]) => `#${i} ${a}`).join(", ")}`); const queries = ["subjects:roman", "language:türkçe", "donor:kaya", "periodicity:monthly"]; console.log(`\n${"query".padEnd(18)}${"dynamic".padStart(9)}${"explicit".padStart(9)}${"diff".padStart(7)}`); for (const s of queries) { const d = dynamic.index.search(s).results.length, e = explicit.index.search(s).results.length; console.log(s.padEnd(18) + String(d).padStart(9) + String(e).padStart(9) + String(d - e).padStart(7)); } ``` ``` seed 20250317; 600 catalog records, three extra sources (donation, periodical, digital copy) field count growing with document count (dynamic mapping) docs processed fields metadata bytes 50 49 1684 150 77 2717 300 85 3013 600 91 3235 mapping fields metadata bytes terms index bytes accepted rejected unindexed fields dynamic 91 3235 660 161329 489 111 0 explicit 7 211 437 171862 600 0 84 type conflict: 111 documents rejected; first three #2 issue (text -> number), #12 issue (text -> number), #22 issue (text -> number) query dynamic explicit diff subjects:roman 70 82 -12 language:türkçe 253 310 -57 donor:kaya 18 0 18 periodicity:monthly 40 0 40 ``` #### Field Count Grows With Documents The first table shows the shape of mapping explosion. Once the first 50 documents are processed, the mapping already has **49 fields**: nearly one new field per record. Because field names are derived from the data — `donation_1994`, `note_kaya`, `census_2019` — every new donation year, every new donor, and every new census year adds a row to the mapping. Growth then slows (77 at 150 documents, 91 at 600) because the pool of years and last names runs out; the pool being limited is a property of the corpus, a real import has no pool that runs out. Metadata tracks this growth directly: from 1,684 bytes to 3,235 bytes. The number can look small, but it must be read at scale — metadata does not grow with document count, it grows with **field count**, and it is held again in every part of the index. In the explicit mapping the same line item is 211 bytes: about fifteen times smaller, and fixed regardless of document count. #### A Type Conflict Drops the Document The second table's harshest number is the `rejected` column. Dynamic mapping accepted **489** of the 600 records and rejected **111**. The reason is a single field: `issue`. In the first periodical record this field arrived as text in the form `1-8`, so the mapping fixed it as text; when the same field later arrived as a number, the document was rejected in its entirety. The loss is not limited to that field — the document's `title`, `summary`, and `subjects` fields never enter the index either. The result shows up in the query table. The `subjects:roman` query returns 70 documents in dynamic mapping and 82 in explicit mapping; for `language:türkçe` the gap is 253 against 310. So because of the `issue` field's type, 57 Turkish books are **unsearchable** in the catalog, and there is no sign of this in the query result at all: the missing documents are silently absent. The reverse direction is also measured. The `donor:kaya` query returns 18 documents in dynamic mapping and 0 in explicit mapping; for `periodicity:monthly` it is 40 against 0. Because explicit mapping does not index 84 separate fields, these queries go unanswered. Explicit mapping's control is not free: every new question the import brings stays unanswered until a row is added to the mapping by hand. Index size sits in a misleading order in this table: dynamic mapping holds 660 terms in 161,329 bytes, explicit mapping holds 437 terms in 171,862 bytes. Explicit mapping is not larger because of its field count, it is larger because **it indexes 111 more documents**. The same number needs to be read per accepted document: 330 bytes per document in dynamic mapping, 286 bytes in explicit mapping. #### Summary - Dynamic mapping adds every incoming field to the mapping and infers its type from the first value; explicit mapping fixes the field list in advance and does not index anything not on the list. - When field names are derived from data, the mapping grows with the documents: 49 fields at the first 50 documents, 91 fields at 600; metadata rises from 1,684 bytes to 3,235 bytes. In explicit mapping the same line item is 211 bytes and stays fixed. - A type conflict drops the document, not the field: because the `issue` field arrives as text in one record and as a number in another, 111 of 600 records are rejected and never enter the index. - The loss is invisible in the query result but measurable: `subjects:roman` returns 70 instead of 82, `language:türkçe` returns 253 instead of 310. Nothing in the result signals that documents are missing. - Explicit mapping's cost is unanswered queries: because 84 fields are not indexed, `donor:kaya` and `periodicity:monthly` return 0 documents. #### Next Step This lesson's two mappings both took in documents once and let them go: a record arrived, and it either entered the index or it did not. The catalog, however, does not stand still — a book's summary gets corrected, a subject tag changes, a lost book is removed from the catalog. The next lesson measures what these three operations do to the index: why an update happens as a delete plus an add, how much room a deleted document keeps in the posting lists, the effect of the delete marker on the query result and the scanned entries, and how bloated the index gets before a cleanup pass. ### Document Lifecycle Source: https://academia.sh/en/courses/search-engines/document-lifecycle A document entering, changing, and leaving the index: an update happening as a delete plus an add, old posting entries staying in place, the room a delete marker takes per internal id, dead entries being filtered out at query time while adding to the scanned count, an updated document moving to the end of the result order, and how bloated the index gets before a cleanup round. The previous two lessons took in documents once and let them go: a record arrived, a mapping was built, the index was written. The catalog, however, does not stand still. A book's summary gets corrected, a subject tag changes, a lost book is removed from the catalog. This lesson measures what these three operations do to the index. The inverted index's structure imposes a constraint. Posting lists are kept per term and sorted by document id; when a document's text changes, that document's entries sit scattered across **dozens of separate lists**. Correcting these entries in place would mean finding every list and removing one record out of each. Instead, the index does the cheaper thing: it marks the old record **dead** and appends the new text as a new record at the end. An update is therefore not a separate operation, it is **a delete plus an add**. #### Internal Id and the Delete Marker This mechanism requires two ids. The **external id** is the catalog's book number and never changes. The **internal id** is the document's entry order in the index, and a new one is given on every write. Posting lists carry the internal id; the mapping between external id and internal id is kept separately. When a document is updated, the mapping turns to the new internal id, and the old one enters the **delete marker**. **IA1:** the corpus is unchanged — 600 records, seed 20250317; the operation sequence is generated from seed 20250318. **IA2:** the byte constants are the same as the previous lessons, and position is not kept. **IA3:** the delete marker is counted as 1 bit per internal id. **IA4:** every round applies 60 updates, 20 deletes, and 20 adds; once the dead-entry ratio exceeds 20 percent, the index is rewritten from live documents. **IA5:** dead entries are eliminated from the query result, but they are included in the scanned entries. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/lifecycle.mjs — document lifecycle: add, update (delete + add), and delete. Dead posting // entries, delete marker bytes, entries eliminated at query time, and cleanup rounds are counted. import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; import { InvertedIndex, SIMPLE, DOC_ID_BYTES, FREQ_BYTES } from "./index.mjs"; let c = SEED + 1; const random = () => (c = (c * 1103515245 + 12345) % 2147483648) / 2147483648; class Store { // document lifecycle layered on top of the inverted index #index = new InvertedIndex({ position: false }); #internal = new Map(); // external id -> internal id (index entry order) #text = new Map(); #deleted = new Set(); // delete marker: dead internal ids counter = 0; add(external, text) { // this is also how an update works: the old internal id is marked dead this.counter += 1; if (this.#internal.has(external)) this.#deleted.add(this.#internal.get(external)); this.#internal.set(external, this.counter); this.#text.set(external, text); this.#index.add(this.counter, text); } delete(external) { this.#deleted.add(this.#internal.get(external)); this.#internal.delete(external); this.#text.delete(external); } search(term) { // dead internal ids are eliminated from the result const l = this.#index.postings(term), reverse = new Map([...this.#internal].map(([d, i]) => [i, d])); const live = l.filter((g) => reverse.has(g.id)); return { results: live.map((g) => reverse.get(g.id)), postings: l.length, eliminated: l.length - live.length }; } rank(external, term) { const k = this.search(term).results.indexOf(external); return k < 0 ? "-" : `${k + 1}`; } measure() { const b = this.#index.bytes(); const dead = [...this.#index.dictionary.values()].flat().filter((g) => this.#deleted.has(g.id)).length; return { live: this.#internal.size, terms: b.terms, postings: b.postings, dead, index: b.total, marker: Math.ceil(this.counter / 8), share: Math.round(dead * 100 / b.postings) }; } cleanup() { // the index is rewritten from live documents only const live = [...this.#text]; this.#index = new InvertedIndex({ position: false }); this.#internal = new Map(); this.#deleted = new Set(); this.counter = 0; for (const [d, m] of live) this.add(d, m); } } const text = (b) => `${b.title} ${b.summary}`; const store = new Store(); for (const b of corpus) store.add(b.id, text(b)); const initial = store.measure(); console.log(`seed ${SEED}; ${DOC_COUNT} docs added to the index: ${initial.terms} terms, ${initial.postings} posting ` + `entries, ${initial.index} bytes, delete marker ${initial.marker} bytes (1 bit per internal id)`); // --- the mechanics of updating a single document --- const target = corpus.find((b) => SIMPLE(text(b)).includes("kuş")); const df = (t) => store.search(t).results.length; const shared = [...new Set(SIMPLE(text(target)))].filter((t) => t !== "kuş").sort((a, b) => df(b) - df(a))[0]; console.log(`\n#${target.id} "${target.title}": the summary's "kuş" token is being replaced with "martı"`); console.log(` before: "kuş" ${df("kuş")} docs (this doc ranks ${store.rank(target.id, "kuş")}), ` + `"martı" ${df("martı")} docs, ranks ${store.rank(target.id, shared)} in the "${shared}" query`); store.add(target.id, SIMPLE(text(target)).map((t) => (t === "kuş" ? "martı" : t)).join(" ")); console.log(` after : "kuş" ${df("kuş")} docs (this doc ranks ${store.rank(target.id, "kuş")}), ` + `"martı" ${df("martı")} docs, ranks ${store.rank(target.id, shared)} in the "${shared}" query`); const k = store.search("kuş"), o = store.search(shared); console.log(` "kuş" posting list carries ${k.postings} entries, ${k.eliminated} of them eliminated and dropped; ` + `"${shared}" list has ${o.eliminated} eliminated entries`); // --- rounds: each round applies 60 updates, 20 deletes, 20 adds; cleanup once the dead share passes 20% --- console.log(`\n${"round".padStart(5)}${"ops".padStart(7)}${"live docs".padStart(13)}${"postings".padStart(9)}` + `${"dead entries".padStart(13)}${"dead %".padStart(7)}${"index bytes".padStart(12)}${"marker".padStart(8)}${"after cleanup".padStart(18)}`); let next = DOC_COUNT + 1, ops = 0; for (let round = 1; round <= 6; round += 1) { for (let i = 0; i < 60; i += 1) { const b = corpus[Math.floor(random() * DOC_COUNT)]; store.add(b.id, `${text(b)} correction ${round}`); } for (let i = 0; i < 20; i += 1) store.delete(1 + Math.floor(random() * DOC_COUNT)); for (let i = 0; i < 20; i += 1) { const b = corpus[Math.floor(random() * DOC_COUNT)]; store.add(next, text(b)); next += 1; } ops += 100; const s = store.measure(), clean = s.share > 20; if (clean) store.cleanup(); console.log(String(round).padStart(5) + String(ops).padStart(7) + String(s.live).padStart(13) + String(s.postings).padStart(9) + String(s.dead).padStart(13) + `%${s.share}`.padStart(7) + String(s.index).padStart(12) + String(s.marker).padStart(8) + (clean ? `${store.measure().index} bytes` : "-").padStart(18)); } const final = store.measure(); console.log(`\nfinal state: ${final.live} live docs, ${final.postings} posting entries, ${final.dead} dead entries, ` + `${final.index} bytes; started at ${DOC_COUNT} docs holding ${initial.index} bytes`); ``` ``` seed 20250317; 600 docs added to the index: 185 terms, 14609 posting entries, 119722 bytes, delete marker 75 bytes (1 bit per internal id) #10 "Denizin Günleri": the summary's "kuş" token is being replaced with "martı" before: "kuş" 92 docs (this doc ranks 1), "martı" 0 docs, ranks 7 in the "üzerine" query after : "kuş" 91 docs (this doc ranks -), "martı" 1 docs, ranks 509 in the "üzerine" query "kuş" posting list carries 92 entries, 1 of them eliminated and dropped; "üzerine" list has 1 eliminated entries round ops live docs postings dead entries dead % index bytes marker after cleanup 1 100 600 16679 1996 %12 136323 86 - 2 200 603 18729 3856 %21 152732 96 121884 bytes 3 300 607 16961 1926 %11 138597 86 - 4 400 616 19029 3699 %19 155150 96 - 5 500 625 21052 5451 %26 171343 106 127721 bytes 6 600 635 17733 1758 %10 144786 89 - final state: 635 live docs, 17733 posting entries, 1758 dead entries, 144786 bytes; started at 600 docs holding 119722 bytes ``` #### The Trace a Single Update Leaves The second section follows a single document. The token `kuş` in record `#10`'s summary is replaced with `martı`, and three numbers move at once. The `kuş` query drops from 92 documents to 91; this document is no longer in the set. The `martı` query rises from 0 documents to 1. Up to this point, this is the expected behavior. The third number is unexpected. In the `üzerine` query, a term of the document that **did not change**, the record drops from rank 7 to **rank 509**. Nothing about that term changed in its text, and the reason it moves to the end of the order is the internal id: an update gives a new internal id, the new id is the largest one, and because posting lists are kept in id order, the new entry is appended to the end of the list. Correcting a single typo changes that document's place in every one of its result lists. The last line shows the price of the dead entry. The `kuş` posting list still carries **92 entries**; the query reads all 92 of them, eliminates 1, and returns 91 documents. The deleted document does not leave the index, it is only eliminated from the result. The delete marker is very cheap in return: 75 bytes for 600 documents, one bit per internal id. #### Rounds and Cleanup The third section follows what the same store does over six rounds. Every round applies 60 updates, 20 deletes, and 20 adds; that is, 80 of the round's operations leave a dead entry in the index. By the end of the first round, the index carries 1,996 of 16,679 entries dead (12 percent) and has grown from 119,722 bytes to 136,323 bytes. The live document count is still 600 — the entire growth is waste. In the second round the ratio rises to 21 percent, and because the threshold is exceeded, the index is rewritten from live documents: 152,732 bytes drops to **121,884 bytes**, recovering 30,848 bytes. The same cycle repeats in the fifth round; there the ratio reaches 26 percent, the index climbs to 171,343 bytes, and cleanup brings it down to 127,721 bytes. In the fourth round the ratio stays at 19 percent, so no cleanup runs, and 3,699 dead entries carry over for another round: the threshold is a decision about how much waste gets allowed to accumulate. By the end of the six rounds the store holds 635 live documents and takes up 144,786 bytes. The initial 600 documents held 119,722 bytes; while the document count grew 6 percent, the index grew 21 percent. The gap is the 1,758 dead entries that will carry over to the next cleanup. Delete and update are not free in the index: their cost is not paid instantly, it is paid **accumulating across rounds**. #### Summary - In the inverted index, an update is not an in-place correction: the old internal id is marked dead, and the new text is appended as a new record with a new internal id at the end. A delete only places a marker. - The delete marker is cheap — 75 bytes for 600 documents, one bit per internal id — but the entries it marks stay in the index: the `kuş` query reads 92 entries and returns 91 documents. - An update changes both the set and the order: record `#10` leaves the `kuş` set, enters the `martı` set, and drops from rank 7 to rank 509 in the `üzerine` query, which never changed at all. - Dead entries accumulate round by round: over 100 operations the index grows from 119,722 bytes to 136,323 bytes and 12 percent of its entries are dead; once the threshold is exceeded, a rewrite brings 152,732 bytes down to 121,884 bytes. - By the end of six rounds the document count grew 6 percent while the index grew 21 percent; the gap is the 1,758 dead entries carried over to the next cleanup. #### Next Step This topic built the index from the ground up: text was turned into terms, the structure mapping term to document was written, field types were decided, and a document became able to enter and leave the index. On the query side, though, only a single pattern was used. Every question in this topic consisted of bare terms, combined by a single rule — that all of them must be present. Whether a condition was optional, whether a condition excluded a document, whether two words were searched for side by side, or whether a year range was given — none of this was asked. Which order the result came back in was never chosen either: the order was, every time, the posting list's own layout, and this lesson's last measurement showed how arbitrary that layout is. The next topic starts from here — how conditions combine, which measure the result is sorted by, and how both decisions change the set and the order. ## Querying and Relevance Which documents come back in which order: the structure of leaf and compound queries, exact-match term queries versus order-sensitive phrase queries, must, must-not, and should context conditions, the distinction between a query that enters scoring and a filter that does not, term-frequency and inverse-document-frequency based relevance scoring, field weight and function-score tuning, metric and bucket aggregations, highlighting and the deep-pagination problem in result presentation, embedding-based semantic retrieval, and the hybrid approach. ### Query Language Structure Source: https://academia.sh/en/courses/search-engines/query-language-structure The same catalog question is asked in plain text and in structured form: the returned set ranges from 4,211 to 71 documents when how the conditions combine goes unsaid, the compound query is evaluated as a tree, and the tree's evaluation order lowers the entries scanned without changing the set. The previous topic built the inverted index and covered how a document enters and leaves it. Every question asked up to that point was a single term: a word was given, the posting list was read, documents came back. Questions asked against a library catalog do not look like that. "Short stories on children's literature" carries three separate conditions, and how those conditions combine is left unsaid: are all three required, does one suffice, is the word "short" searched in the book's title or in its summary? A quieter second question follows: in what order will the matching documents come back? This lesson separates two layers. A **leaf query** searches a single term in a single field and its answer is a posting list. A **compound query** combines leaves in a tree; the tree's root node is a connective, its leaves are terms. The same catalog question can be written as plain text or as a tree — the two return different numbers of documents and read different numbers of posting entries. #### Corpus **QR1 (assumption):** the corpus is 6,000 book records generated from a library catalog; the seed is 271828 and generation is deterministic. Each record carries a title, a summary, two tags, an author, a year, and a language. `title` and `summary` are text fields, `tag` is a keyword field. **QR2 (assumption):** query terms are written in the form they take in the index; suffixes and case are the analyzer chain's job, and the analyzer chain is fixed in this topic. ```js // corpus.mjs — library catalog corpus and position-aware inverted index. // Seed 271828, 6000 documents; every measurement in the topic shares this module. export const SEED = 271828, N = 6000; let c = SEED; // visible seed, deterministic generator const r = () => { c = (c + 0x6d2b79f5) | 0; let t = Math.imul(c ^ (c >>> 15), 1 | c); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; const pick = (d, e = 1) => d[Math.floor(r() ** e * d.length)]; // e>1: early items frequent, late items sparse const ADJECTIVE = ["critical", "short", "illustrated", "selected", "comparative"]; const GENRE = ["essay", "story", "novel", "study", "anthology"]; const CATEGORY = ["children", "history", "science", "philosophy", "society"]; const SUFFIX = ["selection", "book", "compilation", "series"]; const QUALIFIER = ["comprehensive", "concise", "introductory", "contentious"]; const SUBJECT = ["education", "migration", "city", "memory", "nature", "music", "law", "identity", "labor", "family", "war", "tradition", "health", "travel", "architecture", "archaeology", "seafaring", "astronomy"]; const AUTHOR = ["James Miller", "Anna Clarke", "Peter Brooks", "Laura Bennett", "Henry Cole"]; const LANGUAGE = ["english", "german", "french"]; export const CATALOG = []; for (let i = 1; i <= N; i += 1) { const s = pick(ADJECTIVE, 1.5), t = pick(GENRE, 1.5), a = pick(CATEGORY, 2), k = r(); const pattern = k < 0.18 ? `${s} ${t}` : k < 0.30 ? `${s} meets ${t}` : k < 0.40 ? `${s} and theoretical ${t}` : k < 0.50 ? `${t} and ${s} study` : k < 0.70 ? `${s} narrative piece` : k < 0.88 ? `a contemporary ${t}` : "review of literature"; CATALOG.push({ id: `K-${String(i).padStart(4, "0")}`, title: `${a} ${t} ${pick(SUFFIX)}`, summary: `${a} field: ${pattern}; readable on ${pick(SUBJECT, 3)} and ${pick(SUBJECT, 3)} as ` + `a ${pick(QUALIFIER)} ${pick(GENRE, 1.5)}.`, tag: [a, t], author: pick(AUTHOR), year: 1990 + Math.floor(r() * 35), language: pick(LANGUAGE, 2) }); } export const tokenize = (m) => m.toLocaleLowerCase("en-US").split(/[^\p{L}\p{N}]+/u).filter(Boolean); // Inverted index: "field|term" -> Map(document index -> positions); length: token count per document field. export function invertedIndex(fields) { const postings = new Map(), length = new Map(); CATALOG.forEach((b, i) => { for (const field of fields) { const tk = tokenize(String(b[field])); length.set(`${field}|${i}`, tk.length); tk.forEach((t, p) => { let list = postings.get(`${field}|${t}`); if (!list) postings.set(`${field}|${t}`, (list = new Map())); let positions = list.get(i); if (!positions) list.set(i, (positions = [])); positions.push(p); }); } }); return { postings, length }; } ``` #### Tree and Evaluation Order The evaluator recognizes three node types: leaf, `and`, `or`. Intersection works by galloping — for each entry of the left list, an exponential jump and a binary search run against the right list. **QR3 (assumption):** the posting list's length is read from the index dictionary and is known without reading its entries; **entries scanned** is counted only during the intersection and union walk. This makes the order in which a tree's children are evaluated a real decision: starting with the short list means only a handful of jumps into the long list. ```js // query.mjs — leaf and compound query evaluator. corpus.mjs is in the same directory. // Entries scanned: entries read from posting lists during intersection and union walks. // List length is known from the dictionary without reading entries. import { CATALOG, SEED, invertedIndex } from "./corpus.mjs"; const { postings } = invertedIndex(["title", "summary", "tag"]); const leaf = (field, term) => [...(postings.get(`${field}|${term}`)?.keys() ?? [])]; function gallop(list, target, from, s) { // exponential jump, then binary search let step = 1, i = from; while (i + step < list.length && list[i + step] < target) { step *= 2; s.scanned += 1; } let lo = i, hi = Math.min(i + step, list.length - 1); while (lo <= hi) { const mid = (lo + hi) >> 1; s.scanned += 1; if (list[mid] === target) return mid; if (list[mid] < target) lo = mid + 1; else hi = mid - 1; } return -lo - 1; } function intersect(a, b, s) { // a is scanned, b is jumped through const c = []; let j = 0; for (const d of a) { s.scanned += 1; const k = gallop(b, d, j, s); if (k >= 0) { c.push(d); j = k + 1; } else j = -k - 1; if (j >= b.length) break; } return c; } function union(lists, s) { const k = new Set(); for (const l of lists) { s.scanned += l.length; for (const d of l) k.add(d); } return [...k].sort((x, y) => x - y); } export function evaluate(node, s, optimize = false) { if (node.term) return leaf(node.field, node.term); if (node.or) return union(node.or.map((d) => evaluate(d, s, optimize)), s); const children = node.and.map((d) => evaluate(d, s, optimize)); if (optimize) children.sort((a, b) => a.length - b.length); return children.reduce((acc, l) => intersect(acc, l, s)); } const measure = (name, node, optimize = false) => { const s = { scanned: 0 }; const k = evaluate(node, s, optimize); console.log(`${name.padEnd(34)} ${String(k.length).padStart(4)} documents entries scanned ${String(s.scanned).padStart(6)}`); return k; }; console.log(`corpus ${CATALOG.length} documents, seed ${SEED}`); const T = (field, term) => ({ field, term }); const FIELDS = ["title", "summary", "tag"]; const flat = (term) => ({ or: FIELDS.map((f) => T(f, term)) }); measure("plain text, any of them", { or: ["children", "short", "story"].map(flat) }); measure("plain text, all required", { and: ["children", "short", "story"].map(flat) }); measure("structured", { and: [T("tag", "children"), T("summary", "short"), T("title", "story")] }); const tree = { and: [{ or: [T("title", "story"), T("title", "novel")] }, T("summary", "short"), T("tag", "children")] }; const r1 = measure("tree, written order", tree); const r2 = measure("tree, shortest list first", tree, true); console.log("same set:", JSON.stringify(r1) === JSON.stringify(r2)); for (const [f, t] of [["tag", "children"], ["summary", "short"], ["title", "story"], ["title", "novel"]]) console.log(` posting list ${f}|${t}`.padEnd(30), String(leaf(f, t).length).padStart(5)); // Order: same set, sorted two ways. How many conditions each document satisfies is counted. const s2 = { scanned: 0 }, tally = new Map(); for (const k of ["children", "short", "story"].map(flat)) for (const d of evaluate(k, s2)) tally.set(d, (tally.get(d) ?? 0) + 1); const indexOrder = [...tally.keys()].sort((a, b) => a - b); const conditionOrder = [...indexOrder].sort((a, b) => tally.get(b) - tally.get(a) || a - b); const topAllThree = (l) => l.slice(0, 10).filter((d) => tally.get(d) === 3).length; console.log(`all three conditions in top ten: index order ${topAllThree(indexOrder)}, condition order ${topAllThree(conditionOrder)}`); const full = conditionOrder.filter((d) => tally.get(d) === 3); const ranks = full.map((d) => indexOrder.indexOf(d) + 1); console.log(`${full.length} documents satisfying all three conditions rank ` + `${Math.round(ranks.reduce((x, y) => x + y, 0) / ranks.length)} on average in index order, the last at ${Math.max(...ranks)}.`); ``` ``` corpus 6000 documents, seed 271828 plain text, any of them 4211 documents entries scanned 19102 plain text, all required 132 documents entries scanned 21303 structured 71 documents entries scanned 7576 tree, written order 138 documents entries scanned 9501 tree, shortest list first 138 documents entries scanned 7876 same set: true posting list tag|children 2678 posting list summary|short 895 posting list title|story 1271 posting list title|novel 1007 all three conditions in top ten: index order 1, condition order 10 132 documents satisfying all three conditions rank 2023 on average in index order, the last at 4185. ``` #### Set: Three Readings of the Same Question Three lines come from the same three words and return three different sets. The "any of them" reading returns 4,211 documents — 70.2% of the corpus. This reading is practically useless in a catalog: the result list is as large as the library itself. The "all required" reading falls to 132 documents; which field a word appears in is still unconstrained, "story" can sit in either the book's title or its summary. The structured reading ties the three conditions to three separate fields — tagged children, "short" in the summary, "story" in the title — and returns 71 documents. The 61-document gap in between is not lost; those are books that carry the word "story" only in their summary, meaning they are not a story collection but a book that talks about one. Which set is correct depends on the question itself, and plain text does not say. Plain text is not a query; **it is a candidate reading of a query**, and the party that picks the reading is the engine. A catalog interface makes that choice, not the user — which is why two users typing the same three words into two different interfaces can see anywhere between 4,211 and 71 results. #### Cost: The Tree's Evaluation Order The bottom three lines hold the set fixed and measure cost. The compound tree is: books whose title carries "story" or "novel", whose summary carries "short", tagged children. Evaluated in written order, the union of the two long lists is taken first (1,271 and 1,007 entries), the result is then carried against the 2,678-entry tag list, and 9,501 entries are read in total. In the same tree, when the children are sorted by length, the intersection starts with the shortest list of 895 entries and the total falls to 7,876 entries — 17.1% less reading. The set returned in both runs is 138 documents; the comparison confirms it. The same effect is larger in the plain-text readings: "all required" reads 21,303 entries, because for every term the union of three fields is built first and those unions are long. The structured form comes out to 7,576 entries — 64.4% less — because here both the set and the cost fall together, since field separation prunes both unneeded documents and unneeded posting lists. Galloping intersection is what produces this difference: if list walking were linear, either order would have to read the entire long list. #### Order: The Set Does Not Give Order The array the evaluator returns is sorted by ascending document number, and that is not a relevance order — it is index insertion order. The last two lines measure this. Within the 4,211 documents of the "any of them" reading there are 132 documents satisfying all three conditions; in index order, only 1 of the top ten results belongs to that group. Sorted by satisfied-condition count, 10 of the top ten come from that group. The set has not changed, only the order has. Rank churn makes this difference larger: the 132 documents sit at position 2,023 on average in index order, the furthest at position 4,185. If a reader gives up after the first page, they never see the book they wanted, even though the set was correct all along. Satisfied-condition count is the crudest sort criterion there is; it does not account for how rare the terms are, which field they appear in, or how long that field is. All of that goes into a relevance score, and that score is built in this topic's fifth lesson. What this lesson leaves behind matters more: deciding the set and deciding the order are separate decisions, and a query form does not fix both at once. #### Summary - A leaf query searches a single term in a single field; a compound query combines leaves in a tree, and the tree's root determines the connective. - In a 6,000-document corpus (seed 271828) the same three words return 4,211, 132, and 71 documents across three readings; plain text does not say which reading applies, the engine picks it. - Field separation both narrows the set and lowers the cost: the structured form reads 7,576 entries, the plain-text "all required" reading reads 21,303. - When a tree's children are sorted by length, the set stays the same (138 documents) and entries scanned falls from 9,501 to 7,876. - The matching set is unordered on its own: the 132 documents satisfying all three conditions sit at position 2,023 on average in index order, but land in the top ten when sorted by satisfied-condition count. #### Next Step The structured query tied terms to fields, but still left the relationship between terms at "occurs in the same document." Part of what catalog questions ask goes further than that: two words inside a book's title, whose full title is not known, sit next to each other with nothing in between. "A critical essay" and "a critical and theoretical essay" carry the same two terms; in the first they are adjacent, in the second a word has come between them. The position information kept in the inverted index's posting lists exists exactly for this distinction, and it has not been used so far. The next lesson builds the phrase query: it turns term order and the distance between terms into a condition, counts how much slop widens the set, and measures how much position information costs the index. ### Term and Phrase Queries Source: https://academia.sh/en/courses/search-engines/term-and-phrase-queries The same two terms are asked first at document level, then at position level: a term query returns 757 documents, an adjacent phrase falls to 144, slop 3 raises that to 377, and the position information that makes this distinction possible grows the index by 51.7%. The previous lesson built the compound query as a tree and left the relationship between terms at "occurs in the same document." Part of catalog questions goes further than that. A reader does not remember a book's full title, but knows two words sit next to each other. "A critical essay" and "critical meets essay" carry the same two terms; in one they are adjacent, in the other a word has come between them. A document-level query cannot tell the two apart — and it also puts documents in the same set where the terms sit at opposite ends of a sentence with no relation between them at all. A **phrase query** makes this distinction: it turns not just the presence of terms but their positions into a condition. The position information kept in the inverted index's posting lists exists exactly for this and has not been used so far. **Slop** loosens the condition: it allows extra words in between. This lesson measures three numbers together — the set the phrase narrows, the documents slop brings back, and the price position information charges the index. #### Corpus and the Slop Definition The corpus is the previous lesson's module: 6,000 book records, seed 271828, deterministic generation. **QR4 (assumption):** slop is the number of extra words between two terms; if the terms match in the order written in the query, it is the word count between them, and if they match in reverse order, two is added to that count — one term crossing to the other side of the other counts as two steps. The measured phrase has two terms; longer phrases are the same rule applied in sequence. ```js // corpus.mjs — library catalog corpus and position-aware inverted index. // Seed 271828, 6000 documents; every measurement in the topic shares this module. export const SEED = 271828, N = 6000; let c = SEED; // visible seed, deterministic generator const r = () => { c = (c + 0x6d2b79f5) | 0; let t = Math.imul(c ^ (c >>> 15), 1 | c); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; const pick = (d, e = 1) => d[Math.floor(r() ** e * d.length)]; // e>1: early items frequent, late items sparse const ADJECTIVE = ["critical", "short", "illustrated", "selected", "comparative"]; const GENRE = ["essay", "story", "novel", "study", "anthology"]; const CATEGORY = ["children", "history", "science", "philosophy", "society"]; const SUFFIX = ["selection", "book", "compilation", "series"]; const QUALIFIER = ["comprehensive", "concise", "introductory", "contentious"]; const SUBJECT = ["education", "migration", "city", "memory", "nature", "music", "law", "identity", "labor", "family", "war", "tradition", "health", "travel", "architecture", "archaeology", "seafaring", "astronomy"]; const AUTHOR = ["James Miller", "Anna Clarke", "Peter Brooks", "Laura Bennett", "Henry Cole"]; const LANGUAGE = ["english", "german", "french"]; export const CATALOG = []; for (let i = 1; i <= N; i += 1) { const s = pick(ADJECTIVE, 1.5), t = pick(GENRE, 1.5), a = pick(CATEGORY, 2), k = r(); const pattern = k < 0.18 ? `${s} ${t}` : k < 0.30 ? `${s} meets ${t}` : k < 0.40 ? `${s} and theoretical ${t}` : k < 0.50 ? `${t} and ${s} study` : k < 0.70 ? `${s} narrative piece` : k < 0.88 ? `a contemporary ${t}` : "review of literature"; CATALOG.push({ id: `K-${String(i).padStart(4, "0")}`, title: `${a} ${t} ${pick(SUFFIX)}`, summary: `${a} field: ${pattern}; readable on ${pick(SUBJECT, 3)} and ${pick(SUBJECT, 3)} as ` + `a ${pick(QUALIFIER)} ${pick(GENRE, 1.5)}.`, tag: [a, t], author: pick(AUTHOR), year: 1990 + Math.floor(r() * 35), language: pick(LANGUAGE, 2) }); } export const tokenize = (m) => m.toLocaleLowerCase("en-US").split(/[^\p{L}\p{N}]+/u).filter(Boolean); // Inverted index: "field|term" -> Map(document index -> positions); length: token count per document field. export function invertedIndex(fields) { const postings = new Map(), length = new Map(); CATALOG.forEach((b, i) => { for (const field of fields) { const tk = tokenize(String(b[field])); length.set(`${field}|${i}`, tk.length); tk.forEach((t, p) => { let list = postings.get(`${field}|${t}`); if (!list) postings.set(`${field}|${t}`, (list = new Map())); let positions = list.get(i); if (!positions) list.set(i, (positions = [])); positions.push(p); }); } }); return { postings, length }; } ``` #### Measurement **QR5 (assumption):** index size is computed from the variable-length integer encoding of the posting lists: document numbers and positions are written as deltas, each number taking 1–4 bytes by its magnitude, and the dictionary counts the term name plus one byte. The index without positions writes the document delta and term frequency; the index with positions additionally writes the position deltas. ```js // phrase.mjs — term query versus phrase query. corpus.mjs is in the same directory. // Slop: the number of extra words allowed between two terms; if the terms match in // reverse order, two is added to this number. The measured phrase has two terms. import { CATALOG, SEED, invertedIndex } from "./corpus.mjs"; const { postings } = invertedIndex(["title", "summary", "tag"]); const list = (field, term) => postings.get(`${field}|${term}`) ?? new Map(); function closest(p1, p2, s) { // smallest cost between two position arrays let best = Infinity; for (const a of p1) for (const b of p2) { s.positions += 1; best = Math.min(best, b > a ? b - a - 1 : a - b + 1); } return best; } function phraseQuery(field, t1, t2, slop, s) { const l2 = list(field, t2), result = []; for (const [d, p1] of list(field, t1)) { const p2 = l2.get(d); if (p2 && closest(p1, p2, s) <= slop) result.push(d); } return result; } const vByte = (n) => (n < 128 ? 1 : n < 16384 ? 2 : n < 2097152 ? 3 : 4); // variable-length integer function indexBytes(withPositions) { let b = 0; for (const [key, l] of postings) { b += Buffer.byteLength(key) + 1; let previous = 0; for (const [d, position] of l) { b += vByte(d - previous) + vByte(position.length); previous = d; if (withPositions) { let p = 0; for (const k of position) { b += vByte(k - p); p = k; } } } } return b; } const report = (name, n, tail) => console.log(`${name.padEnd(30)} ${String(n).padStart(4)} documents ${tail}`); console.log(`corpus ${CATALOG.length} documents, seed ${SEED}`); console.log(`summary|critical ${list("summary", "critical").size} documents, summary|essay ${list("summary", "essay").size} documents`); const term = [...list("summary", "critical").keys()].filter((d) => list("summary", "essay").has(d)); report("term query, both present", term.length, `${" ".repeat(28)}positions read 0`); let previous = 0; for (const slop of [0, 1, 2, 3, 4]) { const s = { positions: 0 }; const k = phraseQuery("summary", "critical", "essay", slop, s); report(`phrase query, slop ${slop}`, k.length, `in ${String(k.length - previous).padStart(3)} out ${String(term.length - k.length).padStart(4)} ` + `positions read ${String(s.positions).padStart(5)}`); previous = k.length; } const noPositions = indexBytes(false), withPositions = indexBytes(true); console.log(`index bytes without positions ${noPositions} with positions ${withPositions} increase %${((withPositions / noPositions - 1) * 100).toFixed(1)}`); // Order: same set, two criteria. The term set is sorted by adjacency distance. const s1 = { positions: 0 }, distance = new Map(); for (const d of term) distance.set(d, closest(list("summary", "critical").get(d), list("summary", "essay").get(d), s1)); const closestOrder = [...term].sort((a, b) => distance.get(a) - distance.get(b) || a - b); const adjacent = (l) => l.slice(0, 10).filter((d) => distance.get(d) === 0).length; console.log(`adjacent match in top ten: index order ${adjacent(term)}, by distance ${adjacent(closestOrder)}`); const farthest = term.find((d) => distance.get(d) > 4); console.log(`adjacent ${CATALOG[closestOrder[0]].id}: ${CATALOG[closestOrder[0]].summary}`); console.log(`farthest ${CATALOG[farthest].id} (distance ${distance.get(farthest)}): ${CATALOG[farthest].summary}`); ``` ``` corpus 6000 documents, seed 271828 summary|critical 1487 documents, summary|essay 2983 documents term query, both present 757 documents positions read 0 phrase query, slop 0 144 documents in 144 out 613 positions read 869 phrase query, slop 1 230 documents in 86 out 527 positions read 869 phrase query, slop 2 303 documents in 73 out 454 positions read 869 phrase query, slop 3 377 documents in 74 out 380 positions read 869 phrase query, slop 4 377 documents in 0 out 380 positions read 869 index bytes without positions 220626 with positions 334724 increase %51.7 adjacent match in top ten: index order 1, by distance 10 adjacent K-0069: history field: critical essay; readable on memory and education as a contentious study. farthest K-0002 (distance 10): science field: critical narrative piece; readable on music and war as a comprehensive essay. ``` #### Set: The Phrase Leaves a Fifth of the Term Query Behind "Critical" occurs in 1,487 summaries, "essay" in 2,983; the count of documents carrying both is 757. The adjacent-phrase condition brings this set down to 144 documents — 19.0% of the term query. The 613 documents left out carry both terms, but not side by side. The output's last line shows one of them: in K-0002's summary "critical" qualifies a narrative, and "essay" names an entirely different genre at the other end of the sentence; the distance between the two terms is 10 and there is no meaning connecting them. The term query brought this document back wrongly; the phrase query does not. Slop loosens this condition step by step. One unit of slop lets in the "critical meets essay" shape and 86 documents enter; two units let in "critical and theoretical essay", 73 more enter; three units let in the reverse-order match — "essay and critical study" — and 74 more enter. At the fourth step, no document enters. This shows that raising slop does not have a constant effect: which distances exist is decided by the corpus's sentence shapes, slop only opens those steps one at a time. In a catalog interface, raising slop from 0 to 3 grows the set here from 144 to 377 — 2.6 times — while raising it to 4 changes nothing. #### Cost: Position Information Is Charged Against Every Document The cost at query time is small. The phrase condition reads positions only in the documents the two posting lists share: 757 candidate documents draw 869 position comparisons, because most documents carry each term once and only a single pair is compared. This number does not change when slop changes — slop is a threshold; it does not decide how many positions are read, only how many documents clear the threshold. The real cost sits in the index. The index without positions is 220,626 bytes, the index with positions is 334,724 bytes: a 51.7% increase, a 114,098-byte difference, roughly 19 bytes per document. The nature of this cost matters — it is paid **per document**, not per query. Position information is kept in every document whether or not a phrase query is ever asked, written at every indexing pass, carried through every segment merge. The decision is this: in a field where phrase queries are never needed, position information can be turned off and the index shrinks by roughly a third; in exchange, phrase queries can no longer be asked on that field, only an approximate result comes back from a term query — and that approximation's error is measured right here: 757 against 144. #### Order: Distance Is a Sort Criterion When the phrase condition is used as a threshold, the set splits sharply: 144 inside, 613 outside. The same distance can also be used as a sort criterion instead of a threshold, and then the set does not narrow at all. The last measurement shows this: when the 757-document term set is given in index order, only 1 of the top ten results is an adjacent match; when the same set is sorted by inter-term distance, all ten of the top ten are adjacent matches. The difference between the two approaches is the plainest example of this course's core distinction. The phrase condition used as a threshold changes the **set** and makes 613 documents entirely invisible; distance used as a sort criterion changes the **order** and leaves those 613 documents at the bottom of the list. In the first case, missing a document has no way back; in the second, the user finds it by scrolling down. Which is correct depends on how certain the question is: a threshold suits a book whose title is fully known, a sort criterion suits a topic search. #### Summary - A phrase query turns term positions, not just their presence, into a condition; in a 6,000-document corpus (seed 271828) it narrows the term query's 757 documents to 144. - Slop loosens the condition step by step: 1 unit adds 86 documents, 2 units add 73, 3 units add 74, 4 units add none. - A reverse-order match counts as two steps, which is why the "essay and critical study" shape only enters the set once slop reaches 3. - Position information is cheap at query time (869 position comparisons for 757 candidates) and expensive in the index: from 220,626 bytes to 334,724 bytes, a 51.7% increase. - The same distance measure splits the set when used as a threshold and only reorders it when used as a sort criterion: adjacent matches in the top ten rise from 1 to 10. #### Next Step Every condition built so far was the same kind: a document that fails the condition drops out of the set. Catalog questions are not this uniform. "It should be tagged children" is a required condition; "it should not be in French" is a condition that narrows the set but works in reverse; "summaries mentioning 'illustrated' should rank higher" is a request that eliminates no document at all, it only affects order. All three sit side by side inside a single compound query, and their effect on the set is entirely different from one another. The next lesson separates these three condition types and counts how an optional condition rewrites the top ten results without changing a single document in the set. ### Context Queries Source: https://academia.sh/en/courses/search-engines/context-queries Required, excluding, and optional conditions are measured in the same compound query: an excluding condition removes 512 documents from the set, an optional condition rewrites nine of the top ten results without removing a single document, but with no required condition present that same optional condition turns into the condition that determines the set. The two previous lessons built every condition the same way: a document that failed the condition dropped out of the set. Catalog questions are not this uniform. A reader says: "It should be a children's book, not in French, and illustrated ones should rank first." All three requests sit in one sentence, but the three do not do the same job. The first determines the set, the second removes from the set, the third removes no one from the set at all — it only changes what comes first. A compound query makes this distinction with **condition context**. A **must** condition eliminates a document that does not satisfy it. A **must-not** condition eliminates a document that does satisfy it. A **should** condition eliminates no document; it adds score to a document that satisfies it. This lesson counts the effect of the three contexts on the set and on order separately, and shows a place where a should condition changes the rule. #### Corpus and the Scoring Rule The corpus is the previous lessons' module: 6,000 book records, seed 271828, deterministic generation. **QR6 (assumption):** in this lesson score is the count of satisfied should conditions; ties are broken by document number. The real scoring model built on term frequency and rarity comes in this topic's fifth lesson — what is measured here is not the size of the score but the fact that a should condition touches order and does not touch the set. ```js // corpus.mjs — library catalog corpus and position-aware inverted index. // Seed 271828, 6000 documents; every measurement in the topic shares this module. export const SEED = 271828, N = 6000; let c = SEED; // visible seed, deterministic generator const r = () => { c = (c + 0x6d2b79f5) | 0; let t = Math.imul(c ^ (c >>> 15), 1 | c); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; const pick = (d, e = 1) => d[Math.floor(r() ** e * d.length)]; // e>1: early items frequent, late items sparse const ADJECTIVE = ["critical", "short", "illustrated", "selected", "comparative"]; const GENRE = ["essay", "story", "novel", "study", "anthology"]; const CATEGORY = ["children", "history", "science", "philosophy", "society"]; const SUFFIX = ["selection", "book", "compilation", "series"]; const QUALIFIER = ["comprehensive", "concise", "introductory", "contentious"]; const SUBJECT = ["education", "migration", "city", "memory", "nature", "music", "law", "identity", "labor", "family", "war", "tradition", "health", "travel", "architecture", "archaeology", "seafaring", "astronomy"]; const AUTHOR = ["James Miller", "Anna Clarke", "Peter Brooks", "Laura Bennett", "Henry Cole"]; const LANGUAGE = ["english", "german", "french"]; export const CATALOG = []; for (let i = 1; i <= N; i += 1) { const s = pick(ADJECTIVE, 1.5), t = pick(GENRE, 1.5), a = pick(CATEGORY, 2), k = r(); const pattern = k < 0.18 ? `${s} ${t}` : k < 0.30 ? `${s} meets ${t}` : k < 0.40 ? `${s} and theoretical ${t}` : k < 0.50 ? `${t} and ${s} study` : k < 0.70 ? `${s} narrative piece` : k < 0.88 ? `a contemporary ${t}` : "review of literature"; CATALOG.push({ id: `K-${String(i).padStart(4, "0")}`, title: `${a} ${t} ${pick(SUFFIX)}`, summary: `${a} field: ${pattern}; readable on ${pick(SUBJECT, 3)} and ${pick(SUBJECT, 3)} as ` + `a ${pick(QUALIFIER)} ${pick(GENRE, 1.5)}.`, tag: [a, t], author: pick(AUTHOR), year: 1990 + Math.floor(r() * 35), language: pick(LANGUAGE, 2) }); } export const tokenize = (m) => m.toLocaleLowerCase("en-US").split(/[^\p{L}\p{N}]+/u).filter(Boolean); // Inverted index: "field|term" -> Map(document index -> positions); length: token count per document field. export function invertedIndex(fields) { const postings = new Map(), length = new Map(); CATALOG.forEach((b, i) => { for (const field of fields) { const tk = tokenize(String(b[field])); length.set(`${field}|${i}`, tk.length); tk.forEach((t, p) => { let list = postings.get(`${field}|${t}`); if (!list) postings.set(`${field}|${t}`, (list = new Map())); let positions = list.get(i); if (!positions) list.set(i, (positions = [])); positions.push(p); }); } }); return { postings, length }; } ``` #### Measuring the Three Contexts ```js // context.mjs — must, must-not, and should conditions. corpus.mjs is in the same directory. // A should condition eliminates no document; it only produces score. Score here is the // count of satisfied should conditions; ties are broken by document number. import { CATALOG, SEED, invertedIndex } from "./corpus.mjs"; const { postings } = invertedIndex(["title", "summary", "tag", "language"]); const matchSet = (field, term) => new Set(postings.get(`${field}|${term}`)?.keys() ?? []); function run(query, s) { let candidate = null; for (const [field, term] of query.must ?? []) { const k = matchSet(field, term); s.scanned += k.size; candidate = candidate === null ? [...k] : candidate.filter((d) => k.has(d)); } const shouldSets = (query.should ?? []).map(([f, t]) => { const k = matchSet(f, t); s.scanned += k.size; return k; }); if (candidate === null) { // no must condition: should conditions determine the set const b = new Set(); for (const k of shouldSets) for (const d of k) b.add(d); candidate = [...b]; } for (const [field, term] of query.mustNot ?? []) { const k = matchSet(field, term); s.scanned += k.size; candidate = candidate.filter((d) => !k.has(d)); } candidate.sort((x, y) => x - y); const score = new Map(candidate.map((d) => [d, shouldSets.filter((k) => k.has(d)).length])); const ranked = [...candidate].sort((x, y) => score.get(y) - score.get(x) || x - y); return { set: candidate, score, ranked }; } const measure = (name, query) => { const s = { scanned: 0 }, c = run(query, s); console.log(`${name.padEnd(38)} ${String(c.set.length).padStart(4)} documents entries scanned ${String(s.scanned).padStart(5)}`); return c; }; console.log(`corpus ${CATALOG.length} documents, seed ${SEED}`); const MUST = [["tag", "children"]], SHOULD = [["summary", "illustrated"], ["title", "story"]], MUST_NOT = [["language", "french"]]; const a = measure("must[tag:children]", { must: MUST }); const b = measure("must + mustNot[language:french]", { must: MUST, mustNot: MUST_NOT }); const c = measure("must + should[illustrated, story]", { must: MUST, should: SHOULD }); const d = measure("must + should + mustNot", { must: MUST, should: SHOULD, mustNot: MUST_NOT }); const e = measure("should only[illustrated, story]", { should: SHOULD }); console.log(`documents removed by mustNot ${a.set.length - b.set.length}`); console.log(`should condition's effect on the set ${c.set.length - a.set.length} documents`); const topTen = (l) => l.slice(0, 10).map((x) => CATALOG[x].id); const changed = topTen(c.ranked).filter((x) => !topTen(a.ranked).includes(x)).length; console.log(`top ten, must only ${topTen(a.ranked).join(" ")}`); console.log(`top ten, +should ${topTen(c.ranked).join(" ")} changed ${changed}/10`); const twoMatches = c.ranked.filter((x) => c.score.get(x) === 2); console.log(`documents satisfying both should conditions ${twoMatches.length}; ` + `the first, ${CATALOG[twoMatches[0]].id}, was at position ${a.set.indexOf(twoMatches[0]) + 1} in the must-only order`); const movement = twoMatches.map((x) => a.set.indexOf(x) + 1 - (c.ranked.indexOf(x) + 1)); console.log(`these documents gain ${Math.round(movement.reduce((x, y) => x + y, 0) / movement.length)} positions on average, ` + `the largest gain ${Math.max(...movement)}`); ``` ``` corpus 6000 documents, seed 271828 must[tag:children] 2678 documents entries scanned 2678 must + mustNot[language:french] 2166 documents entries scanned 3800 must + should[illustrated, story] 2678 documents entries scanned 4638 must + should + mustNot 2166 documents entries scanned 5760 should only[illustrated, story] 1802 documents entries scanned 1960 documents removed by mustNot 512 should condition's effect on the set 0 documents top ten, must only K-0004 K-0008 K-0016 K-0018 K-0020 K-0023 K-0024 K-0026 K-0033 K-0037 top ten, +should K-0037 K-0072 K-0078 K-0080 K-0216 K-0345 K-0371 K-0518 K-0615 K-0712 changed 9/10 documents satisfying both should conditions 82; the first, K-0037, was at position 10 in the must-only order these documents gain 1263 positions on average, the largest gain 2574 ``` #### Set: Two Kinds of Condition Eliminate, One Does Not The must condition alone returns 2,678 documents — the tagged-children books. Once the must-not condition is added, the set falls to 2,166: 512 documents drop out, all of them French-language records. The number a must-not condition produces is always a subtraction; it never adds a document to the set, and used alone its meaning is unclear, because "not French" does not say from which set it should be subtracted. The third row is this lesson's core. The same must condition gained two should conditions and the set **stayed at 2,678 documents**: nothing entered, nothing left, the difference is zero. A book that is not illustrated and does not carry "story" in its title did not drop out of the result list; only its score stayed at zero. The fourth row puts the must-not condition back on the same query and the set falls to 2,166 again — because what changes the set is not the should condition, it is the must-not condition. The fifth row shows the boundary of the rule, and it is the point most often misunderstood in practice. Once the must condition is removed, the same two should conditions bring the set down to 1,802 documents. The should conditions have not suddenly started eliminating; what happens is this: once no condition remains to determine the set, **at least one should condition must be satisfied**, or the query would return the entire corpus. So the sentence "a should condition does not change the set" is not unconditionally true; the correct statement is: **as long as some other condition determines the set**, a should condition does not touch it. When an interface adds a condition for weighting purposes and the must condition is later removed, that condition dropping the result from 6,000 to 1,802 comes from this rule. #### Order and Cost: What the Non-Eliminating Condition Charges The set stays the same, yet the list becomes unrecognizable. The top ten results the must condition alone gives start with K-0004; once the should conditions are added, nine of the top ten change. The one document that survives is K-0037, and it climbs from tenth position to first. There are 82 documents that satisfy both should conditions; these documents climb 1,263 positions on average, the largest climbing 2,574 positions. In a 2,678-document list, a 2,574-position jump means the document moves from the very end of the list to the very front. The cost side shows that a non-eliminating condition is not free either. The must condition alone reads 2,678 entries. Once the two should conditions are added, reading rises to 4,638 entries: a 73.2% increase, without eliminating a single document. The reason is plain — computing a should condition's contribution to score requires reading the entire posting list, while a must condition does less work as the candidate set shrinks. With all three contexts together, reading reaches 5,760 entries; 1,122 of that is the must-not condition's list, read only to remove 512 documents. The operating rule that follows from this is: whether a condition is cheap or not is decided not by how many documents it eliminates, but by how many entries it makes you read. A condition that changes the set not at all can be the most expensive condition of all. #### Summary - A must condition determines the set, a must-not condition removes from it, a should condition only produces score; in a 6,000-document corpus (seed 271828) the must-not condition removes 512 documents, the should condition removes 0. - The should condition's neutrality toward the set is conditional: if no other condition determines the set, at least one should condition must be satisfied, and the set becomes 1,802 documents instead of 6,000. - Order is rewritten while the set stays fixed: nine of the top ten results change, the 82 documents satisfying both should conditions climb 1,263 positions on average, the largest climbing 2,574 positions. - A non-eliminating condition does not run for free: the two should conditions raise entries scanned from 2,678 to 4,638 without eliminating a single document. - A condition's cost is measured by the posting entries it makes you read, not by how many documents it eliminates. #### Next Step All three contexts meet at one point: every one of them passes in front of the scorer. The must condition both determines the set and contributes to score at the same time; the should condition exists for scoring alone. Yet part of catalog conditions should never enter the score at all. "Publication year 2010 or later" or "language is english" is a condition that is either satisfied or not; a book being a 2015 printing does not make it more relevant than a 2011 printing. These conditions are evaluated in a separate context, and that context has two consequences: the score computation shortens, and the result becomes reusable, because the same condition returns the same documents on every query. The next lesson runs the same condition in two contexts, shows that the set stays the same while the order changes, and counts how many entries the cached condition reads on its second run. ### Filtering and Query Distinction Source: https://academia.sh/en/courses/search-engines/filtering-and-query-distinction The same condition is run first in query context, then in filter context: the returned set stays fixed at 1,138 documents while nine of the top ten results change, only four of 1,138 documents stay in place, score computation is cut in half, and the filter's result can be reused as a 750-byte bit set. The previous lesson separated the three condition contexts, but all three shared one thing: all of them passed in front of the scorer. The must condition both determines the set and contributes to score at the same time; the should condition exists for scoring alone. Part of catalog conditions is wrong for this. "Publication year 2010 or later", "language is english", "subject heading mentions education" — conditions like these are either satisfied or not; a book being a 2015 printing does not make it more relevant than a 2011 printing. This distinction is named **filter context** and **query context**. The same condition passes the same documents in both contexts — the set does not change. What changes is whether the condition contributes to score, and that has two consequences: order comes out different, cost comes out different. This lesson runs the same condition in both contexts and compares three numbers: the documents returned, the top ten results, and the entries read. #### Corpus and the Scoring Rule The corpus is the previous lessons' module: 6,000 book records, seed 271828, deterministic generation. **QR7 (assumption):** in this lesson score is the sum, over every scored condition, of that condition's term frequency divided by the token count of that field. There is no rarity multiplier yet; the full model is built in this topic's fifth lesson. What is measured here is not the size of the score, but the effect on order of a condition entering or not entering scoring. ```js // corpus.mjs — library catalog corpus and position-aware inverted index. // Seed 271828, 6000 documents; every measurement in the topic shares this module. export const SEED = 271828, N = 6000; let c = SEED; // visible seed, deterministic generator const r = () => { c = (c + 0x6d2b79f5) | 0; let t = Math.imul(c ^ (c >>> 15), 1 | c); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; const pick = (d, e = 1) => d[Math.floor(r() ** e * d.length)]; // e>1: early items frequent, late items sparse const ADJECTIVE = ["critical", "short", "illustrated", "selected", "comparative"]; const GENRE = ["essay", "story", "novel", "study", "anthology"]; const CATEGORY = ["children", "history", "science", "philosophy", "society"]; const SUFFIX = ["selection", "book", "compilation", "series"]; const QUALIFIER = ["comprehensive", "concise", "introductory", "contentious"]; const SUBJECT = ["education", "migration", "city", "memory", "nature", "music", "law", "identity", "labor", "family", "war", "tradition", "health", "travel", "architecture", "archaeology", "seafaring", "astronomy"]; const AUTHOR = ["James Miller", "Anna Clarke", "Peter Brooks", "Laura Bennett", "Henry Cole"]; const LANGUAGE = ["english", "german", "french"]; export const CATALOG = []; for (let i = 1; i <= N; i += 1) { const s = pick(ADJECTIVE, 1.5), t = pick(GENRE, 1.5), a = pick(CATEGORY, 2), k = r(); const pattern = k < 0.18 ? `${s} ${t}` : k < 0.30 ? `${s} meets ${t}` : k < 0.40 ? `${s} and theoretical ${t}` : k < 0.50 ? `${t} and ${s} study` : k < 0.70 ? `${s} narrative piece` : k < 0.88 ? `a contemporary ${t}` : "review of literature"; CATALOG.push({ id: `K-${String(i).padStart(4, "0")}`, title: `${a} ${t} ${pick(SUFFIX)}`, summary: `${a} field: ${pattern}; readable on ${pick(SUBJECT, 3)} and ${pick(SUBJECT, 3)} as ` + `a ${pick(QUALIFIER)} ${pick(GENRE, 1.5)}.`, tag: [a, t], author: pick(AUTHOR), year: 1990 + Math.floor(r() * 35), language: pick(LANGUAGE, 2) }); } export const tokenize = (m) => m.toLocaleLowerCase("en-US").split(/[^\p{L}\p{N}]+/u).filter(Boolean); // Inverted index: "field|term" -> Map(document index -> positions); length: token count per document field. export function invertedIndex(fields) { const postings = new Map(), length = new Map(); CATALOG.forEach((b, i) => { for (const field of fields) { const tk = tokenize(String(b[field])); length.set(`${field}|${i}`, tk.length); tk.forEach((t, p) => { let list = postings.get(`${field}|${t}`); if (!list) postings.set(`${field}|${t}`, (list = new Map())); let positions = list.get(i); if (!positions) list.set(i, (positions = [])); positions.push(p); }); } }); return { postings, length }; } ``` #### Measuring the Two Contexts ```js // filter.mjs — the same condition, first in query context, then in filter context. Corpus: corpus.mjs. // Score here is term frequency divided by field length; the full model is built in the next lesson. import { CATALOG, SEED, invertedIndex } from "./corpus.mjs"; const { postings, length } = invertedIndex(["title", "summary", "tag", "language"]); const list = (field, term) => postings.get(`${field}|${term}`) ?? new Map(); function search(scored, filters, s) { let candidate = null; for (const [field, term] of [...scored, ...filters]) { const l = list(field, term); s.scanned += l.size; candidate = candidate === null ? [...l.keys()] : candidate.filter((d) => l.has(d)); } const score = new Map(); for (const d of candidate) { let p = 0; for (const [field, term] of scored) { s.computed += 1; p += list(field, term).get(d).length / length.get(`${field}|${d}`); } score.set(d, p); } return { set: candidate, score, ranked: [...candidate].sort((x, y) => score.get(y) - score.get(x) || x - y) }; } const measure = (name, scored, filters) => { const s = { scanned: 0, computed: 0 }, c = search(scored, filters, s); console.log(`${name.padEnd(40)} ${String(c.set.length).padStart(4)} documents score computed ${String(s.computed).padStart(4)}` + ` entries scanned ${String(s.scanned).padStart(5)}`); return c; }; console.log(`corpus ${CATALOG.length} documents, seed ${SEED}`); const QUERY = [["summary", "story"]], CONDITION = [["summary", "education"]]; const a = measure("query context: story + education scored", [...QUERY, ...CONDITION], []); const b = measure("filter context: education only filters", QUERY, CONDITION); console.log(`same set: ${JSON.stringify(a.set) === JSON.stringify(b.set)}`); const topTen = (c) => c.ranked.slice(0, 10).map((d) => CATALOG[d].id); console.log(`top ten, query context ${topTen(a).join(" ")}`); console.log(`top ten, filter context ${topTen(b).join(" ")}`); console.log(`changed in top ten ${topTen(a).filter((x) => !topTen(b).includes(x)).length}/10`); console.log(`distinct score values: query context ${new Set(a.score.values()).size}, ` + `filter context ${new Set(b.score.values()).size}`); const movement = a.set.map((d) => Math.abs(a.ranked.indexOf(d) - b.ranked.indexOf(d))); console.log(`position movement: average ${Math.round(movement.reduce((x, y) => x + y, 0) / movement.length)}, ` + `largest ${Math.max(...movement)}, unchanged ${movement.filter((o) => o === 0).length} documents`); // Cache: the filter's result is independent of the query, stored as a bit set. const cache = new Map(); function filterSet(field, term, s) { const key = `${field}|${term}`; if (cache.has(key)) { s.hits += 1; return cache.get(key); } const l = list(field, term); s.scanned += l.size; const bits = new Uint8Array(Math.ceil(CATALOG.length / 8)); for (const d of l.keys()) bits[d >> 3] |= 1 << (d & 7); cache.set(key, bits); return bits; } const FIVE = ["story", "novel", "essay", "study", "anthology"]; for (const cached of [false, true]) { cache.clear(); const s = { scanned: 0, hits: 0 }; for (const t of FIVE) { list("summary", t); s.scanned += list("summary", t).size; if (cached) filterSet("summary", "education", s); else s.scanned += list("summary", "education").size; } console.log(`five queries, same filter ${cached ? "cached " : "uncached "} ` + `entries scanned ${String(s.scanned).padStart(6)} cache hits ${s.hits}`); } console.log(`filter bit set ${Math.ceil(CATALOG.length / 8)} bytes; ` + `the same condition's score array ${list("summary", "education").size * 8} bytes`); ``` ``` corpus 6000 documents, seed 271828 query context: story + education scored 1138 documents score computed 2276 entries scanned 5513 filter context: education only filters 1138 documents score computed 1138 entries scanned 5513 same set: true top ten, query context K-0836 K-2188 K-4924 K-5913 K-1025 K-1207 K-2800 K-3095 K-3253 K-3683 top ten, filter context K-0171 K-0360 K-0417 K-0709 K-0836 K-0958 K-1185 K-1325 K-1520 K-1670 changed in top ten 9/10 distinct score values: query context 9, filter context 6 position movement: average 173, largest 791, unchanged 4 documents five queries, same filter uncached entries scanned 27836 cache hits 0 five queries, same filter cached entries scanned 13260 cache hits 4 filter bit set 750 bytes; the same condition's score array 29152 bytes ``` #### Set Stays, Order Does Not Both runs return 1,138 documents, and the comparison confirms the sets are literally identical. This is the expected result: in both contexts the condition is required, the documents it passes are the same. On the set side, filter context has no effect at all. On the order side, the effect is large. Nine of the top ten results change; the only record surviving between the two lists is K-0836. Only **four** of the 1,138 documents stay in place, the average position movement is 173, the largest is 791. The source of the difference is score resolution: in filter context, score comes from a single condition and takes 6 distinct values; in query context, score comes from two conditions and takes 9 distinct values. As the number of distinct values grows, ties get broken and the list gets reshuffled. The decision that follows is clear. "It should be about education" is not a relevance measure, it is a scope restriction; left in query context, a book with a short summary or one that mentions "education" twice climbs to the top of the list. This ordering has nothing to do with what the user asked — the user is not asking how many times the word education occurs, they are looking for a story book. Filter context leaves ordering to the user's actual question. #### Cost: Computation, Reading, and Cache Score computation is cut in half: from 2,276 to 1,138. For every candidate document, one multiplication and one division are done instead of two, and this saving grows in proportion to the candidate count. Entries scanned does not change on the first run (5,513) — a filter still reads a posting list in the end. The real gain shows up on the second run. The filter's result is **independent of the query**: whatever free-text query is asked, the set "summaries mentioning education" is the same. This means the result can be stored as a bit set, and later queries never touch the posting list again. When five different queries share the same filter, an uncached run reads 27,836 entries; a cached run reads 13,260 entries — 52.4% less, with four cache hits. The cache's size makes this decision easy. For a 6,000-document corpus, the bit set is 750 bytes. Storing the same condition's score contribution instead would need an eight-byte decimal for each of 3,644 documents, which is 29,152 bytes — thirty-nine times as much. Storing the score is also pointless besides, because the final score is computed together with the query's other conditions and comes out different on every query. The filter owes its cacheability entirely to the fact that it does not enter scoring: the result of a condition that does not enter scoring is a boolean value, and a boolean value does not change from query to query. The filter is not free of cost. The cache consumes memory and invalidates on every document insert or delete; keeping a rarely asked condition in the cache wastes memory for nothing. The measure is simple: how many separate queries repeat the same condition. The language and subject restrictions in a library interface repeat on every query, the free text the user types does not. #### Summary - The same required condition returns the same set in both contexts: in a 6,000-document corpus (seed 271828), both runs return 1,138 documents. - Order changes entirely: nine of the top ten results differ, only four of the 1,138 documents stay in place, average position movement is 173, the largest is 791. - The source of the difference is score resolution: filter context produces 6 distinct score values, query context produces 9. - Filter context cuts score computation in half (2,276 to 1,138) and, by making the result independent of the query, makes it cacheable: 13,260 entries instead of 27,836 across five queries. - The filter cache is 750 bytes, the same condition's score array is 29,152 bytes; storing the score is pointless besides, because the final score comes out different on every query. #### Next Step Up to this point score was always computed with a placeholder formula: first satisfied-condition count, then term frequency divided by field length. Both produced an ordering, but both skipped the real other half of the question. A term occurring in a document carries no meaning without knowing how rare that term is across the corpus: the word "and" occurs in every one of the 6,000 documents and distinguishes nothing, while "astronomy" occurs in about two hundred documents and nearly determines the document by itself. The next lesson splits score into three components — term frequency, inverse document frequency, and field length — shows which component a document's position comes from, and defines precision and recall to measure how much of the result is correct. ### Relevance Scoring Source: https://academia.sh/en/courses/search-engines/relevance-scoring The scorer is built from three components — term frequency, inverse document frequency, and field length — and which component a document's position comes from is shown one by one; precision and recall are defined here, and the measurement shows ranking raising precision from 0.353 to 0.640 while hitting a ceiling on a measure the model cannot see. The previous lesson always computed score with a placeholder formula: first satisfied-condition count, then term frequency divided by field length. Both produced an ordering, but both skipped half the question. A term occurring in a document carries no meaning without knowing how rare that term is across the corpus: the word "and" occurs in every one of the catalog's 6,000 summaries and distinguishes nothing, "astronomy" occurs in about two hundred summaries and nearly determines the document by itself. **Relevance score** is built from three components. **Term frequency** is how many times the term occurs in the document, and it saturates — a fifth occurrence says less than the first. **Inverse document frequency** is the term's rarity across the corpus; a rare term carries much weight, a common term carries little. **Field length** normalization distinguishes a match in a three-word title from a match in a fourteen-word summary. This lesson builds the scorer, shows which component a document's position comes from, and defines two measures for how correct the result is. #### Corpus, Model, and Requirement The corpus is the previous lessons' module: 6,000 book records, seed 271828, deterministic generation. **QR8 (assumption):** the scored fields are `title` and `summary`; there is no field weight, the two are summed one-to-one. The term frequency saturation constant is 1.2, the field length normalization constant is 0.75; inverse document frequency is found from the logarithm of the ratio between corpus size and document frequency. **QR9 (assumption):** the reader's requirement is broader than the query — they are looking for "a children's book tagged story, in english", but type only "children story" into the search box. The `tag`, `language`, and `year` fields are not indexed in this lesson; the scorer cannot see the requirement's language criterion. This is the ordinary state of affairs in a real catalog. ```js // corpus.mjs — library catalog corpus and position-aware inverted index. // Seed 271828, 6000 documents; every measurement in the topic shares this module. export const SEED = 271828, N = 6000; let c = SEED; // visible seed, deterministic generator const r = () => { c = (c + 0x6d2b79f5) | 0; let t = Math.imul(c ^ (c >>> 15), 1 | c); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; const pick = (d, e = 1) => d[Math.floor(r() ** e * d.length)]; // e>1: early items frequent, late items sparse const ADJECTIVE = ["critical", "short", "illustrated", "selected", "comparative"]; const GENRE = ["essay", "story", "novel", "study", "anthology"]; const CATEGORY = ["children", "history", "science", "philosophy", "society"]; const SUFFIX = ["selection", "book", "compilation", "series"]; const QUALIFIER = ["comprehensive", "concise", "introductory", "contentious"]; const SUBJECT = ["education", "migration", "city", "memory", "nature", "music", "law", "identity", "labor", "family", "war", "tradition", "health", "travel", "architecture", "archaeology", "seafaring", "astronomy"]; const AUTHOR = ["James Miller", "Anna Clarke", "Peter Brooks", "Laura Bennett", "Henry Cole"]; const LANGUAGE = ["english", "german", "french"]; export const CATALOG = []; for (let i = 1; i <= N; i += 1) { const s = pick(ADJECTIVE, 1.5), t = pick(GENRE, 1.5), a = pick(CATEGORY, 2), k = r(); const pattern = k < 0.18 ? `${s} ${t}` : k < 0.30 ? `${s} meets ${t}` : k < 0.40 ? `${s} and theoretical ${t}` : k < 0.50 ? `${t} and ${s} study` : k < 0.70 ? `${s} narrative piece` : k < 0.88 ? `a contemporary ${t}` : "review of literature"; CATALOG.push({ id: `K-${String(i).padStart(4, "0")}`, title: `${a} ${t} ${pick(SUFFIX)}`, summary: `${a} field: ${pattern}; readable on ${pick(SUBJECT, 3)} and ${pick(SUBJECT, 3)} as ` + `a ${pick(QUALIFIER)} ${pick(GENRE, 1.5)}.`, tag: [a, t], author: pick(AUTHOR), year: 1990 + Math.floor(r() * 35), language: pick(LANGUAGE, 2) }); } export const tokenize = (m) => m.toLocaleLowerCase("en-US").split(/[^\p{L}\p{N}]+/u).filter(Boolean); // Inverted index: "field|term" -> Map(document index -> positions); length: token count per document field. export function invertedIndex(fields) { const postings = new Map(), length = new Map(); CATALOG.forEach((b, i) => { for (const field of fields) { const tk = tokenize(String(b[field])); length.set(`${field}|${i}`, tk.length); tk.forEach((t, p) => { let list = postings.get(`${field}|${t}`); if (!list) postings.set(`${field}|${t}`, (list = new Map())); let positions = list.get(i); if (!positions) list.set(i, (positions = [])); positions.push(p); }); } }); return { postings, length }; } ``` #### Scorer ```js // score.mjs — relevance scorer. corpus.mjs is in the same directory. // Score comes from three components: term frequency (which saturates), inverse document // frequency, and field length normalization. import { CATALOG, SEED, invertedIndex } from "./corpus.mjs"; const { postings, length } = invertedIndex(["title", "summary"]); // tag, language, and year are not indexed const N = CATALOG.length, FIELDS = ["title", "summary"], K = 1.2, B = 0.75; const list = (field, term) => postings.get(`${field}|${term}`) ?? new Map(); const avgLength = Object.fromEntries(FIELDS.map((f) => { let t = 0; for (let d = 0; d < N; d += 1) t += length.get(`${f}|${d}`); return [f, t / N]; })); const idf = (field, term) => { const df = list(field, term).size; return Math.log(1 + (N - df + 0.5) / (df + 0.5)); }; function component(field, term, d, mode) { const tf = (list(field, term).get(d) ?? []).length; if (tf === 0) return 0; if (mode === "term frequency only") return tf; if (mode === "term x inverse document frequency") return tf * idf(field, term); const len = length.get(`${field}|${d}`) / avgLength[field]; // field length normalization return idf(field, term) * (tf * (K + 1)) / (tf + K * (1 - B + B * len)); } const scoreDoc = (d, mode) => TERM.reduce((p, t) => p + FIELDS.reduce((q, f) => q + component(f, t, d, mode), 0), 0); const TERM = ["children", "story"], MODES = ["term frequency only", "term x inverse document frequency", "full model"]; const candidate = [...Array(N).keys()].filter((d) => TERM.every((t) => FIELDS.some((f) => list(f, t).has(d)))); const relevant = new Set(candidate.filter((d) => CATALOG[d].tag.includes("children") && CATALOG[d].tag.includes("story") && CATALOG[d].language === "english")); console.log(`corpus ${N} documents, seed ${SEED}; question "children story"`); console.log(`candidate set ${candidate.length} documents; requirement: tagged children+story and english -> ${relevant.size} documents relevant`); for (const t of TERM) for (const f of FIELDS) console.log(` ${f}|${t}`.padEnd(21) + `document frequency ${String(list(f, t).size).padStart(4)} ` + `inverse document frequency ${idf(f, t).toFixed(3)} average field length ${avgLength[f].toFixed(2)}`); const rankedBy = (mode) => [...candidate].sort((x, y) => scoreDoc(y, mode) - scoreDoc(x, mode) || x - y); const full = rankedBy("full model"); console.log("score components: first and five-hundredth position"); for (const d of [full[0], full[499]]) { const parts = []; for (const t of TERM) for (const f of FIELDS) { const k = component(f, t, d, "full model"); if (k > 0) parts.push(`${f}|${t} ${k.toFixed(2)} (frequency ${(list(f, t).get(d) ?? []).length}, length ${length.get(`${f}|${d}`)})`); } console.log(` ${String(full.indexOf(d) + 1).padStart(3)}. ${CATALOG[d].id} score ${scoreDoc(d, "full model").toFixed(2)} = ${parts.join(" + ")}`); } const precision = (l, k) => l.slice(0, k).filter((d) => relevant.has(d)).length / Math.min(k, l.length); const recall = (l, k) => l.slice(0, k).filter((d) => relevant.has(d)).length / relevant.size; console.log("ranking model".padEnd(36) + "distinct scores top ten shared largest movement precision@10 precision@50"); for (const mode of MODES) { const l = rankedBy(mode); const values = new Set(candidate.map((d) => scoreDoc(d, mode).toFixed(6))).size; const shared = l.slice(0, 10).filter((d) => full.slice(0, 10).includes(d)).length; const movement = Math.max(...candidate.map((d) => Math.abs(l.indexOf(d) - full.indexOf(d)))); console.log(mode.padEnd(36) + String(values).padStart(11) + String(shared).padStart(14) + String(movement).padStart(17) + precision(l, 10).toFixed(3).padStart(13) + precision(l, 50).toFixed(3).padStart(13)); } console.log(`unranked set: precision ${(relevant.size / candidate.length).toFixed(3)}, recall 1.000`); for (const k of [10, 50, 200, 567, 984]) console.log(` top ${String(k).padStart(3)} results: precision ${precision(full, k).toFixed(3)} recall ${recall(full, k).toFixed(3)}`); const wrong = full.slice(0, 50).filter((d) => !relevant.has(d)); console.log(`the first of the ${wrong.length} wrong documents in the top 50 is ${CATALOG[wrong[0]].id}: ` + `${CATALOG[wrong[0]].title}, language ${CATALOG[wrong[0]].language}, tag ${CATALOG[wrong[0]].tag.join("+")}`); ``` ``` corpus 6000 documents, seed 271828; question "children story" candidate set 984 documents; requirement: tagged children+story and english -> 347 documents relevant title|children document frequency 2678 inverse document frequency 0.807 average field length 3.00 summary|children document frequency 2678 inverse document frequency 0.807 average field length 14.02 title|story document frequency 1271 inverse document frequency 1.552 average field length 3.00 summary|story document frequency 1869 inverse document frequency 1.166 average field length 14.02 score components: first and five-hundredth position 1. K-0078 score 4.83 = title|children 0.81 (frequency 1, length 3) + summary|children 0.83 (frequency 1, length 13) + title|story 1.55 (frequency 1, length 3) + summary|story 1.64 (frequency 2, length 13) 500. K-3736 score 3.17 = title|children 0.81 (frequency 1, length 3) + summary|children 0.81 (frequency 1, length 14) + title|story 1.55 (frequency 1, length 3) ranking model distinct scores top ten shared largest movement precision@10 precision@50 term frequency only 3 3 468 0.700 0.640 term x inverse document frequency 4 3 342 0.700 0.640 full model 10 10 0 0.600 0.640 unranked set: precision 0.353, recall 1.000 top 10 results: precision 0.600 recall 0.017 top 50 results: precision 0.640 recall 0.092 top 200 results: precision 0.615 recall 0.354 top 567 results: precision 0.612 recall 1.000 top 984 results: precision 0.353 recall 1.000 the first of the 18 wrong documents in the top 50 is K-0171: children story selection, language french, tag children+story ``` #### Where a Position Comes From The first table gives the source of the components. "Children" occurs in 2,678 documents and its inverse document frequency is 0.807; "story" occurs in 1,271 book titles and carries 1.552. The same term occurs in 1,869 summaries, and there it is only 1.166 — **the same word's weight changes from field to field**, because rarity is measured within the field. The first-position document, K-0078, has a score of 4.83 and is the sum of four components, each greater than zero. Of that score, 3.19 — 66% — comes from the term "story"; "children" contributes 1.64. What determines order is the query's **rare** term. The five-hundredth-position document, K-3736, has the same three components and its only difference is that the fourth component is missing: "story" never occurs in its summary. The absence of one component costs 1.66 points and 499 positions. Both are children's story books in the catalog; the only difference is that one's summary mentions the genre twice. Field length's contribution is smaller but visible. K-0078's summary is 13 tokens, K-3736's is 14; the average is 14.02. This one-token difference lowers the `summary|children` component from 0.83 to 0.81. In the title field, average length is exactly 3.00, because every title is three words, so the normalization has no effect there, so field length distinguishes nothing where every title is the same length; normalization only works where length varies. #### Three Models, Three Orders The model table measures what the components do to order. With term frequency alone, the 984 candidate documents take only **3 distinct score values**: the list splits into three piles and order within a pile falls to document number. Once inverse document frequency is added, the value count rises to 4; once field length normalization is also added, it rises to 10. As resolution rises, ties get broken and the list is reshuffled. Against the full model, the largest position movement is 468 in the term-frequency-only model, 342 in the intermediate model — in a 984-document list, this means a document moves from the middle to the front. In the top ten results, the three models share only 3 documents; the full model's top-ten precision is 0.600, below the other two models' 0.700. This does not show the full model is worse: in the top fifty all three models come to 0.640, and the difference is noise from a measure the model cannot see. Choosing a model from a ten-result sample misuses the measure itself. #### Precision and Recall Two measures ask how correct the result is from different directions. **Precision** is how much of the returned result is relevant: it penalizes a document wrongly returned. **Recall** is how much of the relevant documents were returned: it penalizes a document missed. The same pair goes by other names in the same field elsewhere in this platform: in the Non-Functional Testing curriculum's Static Application Security Testing lesson, a finding wrongly reported is a **false positive**, a real vulnerability never reported is a **miss**; precision measures false positives, recall measures misses. The unranked set is 984 documents and holds 347 relevant documents: precision 0.353, recall 1.000. This is the typical balance of a Boolean query — no relevant document is missed, but two-thirds are unneeded. Ranking ties this balance to the cutoff point. In the top 50 results precision rises to 0.640, recall falls to 0.092. In the top 200, precision is 0.615, recall 0.354. At the top 567, recall reaches 1.000: all 347 relevant documents sit within the first 58% of the list, meaning the scorer has never pushed a relevant document to the bottom of the list. Precision sticking around 0.64 is this lesson's final finding. The first of the 18 wrong documents in the top 50 is K-0171, and its title is "children story selection" — exactly the kind of book the reader is looking for, only it is in French. The scorer has ranked it high correctly, since it is a textually flawless match; what fails the requirement sits in a field the scorer never sees. The rule is this: **when a criterion sits in a field the scorer cannot see, no weight adjustment raises precision.** The right tool is the previous lesson's filter; a score adjustment only works in fields the model can see. #### Summary - Relevance score comes from three components: saturating term frequency, inverse document frequency, and field length normalization; the same term carries 1.552 weight in a book title, 1.166 in a summary. - 66% of the first-position document's 4.83 score comes from the query's rare term; the absence of one component costs 1.66 points and 499 positions. - Resolution rises as components are added: the 984 candidate documents take 3, 4, and 10 distinct score values; the largest position movement is 468. - Precision is how much of the returned result is relevant, recall is how much of the relevant documents were returned; in the unranked set precision is 0.353 and recall is 1.000. - Ranking raises precision to 0.640 in the top 50 results and carries recall to 1.000 by the top 567; but because the requirement's language criterion is not indexed, precision hits a ceiling around 0.64. #### Next Step The scorer works, and which component a produced order comes from can now be seen. This does not mean the order is the wanted order. For a library, a match in a book's title is a stronger signal than a match in its summary, yet in the current model the two are just summed one-to-one. Newer printings might be wanted to rank ahead of older ones, but publication year does not enter the score at all. Loan count, a quantity unrelated to the text, might enter the ranking too. All of these are outside interventions on the score's components, each changing order without changing the set — like the should condition, but now with a measured coefficient. The next lesson builds these adjustments and counts how many documents each adjustment moves in the top ten results and by how many points it shifts precision. ### Score Tuning Source: https://academia.sh/en/courses/search-engines/score-tuning Two adjustments that rewrite the same query's order: how many documents field weight changes in the top ten and how many ranks it moves a document, how field selection — unlike weight — narrows the matching set, at what weight function-based scoring lets recency and popularity override relevance, and the difference between the two combination modes. The previous lesson broke the relevance score into its components: how many times a term appears in a document, how many documents it appears in, and how long the document is. The decomposition was done, but the weight of the components relative to one another was taken as given; every field counted the same, and that was never questioned as a decision. In the catalog, the question is concrete: for a reader searching "ocean ship," is a word appearing in a book's **title** the same as appearing in its **summary**? What does appearing in the author's name mean? This lesson measures two adjustments. **Field weight** is the coefficient that enters the score depending on which field the same term is found in. **Function-based scoring** is the inclusion in the score of a quantity unrelated to the text — publication year, loan count. Both are defended with the justification of "a more relevant result"; here that justification is converted into a number. #### Corpus and Four Fields The entire measurement runs on a single corpus: 1,200 book records generated from a library catalog, seed 20260801. Each record is indexed in four fields. | Code | Assumption | Value | Rationale | |---|---|---|---| | QR10 | analyzer chain | established in the previous topic | tokens arrive normalized | | QR11 | indexed fields | title, summary, subject, author | each with its own posting list | | QR12 | subject tag | differs from the text's theme in a quarter of records | the tag is a publisher decision | | QR13 | loan count and year | sits in the index as a document field | input to function-based scoring | QR12 is this lesson's distinguishing assumption: if the fields were copies of one another, weight would move nothing. ```js // corpus.mjs — library catalog: seeded corpus, inverted index, and per-field scorer. export const N = 1200, SEED = 20260801, FIELDS = ["title", "summary", "subject", "author"]; let d = SEED; // 32-bit generator, no overflow export const random = () => { d = (d + 0x6D2B79F5) | 0; let t = Math.imul(d ^ (d >>> 15), 1 | d); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32; }; const pick = (a) => a[Math.floor(random() * a.length)]; const THEMES = [["story|tale", "literature", "short selection collection narrative fiction rural"], ["fable", "children's literature", "child illustrated school youth forest sleep"], ["history", "history", "ottoman republic archive document chronicle foundation"], ["ocean", "travel", "coast fisherman harbor ship island lighthouse"], ["mathematics", "science", "geometry number proof theory solution probability"], ["poetry", "poetry", "verse collected divan translation selection meter"]] .map(([k, subject, a]) => ({ k: k.split("|"), subject, core: a.split(" ") })); const COMMON = "book volume edition publication review notes introduction glossary".split(" "); const FIRST_NAMES = "James Emma Simon Nora Kevin Ocean Grace Owen Meredith Blake".split(" "); const SURNAMES = "Stone Bishop Miller Hayes Reed Frost".split(" "); export const documents = Array.from({ length: N }, (_, id) => { const t = pick(THEMES), draw = t.k[t.k.length > 1 && random() < 0.5 ? 1 : 0], sm = [draw, draw]; const title = [...new Set([draw, pick(t.core), pick(t.core)])]; t.core.forEach((s, j) => { if (random() < 1 / (1 + j * 0.42)) sm.push(s); }); COMMON.forEach((s, j) => { if (random() < 0.55 / (1 + j * 0.28)) sm.push(s); }); if (random() < 0.5) sm.push(pick(pick(THEMES).core)); // word leaking from another theme for (let k = sm.length - 1; k > 0; k -= 1) { const j = Math.floor(random() * (k + 1)); [sm[k], sm[j]] = [sm[j], sm[k]]; } // the subject tag is a publisher decision: a quarter of documents do not match the text's theme return { id, title: title.join(" "), summary: sm.join(" "), subject: (random() < 0.25 ? pick(THEMES) : t).subject, author: `${pick(FIRST_NAMES)} ${pick(SURNAMES)}`, year: 1975 + Math.floor(random() ** 0.6 * 50), loans: Math.floor(random() ** 3 * 400) }; }); export const tokenize = (s) => s.toLocaleLowerCase("en-US").match(/[\p{L}\p{N}]+/gu) ?? []; export function buildIndex(bs) { // per-field inverted index const idx = {}; for (const a of FIELDS) { const postings = new Map(), lengths = new Float64Array(bs.length); for (const b of bs) { const ts = tokenize(b[a]), count = new Map(); lengths[b.id] = ts.length; for (const t of ts) count.set(t, (count.get(t) ?? 0) + 1); for (const [t, tf] of count) { if (!postings.has(t)) postings.set(t, []); postings.get(t).push({ id: b.id, tf }); } } idx[a] = { postings, lengths, avgLength: lengths.reduce((x, y) => x + y, 0) / bs.length }; } return idx; } export function fieldScore(idx, a, terms) { // tf, inverse document frequency, length const k1 = 1.2, b = 0.75, p = new Map(); for (const t of terms) { const g = idx[a].postings.get(t); if (!g) continue; const idf = Math.log(1 + (N - g.length + 0.5) / (g.length + 0.5)); for (const e of g) { const norm = 1 - b + (b * idx[a].lengths[e.id]) / idx[a].avgLength; p.set(e.id, (p.get(e.id) ?? 0) + (idf * e.tf * (k1 + 1)) / (e.tf + k1 * norm)); } } return p; } export function search(idx, query, weight) { // total score with field weights const terms = tokenize(query), totals = new Map(); for (const a of FIELDS) { if (!weight[a]) continue; for (const [id, v] of fieldScore(idx, a, terms)) totals.set(id, (totals.get(id) ?? 0) + weight[a] * v); } return [...totals].sort((x, y) => y[1] - x[1] || x[0] - y[0]); } ``` The scorer runs separately per field: each field has its own posting list, its own average length, and its own inverse document frequency; the total score is the weighted sum of the field scores. If the weight is zero, that field is not scanned at all. #### Weight Changes Order, Field Selection Changes the Set Six configurations run the same query. Four only change weight; two narrow the set of fields scanned. ```js // score.mjs — same query, changing field weights: set, top ten, rank shift, and cost. import { documents, buildIndex, search, fieldScore, tokenize, FIELDS, N, SEED } from "./corpus.mjs"; const idx = buildIndex(documents), QUERY = "ocean ship", terms = tokenize(QUERY); const touched = (a) => terms.reduce((s, t) => s + (idx[a].postings.get(t)?.length ?? 0), 0); const CONFIGS = { "equal": { title: 1, summary: 1, subject: 1, author: 1 }, "title 5x": { title: 5, summary: 1, subject: 1, author: 1 }, "title 12x": { title: 12, summary: 1, subject: 1, author: 1 }, "author 5x": { title: 1, summary: 1, subject: 1, author: 5 }, "title only": { title: 1 }, "summary only": { summary: 1 } }; const base = search(idx, QUERY, CONFIGS["equal"]); const rank = (r) => new Map(r.map(([id], i) => [id, i + 1])), baseRank = rank(base); const baseTop = new Set(base.slice(0, 10).map(([id]) => id)); console.log(`corpus ${N} documents, seed ${SEED}; query "${QUERY}"`); console.log("field term count / average length: " + FIELDS.map((a) => `${a} ${idx[a].postings.size}/${idx[a].avgLength.toFixed(1)}`).join(", ")); console.log("posting entries touched: " + FIELDS.map((a) => `${a} ${touched(a)}`).join(", ") + "\n"); console.log("configuration".padEnd(14) + "matched".padStart(9) + "entries".padStart(9) + "top10 changed".padStart(15) + "largest jump".padStart(21) + " top result"); for (const [label, w] of Object.entries(CONFIGS)) { const r = search(idx, QUERY, w), y = rank(r); const changed = r.slice(0, 10).filter(([id]) => !baseTop.has(id)).length; let maxJump = 0, jumpDoc = -1; for (const [id] of base.slice(0, 50)) if (y.has(id) && Math.abs(baseRank.get(id) - y.get(id)) > maxJump) { maxJump = Math.abs(baseRank.get(id) - y.get(id)); jumpDoc = id; } const b = documents[r[0][0]]; console.log(label.padEnd(14) + `${r.length}`.padStart(9) + `${Object.keys(w).reduce((s, a) => s + touched(a), 0)}`.padStart(9) + `${changed}`.padStart(15) + (jumpDoc < 0 ? "base" : `${maxJump} (document ${jumpDoc})`).padStart(21) + ` ${b.title} / ${b.author}`); } const components = (id) => FIELDS .map((a) => `${a} ${(fieldScore(idx, a, terms).get(id) ?? 0).toFixed(2)}`).join(", "); const authorTop = search(idx, QUERY, CONFIGS["author 5x"])[0][0]; console.log(`\nwith author 5x, document ${authorTop} reaches the top: ${documents[authorTop].title} / ` + `${documents[authorTop].author}\n ranked ${baseRank.get(authorTop)} under equal weight; ` + `components: ${components(authorTop)}`); console.log(`top document under equal weight ${base[0][0]}: ${documents[base[0][0]].title} / ` + `${documents[base[0][0]].author}\n components: ${components(base[0][0])}`); ``` ``` corpus 1200 documents, seed 20260801; query "ocean ship" field term count / average length: title 42/2.9, summary 50/8.4, subject 7/1.3, author 16/2.0 posting entries touched: title 265, summary 302, subject 0, author 131 configuration matched entries top10 changed largest jump top result equal 325 698 0 base ocean coast ship / Owen Miller title 5x 325 698 4 44 (document 365) ocean ship / Blake Reed title 12x 325 698 4 44 (document 365) ocean ship / Blake Reed author 5x 325 698 10 131 (document 143) ocean ship coast / Ocean Bishop title only 199 265 9 144 (document 1122) ocean ship / Simon Miller summary only 211 302 8 159 (document 852) ocean lighthouse fisherman / Owen Reed with author 5x, document 622 reaches the top: ocean ship coast / Ocean Bishop ranked 17 under equal weight; components: title 4.59, summary 2.69, subject 0.00, author 2.21 top document under equal weight 143: ocean coast ship / Owen Miller components: title 4.59, summary 5.46, subject 0.00, author 0.00 ``` Document contents depend on the seed; the number of posting entries touched does not — that number comes directly from the length of the posting lists. The first column is the same across four rows: **325**. Changing weight does not touch the matching set, because weight is a multiplier that enters the score, not a matching condition. A document is in the set if it carries at least one query term; weight only decides where it lands. In the last two rows the set drops to 199 and 211, because there what changes is not weight but **which fields are scanned**. Searching only the title field drops 126 documents whose title carries none of the query terms. Weight and field selection are talked about as if they were the same adjustment, but one changes order and the other changes the set. On the order side, the first surprise is that the `title 5x` and `title 12x` rows are identical: four documents change in the top ten, the largest jump is forty-four ranks, and raising the coefficient two and a half times further adds nothing. As weight grows, the ranking converges on the title field's own internal order and stops there; past a certain point, an instruction to raise it further has no effect. The second surprise is in the `author 5x` row: **all ten** of the top ten change. The reason appears in the output's lower section. In the author field, "Ocean" is a name; document 622 sits seventeenth under equal weight, but multiplying the author component by five carries it to first place, and document 143 — first under equal weight — has an author component of zero, so it drops to rank 132. A single coefficient, a hundred thirty-one ranks. Field weight is not increasing relevance here; it is amplifying **field collision**. The subject field stays silent throughout the row: posting entries touched is zero, because "ocean" and "ship" appear in no subject tag. Raising the subject weight to a hundred changes nothing in this query; a weight's effect depends on the query term being present in that field. #### Function-Based Scoring and the Suppression Point The second adjustment comes from outside the text. The library wants "new and heavily borrowed books to surface higher." This request can enter the score in two forms: multiplied with the text score or added to the text score. Both are implementations of the same request, and their results are not the same. ```js // function-score.mjs — function-based score: two combination modes and relevance's suppression point. import { documents, buildIndex, search } from "./corpus.mjs"; const idx = buildIndex(documents), QUERY = "ocean ship"; const textScore = search(idx, QUERY, { title: 1, summary: 1, subject: 1, author: 1 }); const PEAK = textScore[0][1]; const textRank = new Map(textScore.map(([id], i) => [id, i + 1])); const textTop = new Set(textScore.slice(0, 10).map(([id]) => id)); const factor = (b) => 0.6 * (Math.log1p(b.loans) / Math.log1p(400)) + 0.4 * ((b.year - 1975) / 49); const topTen = (f) => textScore.map(([id, p]) => [id, f(p, documents[id])]) .sort((x, y) => y[1] - x[1] || x[0] - y[0]).slice(0, 10); const counts = (r) => `${r.filter(([id]) => !textTop.has(id)).length}/` + `${r.filter(([id]) => textRank.get(id) > 50).length}`; const avg = (r, f) => (r.reduce((s, [id]) => s + f(documents[id]), 0) / r.length).toFixed(0); console.log(`query "${QUERY}", ${textScore.length} matching documents; set is the same at every weight`); console.log(`highest text score ${PEAK.toFixed(2)}; top ten average loan count ` + `${avg(textScore.slice(0, 10), (b) => b.loans)}, average year ` + `${avg(textScore.slice(0, 10), (b) => b.year)}\n`); console.log("w".padEnd(6) + "multiplicative".padStart(16) + "additive".padStart(11) + "avg loans".padStart(11) + "avg year".padStart(9) + " top result in additive mode"); for (const w of [0.25, 0.5, 1, 2, 4, 8]) { const c = topTen((p, b) => p * (1 + w * factor(b))), t = topTen((p, b) => p + w * PEAK * factor(b)); const b = documents[t[0][0]]; console.log(`${w}`.padEnd(6) + counts(c).padStart(16) + counts(t).padStart(11) + avg(t, (x) => x.loans).padStart(11) + avg(t, (x) => x.year).padStart(9) + ` ${b.title} (${b.year}, ${b.loans} loans, text rank ${textRank.get(b.id)})`); } ``` ``` query "ocean ship", 325 matching documents; set is the same at every weight highest text score 10.05; top ten average loan count 100, average year 2006 w multiplicative additive avg loans avg year top result in additive mode 0.25 4/0 5/0 185 2013 ocean coast ship (2008, 235 loans, text rank 1) 0.5 5/0 6/0 191 2013 ocean ship fisherman (2013, 398 loans, text rank 14) 1 6/0 6/0 191 2013 ocean ship fisherman (2013, 398 loans, text rank 14) 2 6/0 7/2 198 2017 ocean ship fisherman (2013, 398 loans, text rank 14) 4 6/0 8/5 257 2018 ocean ship fisherman (2013, 398 loans, text rank 14) 8 6/0 10/8 324 2021 ocean coast fisherman (2024, 301 loans, text rank 123) ``` The two numbers in the columns read as follows: how many of the top ten results were not in the text ranking's top ten, and how many of those came after rank fifty in the text ranking. The multiplicative mode freezes once w passes one: four, five, six, and it stays there; the second number never leaves zero at any weight. The reason lies in the multiplier's structure: because the text score remains a factor, the ranking converges on the product of the text score and the coefficient, and a document that is very weak in the text cannot win that product. The multiplicative mode **reorders** the top ten; it does not refill it. In additive mode the contribution is added as a magnitude independent of the text score, and as w grows the text score dissolves into the sum. At w four, five of the top ten come from beyond rank fifty in the text ranking; if a countable criterion is needed, this is the **suppression point** — half the top ten is now selected not by the query but by loan count. At w eight the top ten changes completely, the top-ranked document sits at rank one hundred twenty-three in the text ranking, the top ten's average loan count rises from 100 to 324, and the average year rises from 2006 to 2021. The list is still the answer to the "ocean ship" query, and every one of its documents carries one of the query terms; but the order is now the order of the library's most heavily borrowed new books. The set stays fixed at 325 across all these weights: function-based scoring adds or removes no document. Once one considers that what the user actually sees is the top ten, the distinction thins — even though the set does not change, the **seen** set has changed completely. #### Summary - Field weight does not touch the matching set: all four weight configurations returned 325 documents. What changes the set is field selection, not weight; searching only the title field dropped the set to 199. - Weight's effect saturates: raising title weight from 5x to 12x left the change in the top ten at four documents and the largest jump at forty-four ranks. Weight does not change the scan cost either; the only decision that drops 698 posting entries to 265 is closing a field. - Weight given to the wrong field amplifies field collision, not relevance: when author weight rose to five, all ten of the top ten changed, and the document that was first under equal weight dropped to rank 132. The subject field, in contrast, does no work at all in this query, because the posting entries touched is zero. - In function-based scoring, multiplicative mode only reorders the top ten; additive mode fills half the top ten from beyond rank fifty in the text ranking at w four, and at w eight the top-ranked document sits at rank one hundred twenty-three in the text ranking. #### Next Step Every question asked so far wanted a document list: which books, in what order. What the loan desk asks is often not a document but a number: how many of this search's results were published after 2015, how many books fall under each subject heading, what is the matching set's average publication year. These questions run on the same inverted index, but their output is not a sorted list, and whether the answer is computed over the matching set or the whole corpus changes the result entirely. The next lesson takes up queries that produce numbers instead of documents, and their memory cost. ### Aggregation Queries Source: https://academia.sh/en/courses/search-engines/aggregation-queries Queries that produce a number instead of a document: measuring metric, bucket, and pipeline aggregations on the same loan record, the byte cost of bucket count, comparing an exact set against an estimate in high-cardinality distinct counting, and how running an aggregation over the whole corpus, the matching set, or the filtered set changes the result. The previous lesson produced a sorted document list and discussed that list's top ten. The questions sitting on the loan desk's board, by contrast, do not want a document: how many books fall under each subject heading, how many times per book have this search's results been borrowed, what does the loan curve look like across years. These questions run on the same inverted index, but their output is not a sorted list. Three forms are distinguished. A **metric aggregation** produces a single number from a set of documents: count, sum, average, maximum. A **bucket aggregation** splits the set into parts by a key and computes a metric within each part. A **pipeline aggregation** takes its input not from documents but from another aggregation's buckets: cumulative share, moving average, difference between buckets. That aggregation ignores order is this lesson's first observation. The query evaluator produces a sorted list; the aggregation reads that list's set, not its order. It has its own order, but that order is bucket order. The lesson measures set and order on this plane, and counts the cost in bytes. #### Corpus, Loan Record, and Three Aggregations | Code | Assumption | Value | Rationale | |---|---|---|---| | QR14 | loan record | each book produces as many events as its loan count | the record is derived from the corpus, no separate seed needed | | QR15 | reader pool | 40,000 readers, skewed selection | so the distinct reader count per bucket produces high cardinality | | QR16 | metric bucket | three numbers per bucket (count, sum, maximum) | a fixed 24-byte bucket state | | QR17 | aggregation input | the entire matching set | the top-N result limit does not apply to the aggregation | QR17 is quiet but decisive: even though the user is shown ten results, the aggregation sees all 325 documents; its cost grows with the size of the matching set, not the number of results shown. ```js // corpus.mjs — library catalog: seeded corpus, inverted index, and matching set. export const N = 1200, SEED = 20260801, FIELDS = ["title", "summary", "subject", "author"]; let d = SEED; // 32-bit generator, no overflow export const random = () => { d = (d + 0x6D2B79F5) | 0; let t = Math.imul(d ^ (d >>> 15), 1 | d); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32; }; const pick = (a) => a[Math.floor(random() * a.length)]; const THEMES = [["story|tale", "literature", "short selection collection narrative fiction rural"], ["fable", "children's literature", "child illustrated school youth forest sleep"], ["history", "history", "ottoman republic archive document chronicle foundation"], ["ocean", "travel", "coast fisherman harbor ship island lighthouse"], ["mathematics", "science", "geometry number proof theory solution probability"], ["poetry", "poetry", "verse collected divan translation selection meter"]] .map(([k, subject, a]) => ({ k: k.split("|"), subject, core: a.split(" ") })); const COMMON = "book volume edition publication review notes introduction glossary".split(" "); const FIRST_NAMES = "James Emma Simon Nora Kevin Ocean Grace Owen Meredith Blake".split(" "); const SURNAMES = "Stone Bishop Miller Hayes Reed Frost".split(" "); export const documents = Array.from({ length: N }, (_, id) => { const t = pick(THEMES), draw = t.k[t.k.length > 1 && random() < 0.5 ? 1 : 0], sm = [draw, draw]; const title = [...new Set([draw, pick(t.core), pick(t.core)])]; t.core.forEach((s, j) => { if (random() < 1 / (1 + j * 0.42)) sm.push(s); }); COMMON.forEach((s, j) => { if (random() < 0.55 / (1 + j * 0.28)) sm.push(s); }); if (random() < 0.5) sm.push(pick(pick(THEMES).core)); // word leaking from another theme for (let k = sm.length - 1; k > 0; k -= 1) { const j = Math.floor(random() * (k + 1)); [sm[k], sm[j]] = [sm[j], sm[k]]; } // the subject tag is a publisher decision: a quarter of documents do not match the text's theme return { id, title: title.join(" "), summary: sm.join(" "), subject: (random() < 0.25 ? pick(THEMES) : t).subject, author: `${pick(FIRST_NAMES)} ${pick(SURNAMES)}`, year: 1975 + Math.floor(random() ** 0.6 * 50), loans: Math.floor(random() ** 3 * 400) }; }); export const tokenize = (s) => s.toLocaleLowerCase("en-US").match(/[\p{L}\p{N}]+/gu) ?? []; export function buildIndex(bs) { // per-field inverted index const idx = {}; for (const a of FIELDS) { const postings = new Map(); for (const b of bs) for (const t of new Set(tokenize(b[a]))) { if (!postings.has(t)) postings.set(t, []); postings.get(t).push(b.id); } idx[a] = postings; } return idx; } export function matching(idx, query) { // documents carrying at least one query term const s = new Set(); for (const a of FIELDS) for (const t of tokenize(query)) for (const id of idx[a].get(t) ?? []) s.add(id); return s; } ``` #### Bucket Count Is a Memory Decision A bucket aggregation is determined by the question "which field do we group by," and that is a memory decision: each bucket must keep its own state to compute its metric. Metrics like count and sum need a fixed amount of space per bucket; distinct counting does not, because knowing how many separate readers there were requires keeping a representation of those readers. The setup measures both for five bucket keys: an exact set and a cardinality estimate. The estimate's structure was built and measured in M17/K06; here it is used as a tool, and the bytes it holds are read from its own buffer. ```js // aggregation.mjs — bucket count and distinct-count byte cost: exact set and estimator. import { documents, random, N, SEED } from "./corpus.mjs"; const READERS = 40_000, events = []; // each book as many times as its loan count for (const b of documents) for (let i = 0; i < b.loans; i += 1) events.push([b.id, 1 + Math.floor(random() ** 2 * READERS)]); // reader selection is skewed class ExactSet { // open addressing, empty slot 0 constructor() { this.table = new Int32Array(16); this.n = 0; } slot(t, x) { let i = (Math.imul(x, 2654435761) >>> 0) & (t.length - 1); while (t[i] !== 0 && t[i] !== x) i = (i + 1) & (t.length - 1); return i; } add(x) { if ((this.n + 1) * 2 > this.table.length) { const y = new Int32Array(this.table.length * 2); for (const v of this.table) if (v !== 0) y[this.slot(y, v)] = v; this.table = y; } const i = this.slot(this.table, x); if (this.table[i] === 0) { this.table[i] = x; this.n += 1; } } } class Estimator { // cardinality estimate: structure was built in M17/K06, used here as a tool constructor(p) { this.p = p; this.R = new Uint8Array(1 << p); } add(x) { let h = Math.imul(x ^ 0x9e37, 0x85ebca6b) >>> 0; h = Math.imul(h ^ (h >>> 13), 0xc2b2ae35) >>> 0; h = (h ^ (h >>> 16)) >>> 0; const i = h >>> (32 - this.p), w = (h << this.p) >>> 0; const r = w === 0 ? 33 - this.p : Math.clz32(w) + 1; if (r > this.R[i]) this.R[i] = r; } estimate() { const m = this.R.length; let z = 0, s = 0; for (const v of this.R) { z += 2 ** -v; if (v === 0) s += 1; } const e = (0.7213 / (1 + 1.079 / m)) * m * m / z; return e <= 2.5 * m && s > 0 ? m * Math.log(m / s) : e; } } const fmt = (x) => Math.round(x).toLocaleString("en-US"); const KEYS = { subject: (b) => b.subject, decade: (b) => `${Math.floor(b.year / 10) * 10}`, year: (b) => `${b.year}`, author: (b) => b.author, book: (b) => `${b.id}` }; console.log(`corpus ${N} documents, seed ${SEED}; loan record ${fmt(events.length)} events, ` + `${fmt(READERS)} reader pool\n`); console.log("bucket field".padEnd(12) + "buckets".padStart(8) + "metric".padStart(8) + "exact distinct".padStart(16) + "estimate p8".padStart(13) + "error".padStart(8) + "estimate p12".padStart(14) + "error".padStart(8) + "largest bucket".padStart(16)); for (const [label, f] of Object.entries(KEYS)) { const exact = new Map(), approx = [new Map(), new Map()], P = [8, 12]; for (const [book, reader] of events) { const k = f(documents[book]); if (!exact.has(k)) { exact.set(k, new ExactSet()); P.forEach((p, j) => approx[j].set(k, new Estimator(p))); } exact.get(k).add(reader); approx.forEach((m) => m.get(k).add(reader)); } let exactBytes = 0, largest = 0; const approxBytes = [0, 0], error = [0, 0]; for (const [k, s] of exact) { exactBytes += s.table.byteLength; largest = Math.max(largest, s.n); approx.forEach((m, j) => { approxBytes[j] += m.get(k).R.byteLength; error[j] = Math.max(error[j], Math.abs(m.get(k).estimate() - s.n) / s.n); }); } console.log(label.padEnd(12) + fmt(exact.size).padStart(8) + fmt(exact.size * 3 * 8).padStart(8) + fmt(exactBytes).padStart(16) + fmt(approxBytes[0]).padStart(13) + `%${(error[0] * 100).toFixed(1)}`.padStart(8) + fmt(approxBytes[1]).padStart(14) + `%${(error[1] * 100).toFixed(1)}`.padStart(8) + `${fmt(largest)} distinct`.padStart(16)); } ``` ``` corpus 1200 documents, seed 20260801; loan record 120,809 events, 40,000 reader pool bucket field buckets metric exact distinct estimate p8 error estimate p12 error largest bucket subject 6 144 786,432 1,536 %11.1 24,576 %1.6 15,257 distinct decade 6 144 819,200 1,536 %6.5 24,576 %2.1 20,949 distinct year 50 1,200 1,202,176 12,800 %18.5 204,800 %2.8 4,392 distinct author 60 1,440 1,318,912 15,360 %14.8 245,760 %3.3 3,908 distinct book 1,045 25,080 1,401,216 267,520 %33.1 4,280,320 %8.2 394 distinct ``` The byte columns are read from the structures' own buffers; the metric column is the bucket count times twenty-four. The metric column shows why a bucket aggregation is assumed to be cheap: 25,080 bytes for 1,045 buckets. Even if the bucket count grew a thousandfold, this column would be twenty-five megabytes; the statement "bucket count takes memory" is nearly false for a metric aggregation. The distinct-count column is a different world. For the same 1,045 buckets, the exact sets hold 1,401,216 bytes — fifty-six times the metric state. The exact column does not grow much with bucket count either — 786,432 at six buckets, 1,401,216 at one thousand forty-five — because what determines the cost is not the bucket count but the number of distinct readers spread across the buckets. The estimate columns show where the decision actually sits. At six buckets, the p8 estimate wants 1,536 bytes — one five-hundred-twelfth of the exact set — for a worst-case bucket error of 11.1 percent. Moving to p12 drops the error to 1.6 percent and raises the cost to 24,576 bytes — still one thirty-second of the exact figure. But in the last row the sign reverses: for one thousand forty-five buckets, the p12 estimate wants 4,280,320 bytes, three times the exact sets. An estimator wants a fixed buffer per bucket; as buckets shrink, that fixed amount exceeds the distinct count it is meant to represent. M17/K06 measured this for a single structure; in a bucket aggregation, the same break comes with the bucket count itself. Error also reads by bucket size: the worst deviation is 11.1 percent at six buckets and 33.1 percent at one thousand forty-five. An estimate's error should be judged not over total cardinality but over the smallest bucket; the report will be wrong not where the buckets are large, but where they are small. #### Which Set an Aggregation Runs On The second decision is not about memory but about meaning: is the aggregation computed over the whole corpus, over the query's matching set, or over the set left after the user's applied filter? The three give three different numbers, and all three can be "correct." ```js // buckets.mjs — which set an aggregation runs on: corpus, matched set, filtered set; and pipeline. import { documents, buildIndex, matching } from "./corpus.mjs"; const idx = buildIndex(documents); const matched = [...matching(idx, "ocean ship")].map((id) => documents[id]); const filtered = matched.filter((b) => b.subject === "travel"); const fmt = (x) => Math.round(x).toLocaleString("en-US"); const bucket = (bs, f) => { // bucket: key -> count and loans const m = new Map(); for (const b of bs) { const k = f(b), v = m.get(k) ?? { n: 0, loans: 0 }; v.n += 1; v.loans += b.loans; m.set(k, v); } return [...m].sort((x, y) => y[1].n - x[1].n || (x[0] < y[0] ? -1 : 1)); }; const corpusBuckets = new Map(bucket(documents, (b) => b.subject)); console.log(`subject buckets — corpus ${documents.length}, matched ${matched.length}, ` + `${filtered.length} documents after the subject filter\n`); console.log("subject".padEnd(23) + "corpus".padStart(8) + "matched".padStart(9) + "matched loans".padStart(15) + "per book".padStart(14)); for (const [k, v] of bucket(matched, (b) => b.subject)) console.log(k.padEnd(23) + fmt(corpusBuckets.get(k).n).padStart(8) + fmt(v.n).padStart(9) + fmt(v.loans).padStart(15) + (v.loans / v.n).toFixed(1).padStart(14)); console.log("bucket count if the same aggregation runs on the filtered set: " + `${bucket(filtered, (b) => b.subject).length}`); const yearBuckets = bucket(matched, (b) => `${b.year}`); const byLoans = [...yearBuckets].sort((x, y) => y[1].loans - x[1].loans); const total = (l) => l.reduce((s, [, v]) => s + v.loans, 0); console.log(`\npipeline: cumulative share over the loan total of year buckets`); console.log("bucket set".padEnd(16) + "buckets".padStart(9) + "loans".padStart(9) + "top three buckets' share".padStart(26) + "top bucket's share".padStart(21)); for (const [label, list] of [["full", byLoans], ["top 10 buckets", byLoans.slice(0, 10)], ["top 3 buckets", byLoans.slice(0, 3)]]) { const t = total(list); console.log(label.padEnd(16) + `${list.length}`.padStart(9) + fmt(t).padStart(9) + `%${((total(list.slice(0, 3)) / t) * 100).toFixed(1)}`.padStart(26) + `%${((list[0][1].loans / t) * 100).toFixed(1)}`.padStart(21)); } ``` ``` subject buckets — corpus 1200, matched 325, 168 documents after the subject filter subject corpus matched matched loans per book travel 208 168 14,842 88.3 poetry 208 35 3,761 107.5 science 185 34 2,886 84.9 history 206 33 2,937 89.0 literature 220 30 3,875 129.2 children's literature 173 25 3,895 155.8 bucket count if the same aggregation runs on the filtered set: 1 pipeline: cumulative share over the loan total of year buckets bucket set buckets loans top three buckets' share top bucket's share full 49 32,196 %15.4 %5.4 top 10 buckets 10 13,181 %37.6 %13.1 top 3 buckets 3 4,961 %100.0 %34.8 ``` The first two columns are two different numbers for the same bucket. The corpus has 208 travel books; the "ocean ship" query's matching set has 168. A surface that shows the user "travel (208)" gives information independent of the query; "travel (168)" describes navigating within the query. Mixing the two breaks the totals: the matched column sums to 325, the corpus column sums to 1,200. The per-book loans column shows the rule for reading a metric: the highest value, 155.8, is in the children's literature bucket, but that bucket has only 25 documents. An average drawn from twenty-five documents and one drawn from a hundred sixty-eight documents are not equally trustworthy. The filter row turns navigation surfaces' classic mistake into a number. When the user selects "travel," the result set drops to 168 documents, and if the same subject aggregation runs on that set, it returns a **single bucket**: no other subject appears on the surface, and the selection cannot be changed. The subject aggregation must be computed on the set before its own filter is applied, and after the other filters are applied. The order between aggregation and filtering is not a display preference; it is a decision that determines the set. The pipeline table shows the last issue. The answer to "what share of total loans do the top three years hold" is 15.4 percent when all forty-nine buckets are seen, 37.6 percent when only the top ten buckets are seen, and 100 percent when three buckets are seen. Because a pipeline aggregation takes its input from buckets, not documents, a truncated input produces a wrong output, and the error cannot be seen by looking at the output — all three numbers are internally consistent. Truncating buckets looks like a display decision, but it changes the pipeline's input. #### Summary - Aggregation does not use document order; it reads the set. Its own order is bucket order, and buckets can be arranged by count, by key, or by metric. - A metric bucket is twenty-four bytes per bucket: 25,080 bytes for 1,045 buckets. A distinct-count bucket wants 1,401,216 bytes with exact sets on the same buckets, and its cost depends on the number of distinct readers in the buckets, not the bucket count. - Cardinality estimation gains five hundred times at few buckets (1,536 bytes against 786,432 at six buckets) and loses at many buckets: the p12 estimate for 1,045 buckets is 4,280,320 bytes, three times the exact figure. - An estimate's error is read at the smallest bucket: the worst deviation is 11.1 percent at six buckets, 33.1 percent at one thousand forty-five. - The same subject aggregation gives 208 over the corpus, 168 over the matching set, and a single bucket after its own filter; the pipeline's answer likewise ranges from 15.4 percent to 100 percent depending on the bucket count in its input. #### Next Step Aggregations produced numbers, the query evaluator produced a sorted list; neither addressed how the result appears to the reader. What sits on the screen is not a document identifier but a snippet drawn from the book's title and summary; which part of that snippet gets picked depends on where the query's word matches, and that information was not kept during scoring. The second question is heavier: when the reader moves to page twenty, how the same ranking is preserved, and how many candidates that requires keeping in memory, was never asked. The next lesson takes up result presentation and the cost of deep pagination. ### Highlighting and Pagination Source: https://academia.sh/en/courses/search-engines/highlighting-and-pagination The result's face shown to the reader: highlighting's per-document re-scan cost against the permanent byte overhead of storing position in the index, the linear growth with page depth of the candidate count pulled from each shard in deep pagination, cursor-based navigation producing the same page at a fixed cost, and the repeats and skips offset pagination produces when the index changes. Every measurement so far has been done on document identifiers and scores. What the reader actually sees is not an identifier but a short snippet drawn from the book's title and summary; within that snippet, the word being searched for is marked. There is a second thing too: the list does not end at ten rows — the reader moves to page two, then to page twenty. These two jobs do not resemble each other. **Highlighting** touches only the shown documents, and its cost is independent of page depth. **Pagination**, by contrast, grows more expensive with depth, and that expense is multiplied by how many shards the index is split into. The lesson measures both on the same corpus. #### Corpus, Page, and Shard | Code | Assumption | Value | Rationale | |---|---|---|---| | QR18 | page size and shown fields | 10 results; title and summary | highlighting is done only on these two fields | | QR19 | highlight window | 30 characters, around the first match | the snippet must fit on one line | | QR20 | index shard | 4 shards, each shard produces its own ranking | merging happens in the same process | | QR21 | candidate entry | 12 bytes (4 for the id, 8 for the score) | size of the candidate kept by the merger | QR20 is this lesson's measurement rig: the shards run in the same process, there is no network between them; what is measured is the number of candidates pulled from a shard, not time. ```js // corpus.mjs — library catalog: seeded corpus, inverted index, and scorer. export const N = 1200, SEED = 20260801, FIELDS = ["title", "summary", "subject", "author"]; let d = SEED; // 32-bit generator, no overflow export const random = () => { d = (d + 0x6D2B79F5) | 0; let t = Math.imul(d ^ (d >>> 15), 1 | d); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32; }; const pick = (a) => a[Math.floor(random() * a.length)]; const THEMES = [["story|tale", "literature", "short selection collection narrative fiction rural"], ["fable", "children's literature", "child illustrated school youth forest sleep"], ["history", "history", "ottoman republic archive document chronicle foundation"], ["ocean", "travel", "coast fisherman harbor ship island lighthouse"], ["mathematics", "science", "geometry number proof theory solution probability"], ["poetry", "poetry", "verse collected divan translation selection meter"]] .map(([k, subject, a]) => ({ k: k.split("|"), subject, core: a.split(" ") })); const COMMON = "book volume edition publication review notes introduction glossary".split(" "); const FIRST_NAMES = "James Emma Simon Nora Kevin Ocean Grace Owen Meredith Blake".split(" "); const SURNAMES = "Stone Bishop Miller Hayes Reed Frost".split(" "); export const documents = Array.from({ length: N }, (_, id) => { const t = pick(THEMES), draw = t.k[t.k.length > 1 && random() < 0.5 ? 1 : 0], sm = [draw, draw]; const title = [...new Set([draw, pick(t.core), pick(t.core)])]; t.core.forEach((s, j) => { if (random() < 1 / (1 + j * 0.42)) sm.push(s); }); COMMON.forEach((s, j) => { if (random() < 0.55 / (1 + j * 0.28)) sm.push(s); }); if (random() < 0.5) sm.push(pick(pick(THEMES).core)); // word leaking from another theme for (let k = sm.length - 1; k > 0; k -= 1) { const j = Math.floor(random() * (k + 1)); [sm[k], sm[j]] = [sm[j], sm[k]]; } // the subject tag is a publisher decision: a quarter of documents do not match the text's theme return { id, title: title.join(" "), summary: sm.join(" "), subject: (random() < 0.25 ? pick(THEMES) : t).subject, author: `${pick(FIRST_NAMES)} ${pick(SURNAMES)}`, year: 1975 + Math.floor(random() ** 0.6 * 50), loans: Math.floor(random() ** 3 * 400) }; }); export const tokenize = (s) => s.toLocaleLowerCase("en-US").match(/[\p{L}\p{N}]+/gu) ?? []; export function buildIndex(bs) { // per-field inverted index const idx = {}; for (const a of FIELDS) { const postings = new Map(), lengths = new Float64Array(bs.length); for (const b of bs) { const ts = tokenize(b[a]), count = new Map(); lengths[b.id] = ts.length; for (const t of ts) count.set(t, (count.get(t) ?? 0) + 1); for (const [t, tf] of count) { if (!postings.has(t)) postings.set(t, []); postings.get(t).push({ id: b.id, tf }); } } idx[a] = { postings, lengths, avgLength: lengths.reduce((x, y) => x + y, 0) / bs.length }; } return idx; } export function search(idx, query) { // result list scored with equal field weight const terms = tokenize(query), k1 = 1.2, b = 0.75, p = new Map(); for (const a of FIELDS) for (const t of terms) { const g = idx[a].postings.get(t); if (!g) continue; const idf = Math.log(1 + (N - g.length + 0.5) / (g.length + 0.5)); for (const e of g) { const norm = 1 - b + (b * idx[a].lengths[e.id]) / idx[a].avgLength; p.set(e.id, (p.get(e.id) ?? 0) + (idf * e.tf * (k1 + 1)) / (e.tf + k1 * norm)); } } return [...p].sort((x, y) => y[1] - x[1] || x[0] - y[0]); } ``` #### Highlighting's Per-Document Cost The inverted index knows which term appears in which document, but not **where** in the text the term appears: what enters the index is the token, not the token's character position. Highlighting wants exactly that information. There are two paths: store the position in the index, or re-scan the shown documents at query time. ```js // highlight.mjs — highlighting cost: re-scan at query time, or keep position in the index. import { documents, buildIndex, search, tokenize, N, SEED } from "./corpus.mjs"; const idx = buildIndex(documents), QUERY = "ocean ship book"; const terms = new Set(tokenize(QUERY)), results = search(idx, QUERY); const TOKEN_RE = /[\p{L}\p{N}]+/gu, fmt = (x) => Math.round(x).toLocaleString("en-US"); function scan(text) { // re-scan: tokenize, mark positions const positions = []; let count = 0; for (const m of text.matchAll(TOKEN_RE)) { count += 1; if (terms.has(m[0].toLocaleLowerCase("en-US"))) positions.push([m.index, m[0].length]); } return { characters: text.length, tokens: count, positions }; } const WINDOW = 30; const snippet = (text, positions, width = WINDOW) => { // window around the first match if (positions.length === 0) return ""; const start = Math.max(0, positions[0][0] - 12), more = start + width < text.length; let s = text.slice(start, start + width); if (more) s = s.slice(0, s.lastIndexOf(" ")); return (start > 0 ? "…" : "") + s.replace(TOKEN_RE, (w) => terms.has(w.toLocaleLowerCase("en-US")) ? `[${w}]` : w) + (more ? "…" : ""); }; console.log(`corpus ${N} documents, seed ${SEED}; query "${QUERY}" matched ${results.length} documents`); console.log("shown".padEnd(12) + "re-scanned characters".padStart(28) + "tokens produced".padStart(19) + "matches found".padStart(17) + "left outside window".padStart(23)); for (const shown of [10, 50, 100]) { let chars = 0, toks = 0, matches = 0, outside = 0; for (const [id] of results.slice(0, shown)) { const titleScan = scan(documents[id].title), summaryScan = scan(documents[id].summary); chars += titleScan.characters + summaryScan.characters; toks += titleScan.tokens + summaryScan.tokens; matches += titleScan.positions.length + summaryScan.positions.length; const base = summaryScan.positions.length ? Math.max(0, summaryScan.positions[0][0] - 12) : 0; outside += summaryScan.positions.filter(([i]) => i < base || i >= base + WINDOW).length; } console.log(`${shown} documents`.padEnd(12) + fmt(chars).padStart(28) + fmt(toks).padStart(19) + fmt(matches).padStart(17) + `${outside}`.padStart(23)); } let occurrences = 0, totalChars = 0; // if a position index were built for the whole corpus for (const b of documents) { occurrences += tokenize(b.title).length + tokenize(b.summary).length; totalChars += b.title.length + b.summary.length; } console.log(`\nif a position index were chosen: ${fmt(occurrences)} token occurrences, 8 bytes per ` + `occurrence (start and length) = ${fmt(occurrences * 8)} bytes of permanent overhead`); console.log(`if the same information is produced at query time there is no permanent overhead: only ` + `the shown documents' characters are scanned, out of the corpus's ${fmt(totalChars)} characters`); for (const [id] of results.slice(0, 3)) { const summaryScan = scan(documents[id].summary), titleScan = scan(documents[id].title); console.log(` ${documents[id].title.padEnd(24)} summary snippet: ` + `${snippet(documents[id].summary, summaryScan.positions) || "(none, from title field: " + snippet(documents[id].title, titleScan.positions) + ")"}`); } ``` ``` corpus 1200 documents, seed 20260801; query "ocean ship book" matched 817 documents shown re-scanned characters tokens produced matches found left outside window 10 documents 656 107 60 13 50 documents 3,913 581 263 65 100 documents 7,476 1,103 493 108 if a position index were chosen: 13,529 token occurrences, 8 bytes per occurrence (start and length) = 108,232 bytes of permanent overhead if the same information is produced at query time there is no permanent overhead: only the shown documents' characters are scanned, out of the corpus's 103,849 characters ocean ship summary snippet: [book] harbor volume coast… ocean coast ship summary snippet: [ocean] [ship] [book] harbor… ocean ship harbor summary snippet: [ocean] notes [ship] fisherman… ``` Highlighting changes neither the set nor the order: the matched 817 documents and their order are the same whether highlighting is on or off. The only thing that changes is the content of the shown line. Its cost, by contrast, is linear and depends on the number of documents shown: 656 characters and 107 tokens for ten documents, 7,476 characters and 1,103 tokens for a hundred documents. Page depth never appears in this table; highlighting the hundredth page's ten documents does about as much work as the first page's ten documents — around 656 characters either way. The position-index row gives the price of the other path: eight bytes per occurrence for the whole corpus's 13,529 token occurrences, a total of 108,232 bytes of permanent overhead. This cost is paid for every document, whereas highlighting is done only for the ten shown documents. If the index holds 1,200 documents while a query looks at ten of them, storing position means paying up front for the work of a hundred twenty documents. The decision follows this asymmetry: a position index wins when the text is long and the same documents are highlighted often; when the field is short, as in a catalog record, re-scanning is cheaper. The last column counts highlighting's own loss. Of the sixty matches found in ten documents, thirteen fall outside the thirty-character window; the reader never sees those matches at all. Enlarging the window lowers this number but lengthens the line. Highlighting does not show that a match **exists**; it shows only the surroundings of one of them. #### Deep Pagination Pagination's cost is talked about as if it were a cursor held in a single sorted list. Once the index is split into shards, that stops being true: because no shard knows the full ranking, which shard the second page's tenth document will come from can only be found by pulling enough candidates from every shard, starting from the beginning. ```js // paginate.mjs — deep pagination: candidates pulled per shard, cursor comparison, and drift. import { documents, buildIndex, search } from "./corpus.mjs"; const idx = buildIndex(documents), QUERY = "ocean ship book", SHARDS = 4, PAGE_SIZE = 10, BYTES = 12; const full = search(idx, QUERY), before = (a, b) => b[1] - a[1] || a[0] - b[0]; let shards = Array.from({ length: SHARDS }, (_, p) => full.filter(([id]) => id % SHARDS === p)); const BASE = shards.map((pz) => [...pz]); const fmt = (x) => Math.round(x).toLocaleString("en-US"); const offsetPage = (page) => { // each shard returns from+size candidates from the start const start = (page - 1) * PAGE_SIZE, requested = start + PAGE_SIZE, pool = []; for (const pz of shards) pool.push(...pz.slice(0, requested)); pool.sort(before); return { candidates: requested * SHARDS, page: pool.slice(start, start + PAGE_SIZE) }; }; const cursorPage = (cursor) => { // each shard returns size candidates after the cursor const pool = []; for (const pz of shards) pool.push(...pz.filter((e) => cursor === null || before(e, cursor) > 0).slice(0, PAGE_SIZE)); pool.sort(before); return pool.slice(0, PAGE_SIZE); }; console.log(`query "${QUERY}": ${full.length} results, ${SHARDS} shards, page size ${PAGE_SIZE}`); console.log("page".padEnd(7) + "offset candidates".padStart(19) + "cursor candidates".padStart(19) + "ratio".padStart(8) + "merge bytes".padStart(19) + "cumulative offset".padStart(20) + "cumulative cursor".padStart(19) + "same page".padStart(12)); let cumulative = 0, cursor = null, cursorResult = []; for (let pageNum = 1; pageNum <= 20; pageNum += 1) { const offsetResult = offsetPage(pageNum); cumulative += offsetResult.candidates; cursorResult = cursorPage(cursor); cursor = cursorResult[cursorResult.length - 1]; if (![1, 5, 10, 20].includes(pageNum)) continue; const same = offsetResult.page.every(([id], i) => id === cursorResult[i][0]); console.log(`${pageNum}`.padEnd(7) + fmt(offsetResult.candidates).padStart(19) + fmt(PAGE_SIZE * SHARDS).padStart(19) + `${(offsetResult.candidates / (PAGE_SIZE * SHARDS)).toFixed(0)}x`.padStart(8) + fmt(offsetResult.candidates * BYTES).padStart(19) + fmt(cumulative).padStart(20) + fmt(pageNum * PAGE_SIZE * SHARDS).padStart(19) + (same ? "yes" : "no").padStart(12)); } const allIds = () => shards.flat().sort(before).map(([id]) => id); const page1 = offsetPage(1).page, shown1 = new Set(page1.map(([id]) => id)), cursor1 = page1[page1.length - 1]; console.log("\nwhat happens on page 2 if the index changes after page 1 is shown"); console.log("change".padEnd(24) + "offset repeat".padStart(18) + "skipped".padStart(9) + "cursor repeat".padStart(16) + "skipped".padStart(9)); for (const [label, change] of [ ["new document at the top", () => shards[0].unshift([-1, full[0][1] + 1])], ["delete from first page", () => { const x = page1[2][0]; shards[x % SHARDS] = shards[x % SHARDS].filter(([id]) => id !== x); }]]) { shards = BASE.map((pz) => [...pz]); change(); const updated = allIds(); // skipped: those left before where page 2 now starts const counts = (p2) => [p2.filter((id) => shown1.has(id)).length, updated.slice(0, updated.indexOf(p2[0])).filter((id) => id >= 0 && !shown1.has(id)).length]; const offsetCounts = counts(offsetPage(2).page.map(([id]) => id)); const cursorCounts = counts(cursorPage(cursor1).map(([id]) => id)); console.log(label.padEnd(24) + `${offsetCounts[0]}`.padStart(18) + `${offsetCounts[1]}`.padStart(9) + `${cursorCounts[0]}`.padStart(16) + `${cursorCounts[1]}`.padStart(9)); } ``` ``` query "ocean ship book": 817 results, 4 shards, page size 10 page offset candidates cursor candidates ratio merge bytes cumulative offset cumulative cursor same page 1 40 40 1x 480 40 40 yes 5 200 40 5x 2,400 600 200 yes 10 400 40 10x 4,800 2,200 400 yes 20 800 40 20x 9,600 8,400 800 yes what happens on page 2 if the index changes after page 1 is shown change offset repeat skipped cursor repeat skipped new document at the top 1 0 0 0 delete from first page 0 1 0 0 ``` The last column is the basis of the whole comparison: both paths produce the **same page**. The documents, and their order, are identical. The only thing that changes is the cost. In offset pagination, the candidates pulled per shard equal the number skipped plus the page size; with four shards, page twenty means 800 candidates, of which 790 are discarded. The cost grows linearly with page depth and is multiplied by the shard count: the same page would want 1,600 candidates in an eight-shard index. In cursor-based navigation the number is fixed: 40 candidates on every page, because each shard is asked "the first ten documents after this score and this id," and no counting from the start is needed. The cumulative columns show the reader's whole journey. A reader who visits twenty pages in sequence generates 8,400 candidates on the offset path and 800 on the cursor path. The offset path's cumulative total grows quadratically, the cursor path's grows linearly; at page twenty the difference is ten and a half times. Cursor navigation, however, cannot do one thing: it cannot jump straight to page twenty, because the cursor is not a rank number but the score and id of the last shown document. If navigation by page number is required, the offset path's cost will be paid. The lower table measures a third difference. If a document that will land at the very top is added to the index after page one is shown, a document the reader has **already seen** reappears on page two in the offset path. If a document on page one is deleted, a document that should have opened page two is never shown to the reader at all. On the cursor path both numbers are zero, because the cursor is not a counter but a value; whatever happens at the start of the list does not shift the window to the right of the cursor. Here, pagination style stops being a display preference and becomes a decision that determines the set the reader sees. #### Summary - Highlighting changes neither the set nor the order; its cost grows only with the number of documents shown: 656 characters and 107 tokens for ten documents, 7,476 characters and 1,103 tokens for a hundred documents. - Keeping position in the index wants 108,232 bytes of permanent overhead for 13,529 token occurrences, and that cost is paid for every document; re-scanning works only for the ones shown. - Highlighting hides matches that do not fit the window: thirteen of the sixty matches in ten documents fell outside the thirty-character window. - In offset pagination, candidates per shard grow linearly with page depth and are multiplied by the shard count: page twenty wants 800 candidates across four shards, 9,600 bytes of merge overhead. - Cursor navigation produces the same page at a fixed 40 candidates and gives no repeats or skips even when the index changes; in exchange, it gives up jumping by page number. #### Next Step The result page is now complete: the matching set has been determined, ranked, tuned, reduced to numbers, and put into the form shown to the reader. All of this work rests on a single assumption: the word in the query is the same word as the word in the document. In the catalog, this assumption often does not hold. A reader searching "tale" cannot see books titled "story"; for the inverted index there is no link at all between these two strings, and no adjustment so far can build that link, because both weight and function-based scoring only change the score of a term that has already matched. The next lesson builds an access method that compares not the word itself but the context the word appears in, and measures the set and order the two paths return, side by side. ### Semantic and Vector Search Source: https://academia.sh/en/courses/search-engines/semantic-and-vector-search An access method that compares the context a word appears in rather than the word itself: building embeddings from term counts, two synonymous terms coming out close without ever appearing together, the documents vector access gains and loses relative to term access, hybrid search's fusion of two lists, and the vector index's byte and scan cost. Every mechanism built so far rests on a single assumption: the word in the query is the same string as the word in the document. In the catalog, this assumption often does not hold. A reader searching "tale" cannot see books whose title carries "story." This is not a scoring problem: for the inverted index there is no link at all between those two strings, so neither field weight nor function-based scoring can bring those documents in — both only change the score of a term that has already matched. This lesson builds another access method. Every term and every document is represented by a fixed-length sequence of numbers, called an **embedding**. The closeness of two documents, or of a query and a document, is a measure computed between these two sequences. No word match is sought; numbers are compared. The term "embedding" appears in two senses in this course: in the document model it described one document being embedded inside another; here it is a text's numeric-sequence form. #### Corpus and Building the Embedding | Code | Assumption | Value | Rationale | |---|---|---|---| | QR22 | fields entering the embedding | title and summary | the subject and author fields do not enter the vector | | QR23 | context window | the whole document | two terms appearing in the same record count as co-occurring | | QR24 | embedding dimension | 32; the projection comes from the same seed | vectors come out the same on every run | | QR25 | synonymous pair | "tale" and "story" in the same context, never together in a document | the measured gain comes from this construction, not from the language | QR25 must be stated explicitly: the corpus is generated so that the two synonyms appear with the same neighboring words. The lesson shows this construction's measurable result; it does not show that an embedding will find every synonym in a language. ```js // corpus.mjs — library catalog: seeded corpus, inverted index, and scorer. export const N = 1200, SEED = 20260801, FIELDS = ["title", "summary", "subject", "author"]; let d = SEED; // 32-bit generator, no overflow export const random = () => { d = (d + 0x6D2B79F5) | 0; let t = Math.imul(d ^ (d >>> 15), 1 | d); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 2 ** 32; }; const pick = (a) => a[Math.floor(random() * a.length)]; const THEMES = [["story|tale", "literature", "short selection collection narrative fiction rural"], ["fable", "children's literature", "child illustrated school youth forest sleep"], ["history", "history", "ottoman republic archive document chronicle foundation"], ["ocean", "travel", "coast fisherman harbor ship island lighthouse"], ["mathematics", "science", "geometry number proof theory solution probability"], ["poetry", "poetry", "verse collected divan translation selection meter"]] .map(([k, subject, a]) => ({ k: k.split("|"), subject, core: a.split(" ") })); const COMMON = "book volume edition publication review notes introduction glossary".split(" "); const FIRST_NAMES = "James Emma Simon Nora Kevin Ocean Grace Owen Meredith Blake".split(" "); const SURNAMES = "Stone Bishop Miller Hayes Reed Frost".split(" "); export const documents = Array.from({ length: N }, (_, id) => { const t = pick(THEMES), draw = t.k[t.k.length > 1 && random() < 0.5 ? 1 : 0], sm = [draw, draw]; const title = [...new Set([draw, pick(t.core), pick(t.core)])]; t.core.forEach((s, j) => { if (random() < 1 / (1 + j * 0.42)) sm.push(s); }); COMMON.forEach((s, j) => { if (random() < 0.55 / (1 + j * 0.28)) sm.push(s); }); if (random() < 0.5) sm.push(pick(pick(THEMES).core)); // word leaking from another theme for (let k = sm.length - 1; k > 0; k -= 1) { const j = Math.floor(random() * (k + 1)); [sm[k], sm[j]] = [sm[j], sm[k]]; } // the subject tag is a publisher decision: a quarter of documents do not match the text's theme return { id, title: title.join(" "), summary: sm.join(" "), subject: (random() < 0.25 ? pick(THEMES) : t).subject, author: `${pick(FIRST_NAMES)} ${pick(SURNAMES)}`, year: 1975 + Math.floor(random() ** 0.6 * 50), loans: Math.floor(random() ** 3 * 400) }; }); export const tokenize = (s) => s.toLocaleLowerCase("en-US").match(/[\p{L}\p{N}]+/gu) ?? []; export function buildIndex(bs) { // per-field inverted index const idx = {}; for (const a of FIELDS) { const postings = new Map(), lengths = new Float64Array(bs.length); for (const b of bs) { const ts = tokenize(b[a]), count = new Map(); lengths[b.id] = ts.length; for (const t of ts) count.set(t, (count.get(t) ?? 0) + 1); for (const [t, tf] of count) { if (!postings.has(t)) postings.set(t, []); postings.get(t).push({ id: b.id, tf }); } } idx[a] = { postings, lengths, avgLength: lengths.reduce((x, y) => x + y, 0) / bs.length }; } return idx; } export function search(idx, query) { // result list scored with equal field weight const terms = tokenize(query), k1 = 1.2, b = 0.75, p = new Map(); for (const a of FIELDS) for (const t of terms) { const g = idx[a].postings.get(t); if (!g) continue; const idf = Math.log(1 + (N - g.length + 0.5) / (g.length + 0.5)); for (const e of g) { const norm = 1 - b + (b * idx[a].lengths[e.id]) / idx[a].avgLength; p.set(e.id, (p.get(e.id) ?? 0) + (idf * e.tf * (k1 + 1)) / (e.tf + k1 * norm)); } } return [...p].sort((x, y) => y[1] - x[1] || x[0] - y[0]); } ``` #### Producing a Vector from Context An embedding is built as follows. For every term, a count is kept of all the terms that appear in the same record; that count is the term's context. The context vector has the length of the 50-word vocabulary and is reduced to 32 dimensions with a seeded projection. A document's embedding is the weighted average of its terms' embeddings. The query goes through the same process, and similarity is the dot product of two unit vectors. ```js // embedding.mjs — embedding built from term counts, vector access, and hybrid search. import { documents, buildIndex, search, tokenize, random, N, SEED, FIELDS } from "./corpus.mjs"; const D = 32, idx = buildIndex(documents), fmt = (x) => Math.round(x).toLocaleString("en-US"); const text = (b) => [...new Set([...tokenize(b.title), ...tokenize(b.summary)])]; const df = new Map(); // vocabulary: term -> document frequency for (const b of documents) for (const t of text(b)) df.set(t, (df.get(t) ?? 0) + 1); const VOCAB = [...df.keys()], pos = new Map(VOCAB.map((t, i) => [t, i])); const CO = VOCAB.map(() => new Float64Array(VOCAB.length)); // co-occurrence count for (const b of documents) { const ts = text(b); for (const a of ts) for (const c of ts) if (a !== c) CO[pos.get(a)][pos.get(c)] += 1; } const idf = VOCAB.map((t) => Math.log(N / df.get(t))); const R = VOCAB.map(() => Float64Array.from({ length: D }, () => (random() < 0.5 ? -1 : 1))); const normalize = (v) => { let s = 0; for (const x of v) s += x * x; s = Math.sqrt(s) || 1; for (let i = 0; i < v.length; i += 1) v[i] /= s; return v; }; const termVec = CO.map((row) => { // context counts -> D-dimensional embedding const p = new Float64Array(D); for (let i = 0; i < row.length; i += 1) if (row[i]) { const w = row[i] * idf[i]; for (let j = 0; j < D; j += 1) p[j] += w * R[i][j]; } return normalize(p); }); const embeddings = new Float32Array(N * D); // document embedding: weighted average of its terms for (const b of documents) { const p = new Float64Array(D); for (const t of text(b)) { const v = termVec[pos.get(t)], w = idf[pos.get(t)]; for (let j = 0; j < D; j += 1) p[j] += w * v[j]; } embeddings.set(normalize(p), b.id * D); } const queryVec = (q) => { const p = new Float64Array(D); for (const t of tokenize(q)) if (pos.has(t)) { const v = termVec[pos.get(t)]; for (let j = 0; j < D; j += 1) p[j] += idf[pos.get(t)] * v[j]; } return normalize(p); }; const vectorSearch = (q) => { const s = queryVec(q), r = []; for (let i = 0; i < N; i += 1) { let d = 0; for (let j = 0; j < D; j += 1) d += s[j] * embeddings[i * D + j]; r.push([i, d]); } return r.sort((a, b) => b[1] - a[1] || a[0] - b[0]); }; const hybrid = (termList, vectorList) => { // rank-based fusion, k = 60 const p = new Map(), add = (l) => l.slice(0, 100).forEach(([id], i) => p.set(id, (p.get(id) ?? 0) + 1 / (60 + i + 1))); add(termList); add(vectorList); return [...p].sort((a, b) => b[1] - a[1] || a[0] - b[0]); }; let entries = 0; for (const a of FIELDS) for (const g of idx[a].postings.values()) entries += g.length; console.log(`corpus ${N} documents, seed ${SEED}; vocabulary ${VOCAB.length} terms, embedding dimension ${D}`); console.log(`vector index ${fmt(embeddings.byteLength)} bytes; inverted index ${fmt(entries)} posting ` + `entries x 8 bytes = ${fmt(entries * 8)} bytes`); console.log(`vector scan per query ${fmt(N * D)} multiply-adds; the set column counts the matching ` + `document on the term path, and the document with similarity 0.5 or above on the vector path\n`); const dot = (a, b) => { let s = 0; for (let i = 0; i < a.length; i += 1) s += a[i] * b[i]; return s; }; for (const t of ["tale", "fable"]) { const k = VOCAB.map((u, i) => [u, dot(termVec[pos.get(t)], termVec[i])]) .sort((a, b) => b[1] - a[1]).slice(1, 5); console.log(`nearest terms to the "${t}" embedding: ` + k.map(([u, d]) => `${u} ${d.toFixed(3)}`).join(", ")); } console.log(`document 0 embedding's first four components: ` + [...embeddings.slice(0, 4)].map((x) => x.toFixed(3)).join(", ") + "\n"); console.log("query".padEnd(13) + "path".padEnd(8) + "set".padStart(7) + "top10 term-bearing".padStart(21) + "top10 synonym".padStart(15) + "overlap with term top10".padStart(26) + " top result"); for (const [q, syn] of [["tale", "story"], ["short story", "tale"]]) { const termRes = search(idx, q), vecRes = vectorSearch(q), hybRes = hybrid(termRes, vecRes); const qTerms = new Set(tokenize(q)), termTop = new Set(termRes.slice(0, 10).map(([id]) => id)); const threshold = vecRes.filter(([, d]) => d >= 0.5).length; for (const [label, list, size] of [["term", termRes, termRes.length], ["vector", vecRes, threshold], ["hybrid", hybRes, hybRes.length]]) { const top = list.slice(0, 10).map(([id]) => documents[id]); const termBearing = top.filter((b) => text(b).some((x) => qTerms.has(x))).length; const synonym = top.filter((b) => text(b).includes(syn)).length; console.log(q.padEnd(13) + label.padEnd(8) + fmt(size).padStart(7) + `${termBearing}`.padStart(21) + `${synonym}`.padStart(15) + `${top.filter((b) => termTop.has(b.id)).length}`.padStart(26) + ` ${top[0].title}`); } } ``` ``` corpus 1200 documents, seed 20260801; vocabulary 50 terms, embedding dimension 32 vector index 153,600 bytes; inverted index 16,216 posting entries x 8 bytes = 129,728 bytes vector scan per query 38,400 multiply-adds; the set column counts the matching document on the term path, and the document with similarity 0.5 or above on the vector path nearest terms to the "tale" embedding: story 0.997, fiction 0.893, narrative 0.882, rural 0.853 nearest terms to the "fable" embedding: child 0.882, sleep 0.867, school 0.833, forest 0.813 document 0 embedding's first four components: -0.186, -0.173, -0.118, 0.220 query path set top10 term-bearing top10 synonym overlap with term top10 top result tale term 108 10 0 10 tale rural tale vector 580 5 5 2 tale narrative short tale hybrid 153 10 0 5 tale narrative short short story term 245 10 0 10 story short short story vector 608 10 5 1 tale narrative short short story hybrid 156 10 0 4 story narrative short ``` The first lines make concrete what an embedding is: document 0's embedding consists of thirty-two decimal numbers, and the first four sit on the screen. The similarity between "tale" and "story" is 0.997 — these two terms never appear together in any document in the corpus; their closeness comes only from appearing with the same neighbors (fiction, narrative, rural). The inverted index holds no measurable relationship at all between these two strings. #### What Is Gained, What Is Lost, and Combining the Two The "tale" query's three rows are this lesson's summary. The term path matches 108 documents, and all ten of the top ten genuinely carry "tale": precision is exact, but not a single document carrying "story" appears — those documents are outside the set, and no weight can bring them in. The vector path fills **five** of the top ten with documents carrying "story." This is the gain, and it is a gain the term path cannot structurally provide. The same row also shows the loss: only five of the top ten carry "tale," and the number of documents shared with the term path's top ten has dropped to two. If the librarian is searching for that exact word, five of the rows are of no use. The set column shows a second difference. In the term path, "matching document" is well defined: 108. There is no such thing in the vector path; every document has a similarity, and the set is chosen only by a threshold. At a threshold of 0.5, 580 documents remain; a threshold of 0.6 would give a different number. In vector search, the boundary of the set does not come from the data — it comes from the chosen threshold. The "short story" rows show how deep the difference can run: the two paths' top tens share **one** common document. Both lists are defensible — one holds the word, the other holds the context — and they are nearly disjoint. The hybrid rows merge the two lists by rank: each document's score comes from the inverse of its ranks in the two lists. The result recovers exact matching in the top ten — all ten carry "tale" — and overlap with the term list rises from two to five. But the synonym column drops to zero: the fusion pushes the documents the vector path gained outside the top ten. Hybrid search favors the two lists' intersection; the gain does not sit in the top ten but deeper in the list. The set narrows too: 153 documents, because only the first hundred documents from each list enter the fusion. On the cost side there are two numbers. The vector index holds 153,600 bytes, the inverted index 129,728; the vector index grows linearly with document count (a fixed 128 bytes per document), while the inverted index depends on document length and term diversity. The real difference is at query time: the term path reads only the posting lists of the query terms — 216 entries for "tale" — while the vector path touches every document and performs 38,400 multiply-adds. Because there is no such thing as a non-matching document in vector search, there is nothing to eliminate either; shortening the scan requires a separate structure, and this lesson does not build that structure. #### Summary - An embedding is a fixed-length sequence of numbers, and similarity is a measure between two sequences: "tale" and "story" come out at a similarity of 0.997 without ever appearing together in a document, because they appear with the same neighbors. - In the "tale" query, the term path gives 108 documents and ten exact matches in the top ten; the vector path fills five of the top ten with documents carrying "story," and the query term is found in only five. - Vector search has no matching set; the set is chosen by a threshold: 580 documents at a threshold of 0.5. - Hybrid search recovers exact matching in the top ten (ten of ten) and raises overlap with the term list from two to five, but drops the synonym-bearing documents out of the top ten. - The vector index is a fixed 128 bytes per document (153,600 total) and wants 38,400 multiply-adds per query by walking the whole corpus; the term path reads only 216 posting entries for "tale." #### Next Step This topic built the query's whole surface: query language, matching, filtering, scoring, weight, aggregation, presentation, and finally an access method that compares context instead of the word itself. Under every number measured there is an unstated condition: all of these queries ran in a single process, on a single index. The inverted index fit in one computer's memory, the scorer read every posting list locally, the aggregation gathered all its buckets in one place, pagination merged shards in the same process, and the vector scan touched every document in sequence. Once the catalog grows to a million records and the index is spread across multiple machines, none of these queries can run the same way: inverse document frequency becomes a number no single machine knows on its own, where a bucket aggregation merges is an open question, and whether the same query gives the same order on two machines is not guaranteed. The next topic takes up distributing the index across machines and what happens to the same query there. ## Cluster Operations The index living on a cluster: the roles of the master, data, and coordinating nodes, the partitioning and redundancy balance between shard and replica count, the split brain problem and the majority requirement, bulk indexing's effect on write throughput, segment merging's background cost, the index lifecycle through rollover, cooling, and deletion policies, snapshot and restore strategy, and user, role, and API key management. ### Node Roles Source: https://academia.sh/en/courses/search-engines/node-roles Moving search from a single process to a cluster: separating the data, coordinating, and master node roles, running the same three hundred queries under two role placements and counting the postings each node reads and the merge steps it performs, showing that as shard count grows the data work splits while the coordinating work does not, and measuring how placing the master role on a loaded data node delays cluster state and changes the returned document set and the order of the top five results. The previous topic ran every query in a single process over a single index: the query evaluator read the posting lists directly, and the returned order was the scorer's own order. As the catalog grows, this arrangement breaks in two places — the index no longer fits in one machine's memory, and a single process cannot keep up with the incoming query rate. Once the index spreads across multiple machines, the query itself does not change; what changes is who answers it, who reads the posting lists, and who merges the returned candidates. This lesson splits those three tasks into three roles and counts what changes as the roles are distributed over the cluster. #### Three Roles A **data node** carries one shard: that shard's inverted index lives in its memory. When a query arrives it scans only its own shard and returns the top candidates from its own local order. A **coordinating node** receives the query, sends one request to each shard, merges the returned candidate lists, and produces the final list; it reads no posting lists itself. A **master node** takes no part in the query path at all. What it carries is cluster state: which shard sits on which node, which mapping is in effect, where shards get assigned when a node is added. It is the only node that can propagate a new version of that state to the cluster. The setup below models these three roles **in process**: the nodes are objects in the same process, and there is no network or message delay. Because it is a model, what it measures is not time but the work each node does — postings read and merge steps performed, both independent of the run. **CO1 — the query mix consists of four catalog questions and holds an equal share across 300 rounds.** **CO2 — applying one cluster state change costs 500 work units, and indexing runs at one document per 20 work units.** Both assumptions have a linear effect: if the values change, the delay and rejected-document count below change in the same proportion. ```js // cluster/corpus.mjs — library catalog corpus and inverted index. The generator is self-written, // the seed is visible: every run produces the same 4000 records. The index holds a dictionary: // term -> posting list; a posting here is a document entry (document id + term frequency). const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"]; const SPECIAL = { child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"], novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"], travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"], essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"], }; const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"]; const TOPIC = Object.keys(SPECIAL); export function corpus({ count = 4000, seed = 20260731 } = {}) { let s = seed % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; const pick = (a) => a[Math.floor(r() * a.length)], docs = []; for (let i = 1; i <= count; i += 1) { const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)]; if (r() < 0.5) terms.push(pick(COMMON)); for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1) terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE)); docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") }); } return docs; } export function buildIndex(docs) { const postings = new Map(), lengths = new Map(); let bytes = 0; for (const d of docs) { const t = d.text.split(" "), counts = new Map(); for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1); lengths.set(d.id, t.length); for (const [x, n] of counts) { if (!postings.has(x)) { postings.set(x, []); bytes += x.length + 4; } postings.get(x).push([d.id, n]); bytes += 8; } } const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1); return { postings, lengths, N: docs.length, avg, bytes }; } // Scoring: term frequency, inverse document frequency, and document length. In LOCAL scoring df // and average length are read only from this index; if global is given, the whole corpus's // statistics are used instead. export function search(ix, terms, k, global = null) { const N = global ? global.N : ix.N, avg = global ? global.avg : ix.avg, scores = new Map(); let scanned = 0; for (const t of terms) { const g = ix.postings.get(t) ?? []; const df = global ? global.df.get(t) ?? 0 : g.length; const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5)); for (const [id, tf] of g) { scanned += 1; const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg); scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm); } } const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k); return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size }; } // k-way merge: at each step the heads of the shard lists are compared, the highest score is // taken. The step count is the real comparison count and is machine-independent. export function merge(lists, k) { const pos = lists.map(() => 0), result = []; let steps = 0, best = 0; while (result.length < k && best >= 0) { best = -1; for (let i = 0; i < lists.length; i += 1) { const a = lists[i][pos[i]], b = best < 0 ? null : lists[best][pos[best]]; if (!a) continue; steps += 1; if (!b || a.p > b.p || (a.p === b.p && a.id < b.id)) best = i; } if (best >= 0) { result.push(lists[best][pos[best]]); pos[best] += 1; } } return { result, steps }; } ``` ```js // cluster/roles.mjs — the work master, data, and coordinating nodes do per query. The cluster is // modeled IN-PROCESS: nodes are objects in the same process, there is no network. What is measured // is not time but the work each node does — the postings read and the merge steps taken, both // independent of the run. import { corpus, buildIndex, search, merge } from "./corpus.mjs"; const P = 4, KP = 10, K = 10, ROUNDS = 300, STATE_WORK = 500, DOC_WORK = 20, NEW = 800; const QUERIES = [["short", "story", "loneliness"], ["sea", "harbor", "ship"], ["tale", "illustrated", "child"], ["archive", "document", "century"]]; const pad = (x, n) => String(x).padStart(n); const shard = (b, p) => Array.from({ length: p }, (_, i) => buildIndex(b.filter((d) => d.id % p === i))); // One query round: the coordinating node goes to every shard, collects KP local candidates per // shard, and merges them. function query(index, terms, coord, work) { const lists = []; for (let i = 0; i < index.length; i += 1) { const r = search(index[i], terms, KP); work[`d${i}`] = (work[`d${i}`] ?? 0) + r.scanned; lists.push(r.candidates); } const b = merge(lists, K); work[coord] = (work[coord] ?? 0) + b.steps; return { result: b.result, candidates: lists.reduce((t, l) => t + l.length, 0), steps: b.steps }; } const docs = corpus(), whole = buildIndex(docs), index = shard(docs, P), load = {}; console.log(`${docs.length} documents, ${P} shards, ${KP} candidates per shard, final list ${K}. Seed 20260731.`); console.log(`single-shard index: ${whole.postings.size} terms, ${whole.bytes} bytes. Query rounds: ${ROUNDS},`); console.log(`candidates reaching the coordinating node: ${ROUNDS * P * KP}.\n`); console.log("placement | node | roles | work units | share"); console.log("-----------------|-------|---------------------------|-----------|------"); for (const separate of [false, true]) { const work = {}, coord = separate ? "e0" : "d0"; for (let t = 0; t < ROUNDS; t += 1) query(index, QUERIES[t % QUERIES.length], coord, work); const nodes = separate ? ["a0", "e0", "d0", "d1", "d2", "d3"] : ["d0", "d1", "d2", "d3"]; const role = (n) => n === "a0" ? "master" : n === "e0" ? "coordinating" : (n === "d0" && !separate) ? "master+data+coordinating" : "data"; const total = nodes.reduce((t, n) => t + (work[n] ?? 0), 0); load[separate ? "separate" : "combined"] = work; for (const n of nodes) console.log(`${(separate ? "roles separate" : "roles combined").padEnd(16)} | ${n.padEnd(5)} | ${role(n).padEnd(25)} | ` + `${pad(work[n] ?? 0, 9)} | ${pad("%" + (100 * (work[n] ?? 0) / total).toFixed(1), 5)}`); } console.log("\ncoordinating node work per query (same corpus, shard count varies):"); console.log("shards | candidates collected | merge steps | coordinating share | entries scanned per data node"); for (const p of [1, 2, 4, 8, 16]) { const work = {}, r = query(shard(docs, p), QUERIES[0], "e0", work); const data = Object.entries(work).filter(([n]) => n !== "e0").reduce((t, [, v]) => t + v, 0); console.log(`${pad(p, 6)} | ${pad(r.candidates, 21)} | ${pad(r.steps, 11)} | ` + `${pad("%" + (100 * r.steps / (data + r.steps)).toFixed(1), 19)} | ${pad(Math.round(data / p), 30)}`); } // The master node takes no part in the query path; it propagates cluster state. If a fifth shard // is requested after query 150, and the master role is placed on a loaded data node, that work // falls behind that node's query queue. Until the assignment is applied, documents routed to the // new shard are rejected. console.log(`\nmaster node placement: after query 150, a fifth shard is requested, ${NEW} new records are waiting`); console.log("placement | master's queue | state delay | rejected | indexed docs | matched | top 5 results"); const wide = corpus({ count: 4000 + NEW }); for (const separate of [false, true]) { const queue = separate ? 0 : Math.round(load.combined.d0 * (ROUNDS - 150) / ROUNDS); const delay = queue + STATE_WORK; const rejected = Math.min(NEW, Math.floor(delay / DOC_WORK)); const kept = wide.filter((d) => d.id <= 4000 || d.id > 4000 + rejected); const ix = shard(kept, rejected === NEW ? P : P + 1), work = {}; const last = query(ix, QUERIES[0], "e0", work); const matched = ix.reduce((t, x) => t + search(x, QUERIES[0], K).matched, 0); console.log(`${(separate ? "roles separate" : "roles combined").padEnd(16)} | ${pad(queue, 14)} | ${pad(delay, 11)} | ` + `${pad(rejected, 8)} | ${pad(kept.length, 12)} | ${pad(matched, 7)} | ${last.result.slice(0, 5).map((x) => x.id).join(" ")}`); } console.log(`\nrun-independent: the coordinating node collects P x ${KP} candidates per query. As shards` + ` are added, per-data-node work is divided; the coordinating node's work is not divided.`); ``` ``` 4000 documents, 4 shards, 10 candidates per shard, final list 10. Seed 20260731. single-shard index: 80 terms, 382832 bytes. Query rounds: 300, candidates reaching the coordinating node: 12000. placement | node | roles | work units | share -----------------|-------|---------------------------|-----------|------ roles combined | d0 | master+data+coordinating | 94875 | %27.4 roles combined | d1 | data | 78375 | %22.6 roles combined | d2 | data | 89100 | %25.7 roles combined | d3 | data | 84150 | %24.3 roles separate | a0 | master | 0 | %0.0 roles separate | e0 | coordinating | 12000 | %3.5 roles separate | d0 | data | 82875 | %23.9 roles separate | d1 | data | 78375 | %22.6 roles separate | d2 | data | 89100 | %25.7 roles separate | d3 | data | 84150 | %24.3 coordinating node work per query (same corpus, shard count varies): shards | candidates collected | merge steps | coordinating share | entries scanned per data node 1 | 10 | 10 | %0.8 | 1238 2 | 20 | 20 | %1.6 | 619 4 | 40 | 40 | %3.1 | 310 8 | 80 | 80 | %6.1 | 155 16 | 160 | 160 | %11.4 | 77 master node placement: after query 150, a fifth shard is requested, 800 new records are waiting placement | master's queue | state delay | rejected | indexed docs | matched | top 5 results roles combined | 47438 | 47938 | 800 | 4000 | 515 | 2689 701 2181 3159 1777 roles separate | 0 | 500 | 25 | 4775 | 624 | 3159 3544 2689 701 2362 run-independent: the coordinating node collects P x 10 candidates per query. As shards are added, per-data-node work is divided; the coordinating node's work is not divided. ``` #### Where Per-Query Work Falls The first table runs the same 300 queries under two role placements and **returns the same result**: since both placements read the same shard indexes, the returned document set and order are identical either way. What changes is where the work falls. With combined roles, queries all land at one entry point, d0; d0 both scans its own shard and does the entire merge. It carries 94,875 work units, 27.4 percent of the cluster's total, while the lightest node, d1, carries 22.6 percent. The 12,000-unit gap between them is the coordinating work itself; once the roles are split, those units move to e0 and d0 drops to 82,875 units. In the second row group, the master node's share comes out at **0.0 percent**: across 300 queries, a0 read not one posting and made not one comparison. The master node's load grows not with query count but with **how often cluster state changes**. #### Coordinating Work Does Not Split The second table splits the same corpus into 1, 2, 4, 8, and 16 shards and runs a single query. Two columns move in opposite directions. Postings scanned per data node drop from 1238 to 77 — dividing exactly by shard count, since the same posting lists are spread across the shards. The coordinating node's collected candidates and merge steps, by contrast, rise from 10 to 160: every shard must send its own top ten candidates, and the merger compares every list head for every result. The coordinating share rising from 0.8 percent to 11.4 percent is the sum of these two curves: adding shards divides the cheap side and gathers the expensive side onto a single node. #### Master Node Placement Changes the Cluster The third table asks for a fifth shard to be added to the cluster after query 150; 800 new catalog records are waiting to be routed to that shard. When the master role is loaded onto d0, the state change falls behind that node's query queue: 47,438 units of accumulated query work, then 500 units of state work. At CO2's indexing rate, that means all 800 records are rejected, and the corpus stays at 4000 documents. With the master role on its own node, the delay is 500 units, rejected records are 25, and the index grows to 4775 documents. The difference shows up in the query result. The same query — records about a short story touching on loneliness — matches 515 documents under combined placement and 624 under separate placement: a gap of 109 documents. The top five results differ too. Under combined placement the order is 2689, 701, 2181, 3159, 1777; under separate placement it is 3159, 3544, 2689, 701, 2362. Three documents are shared, but none at the same rank, and two list members change entirely. Role placement is not a query setting; it nonetheless changes the returned set and its order, because it determines which documents could even enter the index. #### Summary - The three roles map to three separate tasks: the data node reads posting lists, the coordinating node collects and merges candidates, and the master node propagates cluster state and takes no part in the query path. - Splitting the roles did not change the returned set or order; across 300 queries it moved 12,000 work units from d0 to the coordinating node, and the heaviest node's share fell from 27.4 percent to 25.7 percent. In the same run the master node's share stayed at 0.0 percent: its load grows not with query count but with state changes. - As shard count rose from 1 to 16, postings scanned per data node fell from 1238 to 77, while the coordinating node's candidates and merge steps rose from 10 to 160, its share from 0.8 percent to 11.4 percent. - Loading the master role onto a busy data node delayed the state change by 47,938 units and rejected all 800 records; on its own node the delay was 500 units and 25 records were rejected. That delay carried into the query: matched documents were 515 versus 624, and the order of the top five changed entirely. #### Next Step This lesson held shard count fixed and pulled a fixed number of candidates from each shard; whether either number was well chosen was never questioned. A heavier question is still open: each shard picks its own top ten candidates using **its own local statistics**, and the merger trusts those local scores. As shard count grows, the document frequency each shard measures drifts further from the global value, and the candidates it selects change. The next lesson raises shard count from 1 to 16 and counts how far the top ten results drift from the single-shard index's top ten, then adds replica count's effect on index size and on node loss. ### Shards and Replicas Source: https://academia.sh/en/courses/search-engines/shards-and-replicas Showing that shard count is a scoring decision: splitting the same corpus into one, two, four, eight, and sixteen shards and comparing each placement's top ten result with the single-shard index's top ten by shared documents and rank shift, measuring that raising candidate depth does not fix the drift, counting that a global document-frequency round restores the top ten exactly and costs one extra round, and separately measuring how replica count changes index bytes and the returned document set under node loss. The previous lesson held shard count at four and pulled a fixed number of candidates from each shard. Both numbers quietly assumed something: that the top ten candidates a shard sends are truly that shard's ten best documents. But a shard computes its score using only the document frequency it can see. In a four-shard cluster each shard sees a quarter of the corpus; in a sixteen-shard cluster, a sixteenth. As a term thins out, its inverse document frequency drifts, and what a shard calls "best" changes. This lesson raises shard count from 1 to 16 and counts how far the same four catalog queries' top ten result drifts, then measures replica count as a separate decision. #### Setup The baseline is fixed: the single-shard index's top ten result — the only placement that sees the whole corpus's statistics. Two accuracy measures are used: shared document count and the average rank shift of shared documents. The cost columns are collected candidates, merge steps, and total index bytes. The cluster is again modeled in process: shards are separate inverted indexes in the same process, and there is no network. What is measured is not time but request counts and countable structure sizes. **CO3 — documents are distributed to shards by id remainder, so the split is content-independent and balanced.** This is the best case: in a corpus split by topic, each shard's word distribution would be more skewed and the drift below would grow larger. **CO4 — a query goes to only one copy of each shard, and copies carry the same documents; replica lag is taken as zero.** ```js // cluster/corpus.mjs — library catalog corpus and inverted index (same generator and seed as the // previous lesson; this lesson carries only the parts it uses). The index holds a dictionary: // term -> posting list; a posting here is a document entry (document id + term frequency). const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"]; const SPECIAL = { child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"], novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"], travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"], essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"], }; const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"]; const TOPIC = Object.keys(SPECIAL); export function corpus({ count = 4000, seed = 20260731 } = {}) { let s = seed % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; const pick = (a) => a[Math.floor(r() * a.length)], docs = []; for (let i = 1; i <= count; i += 1) { const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)]; if (r() < 0.5) terms.push(pick(COMMON)); for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1) terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE)); docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") }); } return docs; } export function buildIndex(docs) { const postings = new Map(), lengths = new Map(); let bytes = 0; for (const d of docs) { const t = d.text.split(" "), counts = new Map(); for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1); lengths.set(d.id, t.length); for (const [x, n] of counts) { if (!postings.has(x)) { postings.set(x, []); bytes += x.length + 4; } // dictionary entry postings.get(x).push([d.id, n]); bytes += 8; } } const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1); return { postings, lengths, N: docs.length, avg, bytes }; } // Scoring: term frequency, inverse document frequency, and document length. In LOCAL scoring df // and average length are read only from this index; if global is given, the whole corpus's // statistics are used instead. export function search(ix, terms, k, global = null) { const N = global ? global.N : ix.N, avg = global ? global.avg : ix.avg, scores = new Map(); let scanned = 0; for (const t of terms) { const g = ix.postings.get(t) ?? []; const df = global ? global.df.get(t) ?? 0 : g.length; const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5)); for (const [id, tf] of g) { scanned += 1; const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg); scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm); } } const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k); return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size }; } // k-way merge: at each step the heads of the shard lists are compared, the highest score is // taken. The step count is the real comparison count and is machine-independent. export function merge(lists, k) { const pos = lists.map(() => 0), result = []; let steps = 0, best = 0; while (result.length < k && best >= 0) { best = -1; for (let i = 0; i < lists.length; i += 1) { const a = lists[i][pos[i]], b = best < 0 ? null : lists[best][pos[best]]; if (!a) continue; steps += 1; if (!b || a.p > b.p || (a.p === b.p && a.id < b.id)) best = i; } if (best >= 0) { result.push(lists[best][pos[best]]); pos[best] += 1; } } return { result, steps }; } ``` ```js // cluster/shards.mjs — the effect of shard count and candidate depth on the top ten result. The // cluster is modeled IN-PROCESS: shards are separate inverted indexes in the same process. The // baseline is the single-shard index's top ten result; every placement is compared against it by // shared documents and rank shift. import { corpus, buildIndex, search, merge } from "./corpus.mjs"; const K = 10, QUERIES = [["short", "story", "loneliness"], ["sea", "harbor", "ship"], ["tale", "illustrated", "child"], ["archive", "document", "century"]]; const pad = (x, n) => String(x).padStart(n); const shard = (b, p) => Array.from({ length: p }, (_, i) => buildIndex(b.filter((d) => d.id % p === i))); // Global statistics: document frequency is summed across every shard. Using this costs one extra // round — the coordinating node first sums df, then has the shards score with it (a two-round query). function global(ix) { const df = new Map(); let N = 0, len = 0; for (const d of ix) { N += d.N; len += d.avg * d.N; for (const [t, g] of d.postings) df.set(t, (df.get(t) ?? 0) + g.length); } return { df, N, avg: len / N }; } // One query: each shard returns KP local candidates, the coordinating node merges. If glo is // given, shards score with global statistics instead (a two-round query). function ask(ix, terms, KP, glo = null) { const lists = ix.map((d) => search(d, terms, KP, glo).candidates); const b = merge(lists, K); return { top: b.result.map((x) => x.id), steps: b.steps, candidates: lists.reduce((t, l) => t + l.length, 0) }; } const docs = corpus(), single = buildIndex(docs); const baseline = QUERIES.map((q) => search(single, q, K).candidates.map((x) => x.id)); const compare = (a, i) => { // shared documents and total rank shift vs. baseline let shared = 0, shift = 0; a.forEach((id, j) => { const k = baseline[i].indexOf(id); if (k >= 0) { shared += 1; shift += Math.abs(k - j); } }); return { shared, shift }; }; const average = (f) => { const t = QUERIES.map(f); return t.reduce((a, b) => a + b, 0) / t.length; }; console.log(`${docs.length} documents, seed 20260731, ${single.postings.size} terms. Single-shard index: ${single.bytes} bytes.`); console.log(`Baseline: single-shard index's top ${K} result, averaged over four catalog queries.\n`); console.log("shards | index bytes | candidates | merge steps | depth 10: shared / shift | depth 50 | global df round"); console.log("-------|-------------|------------|-------------|---------------------------|----------|----------------"); for (const P of [1, 2, 4, 8, 16]) { const ix = shard(docs, P), glo = global(ix); const bytes = ix.reduce((t, d) => t + d.bytes, 0); const d10 = average((q, i) => compare(ask(ix, q, 10).top, i).shared); const k10 = average((q, i) => { const x = compare(ask(ix, q, 10).top, i); return x.shift / x.shared; }); const d50 = average((q, i) => compare(ask(ix, q, 50).top, i).shared); const dglo = average((q, i) => compare(ask(ix, q, 10, glo).top, i).shared); const r = ask(ix, QUERIES[0], 10); console.log(`${pad(P, 6)} | ${pad(bytes, 11)} | ${pad(r.candidates, 10)} | ${pad(r.steps, 11)} | ` + `${pad(d10.toFixed(2) + " / " + k10.toFixed(2), 25)} | ${pad(d50.toFixed(2), 8)} | ${pad(dglo.toFixed(2), 15)}`); } console.log("\ntop 10 result (query: short story loneliness):"); console.log(`${"single shard".padEnd(23)}: ${baseline[0].join(" ")}`); for (const P of [4, 16]) { const ix = shard(docs, P); console.log(`${(P + " shards, depth 10").padEnd(23)}: ${ask(ix, QUERIES[0], 10).top.join(" ")}`); console.log(`${(P + " shards, global df").padEnd(23)}: ${ask(ix, QUERIES[0], 10, global(ix)).top.join(" ")}`); } // Replica: R+1 copies of each shard sit on separate nodes. A query goes to ONE copy of each // shard; replica count does not change query width, it changes index bytes and node loss. const P = 4, ix = shard(docs, P), bytes = ix.reduce((t, d) => t + d.bytes, 0); console.log("\nreplica count (4 shards, one node lost):"); console.log("replicas | nodes | total index bytes | nodes touched | unreachable shards | matched | remaining in top 10"); for (const R of [0, 1, 2]) { const lost = R === 0 ? 1 : 0; // R=0: one node loss takes an entire shard down const remaining = ix.filter((_, i) => i >= lost); const matched = remaining.reduce((t, d) => t + search(d, QUERIES[0], K).matched, 0); const top = ask(remaining, QUERIES[0], 10).top; console.log(`${pad(R, 8)} | ${pad(P * (R + 1), 5)} | ${pad(bytes * (R + 1), 18)} | ${pad(P, 13)} | ` + `${pad(lost, 19)} | ${pad(matched, 7)} | ${pad(top.filter((x) => baseline[0].includes(x)).length + "/10", 19)}`); } console.log(`\nrun-independent: collected candidates are P x depth, merge steps are ${K} x P, index bytes` + ` scale by (R+1). A single-round query costs P requests, the global df round costs 2P. The` + ` number of nodes a query touches does not change with replica count, only with shard count.`); ``` ``` 4000 documents, seed 20260731, 80 terms. Single-shard index: 382832 bytes. Baseline: single-shard index's top 10 result, averaged over four catalog queries. shards | index bytes | candidates | merge steps | depth 10: shared / shift | depth 50 | global df round -------|-------------|------------|-------------|---------------------------|----------|---------------- 1 | 382832 | 10 | 10 | 10.00 / 0.00 | 10.00 | 10.00 2 | 383656 | 20 | 20 | 8.50 / 1.28 | 8.50 | 10.00 4 | 385304 | 40 | 40 | 6.25 / 2.50 | 6.25 | 10.00 8 | 388600 | 80 | 80 | 4.75 / 2.04 | 4.75 | 10.00 16 | 395192 | 160 | 160 | 4.75 / 4.01 | 4.75 | 10.00 top 10 result (query: short story loneliness): single shard : 3159 3544 2689 2362 3370 701 2448 502 2181 2320 4 shards, depth 10 : 2689 701 2181 3159 1777 549 3544 2362 3370 565 4 shards, global df : 3159 3544 2689 2362 3370 701 2448 502 2181 2320 16 shards, depth 10 : 2689 1777 3265 2323 2180 2529 3544 502 2181 1150 16 shards, global df : 3159 3544 2689 2362 3370 701 2448 502 2181 2320 replica count (4 shards, one node lost): replicas | nodes | total index bytes | nodes touched | unreachable shards | matched | remaining in top 10 0 | 4 | 385304 | 4 | 1 | 379 | 6/10 1 | 8 | 770608 | 4 | 0 | 515 | 7/10 2 | 12 | 1155912 | 4 | 0 | 515 | 7/10 run-independent: collected candidates are P x depth, merge steps are 10 x P, index bytes scale by (R+1). A single-round query costs P requests, the global df round costs 2P. The number of nodes a query touches does not change with replica count, only with shard count. ``` #### Shard Count Corrupts the Top Ten The first table's cost columns behave as expected: collected candidates and merge steps grow linearly with shard count, from 10 to 160. Total index bytes rise from 382,832 to 395,192, a 3.2 percent increase; the source is not the posting lists but the dictionary — the same 80 terms are repeated in every shard. The real number is in the accuracy columns. At two shards, the top ten shares an average of 8.5 documents with the baseline; at four, 6.25; at eight and sixteen, 4.75. At sixteen shards, more than half the top ten result is **absent from the baseline entirely**, and the average rank shift of the shared documents rises from 1.28 to 4.01. The lists show this directly. The single-shard index orders 3159, 3544, 2689 first; the same query under four shards returns 2689, 701, 2181 — the baseline's first result, 3159, drops to fourth place, and 1777, 549, and 565, none in the baseline, enter the list. At sixteen shards, 3159 **drops out of the top ten entirely** and only four documents remain shared with the baseline. No query changed, no document was deleted; the only thing that changed is how many documents each shard can see. #### Depth Does Not Fix It, an Extra Round Does The obvious way to close the gap is to pull more candidates from each shard. The table refutes this: raising candidate depth from 10 to 50 leaves the shared document count **unchanged** (8.50 / 6.25 / 4.75 / 4.75). The error is not coming from truncation: each shard sorts its list by its own local score, and the merger trusts those scores. What is wrong is not the length of the ranking but the ranking itself. What fixes it is in the last column: the coordinating node first sums per-term document frequency across every shard, then has the shards score using that global statistic. Shared document count returns to 10.00 at every shard count; the four- and sixteen-shard lists are identical to the single-shard index's. The cost is that the query goes from one round to two, from P requests to 2P. #### Replica Count Preserves the Set, Not the Order The second table measures replicas as a separate decision. In a replica-free, four-shard cluster, losing one node takes an entire shard down with it: matched documents fall from 515 to 379 — 136 catalog records never appear to the query — and shared top ten documents drop from 7 to 6. Adding one replica means the same node loss changes nothing. The cost is index bytes, and it is linear: from 385,304 to 770,608, and to 1,155,912 at two replicas. The number of nodes a query touches stays at 4 across all three rows — adding a replica does not widen the query. The last column staying at 7/10 is this lesson's distinction: replicas brought back the **lost documents**, not the **wrong order**. That three-document gap comes from local scoring, and only a global-statistics round closes it. Redundancy is a durability decision; sharding, at the same time, is a scoring decision. #### Summary - As shard count rose from 1 to 16, collected candidates and merge steps rose from 10 to 160, and total index bytes rose from 382,832 to 395,192; the increase comes not from the posting lists but from the dictionary repeated in every shard. - The top ten's shared document count with the baseline fell to 6.25 at four shards and 4.75 at sixteen; the shared documents' average rank shift rose to 4.01. - Raising candidate depth from 10 to 50 changed accuracy not at all: the error does not come from truncation but from per-shard local document frequency. - A global document-frequency round returned the top ten to 10.00 at every shard count and made the lists identical to the single-shard index's; the cost is the query rising from P requests to 2P. - In a replica-free cluster, one node loss dropped matched documents from 515 to 379; one replica prevented this entirely, doubled index bytes, and did not change the number of nodes touched. Replicas bring back the lost set, not the corrupted order: the top ten's shared document count stayed at 7/10. #### Next Step This lesson measured replicas as a durability decision and assumed replicas carry the same documents; that holds while the cluster is healthy. When the network splits in two, replicas can no longer see each other and each half can accept writes on its own: documents enter the same index from two places, and the two halves' posting lists diverge. The next lesson splits a five-replica index into a three-node half and a two-node half, toggles the majority requirement, and counts two things — how many of the indexing operations accepted in the minority are lost once the split is repaired, and how many documents each half's query returns short of the full set while the split lasts. ### Split Brain Problem Source: https://academia.sh/en/courses/search-engines/split-brain-problem Splitting a five-replica index into a three-node and a two-node half: counting the indexing operations that are accepted in the minority half while the majority requirement is off and lost on repair, measuring that the same query returns a different document set and different top five in each half, comparing the cost of the minority rejecting writes and answering queries short — under the majority requirement on — in write messages and missing documents, and comparing the post-repair index with the index that would exist had the split never happened. The previous lesson assumed replicas carry the same documents, and measured on that assumption that a replica brings back a lost set. The assumption holds only as long as replicas can see each other. When the network splits in two, a five-replica index turns into two separate indexes: each half keeps growing its own inverted index, the two halves' posting lists diverge, and the same query starts returning two different orders. This is called **split brain**. The same phenomenon was measured in the leader election narrative under the name **two-leader round**; the election mechanics and automatic failover were built there and are not repeated here. The question here is on the search side: what is left of an indexing operation accepted in the minority, and which documents a query fails to return while the split lasts. #### Setup The index has five replicas, and the network splits into a three-node A half and a two-node B half. While the split lasts, 600 new records arrive at the catalog — a single-topic donation batch, meaning documents that fall squarely inside the query. How many replicas must approve a write for it to be accepted is a setting: with the **majority requirement off**, one approval is enough; **on**, three approvals — a majority of five replicas — are required. The baseline is the response the 4600-document index would give to the same query had the split never happened. The cluster is again modeled in process here: the halves are separate inverted indexes in the same process, and a routing rule stands in for the network. **CO5 — of the 600 records arriving during the split, 65 percent reach the three-node half and 35 percent reach the two-node half.** Rationale: if clients are distributed across nodes, the split divides them by node count. The effect is linear. **CO6 — once the split is repaired, the majority half's index is taken as canonical.** One side has to win; whichever side wins, the writes the other side accepted are lost. ```js // cluster/corpus.mjs — library catalog corpus and inverted index (same generator and seed as the // earlier lessons; this lesson carries only the parts it uses). The index holds a dictionary: // term -> posting list; a posting here is a document entry (document id + term frequency). const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"]; const SPECIAL = { child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"], novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"], travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"], essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"], }; const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"]; const TOPIC = Object.keys(SPECIAL); export function corpus({ count = 4000, seed = 20260731 } = {}) { let s = seed % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; const pick = (a) => a[Math.floor(r() * a.length)], docs = []; for (let i = 1; i <= count; i += 1) { const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)]; if (r() < 0.5) terms.push(pick(COMMON)); for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1) terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE)); docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") }); } return docs; } export function buildIndex(docs) { const postings = new Map(), lengths = new Map(); let bytes = 0; for (const d of docs) { const t = d.text.split(" "), counts = new Map(); for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1); lengths.set(d.id, t.length); for (const [x, n] of counts) { if (!postings.has(x)) { postings.set(x, []); bytes += x.length + 4; } postings.get(x).push([d.id, n]); bytes += 8; // id + frequency } } const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1); return { postings, lengths, N: docs.length, avg, bytes }; } // Scoring: term frequency, inverse document frequency, and document length; statistics are read // from this index. export function search(ix, terms, k) { const N = ix.N, avg = ix.avg, scores = new Map(); let scanned = 0; for (const t of terms) { const g = ix.postings.get(t) ?? [], df = g.length; const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5)); for (const [id, tf] of g) { scanned += 1; const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg); scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm); } } const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k); return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size }; } ``` ```js // cluster/split.mjs — a five-replica index split into a three-node and a two-node half. The // cluster is modeled IN-PROCESS: the halves are separate inverted indexes in the same process, // and a routing rule stands in for the network. The baseline is the top ten result the index // would return had the split never happened. import { corpus, buildIndex, search } from "./corpus.mjs"; const N0 = 4000, NEW = 600, REPLICAS = 5, A_NODES = 3, B_NODES = 2, K = 10; const QUERY = ["short", "story", "loneliness"]; const pad = (x, n) => String(x).padStart(n); const base = corpus({ count: N0 }); // The batch cataloged during the split: a single-topic donation. Generated with a separate // seed, ids moved past N0; its content is left untouched. const batch = corpus({ count: 9000, seed: 20260801 }).filter((d) => d.topic === "story") .slice(0, NEW).map((d, i) => ({ ...d, id: N0 + i + 1 })); const aNew = batch.filter((d) => d.id % 20 < 13); // 65% of clients reach the three-node half const bNew = batch.filter((d) => d.id % 20 >= 13); // 35% reach the two-node half const baseline = buildIndex([...base, ...batch]), baseResult = search(baseline, QUERY, K); const baseTop = baseResult.candidates.map((x) => x.id), target = baseResult.matched; const look = (docs) => { // the set and order one index returns for this query const ix = buildIndex(docs), r = search(ix, QUERY, K); const top = r.candidates.map((x) => x.id); return { count: docs.length, matched: r.matched, top, shared: top.filter((x) => baseTop.includes(x)).length }; }; console.log(`${N0} documents in the index, ${NEW} new records arrive during the split. ${REPLICAS} replicas,`); console.log(`halves ${A_NODES} and ${B_NODES} nodes. Seed 20260731. Baseline: the ${N0 + NEW}-document index had the split`); console.log(`never happened, query "short story loneliness", matched ${target} documents.\n`); console.log("majority | W | A accepted | B accepted | B rejected | lost on repair | write messages | post-repair documents"); console.log("---------|---|------------|------------|------------|-----------------|----------------|----------------------"); const result = {}; for (const majority of [false, true]) { const W = majority ? Math.floor(REPLICAS / 2) + 1 : 1; const bAccepted = majority && B_NODES < W ? 0 : bNew.length; const bRejected = bNew.length - bAccepted; const lost = majority ? 0 : bAccepted; // on repair the majority half's index is canonical const messages = aNew.length * Math.min(W, A_NODES) + bAccepted * Math.min(W, B_NODES) + bRejected * B_NODES; const after = base.length + aNew.length + (majority ? bRejected : 0); // rejected writes are resubmitted result[majority] = { W, bAccepted, after }; console.log(`${(majority ? "on" : "off").padEnd(8)} | ${pad(W, 1)} | ${pad(aNew.length, 10)} | ${pad(bAccepted, 10)} | ` + `${pad(bRejected, 10)} | ${pad(lost, 15)} | ${pad(messages, 14)} | ${pad(after, 22)}`); } console.log("\nsame query while the split lasts (baseline: the top 10 that would return had the split never happened):"); console.log("majority | half | nodes | documents in index | matched | missing | top 10 shared | top 5 result"); for (const majority of [false, true]) { for (const [name, nodes, add] of [["A", A_NODES, aNew], ["B", B_NODES, result[majority].bAccepted ? bNew : []]]) { const r = look([...base, ...add]); console.log(`${(majority ? "on" : "off").padEnd(8)} | ${name.padEnd(4)} | ${pad(nodes, 5)} | ${pad(r.count, 19)} | ` + `${pad(r.matched, 7)} | ${pad(target - r.matched, 7)} | ${pad(r.shared + "/10", 14)} | ${r.top.slice(0, 5).join(" ")}`); } } console.log("\nafter repair:"); console.log("majority | documents in index | matched | top 10 shared | top 5 result"); for (const majority of [false, true]) { const add = majority ? [...aNew, ...bNew] : aNew; const r = look([...base, ...add]); console.log(`${(majority ? "on" : "off").padEnd(8)} | ${pad(r.count, 19)} | ${pad(r.matched, 7)} | ` + `${pad(r.shared + "/10", 14)} | ${r.top.slice(0, 5).join(" ")}`); } console.log(`baseline | ${pad(baseline.N, 19)} | ${pad(target, 7)} | ${pad("10/10", 14)} | ${baseTop.slice(0, 5).join(" ")}`); console.log(`\nrun-independent: at ${REPLICAS} replicas the majority is ${Math.floor(REPLICAS / 2) + 1}, and two disjoint halves` + ` cannot both hold a majority at once, because ${Math.floor(REPLICAS / 2) + 1} + ${Math.floor(REPLICAS / 2) + 1} > ${REPLICAS}.`); ``` ``` 4000 documents in the index, 600 new records arrive during the split. 5 replicas, halves 3 and 2 nodes. Seed 20260731. Baseline: the 4600-document index had the split never happened, query "short story loneliness", matched 1115 documents. majority | W | A accepted | B accepted | B rejected | lost on repair | write messages | post-repair documents ---------|---|------------|------------|------------|-----------------|----------------|---------------------- off | 1 | 390 | 210 | 0 | 210 | 600 | 4390 on | 3 | 390 | 0 | 210 | 0 | 1590 | 4600 same query while the split lasts (baseline: the top 10 that would return had the split never happened): majority | half | nodes | documents in index | matched | missing | top 10 shared | top 5 result off | A | 3 | 4390 | 905 | 210 | 8/10 | 4544 3159 3544 2689 2362 off | B | 2 | 4210 | 725 | 390 | 8/10 | 4457 4534 3159 3544 2689 on | A | 3 | 4390 | 905 | 210 | 8/10 | 4544 3159 3544 2689 2362 on | B | 2 | 4000 | 515 | 600 | 6/10 | 3159 3544 2689 2362 3370 after repair: majority | documents in index | matched | top 10 shared | top 5 result off | 4390 | 905 | 8/10 | 4544 3159 3544 2689 2362 on | 4600 | 1115 | 10/10 | 4457 4534 4544 3159 3544 baseline | 4600 | 1115 | 10/10 | 4457 4534 4544 3159 3544 run-independent: at 5 replicas the majority is 3, and two disjoint halves cannot both hold a majority at once, because 3 + 3 > 5. ``` #### Writes Accepted in the Minority The first table shows the write side of the two settings. With the majority requirement off, both halves accept writes: A takes 390, B takes 210. From the librarian's view, all 600 of these records were cataloged successfully — each one received an acknowledgment. On repair, CO6 takes effect and the **210 records the minority half accepted are lost**. These records did not return an error, are not waiting in a queue, and are not on a list to be resubmitted; they were acknowledged and then ceased to exist. With the majority requirement on, the same 210 requests are rejected from the start. The difference is where the loss falls: a rejected request stays in the client's hands and can be resubmitted once the split is repaired. The lost column reads 0, and the post-repair index holds 4600 documents. Both settings lived through the same network fault; one swallowed 210 records silently, the other rejected them 210 times, visibly. The cost is in the write-message column: from 600 messages to 1590. The majority requirement asks for three approvals for every accepted write, and even the rejected requests fail only after touching two nodes in the minority half. That is roughly a 2.65-fold increase in messages per write, and it feeds straight into latency. The run-independent number on the last line says why this setting works at all: at five replicas the majority is three, and two disjoint halves cannot both hold a majority at once, because 3 + 3 is greater than five. This is not a setting; it is arithmetic. #### The Query While the Split Lasts The second table counts what the same query returns while the split lasts. Had the split never happened, 1115 documents would have matched. With the majority requirement off, both halves answer, and **both fall short**: A returns 905 documents (210 short), B returns 725 (390 short). Their top five orders diverge too. Half A opens with 4544; half B opens with 4457 and 4534. All three of these records come from the donation batch, and each exists only in its own half. Two librarians asking the same catalog question from two different ends, at the same time, get two contradicting lists. This is the search-side counterpart of split brain: not a wrong answer, but **two separate right ones**. With the majority requirement on, half B's index freezes at 4000 documents: 515 matches, 600 missing documents, and only 6 documents shared with the baseline's top ten. The minority half keeps answering, but it answers from a **stale index**. This setting does not guarantee a complete returned set; it guarantees that the minority does not carry its index forward. #### After Repair The third table compares the final state with the baseline. With the majority requirement off, the index stays at 4390 documents: 905 matches, 8/10 in the top ten, and the baseline's first two records (4457, 4534) missing entirely from the top five. These two records do not come back; the only way to reindex them is to return to the donation batch's source and recatalog the 210 records by hand. With the majority requirement on, the post-repair index holds 4600 documents, 1115 matches, and a top ten identical to the baseline's. Because the 210 rejected requests are resubmitted, the result looks as if the split never happened. The majority requirement did not prevent the fault — the network still split, and the minority half answered short for all 600 records. What it prevented was the fault becoming **permanent**. #### Summary - With the majority requirement off, both halves accepted writes (390 and 210) and on repair the 210 records the minority accepted were lost; these are records the client had already been acknowledged for. - With the majority requirement on, the same 210 requests were rejected, the lost count was 0, and resubmitting them after repair brought the index to 4600 documents. - The majority requirement's cost is write messages: from 600 to 1590, roughly a 2.65-fold increase per write. - While the split lasted, with the majority requirement off, half A returned 905 documents and half B 725, and their top five lists diverged: A opened with 4544, B with 4457. The same question got two contradicting answers. - With the majority requirement on, the minority half returned 515 matches from its 4000-document stale index: 600 missing documents and 6/10 in the top ten. The guarantee is not a complete answer but that the minority does not advance its index. - After repair, the majority-off setting stayed at 905 matches and 8/10; the majority-on setting produced a result as if the split never happened — 1115 matches and 10/10. #### Next Step This lesson counted the 600 records one by one, as if each were a separate request and became searchable the moment it was accepted. Neither assumption was measured. A catalog batch actually arrives in bulk, not one record at a time, and sending each record as its own request is not the same as sending all of them in one request: request count, the number of segments produced, and total time all change. A document entering the index does not become searchable instantly either; it is first written to a buffer and becomes visible to queries only once a **refresh** operation runs. The next lesson indexes the same 4000-document corpus one record at a time and in bulk, compares request count against segment count, then varies the refresh interval and counts the trade-off between visibility lag and write throughput. ### Bulk Indexing Optimization Source: https://academia.sh/en/courses/search-engines/bulk-indexing-optimization Indexing the same four-thousand-record corpus one at a time and in bulk: comparing them by request count, segments produced, total index bytes, and dictionary lookups per query, showing that batch size and refresh interval are two independent axes, and measuring the trade-off between visibility lag and query cost as the refresh interval changes, counted on a six-hundred-record donation batch by matched documents, missing documents, and the order of the top five results. The previous lesson counted the 600 records arriving during the split one by one, quietly assuming two things: that each record is a separate request, and that it becomes searchable the moment it is accepted. Neither was measured. A catalog batch arrives in bulk; sending 600 records as 600 requests is not the same as sending them as 12. A document entering the index does not become searchable instantly either: it is first written to a buffer, and only becomes visible to queries once a refresh operation turns the buffer into a segment. This lesson measures two settings separately — batch size and refresh interval. #### Setup A **segment** is an immutable inverted index built from one buffer's worth of documents; a new document is never added to an existing segment, a new segment is born instead. A query must look at every segment: it does one dictionary lookup per term in each segment and reads the posting lists. Because scoring statistics are collected across every segment, segment count changes cost, not order; the only effect on order comes from **visibility**. Time is measured too, but raw time is not printed: time depends on the environment, and because the in-process model carries no network, it cannot measure the per-request network cost. What is printed is whether time sits above or below a threshold; the numbers carrying the real weight are requests, segments, bytes, and dictionary lookups. **CO7 — the 600-record donation batch enters the catalog in requests of 50, and the same query runs right after every request.** The effect is linear: if request size changes, the invisible-document count changes in the same direction. **CO8 — refresh runs in the background every `interval` documents and turns the buffer into a single segment.** A document waiting in the buffer is invisible to the query; that is the model's rule. ```js // cluster/corpus.mjs — library catalog corpus and inverted index (same generator and seed as the // earlier lessons). buildIndex() here builds A SINGLE SEGMENT: a segment is an immutable inverted // index built from one buffer's worth of documents. const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"]; const SPECIAL = { child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"], novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"], travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"], essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"], }; const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"]; const TOPIC = Object.keys(SPECIAL); export function corpus({ count = 4000, seed = 20260731 } = {}) { let s = seed % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; const pick = (a) => a[Math.floor(r() * a.length)], docs = []; for (let i = 1; i <= count; i += 1) { const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)]; if (r() < 0.5) terms.push(pick(COMMON)); for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1) terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE)); docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") }); } return docs; } export function buildIndex(docs) { const postings = new Map(), lengths = new Map(); let bytes = 0; for (const d of docs) { const t = d.text.split(" "), counts = new Map(); for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1); lengths.set(d.id, t.length); for (const [x, n] of counts) { if (!postings.has(x)) { postings.set(x, []); bytes += x.length + 4; } // dictionary entry postings.get(x).push([d.id, n]); bytes += 8; } } const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1); return { postings, lengths, N: docs.length, avg, bytes }; } // Scoring: term frequency, inverse document frequency, and document length. In LOCAL scoring df // and average length are read only from this index; if global is given, the whole corpus's // statistics are used instead. export function search(ix, terms, k, global = null) { const N = global ? global.N : ix.N, avg = global ? global.avg : ix.avg, scores = new Map(); let scanned = 0; for (const t of terms) { const g = ix.postings.get(t) ?? []; const df = global ? global.df.get(t) ?? 0 : g.length; const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5)); for (const [id, tf] of g) { scanned += 1; const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg); scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm); } } const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k); return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size }; } ``` ```js // cluster/bulk.mjs — measuring one-at-a-time versus bulk indexing, and the refresh interval. The // index is modeled IN-PROCESS: documents are first written to a buffer, REFRESH turns the buffer // into an immutable segment, and only then does the document become visible to queries. Scoring // statistics are collected across every segment, so segment count changes only cost, not order. import { corpus, buildIndex, search } from "./corpus.mjs"; const N = 4000, NEW = 600, REQUEST = 50, K = 10, MEASURE = 100, QUERY = ["short", "story", "loneliness"]; const pad = (x, n) => String(x).padStart(n); const empty = () => ({ segments: [], buffer: [], requests: 0, lookups: 0 }); const add = (D, batch) => { D.requests += 1; D.buffer.push(...batch); }; const refresh = (D) => { if (D.buffer.length) { D.segments.push(buildIndex(D.buffer)); D.buffer = []; } }; function ask(D, terms, k) { const df = new Map(); let n = 0, len = 0; for (const b of D.segments) { // first pass: cross-segment statistics n += b.N; len += b.avg * b.N; for (const t of terms) { D.lookups += 1; const g = b.postings.get(t); if (g) df.set(t, (df.get(t) ?? 0) + g.length); } } const glo = { df, N: n, avg: len / n }, scores = []; for (const b of D.segments) { D.lookups += terms.length; scores.push(...search(b, terms, Infinity, glo).candidates); } scores.sort((a, b) => b.p - a.p || a.id - b.id); return { top: scores.slice(0, k).map((x) => x.id), matched: scores.length, visible: n }; } // Send N documents in batches of `batch`; a refresh runs every `interval` documents. function build(docs, batch, interval) { const D = empty(); for (let i = 0; i < docs.length; i += batch) { add(D, docs.slice(i, i + batch)); if ((i + batch) % interval === 0 || i + batch >= docs.length) refresh(D); } return D; } // Time measurement: indexing + MEASURE queries. Raw time DEPENDS ON THE ENVIRONMENT, so below // only whether it is above or below a threshold is printed. The smallest of three runs is kept. function time(batch, interval) { let best = Infinity; for (let k = 0; k < 3; k += 1) { const t0 = process.hrtime.bigint(); const D = build(base, batch, interval); for (let q = 0; q < MEASURE; q += 1) ask(D, QUERY, K); best = Math.min(best, Number(process.hrtime.bigint() - t0)); } return best; } const base = corpus({ count: N }); console.log(`${N} documents, seed 20260731. Query "short story loneliness", final list ${K}.`); console.log("Time DEPENDS ON THE ENVIRONMENT; only a thresholded ratio is printed instead of raw time.\n"); console.log("batch | refresh interval | requests | segments | index bytes | dictionary lookups per query"); console.log("------|------------------|----------|----------|-------------|------------------------------"); for (const [batch, interval] of [[1, 1], [1, 500], [500, 500], [1000, 1000], [500, N]]) { const D = build(base, batch, interval); D.lookups = 0; ask(D, QUERY, K); console.log(`${pad(batch, 5)} | ${pad(interval, 16)} | ${pad(D.requests, 8)} | ${pad(D.segments.length, 8)} | ` + `${pad(D.segments.reduce((t, b) => t + b.bytes, 0), 11)} | ${pad(D.lookups, 30)}`); } time(500, 500); // warm-up run const ratio = time(1, 1) / time(500, 500); console.log(`\nindexing + ${MEASURE} query time, one-at-a-time (1/1) over bulk (500/500): ` + `${ratio > 2 ? "more than 2x" : "2x or under"} (raw time depends on the environment, not printed)`); // Visibility: a 600-record donation batch enters the 4000-document index in batches of 50, and the // same query runs right after each request. Refresh runs in the background every `interval` // documents; unrefreshed documents wait in the buffer and are invisible to the query. const donation = corpus({ count: 9000, seed: 20260801 }).filter((d) => d.topic === "story") .slice(0, NEW).map((d, i) => ({ ...d, id: N + i + 1 })); const baseline = ask(build([...base, ...donation], 500, 500), QUERY, K); console.log(`\nrefresh interval (a ${NEW}-record batch enters in requests of ${REQUEST}, one query after each request):`); console.log(`baseline: matched ${baseline.matched} once the batch is fully visible, top 5: ${baseline.top.slice(0, 5).join(" ")}`); console.log("interval | segments added | avg. invisible | last query matched | missing | top 10 shared | lookups | top 5 result"); for (const interval of [1, 75, 250, NEW * 2]) { const D = build(base, 500, 500), baseSegments = D.segments.length; let invisible = 0, visible = 0, last = null; for (let g = REQUEST; g <= NEW; g += REQUEST) { const target = Math.floor(g / interval) * interval; while (visible < target) { // every refresh creates one segment add(D, donation.slice(visible, visible + interval)); refresh(D); visible += interval; } D.lookups = 0; last = ask(D, QUERY, K); invisible += g - visible; } console.log(`${pad(interval > NEW ? "none" : interval, 8)} | ${pad(D.segments.length - baseSegments, 15)} | ` + `${pad((invisible / (NEW / REQUEST)).toFixed(1), 14)} | ${pad(last.matched, 19)} | ${pad(baseline.matched - last.matched, 7)} | ` + `${pad(last.top.filter((x) => baseline.top.includes(x)).length + "/10", 14)} | ${pad(D.lookups, 7)} | ${last.top.slice(0, 5).join(" ")}`); } console.log(`\nrun-independent: request count = documents / batch size, segment count = documents / refresh` + ` interval, dictionary lookups per query = segments x terms x 2.`); ``` ``` 4000 documents, seed 20260731. Query "short story loneliness", final list 10. Time DEPENDS ON THE ENVIRONMENT; only a thresholded ratio is printed instead of raw time. batch | refresh interval | requests | segments | index bytes | dictionary lookups per query ------|------------------|----------|----------|-------------|------------------------------ 1 | 1 | 4000 | 4000 | 869327 | 24000 1 | 500 | 4000 | 8 | 388600 | 48 500 | 500 | 8 | 8 | 388600 | 48 1000 | 1000 | 4 | 4 | 385304 | 24 500 | 4000 | 8 | 1 | 382832 | 6 indexing + 100 query time, one-at-a-time (1/1) over bulk (500/500): more than 2x (raw time depends on the environment, not printed) refresh interval (a 600-record batch enters in requests of 50, one query after each request): baseline: matched 1115 once the batch is fully visible, top 5: 4457 4534 4544 3159 3544 interval | segments added | avg. invisible | last query matched | missing | top 10 shared | lookups | top 5 result 1 | 600 | 0.0 | 1115 | 0 | 10/10 | 3648 | 4457 4534 4544 3159 3544 75 | 8 | 25.0 | 1115 | 0 | 10/10 | 96 | 4457 4534 4544 3159 3544 250 | 2 | 95.8 | 1015 | 100 | 8/10 | 60 | 4457 3159 3544 2689 2362 none | 0 | 325.0 | 515 | 600 | 6/10 | 48 | 3159 3544 2689 2362 3370 run-independent: request count = documents / batch size, segment count = documents / refresh interval, dictionary lookups per query = segments x terms x 2. ``` #### Two Separate Axes The first table's first row is the worst case: every document arrives as its own request and a refresh runs after every request. The result is 4000 requests, 4000 segments, an 869,327-byte index, and 24,000 dictionary lookups for a single query. The fifth row indexes the same corpus with 8 requests and a single segment: 382,832 bytes and 6 lookups per query. Same 4000 documents, same order, four thousand times fewer dictionary lookups. The byte count more than doubling is not caused by the posting lists but by the dictionary: the corpus's 80 terms are rewritten 4000 times, once per segment. There is a fixed cost per segment, and as segment count grows this cost accumulates. The second row gives this lesson's real distinction. Batch size is 1 but refresh interval is 500: request count is still 4000, yet segment count is 8, bytes are 388,600, and lookups are 48. **Batch size determines request count, refresh interval determines segment count.** They are two separate settings and cost two separate things. Bulk indexing reduces the per-request network and parsing overhead; because the in-process model does not carry that overhead, the time ratio printed here reflects only the segment cost — measured together, indexing plus a hundred queries under one-at-a-time placement takes more than twice as long as under bulk placement. #### Refresh Interval: Visibility and Throughput The second table feeds a 600-record donation batch into the 4000-document index in requests of 50 and runs the same query right after every request. The baseline is the response once the whole batch is visible: 1115 matches, with 4457, 4534, 4544 in the top five. At interval 1, no document waits: average invisible is 0.0 and the last query is identical to the baseline. The cost is 600 new segments and 3648 dictionary lookups per query — 76 times the no-refresh case, because every query looks individually at six hundred one-document segments. Interval 75 is this lesson's best-balanced row: 8 segments, an average of 25 documents waiting in the buffer, yet the last query still returns 1115 matches and 10/10. Visibility lag exists without destroying result accuracy; query cost stays at 96 lookups. At interval 250 the lag reaches the returned set. An average of 95.8 documents are invisible and the last query returns 1015 documents: **100 records missing**. The top ten drops to 8/10 and the top five changes — 4534 and 4544 drop off the list, replaced by older records. With refresh off the table reaches its extreme: an average of 325 documents are invisible, the last query answers with 515 documents, **all 600 records are missing**, and not one book from the donation batch appears in the top five. The query is at its cheapest, 48 lookups; the returned list is the list of a catalog the batch never reached. The trade-off does not run in one direction only. Shortening the interval lowers visibility lag but produces segments; lengthening it cheapens the query but pulls the returned set backward. Somewhere in between sits an interval where lag is measurable but does not reach the result — in this corpus, 75. #### Summary - Indexing the same 4000 documents one at a time produced 4000 requests, 4000 segments, 869,327 bytes, and 24,000 dictionary lookups per query; with batches of 500 and a single refresh it produced 8 requests, 1 segment, 382,832 bytes, and 6 lookups. - The byte increase comes not from the posting lists but from the dictionary: 80 terms are repeated across 4000 segments. - Batch size determines request count, refresh interval determines segment count; combining a batch of 1 with an interval of 500 keeps requests at 4000 while segments drop to 8. - Measured together, indexing plus a hundred queries took more than twice as long under one-at-a-time placement as under bulk placement; because raw time depends on the environment, only the threshold was printed. - At a refresh interval of 1, invisible documents were 0 but lookups per query were 3648; at interval 75, an average of 25 documents waited and the result was still 10/10. - At interval 250 the last query missed 100 records and 4534 and 4544 dropped from the top five; with refresh off, all 600 records stayed missing and the top ten fell to 6/10. #### Next Step This lesson treated the segment as a cheap byproduct and only counted how many there were. The tables show this cannot last: one-at-a-time indexing left 4000 segments behind, the interval-1 placement added 600 segments, and every new segment brought one more fixed cost per query. Because segments are immutable, a deleted document also stays in place — it is only marked deleted — and as the index grows, unread postings accumulate. The only way to reclaim this overhead is to **merge** segments. The next lesson measures the background cost of turning small segments into large ones: bytes read and rewritten during a merge, the space deleted documents give back, and how the number of segments per query changes while a merge is running. ### Segment Merging Source: https://academia.sh/en/courses/search-engines/segment-merging The cost every batch's leftover segments charge a query: how segment count affects dictionary lookups, scanned postings, and index size, clearing delete markers, the bytes a merge copies, and comparing two factor settings of the tiered merge policy. Bulk indexing raised write throughput: documents entered the index in batches, not one at a time. That gain has a quiet byproduct. Every batch leaves a new **segment** in the index: a unit with its own dictionary, its own posting lists, and its own document count, never changed again once written. Because a segment is immutable, a deleted document does not move; it only receives a **delete marker** and keeps sitting there. After forty-six batches the index is forty-six separate segments, and every query has to search each one individually. This lesson has two questions. What exactly does segment count charge a query, and how much of that does a background **merge** claw back, at what cost in copied bytes. #### Setting Up the Segment Stack For this measurement, the catalog corpus, the segmented inverted index, and the merge itself are self-written. The corpus is 12,000 book records, seed 20260731; forty batches of 300 documents enter the index, then six correction batches rewrite records already indexed and mark their old copies deleted. **CO10.** Because each segment is searched independently, scoring uses that segment's own document count and document frequency. This is how a real segmented index behaves: a global statistic only emerges once segments merge. **CO11.** A segment's on-disk size is its dictionary and posting lists serialized as text. A real implementation compresses this; the ratio changes, the direction of the measurement does not. **CO12.** A merge's cost is the sum of the input bytes read and the output bytes written. ```js // corpus.mjs — seeded catalog corpus, segmented inverted index, and tiered merging export function rng(seed) { // deterministic pseudo-random generator let a = seed >>> 0; return () => { a = (a + 0x6d2b79f5) >>> 0; let t = a; t = Math.imul(t ^ (t >>> 15), t | 1); t ^= t + Math.imul(t ^ (t >>> 7), t | 61); return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; } const ADJ = 'silent distant broken lost white black thin deep yellow long late final old new lonely'.split(' '); const NOUN = 'door sea road house city garden island notebook river well clock forest bridge tower spring'.split(' '); const TOPIC = ['juvenile fiction', 'short story', 'novel', 'essay', 'poetry', 'history', 'geography', 'philosophy', 'psychology', 'economics', 'architecture', 'music', 'cinema', 'travel', 'biography', 'memoir', 'science fiction', 'mystery', 'folklore', 'linguistics']; const ROOT = ('book author narrative story child city memory journey generation war migration family ' + 'sea winter morning letter photograph train street school library wind silence hope island rain ' + 'forest village mountain river border century woman soldier teacher doctor painter song verse cover edition').split(' '); const FIRST = 'Adam Zoe Matthew Ella Cole Alice Martin Sylvia Kyle Dean Nadia Owen'.split(' '); const LAST = 'Miller Steel Stone Frost Hawke Bright Turner Lyon Fowler Cole'.split(' '); function form(root, i) { // surface form: eight suffix variants const SUF = ['', 's', "'s", 'less', 'like', 'ish', 'ward', 'ful']; return root + SUF[i]; } export function corpusBuild(count, seed) { // library catalog book record const r = rng(seed), list = []; for (let i = 0; i < count; i++) { const summary = []; for (let j = 0; j < 16; j++) // frequent roots occur more often: skewed pick summary.push(form(ROOT[Math.floor(r() ** 2 * ROOT.length)], Math.floor(r() ** 2 * 8))); const topics = [TOPIC[Math.floor(r() * 20)]]; if (r() < 0.45) topics.push(TOPIC[Math.floor(r() * 20)]); list.push({ id: 'k' + String(i).padStart(5, '0'), summary: summary.join(' '), topics, title: `${ADJ[Math.floor(r() * 15)]} ${NOUN[Math.floor(r() * 15)]}`, author: `${FIRST[Math.floor(r() * 12)]} ${LAST[Math.floor(r() * 10)]}`, year: 1990 + Math.floor(r() * 35), shelf: 'pl' + (100 + Math.floor(r() * 900)) + '.' + (1 + Math.floor(r() * 40)), }); } return list; } export const text = (b) => `${b.title} ${b.summary} ${b.topics.join(' ')} ${b.author} ${b.year} ${b.shelf}`; export const tokenize = (s) => // lowercase, split on non-letter/non-number s.toLocaleLowerCase('en').split(/[^\p{L}\p{N}]+/u).filter((t) => t.length > 1); export class Segment { // an immutable segment born from one batch constructor(docs) { this.dict = new Map(); // term -> [[document id, term frequency], ...] this.docs = new Set(); this.deleted = new Set(); for (const b of docs) { this.docs.add(b.id); const count = new Map(); for (const t of tokenize(text(b))) count.set(t, (count.get(t) || 0) + 1); for (const [t, tf] of count) { if (!this.dict.has(t)) this.dict.set(t, []); this.dict.get(t).push([b.id, tf]); } } } serialize() { // the form written to disk: dictionary + posting lists let s = ''; for (const [t, g] of this.dict) s += t + '\t' + g.map(([i, f]) => i + ':' + f).join(' ') + '\n'; for (const id of this.deleted) s += '-\t' + id + '\n'; return s; } get bytes() { return Buffer.byteLength(this.serialize()); } } // search over segments: each segment is searched separately, scores come from the segment's own statistics export function search(segments, terms, k = 10) { const scores = new Map(); let lookups = 0, scanned = 0, dropped = 0; for (const sg of segments) { const N = sg.docs.size; // a deleted document still counts in the segment's statistics for (const t of terms) { lookups++; const g = sg.dict.get(t); if (!g) continue; const idf = Math.log(1 + N / g.length); for (const [id, tf] of g) { scanned++; if (sg.deleted.has(id)) { dropped++; continue; } scores.set(id, (scores.get(id) || 0) + tf * idf); } } } const order = [...scores].sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)); return { top: order.slice(0, k).map(([id]) => id), matched: order.length, lookups, scanned, dropped }; } export function merge(segments) { // drops deleted documents, reduces to one segment const out = new Segment([]); for (const sg of segments) { for (const id of sg.docs) if (!sg.deleted.has(id)) out.docs.add(id); for (const [t, g] of sg.dict) { const kept = g.filter(([id]) => !sg.deleted.has(id)); if (kept.length === 0) continue; if (!out.dict.has(t)) out.dict.set(t, []); for (const p of kept) out.dict.get(t).push(p); } } return out; } // tiered policy: once `factor` segments accumulate in the same tier, they all merge into the tier above export function buildTiers(factor, count = 12000, seed = 20260731) { const corpus = corpusBuild(count, seed); const tier = [], location = new Map(); let copied = 0, merges = 0, totalSegments = 0, batches = 0; const total = () => tier.reduce((n, t) => n + t.length, 0); const place = (b, k) => { while (tier.length <= k) tier.push([]); tier[k].push(b); if (tier[k].length < factor) return; const group = tier[k].splice(0, factor), merged = merge(group); for (const g of group) copied += g.bytes; // read copied += merged.bytes; // written merges++; for (const id of merged.docs) location.set(id, merged); place(merged, k + 1); }; const add = (docs) => { const b = new Segment(docs); for (const id of b.docs) location.set(id, b); place(b, 0); totalSegments += total(); batches++; }; for (let i = 0; i < count; i += 300) add(corpus.slice(i, i + 300)); const r = rng(907); for (let p = 0; p < 6; p++) { // correction batch: a record is reindexed const correction = []; while (correction.length < 300) { const b = corpus[Math.floor(r() * count)], old = location.get(b.id); if (!old || old.deleted.has(b.id)) continue; old.deleted.add(b.id); // the old copy is marked deleted correction.push(b); } add(correction); } return { corpus, segments: tier.flat(), copied, merges, average: totalSegments / batches }; } ``` #### What Segment Count Charges a Query The same stack is searched under three placements: forty-six unmerged segments, reduced to five segments, and a single segment. Five catalog questions run; each placement counts dictionary lookups, scanned postings, postings dropped for a delete marker, and matched documents. The top ten result is then compared against the single-segment placement. ```js // segments.mjs — the effect of segment count on query cost, the set, and the order import { search, merge, buildTiers } from './corpus.mjs'; const QUERIES = [ ['juvenile', 'fiction', 'short', 'story'], ['lost', 'door'], ['library', 'memory', 'journey'], ['sea', 'island', 'rain'], ['mystery', 'city'], ]; const { segments } = buildTiers(Infinity); // Infinity: no merging happens const group = (sg, n) => { // reduces consecutive segments to n groups const step = Math.ceil(sg.length / n), c = []; for (let i = 0; i < sg.length; i += step) c.push(merge(sg.slice(i, i + step))); return c; }; const placements = [['no merging', segments], ['five segments', group(segments, 5)], ['one segment', group(segments, 1)]]; console.log('corpus: 12000 documents, seed 20260731 | 300-document x 40 batches + 6 correction batches'); console.log('placement segments dict entries index bytes lookups scanned dropped matched'); const topResults = new Map(); for (const [name, sg] of placements) { let dict = 0, bytes = 0, lk = 0, sc = 0, dr = 0, ma = 0; for (const s of sg) { dict += s.dict.size; bytes += s.bytes; } const top = []; for (const q of QUERIES) { const c = search(sg, q); lk += c.lookups; sc += c.scanned; dr += c.dropped; ma += c.matched; top.push(c.top); } topResults.set(name, top); console.log(name.padEnd(15), String(sg.length).padStart(5), String(dict).padStart(13), String(bytes).padStart(11), String(lk).padStart(15), String(sc).padStart(8), String(dr).padStart(7), String(ma).padStart(8)); } console.log('\ntop ten result, relative to the one-segment placement (5 queries, 50 ranks)'); console.log('placement dropped from top 10 rank shifted queries with a different first result'); const reference = topResults.get('one segment'); for (const [name, top] of topResults) { let dropped = 0, shifted = 0, first = 0; for (let q = 0; q < QUERIES.length; q++) { const t = reference[q], y = top[q]; for (const id of t) if (!y.includes(id)) dropped++; for (let i = 0; i < t.length; i++) if (y[i] !== t[i]) shifted++; if (y[0] !== t[0]) first++; } console.log(name.padEnd(15), String(dropped).padStart(20), String(shifted).padStart(13), String(first).padStart(37)); } ``` ``` corpus: 12000 documents, seed 20260731 | 300-document x 40 batches + 6 correction batches placement segments dict entries index bytes lookups scanned dropped matched no merging 46 31075 3111596 644 28767 3768 20637 five segments 5 6238 2532371 70 24999 0 20637 one segment 1 1320 2498875 14 24999 0 20637 top ten result, relative to the one-segment placement (5 queries, 50 ranks) placement dropped from top 10 rank shifted queries with a different first result no merging 15 33 1 five segments 12 29 1 one segment 0 0 0 ``` The first column is the source of both numbers. The corpus has 1,320 distinct terms; spread across forty-six segments, the dictionary holds 31,075 entries. The same term is rewritten in every segment, because each segment must be searchable on its own. This is the main reason index size climbs from 2,498,875 bytes to 3,111,596 bytes: roughly a quarter more space for the same data. There are two separate costs on the query side. Dictionary lookups grow linearly with segment count: the fourteen-term query set does 14 lookups at one segment and 644 at forty-six. This is a fixed cost paid per lookup, and it is paid even when the posting list comes back empty. Scanned postings, in turn, rise from 24,999 to 28,767; the 3,768-entry gap is records whose old, delete-marked copies still sit in the posting lists. They are scanned, dropped before scoring, and still force the scan to do its work. The last column shows the most important side of the decision: matched documents come out at **20,637 in all three placements**. Segment count does not change the returned set; because delete-marked copies are filtered out, the answer is the same answer. What changes is the order. In the forty-six-segment placement, 33 of the top ten result's fifty ranks carry a document different from the single-segment placement, and 15 documents drop out of the top ten entirely. The reason is CO10: each segment computes inverse document frequency by looking only at its own 300 documents, so the same term looks rare in one segment and ordinary in another. Merging down to five segments narrows this distortion — 12 documents drop out instead of 15, and 29 ranks shift instead of 33 — but does not remove it. A global statistic only emerges once every segment collapses into one; merging does not correct the order gradually, it corrects it once it crosses that threshold. #### Two Settings of the Merge Policy The merge decision is not made by hand; a policy runs continuously. The **tiered policy** places segments into tiers by size: once `factor` segments accumulate in the same tier, all of them merge into a single segment, and the result lands in the tier above. Factor is the only setting. The same forty-six batches are replayed under two factors. ```js // policy.mjs — two settings of the tiered merge policy: factor 3 and factor 10 import { search, buildTiers } from './corpus.mjs'; const QUERIES = [ ['juvenile', 'fiction', 'short', 'story'], ['lost', 'door'], ['library', 'memory', 'journey'], ['sea', 'island', 'rain'], ['mystery', 'city'], ]; console.log('same 46 batches, same seed: 12000 documents, seed 20260731'); console.log('factor merges bytes copied avg. segments final segments index bytes lookups scanned'); for (const factor of [3, 10]) { const y = buildTiers(factor); let bytes = 0, lk = 0, sc = 0; for (const b of y.segments) bytes += b.bytes; for (const q of QUERIES) { const c = search(y.segments, q); lk += c.lookups; sc += c.scanned; } console.log(String(factor).padStart(6), String(y.merges).padStart(7), String(y.copied).padStart(13), y.average.toFixed(1).padStart(15), String(y.segments.length).padStart(15), String(bytes).padStart(12), String(lk).padStart(9), String(sc).padStart(8)); } ``` ``` same 46 batches, same seed: 12000 documents, seed 20260731 factor merges bytes copied avg. segments final segments index bytes lookups scanned 3 21 15009319 3.3 4 2860909 56 28323 10 4 5216558 6.3 10 2944442 140 28767 ``` The difference between the two settings is measurable on the query side but small: dictionary lookups fall from 140 to 56, scanned postings from 28,767 to 28,323. Index size drops from 2,944,442 bytes to 2,860,909 bytes. The difference paid in the background, though, is large: factor 3 runs twenty-one merges and copies 15,009,319 bytes, factor 10 runs four merges and copies 5,216,558 bytes. Turning the measurement into a ratio sharpens the contrast. Factor 3 reads and writes 15.0 MB to end up with an index that holds 2.9 MB: **write amplification of 5.25x**. For factor 10 the same ratio is 1.77x. This is the real trade-off of merging. The write throughput bulk indexing gains can be clawed back by the merge running in the background; while bytes written per batch stay low, bytes copied can climb to several times that. Incoming batches and merging draw on the same disk and the same channel. Lowering the factor speeds up the query a little, shrinks the index a little, and triples the background load; the query cost's real drop happened earlier, in the first measurement's fall from 644 to 140 — between **no merging at all** and **merging at all**. #### Summary - Every batch leaves one segment; because a segment is immutable, a deleted document stays in place and only receives a delete marker. - At forty-six segments the dictionary holds 31,075 entries, at one segment 1,320; for the same data, index size falls from 3,111,596 bytes to 2,498,875 bytes. - Segment count does not change the set (20,637 matched documents in all three placements); it changes the order: 33 of the top ten result's fifty ranks differ from the single-segment placement, because each segment computes inverse document frequency from its own 300 documents. - Merging removes 3,768 delete-marked entries from the scan and drops dictionary lookups from 644 to 14. - In the tiered policy, factor 3 drops dictionary lookups to 56 from factor 10's 140, but raises copied bytes from 5.2 MB to 15.0 MB: write amplification rises from 1.77x to 5.25x. #### Next Step Merging controls segment count, but it does not control the index's **growth**. The 12,000 records in this measurement are a fixed catalog; in an index with a constant stream of new records, each merge recopies an ever-larger segment, and the cost of deleting old data ties back to that same copying. The next lesson stops treating the index as a single, endlessly growing object: opening a new index at a fixed threshold — a **rollover** — changes how many indexes a query must reach and how many bytes deleting old data costs. What gets measured is rollover's effect on query scope over time, the cost gap between deleting documents in one large index and dropping an entire index, and the memory a hot–warm–cold tier placement holds. ### Index Lifecycle Source: https://academia.sh/en/courses/search-engines/index-lifecycle The measured effect of rolling an endlessly growing index over by time: the rollover interval's effect on query scope and scanned postings, the cost gap between deleting documents in one large index and dropping an index, the data rollover granularity keeps outside the retention window, and the memory a hot, warm, and cold tier placement holds. Merging controls segment count, not the index's growth. The single segment left after forty-six batches becomes a four-hundred-batch segment after four hundred batches, and every merge recopies the whole thing. In a continuously flowing stream, the real question is why the index has to be a single object at all. This lesson measures the effect of splitting the index by time. Opening a new index at a fixed interval is called **rollover**; old indexes stay in place, queries reach them when needed, and once the retention period expires an index is dropped whole. Three things are counted: how many indexes a query has to reach, how many bytes deleting old data costs, and what a tiered placement holds in memory. #### Timestamped Index and Rollover **CO13.** A lifecycle decision only makes sense on timestamped data, so this lesson indexes not the catalog records themselves but the **access log** kept on top of the catalog. Each record carries an access made on a given day, at a given branch, with given terms, and that record's view count. Query order follows this **metric**, not a relevance score: the most-viewed record comes first. **CO16.** Posting lists are not separately indexed by time; applying a day window on a single index requires scanning every entry. A real implementation could keep a minimum and maximum day at the segment level and skip within the index too — that would be a finer-grained version of the skipping measured below. The corpus is 400 records a day for 180 days: 72,000 documents, seed 20260731. ```js // log.mjs — timestamped catalog access log and a rollover-capable index export function rng(seed) { // deterministic pseudo-random generator let a = seed >>> 0; return () => { a = (a + 0x6d2b79f5) >>> 0; let t = a; t = Math.imul(t ^ (t >>> 15), t | 1); t ^= t + Math.imul(t ^ (t >>> 7), t | 61); return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; } const TOPIC = ('literature short-story novel essay poetry history geography philosophy psychology ' + 'economics architecture music cinema travel biography memoir fiction mystery folklore linguistics').split(' '); const ADJ = 'silent distant broken lost white black thin deep yellow long'.split(' '); const NOUN = 'door sea road house city garden island notebook river well'.split(' '); const BRANCH = 'central kadikoy bornova cankaya nilufer'.split(' '); // each record is one catalog access: which day, which branch, which terms, how many views export function logGenerate(days, perDay, seed) { const r = rng(seed), records = []; for (let d = 0; d < days; d++) for (let i = 0; i < perDay; i++) { const terms = [TOPIC[Math.floor(r() * 20)], ADJ[Math.floor(r() * 10)], NOUN[Math.floor(r() * 10)], BRANCH[Math.floor(r() * 5)]]; if (r() < 0.4) terms.push(TOPIC[Math.floor(r() * 20)]); records.push({ id: 'e' + String(records.length).padStart(6, '0'), day: d, terms, views: Math.ceil(2 / Math.sqrt(r())) }); // CO14: long-tailed access counts } return records; } export class Index { constructor(name, firstDay) { this.name = name; this.firstDay = firstDay; this.lastDay = firstDay; this.dict = new Map(); this.docs = new Map(); this.deleted = new Set(); } add(k) { this.lastDay = k.day; this.docs.set(k.id, k); for (const t of new Set(k.terms)) { if (!this.dict.has(t)) this.dict.set(t, []); this.dict.get(t).push([k.id, k.views, k.day]); // posting: id, metric, day } } get entries() { let n = 0; for (const g of this.dict.values()) n += g.length; return n; } get bytes() { let s = ''; for (const [t, g] of this.dict) s += t + '\t' + g.map((p) => p.join(':')).join(' ') + '\n'; for (const id of this.deleted) s += '-\t' + id + '\n'; return Buffer.byteLength(s); } residentBytes(tier) { // CO15: dictionary entry is term bytes + 32, if (tier === 'cold') return 128; // a posting entry is 16 bytes; a closed index is 128 bytes let b = 0; for (const t of this.dict.keys()) b += Buffer.byteLength(t) + 32; return tier === 'warm' ? b : b + this.entries * 16; } } export function rollover(records, dayStep) { // dayStep days full, a new index opens const indexes = []; for (const k of records) { const no = Math.floor(k.day / dayStep); if (!indexes[no]) indexes[no] = new Index('daily-' + no, k.day); indexes[no].add(k); } return indexes; } // search: an index that does not overlap the day window is never opened; order is by access count export function search(indexes, terms, window = null, k = 10) { let touched = 0, lookups = 0, scanned = 0; const candidates = new Map(); for (const d of indexes) { if (window && (d.lastDay < window[0] || d.firstDay > window[1])) continue; touched++; for (const t of terms) { lookups++; const g = d.dict.get(t); if (!g) continue; for (const [id, views, day] of g) { scanned++; if (window && (day < window[0] || day > window[1])) continue; if (d.deleted.has(id)) continue; candidates.set(id, [views, day]); } } } const order = [...candidates].sort((a, b) => b[1][0] - a[1][0] || b[1][1] - a[1][1] || (a[0] < b[0] ? -1 : 1)); return { touched, lookups, scanned, matched: order.length, top: order.slice(0, k).map(([id, v]) => [id, v[1]]) }; } ``` #### Rollover's Effect on Query Scope The same 72,000 records are placed under three placements: a single index, 30-day rollover (6 indexes), and 7-day rollover (26 indexes). Under each placement the same query runs twice — once restricted to the last fourteen days, once with no window. ```js // rollover.mjs — rollover interval's effect on query scope import { logGenerate, rollover, search } from './log.mjs'; const RECORDS = logGenerate(180, 400, 20260731); const QUERY = ['mystery', 'kadikoy']; console.log('access log: 180 days x 400 records = 72000 records, seed 20260731 | query: mystery kadikoy'); console.log('placement indexes query touched lookups scanned matched'); for (const [name, step] of [['single index', 180], ['30-day', 30], ['7-day', 7]]) { const d = rollover(RECORDS, step); for (const [label, window] of [['last 14 days', [166, 179]], ['no window', null]]) { const c = search(d, QUERY, window); console.log(name.padEnd(12), String(d.length).padStart(7), '', label.padEnd(12), String(c.touched).padStart(7), String(c.lookups).padStart(9), String(c.scanned).padStart(8), String(c.matched).padStart(8)); } } ``` ``` access log: 180 days x 400 records = 72000 records, seed 20260731 | query: mystery kadikoy placement indexes query touched lookups scanned matched single index 1 last 14 days 1 2 19476 1406 single index 1 no window 1 2 19476 18456 30-day 6 last 14 days 1 2 3188 1406 30-day 6 no window 6 12 19476 18456 7-day 26 last 14 days 3 6 1993 1406 7-day 26 no window 26 52 19476 18456 ``` The windowed query returns 1,406 records under all three placements: the set does not change. What changes is the work done to reach that set. In the single index, 19,476 postings are scanned and 93 percent of them are eliminated by the day filter. Under 7-day rollover, the query opens only 3 of 26 indexes and scans 1,993 entries — a tenth as many. The gain does not come from scanning faster; it comes from an index that is **never opened at all**: a non-overlapping index's dictionary is not read, its posting lists are not loaded into memory. The same table's second rows show the cost. For the unwindowed query, scanned entries stay at 19,476 across all three placements, but dictionary lookups rise from 2 to 52. Rollover adds a fixed cost equal to the index count to every query that carries no time window: each index searches its terms separately, and each index's partial result must be merged separately, too. The rollover interval is a choice between these two query shapes: as windowed queries grow more frequent, a short interval wins; as queries that do not fit the window dominate, a long interval wins. #### Two Ways to Delete Old Data The retention window is the last 100 days (day 80–179). In a single index, the only way to apply this is to mark every document before day 80 as deleted and clean the index. Under rollover, old indexes are dropped whole — but only the ones **entirely** outside the window. ```js // retention.mjs — two ways to apply a 100-day retention window, and tier placement import { logGenerate, rollover, search } from './log.mjs'; const RECORDS = logGenerate(180, 400, 20260731); const THRESHOLD = 80; // last 100 days are kept: 80..179 const QUERY = ['mystery', 'kadikoy']; function clean(step) { const dz = rollover(RECORDS, step); if (step === 180) { // single index: every old document is marked deleted const d = dz[0]; for (const k of RECORDS) if (k.day < THRESHOLD) d.deleted.add(k.id); const read = d.bytes; const fresh = rollover(RECORDS.filter((k) => k.day >= THRESHOLD), 180); return { remaining: fresh, dropped: 0, marked: d.deleted.size, copied: read + fresh[0].bytes, extra: 0 }; } const remaining = dz.filter((d) => d.lastDay >= THRESHOLD); // an index entirely old is dropped return { remaining, dropped: dz.length - remaining.length, marked: 0, copied: 0, extra: remaining.reduce((n, d) => n + [...d.docs.values()].filter((k) => k.day < THRESHOLD).length, 0) }; } console.log(`retention window: last 100 days (day 80-179), 180 days x 400 records, seed 20260731`); console.log('placement dropped marked copied bytes extra retained matched top 10 outside window'); const placement = new Map(); for (const [name, step] of [['single index', 180], ['30-day', 30], ['7-day', 7]]) { const t = clean(step); placement.set(name, t.remaining); const c = search(t.remaining, QUERY, null); console.log(name.padEnd(12), String(t.dropped).padStart(7), String(t.marked).padStart(7), String(t.copied).padStart(13), String(t.extra).padStart(15), String(c.matched).padStart(8), String(c.top.filter(([, g]) => g < THRESHOLD).length).padStart(22)); } const remaining = placement.get('7-day'); // tier placement over the 7-day rollover const age = (d) => 179 - d.lastDay; console.log('\ntier placement (7-day rollover, after cleanup):'); console.log('placement hot warm cold resident bytes bytes to reopen'); for (const [name, tierOf] of [['all hot', () => 'hot'], ['three tiers', (d) => (age(d) < 7 ? 'hot' : age(d) < 30 ? 'warm' : 'cold')]]) { const count = { hot: 0, warm: 0, cold: 0 }; let bytes = 0, toReopen = 0; for (const d of remaining) { const t = tierOf(d); count[t]++; bytes += d.residentBytes(t); if (t === 'cold') toReopen += d.bytes; } console.log(name.padEnd(12), String(count.hot).padStart(4), String(count.warm).padStart(5), String(count.cold).padStart(5), String(bytes).padStart(15), String(toReopen).padStart(16)); } ``` ``` retention window: last 100 days (day 80-179), 180 days x 400 records, seed 20260731 placement dropped marked copied bytes extra retained matched top 10 outside window single index 0 32000 6985651 0 10262 0 30-day 2 0 0 8000 12274 2 7-day 11 0 0 1200 10545 0 tier placement (7-day rollover, after cleanup): placement hot warm cold resident bytes bytes to reopen all hot 15 0 0 2914122 0 three tiers 2 3 10 346942 1686139 ``` The cost gap shows up in a single row. In the single index, deletion marks 32,000 documents, and cleanup reads and writes 6,985,651 bytes; the only way to get rid of deleted data is to rewrite the entirety of what remains. Under rollover, the same deletion copies **zero bytes**: the index is dropped, the file goes whole. This is the direct sibling of the previous lesson's merge cost — there, copied bytes were paid per segment; here, they are paid per deletion. Rollover's cost is granularity. In the 30-day interval, the index spanning days 60–89 has part of its range inside the window, so it cannot be dropped, and 8,000 records keep being retained outside the retention window; at the 7-day interval this number falls to 1,200. The gap measurably leaks into the answer. The same query returns 10,262 records in the single index, 10,545 under 7-day rollover, and 12,274 under 30-day rollover: the set has grown by 20 percent. The order changes too: under 30-day rollover, two of the top ten results come from outside the retention window, from days that should have been deleted. The expected magnitude confirms this — roughly a sixth of the matched set is outside the window, and two records in the top ten is consistent with that ratio. The sentence "we retain a hundred days of data" means, once the rollover interval is thirty days, retaining data for up to 120 days. #### Hot, Warm, and Cold Tiers The remaining fifteen indexes do not get equal attention: the past week's index is queried constantly, an index from three months back only a few times a month. A tiered placement turns this into a resource decision. In the **hot** tier, the dictionary and posting lists sit in memory; in the **warm** tier, only the dictionary is kept, and posting lists are read from disk on every query; in the **cold** tier, the index is closed and all that remains is metadata carrying the day range. The second table gives the cost of this. Under the all-hot placement, fifteen indexes hold 2,914,122 bytes of resident memory. Under the three-tier placement, two indexes stay hot, three warm, ten cold, and resident memory falls to 346,942 bytes: an eighth as much. In return, a query reaching the ten cold-tier indexes has to open 1,686,139 bytes from disk. The tier decision sits between these two numbers, and it depends directly on the query shape from the previous section: windowed queries never open the cold indexes, an unwindowed query opens all of them every time. #### Summary - Rollover does not change the set; the windowed query returned 1,406 records under all three placements, but scanned postings fell from 19,476 to 1,993 because a non-overlapping index was never opened. - Rollover's fixed cost falls on the unwindowed query: dictionary lookups rose from 2 to 52, because each index searches its terms separately. - Deleting 32,000 old documents in the single index copied 6,985,651 bytes; under rollover, the same deletion cost zero bytes by dropping the index. - The rollover interval is retention granularity: the 30-day interval kept 8,000 records outside the window, the set grew from 10,262 to 12,274, and two of the top ten results came from days that should have been deleted; at the 7-day interval, extra retained records fell to 1,200. - The three-tier placement lowered resident memory from 2,914,122 bytes to 346,942 bytes; the cost is the 1,686,139 bytes a query reaching the cold tier has to open. #### Next Step Rollover, just as it cheapens dropping an index, also changes another task. Segments and completed indexes are never written again; what needs backing up is not a single, constantly changing file but a set of immutable files. The next lesson takes this up: a snapshot of an index can run **incrementally** at the segment level, but a merge running in the background breaks that incrementality. The same lesson compares restoring from a snapshot against reindexing from the source; what gets measured is bytes copied per snapshot, the bytes a restore moves against the documents a reindex processes, and how reindexing shifts the set and the order once the analyzer changes. ### Snapshot and Restore Source: https://academia.sh/en/courses/search-engines/snapshot-and-restore A backup decision specific to the index: the bytes an incremental snapshot built on segment immutability copies, the background merge breaking that incrementality, the store holding more than the live index, and comparing restoring from a snapshot against reindexing from the source by copied bytes, documents processed, the set, and the order. Rollover reduced getting rid of old data to dropping a file. The same structure changes backups too: segments that never change once written turn what needs backing up from "a state constantly in flux" into a **set of immutable files**. An in-memory store's snapshot was measured earlier: the relationship between the interval and the loss window, the cost taking a snapshot places on the running store. That measurement is not repeated here. The question here is specific to the index. If segments are immutable, a snapshot can be **incremental**; what does a background merge do to that, and when the index has to come back, is moving it from the store cheaper, or reindexing from the source. #### Segment-Level Snapshot **CO18.** A segment file is immutable along with its name: a file with a given name in the store always carries the same content. An incremental snapshot is built on this — if that name already exists in the store, the file is not copied again. **CO19.** The snapshot's cost is copied bytes; what the store holds is the sum of its distinct segment files. **CO20.** Reindexing assumes the source records are on hand, and that the same daily batches and the same merge policy are applied. Time depends on the run environment and so is not printed; instead, two run-independent quantities are counted: copied bytes, and documents and tokens processed. The corpus is 8,000 book records, seed 20260731. ```js // backup.mjs — catalog corpus, analyzers, segmented index, and segment-level snapshot store import { writeFileSync, existsSync, mkdirSync, statSync } from 'node:fs'; export function rng(seed) { // deterministic pseudo-random generator let a = seed >>> 0; return () => { a = (a + 0x6d2b79f5) >>> 0; let t = a; t = Math.imul(t ^ (t >>> 15), t | 1); t ^= t + Math.imul(t ^ (t >>> 7), t | 61); return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; } const ADJ = 'silent distant broken lost white black thin deep yellow long'.split(' '); const NOUN = 'door sea road house city garden island notebook river well'.split(' '); const TOPIC = ('juvenile fiction|short story|novel|essay|poetry|history|geography|philosophy|psychology|' + 'economics|architecture|music|cinema|travel|biography|memoir|science fiction|mystery|folklore').split('|'); const ROOT = ('book author narrative story child city memory journey generation war migration family ' + 'sea winter morning letter photograph train street school library wind silence hope island rain ' + 'forest village mountain river border century woman soldier teacher doctor painter song verse cover').split(' '); const FILLER = 'and with a or'.split(' '); function form(root, i) { // surface form: eight suffix variants const SUF = ['', 's', "'s", 'less', 'like', 'ish', 'ward', 'ful']; return root + SUF[i]; } export function corpusBuild(count, seed) { const r = rng(seed), list = []; for (let i = 0; i < count; i++) { const summary = []; for (let j = 0; j < 16; j++) summary.push(j % 4 === 3 ? FILLER[Math.floor(r() * 4)] : form(ROOT[Math.floor(r() ** 2 * ROOT.length)], Math.floor(r() ** 2 * 8))); list.push({ id: 'k' + String(i).padStart(5, '0'), title: `${ADJ[Math.floor(r() * 10)]} ${NOUN[Math.floor(r() * 10)]}`, summary: summary.join(' '), topic: TOPIC[Math.floor(r() * 19)] }); } return list; } export const text = (b) => `${b.title} ${b.summary} ${b.topic}`; const STOP = new Set(FILLER); const SUFFIX = ['less', 'like', 'ward', 'ish', 'ful', "'s", 's']; // longest suffix first const stem = (t) => { // strips the longest matching known ending for (const e of SUFFIX) if (t.length > e.length + 2 && t.endsWith(e)) return t.slice(0, -e.length); return t; }; export const basic = (s) => // lowercase + split on non-letter/non-number s.toLocaleLowerCase('en').split(/[^\p{L}\p{N}]+/u).filter((t) => t.length > 1); export const advanced = (s) => // + stop-word removal + stemming basic(s).filter((t) => !STOP.has(t)).map(stem); let counter = 0; export class Segment { // an immutable segment; its name never changes once written constructor(docs = [], analyze = basic) { this.name = 'b' + String(++counter).padStart(3, '0'); this.dict = new Map(); this.docs = new Set(); for (const b of docs) { this.docs.add(b.id); const count = new Map(); for (const t of analyze(text(b))) count.set(t, (count.get(t) || 0) + 1); for (const [t, tf] of count) { if (!this.dict.has(t)) this.dict.set(t, []); this.dict.get(t).push([b.id, tf]); } } } serialize() { let s = ''; for (const [t, g] of this.dict) s += t + '\t' + g.map(([i, f]) => i + ':' + f).join(' ') + '\n'; return s; } get bytes() { return Buffer.byteLength(this.serialize()); } } export function tiers(factor = 4) { // tiered merge heap const tier = []; let merges = 0; const place = (b, k) => { while (tier.length <= k) tier.push([]); tier[k].push(b); if (tier[k].length < factor) return; merges++; place(merge(tier[k].splice(0, factor)), k + 1); }; return { add(docs, analyze) { merges = 0; place(new Segment(docs, analyze), 0); }, get segments() { return tier.flat(); }, get merges() { return merges; }, }; } export function merge(group) { const y = new Segment(); for (const sg of group) { for (const id of sg.docs) y.docs.add(id); for (const [t, g] of sg.dict) { if (!y.dict.has(t)) y.dict.set(t, []); for (const p of g) y.dict.get(t).push(p); } } return y; } export function search(segments, terms, k = 10) { const scores = new Map(); for (const sg of segments) { const N = sg.docs.size; for (const t of terms) { const g = sg.dict.get(t); if (!g) continue; const idf = Math.log(1 + N / g.length); for (const [id, tf] of g) scores.set(id, (scores.get(id) || 0) + tf * idf); } } const order = [...scores].sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)); return { top: order.slice(0, k).map(([id]) => id), matched: order.length }; } // snapshot: a segment file not yet in the store is written; a non-incremental snapshot rewrites all of them export function snapshot(segments, store, incremental) { mkdirSync(store, { recursive: true }); let copied = 0, files = 0; for (const b of segments) { const path = `${store}/${b.name}`; if (incremental && existsSync(path)) continue; writeFileSync(path, b.serialize()); copied += statSync(path).size; files++; } return { copied, files, list: segments.map((b) => b.name) }; } ``` #### Incremental Snapshots and What Merging Breaks The catalog takes in 500 records a day for sixteen days, and a snapshot is taken at the end of each day. Each day is measured twice: a full snapshot that copies the whole index, and an incremental snapshot that copies only the segment files not yet in the store. ```js // snapshot.mjs — bytes copied by full and incremental snapshots over a sixteen-day index import { corpusBuild, tiers, snapshot } from './backup.mjs'; const corpus = corpusBuild(8000, 20260731); const y = tiers(4); console.log('corpus: 8000 documents, seed 20260731 | one 500-document batch a day, factor 4'); console.log('day segments merges full snapshot bytes incremental bytes incr. files store total'); let fullTotal = 0, storeBytes = 0; for (let day = 1; day <= 16; day++) { y.add(corpus.slice((day - 1) * 500, day * 500)); const full = snapshot(y.segments, `full/d${day}`, false); const incr = snapshot(y.segments, 'store', true); fullTotal += full.copied; storeBytes += incr.copied; console.log(String(day).padStart(3), String(y.segments.length).padStart(8), String(y.merges).padStart(8), String(full.copied).padStart(19), String(incr.copied).padStart(18), String(incr.files).padStart(12), String(storeBytes).padStart(12)); } console.log('total: full snapshots', fullTotal, 'bytes, incremental snapshots', storeBytes, 'bytes'); ``` ``` corpus: 8000 documents, seed 20260731 | one 500-document batch a day, factor 4 day segments merges full snapshot bytes incremental bytes incr. files store total 1 1 0 76788 76788 1 76788 2 2 0 153495 76707 1 153495 3 3 0 230031 76536 1 230031 4 1 1 297765 297765 1 527796 5 2 0 374913 77148 1 604944 6 3 0 451624 76711 1 681655 7 4 0 528250 76626 1 758281 8 2 1 596322 298557 1 1056838 9 3 0 673056 76734 1 1133572 10 4 0 749772 76716 1 1210288 11 5 0 826992 77220 1 1287508 12 3 1 894582 298260 1 1585768 13 4 0 971235 76653 1 1662421 14 5 0 1048455 77220 1 1739641 15 6 0 1125135 76680 1 1816321 16 1 2 1184013 1184013 1 3000334 total: full snapshots 10182428 bytes, incremental snapshots 3000334 bytes ``` On days without a merge, the incremental snapshot pays a fixed cost: roughly 76,800 bytes, the single segment added that day. The full snapshot, by contrast, copies the whole index every day, and by day fifteen a single backup reaches 1,125,135 bytes. Across the sixteen days the gap is more than threefold: 10,182,428 bytes against 3,000,334 bytes. The critical rows are days 4, 8, 12, and 16. On a day when a merge runs, the incremental snapshot copies not the segment added that day but the whole segment the merge produced — about 298,000 bytes instead of 76,800 — because the segment a merge produces is a **new file**: even though its content already sits entirely in the store, it is the file itself that gets copied, not its inputs. When a single segment is left, this amounts to the same work as the full snapshot; on day four both are 297,765 bytes. On days eight and twelve more than one segment remains, so the incremental snapshot is half and a third of the full one. Day sixteen is the extreme case — two merges reduce the index to a single segment, and the incremental snapshot rewrites all 1,184,013 bytes. Incrementality is a gain that comes from segment immutability, and merging resets it at regular intervals precisely because it uses that same immutability to produce a new file. The last column gives a third result. The store holds 3,000,334 bytes, while the live index is 1,184,013 bytes. The store holds about two and a half times the live index; the excess is segment files invalidated by merging but not deletable, because old snapshots still reference them. That space does not come back until the old snapshots are dropped. Lowering the merge factor — a small win on the query side in the previous lesson — directly grows both incremental snapshot traffic and store size here. #### Restore or Reindex When an index is lost, there are two paths. The files from the store's last snapshot can be moved into place, or the 8,000 records at the catalog source can be reindexed. The second, at first glance, makes a backup unnecessary at all. The measurement separates what each path pays and what it **delivers**: in the third row, reindexing runs on a system whose analyzer chain has since changed. ```js // recovery.mjs — comparing restore from snapshot against reindexing from source import { mkdirSync, readFileSync, writeFileSync, statSync } from 'node:fs'; import { corpusBuild, tiers, snapshot, search, text, basic, advanced } from './backup.mjs'; const corpus = corpusBuild(8000, 20260731); const QUERIES = [['juvenile', 'fiction', 'story'], ['lost', 'doors'], ['library', 'memory'], ['seaward', 'island'], ['mystery', 'city']]; function build(analyze) { // same daily batches, same merge policy const y = tiers(4); let tokens = 0; for (let d = 0; d < 16; d++) { const batch = corpus.slice(d * 500, (d + 1) * 500); for (const b of batch) tokens += analyze(text(b)).length; y.add(batch, analyze); } return { segments: y.segments, tokens }; } const original = build(basic); // original index and the content of its last snapshot const snap = snapshot(original.segments, 'store', true); mkdirSync('index', { recursive: true }); // restore: files from the store move into place let restoredBytes = 0; for (const name of snap.list) { writeFileSync(`index/${name}`, readFileSync(`store/${name}`)); restoredBytes += statSync(`index/${name}`).size; } const measure = (segments, analyze) => { let bytes = 0, dict = 0, matched = 0; const top = []; for (const b of segments) { bytes += b.bytes; dict += b.dict.size; } for (const q of QUERIES) { const c = search(segments, analyze(q.join(' '))); matched += c.matched; top.push(c.top); } return { bytes, dict, matched, top }; }; const originalMeasure = measure(original.segments, basic); const diff = (o) => { // top ten change against the original index let dropped = 0, shifted = 0; for (let q = 0; q < QUERIES.length; q++) { for (const id of originalMeasure.top[q]) if (!o.top[q].includes(id)) dropped++; for (let i = 0; i < 10; i++) if (o.top[q][i] !== originalMeasure.top[q][i]) shifted++; } return [dropped, shifted]; }; const reindexed = build(advanced); const reindexedMeasure = measure(reindexed.segments, advanced); console.log('corpus: 8000 documents, seed 20260731 | 5 queries, top ten: 50 ranks'); console.log('path copied docs processed tokens index bytes dict matched dropped shifted'); const row = (name, copied, docs, tokens, o) => console.log(name.padEnd(32), String(copied).padStart(9), String(docs).padStart(14), String(tokens).padStart(9), String(o.bytes).padStart(11), String(o.dict).padStart(6), String(o.matched).padStart(8), String(diff(o)[0]).padStart(6), String(diff(o)[1]).padStart(8)); row('restore', restoredBytes, 0, 0, originalMeasure); row('reindex (same analyzer)', 0, 8000, original.tokens, originalMeasure); row('reindex (new analyzer)', 0, 8000, reindexed.tokens, reindexedMeasure); ``` ``` corpus: 8000 documents, seed 20260731 | 5 queries, top ten: 50 ranks path copied docs processed tokens index bytes dict matched dropped shifted restore 1184013 0 0 1184013 319 9286 0 0 reindex (same analyzer) 0 8000 145239 1184013 319 9286 0 0 reindex (new analyzer) 0 8000 121252 913738 76 16517 40 50 ``` The first two rows produce the identical index: 1,184,013 bytes, 319 terms, 9,286 matched documents across five queries, and not one rank different in the top ten. What the two paths to that same result pay, though, is entirely different. Restoring moves 1,184,013 bytes and touches no document; reindexing moves not one byte but reads 8,000 documents and produces 145,239 tokens. Time depends on the run environment and is not printed here; the decision is made by how these two quantities scale. Restoring's cost grows with index size, reindexing's with source document count and tokens per document. Reindexing also carries a silent condition: the source records have to still be complete. If the index also holds a field no longer present in the catalog, reindexing cannot bring it back. The third row shows why reindexing does not stand in for a backup. Once the analyzer chain is extended with stop-word removal and stemming, the same 8,000 records produce a different index: 121,252 tokens instead of 145,239 (the 23,987-token gap is the discarded filler words), 76 terms instead of 319, 913,738 bytes instead of 1,184,013. The effect shows up in the response. The five queries' total matched documents rises from 9,286 to **16,517**, because "doors" now collapses onto the same term as "door" and inflected forms pool into one posting list — recall goes up. The order is rebuilt too: all fifty ranks across the top ten results change, and 40 documents drop out of the top ten entirely. This is not a malfunction; it is the decision itself, but it cannot be counted as the same operation as restoring. Restoring brings the index back **as it was**; reindexing **rebuilds** it under that day's configuration. #### Summary - Because segments are immutable, a snapshot can be taken incrementally: on days without a merge, copied bytes are that day's single segment (roughly 76,800), while a full snapshot copies the whole index. - Across the sixteen days, full snapshots copied 10,182,428 bytes, incremental snapshots 3,000,334 bytes. - Merging breaks incrementality: on a merge day the incremental snapshot copies not the segment added that day but the whole merged segment (about 298,000 bytes instead of 76,800), and it equals the full snapshot once a single segment is left; on day sixteen all 1,184,013 bytes were recopied. - The store holds 3,000,334 bytes, the live index 1,184,013 bytes: the gap is invalidated segment files that cannot be deleted because old snapshots still reference them. - Restoring moves 1,184,013 bytes and processes zero documents; reindexing moves zero bytes and processes 8,000 documents and 145,239 tokens; both produce the identical index. - If the analyzer chain has changed, reindexing does not produce the same index: matched documents rose from 9,286 to 16,517, all fifty ranks of the top ten results changed, and 40 documents dropped out of the top ten. #### Next Step This lesson covered copying, moving, and rebuilding an index as a whole. All of it shared one assumption: that everyone with access to the cluster is authorized to see the entire index — the one taking the snapshot, the one restoring it, and the one querying it all saw the same documents. In a library catalog this is not true — personal loan records, acquisition notes, and donation correspondence can sit in the same index without being open to every user. The next lesson measures this distinction: how the returned set narrows when the same query runs under two different roles, what document- and field-level filtering does to the order, and what that filtering adds to query cost. ### Access Control Source: https://academia.sh/en/courses/search-engines/access-control The search cluster's authorization surface: counting the indexes, documents, and fields the same query returns under a reader, staff, and unscoped key, measuring how applying document-level filtering before or after scoring affects the top ten result and the deep ranking, showing the information filtered documents leak through a counter, counting the index count an unscoped key reaches against the queries an expired key leaves unanswered, and gathering the course's twenty-three lessons into a set, order, and cost table. The previous lesson covered copying and rebuilding an index as a whole; its shared assumption was that everyone with access to the cluster is authorized to see the entire index. In a library catalog this is not true: records closed off by a donation condition, loan history, and acquisition correspondence sit in the same cluster without being open to everyone. This lesson does not measure authentication; it measures the search cluster's **authorization surface**. #### Authorization Surface A role here is the product of three things: which **indexes** are reachable, which **document filter** applies, and which **fields** are visible. The setup has three indexes — the open catalog, loan records, the acquisition source — and a portion of the catalog records carries an internal **note** field only staff can see. The cluster is again modeled in process: a role is nothing more than this triple. **CO21 — the donation batch is 1500 records, comes from a single topic, and its access is restricted by the donation condition.** The effect is linear: as the batch grows, the statistical drift below grows with it. **CO22 — loan and source records carry the title and topic terms of the book in question. CO23 — a key expires at the sixtieth query of a hundred-query stream.** ```js // cluster/access.mjs — index, document, and field-level access control. Generator and index are // the same as earlier lessons. Access is modeled IN-PROCESS: there is no authentication, a role is // only the triple (indexes, document filter, fields). const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"]; const SPECIAL = { child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"], novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"], travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"], essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"], }; const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"]; const TOPIC = Object.keys(SPECIAL); function corpus({ count = 4000, seed = 20260731 } = {}) { let s = seed % 2147483647; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; const pick = (a) => a[Math.floor(r() * a.length)], docs = []; for (let i = 1; i <= count; i += 1) { const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)]; if (r() < 0.5) terms.push(pick(COMMON)); for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1) terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE)); docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") }); } return docs; } function buildIndex(docs) { const postings = new Map(), lengths = new Map(); for (const d of docs) { const t = d.text.split(" "), counts = new Map(); for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1); lengths.set(d.id, t.length); for (const [x, n] of counts) { if (!postings.has(x)) postings.set(x, []); postings.get(x).push([d.id, n]); // posting: id + frequency } } const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1); return { postings, lengths, N: docs.length, avg }; } // Scoring: term frequency, inverse document frequency, and length. The score comes from // WHICHEVER INDEX PROVIDES THE STATISTIC; this lesson's second measurement compares exactly that choice. function search(ix, terms, k) { const N = ix.N, avg = ix.avg, scores = new Map(); let scanned = 0; for (const t of terms) { const g = ix.postings.get(t) ?? [], df = g.length; const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5)); for (const [id, tf] of g) { scanned += 1; const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg); scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm); } } const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k); return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size }; } const K = 10, STREAM = 100, LIFETIME = 60, RESTRICTED = 1500; const QUERY = ["story", "lighthouse", "book", "donation"]; // a story with a donation record, its title mentions a lighthouse const pad = (x, n) => String(x).padStart(n); // Catalog: 4000 open records + a single-topic batch restricted by donation terms (separate // seed, ids moved past 4000). Loans and the source are separate indexes. const OPEN = corpus({ count: 4000 }); const CATALOG = [...OPEN, ...corpus({ count: 20000, seed: 20260801 }).filter((d) => d.topic === "story") .slice(0, RESTRICTED).map((d, i) => ({ ...d, id: 4001 + i, restricted: true }))]; const LOANS = corpus({ count: 900, seed: 20260901 }).map((d) => ({ ...d, id: 20000 + d.id })); const SOURCE = corpus({ count: 400, seed: 20261001 }).map((d) => ({ ...d, id: 30000 + d.id })); // Internal note field: acquisition correspondence, its own vocabulary and seed. const NOTE_WORDS = ["donation", "condition", "clause", "acquisition", "purchase", "buying", "correspondence", "invoice", "testament"]; let n = 20261101; const rr = () => (n = (n * 48271) % 2147483647) / 2147483647; for (const d of CATALOG) if (rr() < 0.35) d.note = Array.from({ length: 3 + Math.floor(rr() * 4) }, () => NOTE_WORDS[Math.floor(rr() * 9)]).join(" "); // An index for one (document set, field) pair; the "note" field has only documents with a note. const build = (docs, field) => buildIndex(field === "note" ? docs.filter((d) => d.note).map((d) => ({ id: d.id, text: d.note })) : docs); // A query is scored on every (index, field) pair and scores are summed; filter is document-level access control. // this is the RAW match BEFORE filtering: if this is what gets written to the counter, restricted documents leak out. function ask(indexes, terms, filter = () => true) { const scores = new Map(); let scanned = 0, raw = 0; for (const ix of indexes) { const c = search(ix, terms, Infinity); scanned += c.scanned; raw += c.matched; for (const a of c.candidates) if (filter(a.id)) scores.set(a.id, (scores.get(a.id) ?? 0) + a.p); } const y = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).map(([id]) => id); return { all: y, top: y.slice(0, K), matched: y.length, scanned, raw }; } const shared = (a, b) => a.filter((x) => b.includes(x)).length; console.log(`catalog ${CATALOG.length} documents (4000 open + ${RESTRICTED} restricted by donation terms), loans ${LOANS.length}, source`); console.log(`${SOURCE.length}, note field on ${CATALOG.filter((d) => d.note).length} documents. Seed 20260731 and 20261101. Query "story lighthouse book donation".\n`); console.log("role / key | indexes | docs | fields | answered | scanned | matched | top 3 result"); console.log("------------------------|---------|------|-------------|----------|---------|---------|------------------"); for (const [name, set, allow, fields, answered] of [ ["reader", [CATALOG], (d) => !d.restricted, ["text"], STREAM], ["staff", [CATALOG, LOANS], () => true, ["text", "note"], STREAM], ["staff, note off", [CATALOG, LOANS], () => true, ["text"], STREAM], ["unscoped key", [CATALOG, LOANS, SOURCE], () => true, ["text", "note"], STREAM], ["expired key", [], () => true, [], LIFETIME], ]) { const c = ask(set.flatMap((x) => fields.map((f) => build(x.filter(allow), f))), QUERY); console.log(`${name.padEnd(23)} | ${pad(set.length, 7)} | ${pad(set.reduce((t, x) => t + x.filter(allow).length, 0), 4)} | ` + `${(fields.join(", ") || "-").padEnd(11)} | ${pad(`${answered}/${STREAM}`, 8)} | ${pad(c.scanned, 7)} | ${pad(c.matched, 7)} | ${c.top.slice(0, 3).join(" ") || "-"}`); } const before = ask([build(OPEN, "text")], QUERY); const after = ask([build(CATALOG, "text")], QUERY, (id) => id <= 4000); console.log("\ndocument-level filtering (reader role, two orderings, same 4000 open documents):"); console.log("ordering | index providing statistics | matched | reported counter | top 10 shared | documents that swapped places"); for (const [name, set, c, counter] of [["filter first", "restricted-out 4000", before, before.matched], ["filter after", "full catalog 5500", after, after.raw]]) { console.log(`${name.padEnd(15)} | ${set.padEnd(27)} | ${pad(c.matched, 7)} | ${pad(counter, 16)} | ` + `${pad(shared(c.top, before.top) + "/10", 12)} | ${pad(c.all.filter((x, i) => before.all[i] !== x).length, 22)}`); } console.log(`top 10, filter first: ${before.top.join(" ")}`); console.log(`top 10, filter after: ${after.top.join(" ")}`); console.log(`\nrun-independent: an unscoped key gets the same triple as the most privileged role and` + ` an expired key leaves ${STREAM - LIFETIME} of ${STREAM} queries unanswered.`); ``` ``` catalog 5500 documents (4000 open + 1500 restricted by donation terms), loans 900, source 400, note field on 1931 documents. Seed 20260731 and 20261101. Query "story lighthouse book donation". role / key | indexes | docs | fields | answered | scanned | matched | top 3 result ------------------------|---------|------|-------------|----------|---------|---------|------------------ reader | 1 | 4000 | text | 100/100 | 2998 | 2463 | 1613 3589 2806 staff | 2 | 6400 | text, note | 100/100 | 6882 | 4743 | 4551 4589 5265 staff, note off | 2 | 6400 | text | 100/100 | 6130 | 4542 | 20332 20557 20765 unscoped key | 3 | 6800 | text, note | 100/100 | 7175 | 4986 | 4551 4589 5265 expired key | 0 | 0 | - | 60/100 | 0 | 0 | - document-level filtering (reader role, two orderings, same 4000 open documents): ordering | index providing statistics | matched | reported counter | top 10 shared | documents that swapped places filter first | restricted-out 4000 | 2463 | 2463 | 10/10 | 0 filter after | full catalog 5500 | 2463 | 3963 | 7/10 | 1433 top 10, filter first: 1613 3589 2806 493 1188 1100 3214 308 476 1012 top 10, filter after: 1613 3589 2806 493 1188 3127 1100 3214 3695 3561 run-independent: an unscoped key gets the same triple as the most privileged role and an expired key leaves 40 of 100 queries unanswered. ``` #### The Same Query at Three Levels The first table runs a single question under five authorizations. The reader reaches one index and 4000 open records, getting 2463 matches; staff reaches two indexes and 6400 records, matched rises to 4743, and all three top results change. What grows the set is not only permission but the other indexes that permission brings into the same ranking. The third row isolates the field level: same staff role, same two indexes, same 6400 records, the only difference is closing the note field. Matched drops to 4542 — 201 records matched only through the note field — and the top three shift from the catalog to loan records. Closing a field does not just remove documents from the list; it also removes one of the scoring components for the ones that remain. The fourth row is the default-open setup: a key with no written scope reaches three indexes and 6800 records, gaining a breadth granted to no role, and pushing scanned entries to 7175. The fifth shows lifetime: an expired key leaves forty of a hundred queries unanswered. #### Filtering Leaking into Scoring The second table is this lesson's real measurement. The reader role can be implemented two ways: the restricted 1500 records are either kept out of the index from the start (**filter first**), or scored over the full catalog and then dropped from the list (**filter after**). Both return the identical 2463 documents — the set is exactly the same. The order is not the same. Only seven of the top ten results are shared: 3127 appears at rank six, 3695 and 3561 at nine and ten, while 308, 476, and 1012 drop off the list; in the deep ranking, 1433 of the 2463 documents have swapped places. The reason is a single number: inverse document frequency. In the filter-after setup, the term "story" is counted across 5500 documents, and because the restricted batch comes from a single topic, that term's distinctiveness drops. Documents the reader cannot see are deciding the order of the documents they can see. The same setup leaks information too: the reported counter is 2463 under filter-first, 3963 under filter-after, and the 1500-record gap between them is the exact size of the restricted batch — the counter is telling the reader the count of records they cannot see. Whether filtering is applied before or after scoring is not a performance detail: it decides both the order and the leak. #### Summary - Authorization makes three separate cuts: the reader got 2463 matches with 1 index and 4000 records, staff 4743 with 2 indexes and 6400 records; all three top results changed entirely. Closing the note field dropped matched to 4542. - A key with no written scope reached three indexes and 6800 records, pushing scanned entries from 2998 to 7175; an expired key left forty of a hundred queries unanswered. - Document-level filtering changed the order without changing the set: across the same 2463 documents, only seven of the top ten were shared, 1433 documents swapped places, and the counter leaked the 1500 restricted records. #### Course Wrap-Up | Lesson | Changed Set | Changed Order | Cost | |---|---|---|---| | Engine vs. Database | LIKE 68, index 12 | plan order, id order | 124,863 → 1,076 bytes | | Inverted Index | position 23 → 10 | posting list in id order | intersection 19,622 → 470 | | Analyzer Chain | 80 / 122 / 463 | all five of the top five changed | 187,780 → 146,080 bytes | | Mappings and Field Types | `child` 93 vs. 0 | all ten of the top ten shifted | `topic` 8,203 → 11,686 bytes | | Dynamic and Explicit Mapping | conflict dropped 111 documents | a dropped document is invisible in the order | metadata 1,684 → 3,235 bytes | | Document Lifecycle | `#10` moved from set to set | from rank 7 to rank 509 | 119,722 → 136,323 bytes | | Query Language Structure | 4,211 / 132 / 71 | from rank 2,023 to the top ten | 21,303 → 7,576 entries | | Term and Phrase Queries | 757 → 144 | adjacent matches 1 → 10 | 220,626 → 334,724 bytes | | Context Queries | must_not 512, should 0 | nine of the top ten changed | entries 2,678 → 4,638 | | Filtering and Query Distinction | 1,138 in both contexts | average 173 ranks | score computations 2,276 → 1,138 | | Relevance Scoring | precision 0.353 → 0.640 | component absence, 499 ranks | 567 results for recall | | Score Tuning | boost 325, field choice 199 | all ten of the top ten changed | 698 → 265 entries | | Aggregation Queries | 208 / 168 / 1 bucket | order is bucket order | 1,045 buckets, 25,080 bytes | | Highlighting and Pagination | 12 of 60 matches excluded | order unchanged | page 20, 800 candidates | | Semantic and Vector Search | 580 at a 0.5 threshold | six of the top ten from a synonym | 153,600 bytes | | Node Roles | delayed state rejected 800 records | the top five changed entirely | coordinating share 0.8% → 11.4% | | Shards and Replicas | 515 → 379 on node loss | top ten shared 10.00 → 4.75 | candidates 10 → 160 | | Split Brain Problem | 210 records lost in the minority | the two halves gave two lists | 600 → 1590 messages | | Bulk Indexing Optimization | 600 missing with refresh off | top ten 6/10 | 4000 segments, 24,000 lookups | | Segment Merging | unchanged, 20,637 | 33 of 50 ranks differed | 3,111,596 → 2,498,875 bytes | | Index Lifecycle | 10,262 → 12,274 | two of the top ten from a deleted day | entries 19,476 → 1,993 | | Snapshot and Restore | 9,286 → 16,517 | 50 ranks changed | 10,182,428 → 3,000,334 bytes | | Access Control | reader 2463, staff 4743 | top ten shared 7/10 | entries 2,998 → 7,175 | The table's three columns are this course's rule: **a search decision is defended only by its effect on the set and the order.** "A more relevant result" is not a decision; an analyzer, a boost, a shard count, or an authorization rule is defended only by stating which documents return, in what order, and at what cost. This is the seventh and last course of M17. Data Modeling and Relational Theory built how data gets modeled, SQL Fundamentals how that model gets queried, Advanced SQL why the same query slows down, Relational Database Administration how a running system gets operated, Non-Relational Data Models what another path gains and gives up, In-Memory Stores and Caching Systems how a memory budget gets spent. This course closed the last gap: reaching data whose key is unknown by going through the text. What comes after this is a habit of measurement, and its starting point is your own data's access pattern: do questions arrive by key or through the text, how many documents return, in what order, and who decides that order. That was the only thing these twenty-three lessons did.