--- title: 'Dynamic and Explicit Mapping' source: 'https://academia.sh/en/courses/search-engines/dynamic-and-explicit-mapping' course: 'Search Engines and Text Retrieval' language: en updated: '2026-08-23T07:00:49+00:00' license: 'CC BY-SA 4.0' --- # Dynamic and Explicit Mapping Who decides the mapping: comparing dynamic mapping, which adds every incoming field to the index, against explicit mapping, whose field list is fixed in advance, on the same catalog import; the field count growing with document diversity; counting the mapping metadata per field in bytes; and measuring the documents rejected when the same name arrives with two types, and their fall from the query set. In the previous lesson, which field would be indexed with which type was known in advance. In a real catalog import, this information is often missing: records arrive from donation lists, periodical registries, and digital copy records, each source carries its own fields, and none of them are declared in advance. In this situation there are two routes. **Dynamic mapping** adds every new field name to the mapping the moment it sees it, and infers its type from that field's **first value**. **Explicit mapping** fixes the field list in advance; a field not on the list does not enter the index. Both take in the same records, but one chooses flexibility, the other control. This lesson measures the difference between them with four numbers: field count, metadata bytes, accepted documents, and returned set. ## The Import's Fields For the measurement, the same 600 records are enriched as if they arrived from three additional sources. Donation records bring, alongside the `donor` field, fields named after the donation year and the donor's last name: `donation_1994`, `note_kaya`. Periodical records carry `volume`, `issue`, and `periodicity`; `issue` is an integer in most records, and a text like `3-7` in combined issues. Digital copy records add `file_format`, `size_bytes`, and `page`. Every record also gets a field named after a census year: `census_2019`. **IA1:** the corpus is unchanged — 600 records, seed 20250317; the extra fields are derived from the same seed. **IA2:** mapping metadata per field costs the field name's bytes plus 24 bytes (type, analyzer pointer, and options). **IA3:** the explicit mapping consists of seven fields, and a field not on the list is not indexed. **IA4:** in dynamic mapping, a field's type is determined by its first value; a conflicting type arriving later rejects the entire document. **IA5:** terms are qualified by field name, meaning `subjects:roman` is a single term. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/mapping.mjs — the same records taken in under two mappings: dynamic (every incoming field // enters the index) and explicit (field list fixed in advance). Field count, metadata bytes, rejected documents, set difference. import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; import { InvertedIndex } from "./index.mjs"; const FIELD_OVERHEAD = 24; // IA2: mapping record per field let c = SEED; const random = () => (c = (c * 1103515245 + 12345) % 2147483648) / 2147483648; const LAST_NAMES = ["Yılmaz", "Kaya", "Demir", "Şahin", "Çelik", "Aydın", "Doğan", "Arslan", "Koç", "Ertem"]; // catalog import: each source brings its own fields (donation, periodical, digital copy) const records = corpus.map((b) => { const k = { title: b.title, summary: b.summary, subjects: b.subjects.join(" "), author: b.author, year: b.year, language: b.language, shelf: b.shelf }; const t = random(); if (t < 0.35) { const year = 1968 + Math.floor(random() * 57), s = LAST_NAMES[Math.floor(random() * 10)]; Object.assign(k, { donor: `${s} family`, [`donation_${year}`]: "accepted", [`note_${s.toLocaleLowerCase("tr")}`]: "archived" }); } else if (t < 0.6) { k.volume = 1 + Math.floor(random() * 12); k.issue = random() < 0.25 ? `${1 + Math.floor(random() * 4)}-${5 + Math.floor(random() * 4)}` : 1 + Math.floor(random() * 12); k.periodicity = "monthly"; } else if (t < 0.8) { Object.assign(k, { file_format: "scan", size_bytes: 1 << (18 + Math.floor(random() * 4)), page: 40 + Math.floor(random() * 400) }); } k[`census_${2015 + Math.floor(random() * 10)}`] = 1 + Math.floor(random() * 3); return k; }); const EXPLICIT = new Map([["title", "text"], ["summary", "text"], ["subjects", "text"], ["author", "text"], ["year", "number"], ["language", "text"], ["shelf", "text"]]); // IA3: explicit mapping is seven fields const typeOf = (v) => (typeof v === "number" ? "number" : "text"); // term qualified by field name: "subjects:roman" is a single term const termOf = (a, v) => String(v).toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u) .filter(Boolean).map((t) => `${a}:${t}`).join(" "); function take(records, explicit) { // build the mapping, accept or reject the document const field = explicit ? new Map(explicit) : new Map(); const index = new InvertedIndex({ analyze: (s) => s.split(" ").filter(Boolean), position: false }); const rejected = [], skipped = new Map(); let accepted = 0; for (const [i, k] of records.entries()) { const conflict = []; for (const [a, v] of Object.entries(k)) { const type = typeOf(v); if (explicit && !field.has(a)) { skipped.set(a, (skipped.get(a) ?? 0) + 1); continue; } if (!field.has(a)) field.set(a, type); else if (field.get(a) !== type) conflict.push(`${a} (${field.get(a)} -> ${type})`); } if (conflict.length) { rejected.push([i + 1, conflict[0]]); continue; } index.add(i + 1, Object.entries(k).filter(([a]) => field.has(a)).map(([a, v]) => termOf(a, v)).join(" ")); accepted += 1; } const metadata = [...field.keys()].reduce((t, a) => t + Buffer.byteLength(a) + FIELD_OVERHEAD, 0); return { field, index, rejected, skipped, accepted, metadata }; } const dynamic = take(records, null), explicit = take(records, EXPLICIT); console.log(`seed ${SEED}; ${DOC_COUNT} catalog records, three extra sources (donation, periodical, digital copy)`); console.log(`\nfield count growing with document count (dynamic mapping)`); console.log(`${"docs processed".padStart(15)}${"fields".padStart(8)}${"metadata bytes".padStart(16)}`); for (const n of [50, 150, 300, 600]) { const d = take(records.slice(0, n), null); console.log(String(n).padStart(15) + String(d.field.size).padStart(8) + String(d.metadata).padStart(16)); } console.log(`\n${"mapping".padEnd(9)}${"fields".padStart(7)}${"metadata bytes".padStart(16)}${"terms".padStart(7)}` + `${"index bytes".padStart(13)}${"accepted".padStart(9)}${"rejected".padStart(9)}${"unindexed fields".padStart(19)}`); for (const [label, r] of [["dynamic", dynamic], ["explicit", explicit]]) console.log(label.padEnd(9) + String(r.field.size).padStart(7) + String(r.metadata).padStart(16) + String(r.index.bytes().terms).padStart(7) + String(r.index.bytes().total).padStart(13) + String(r.accepted).padStart(9) + String(r.rejected.length).padStart(9) + String(r.skipped.size).padStart(19)); console.log(`\ntype conflict: ${dynamic.rejected.length} documents rejected; first three ` + `${dynamic.rejected.slice(0, 3).map(([i, a]) => `#${i} ${a}`).join(", ")}`); const queries = ["subjects:roman", "language:türkçe", "donor:kaya", "periodicity:monthly"]; console.log(`\n${"query".padEnd(18)}${"dynamic".padStart(9)}${"explicit".padStart(9)}${"diff".padStart(7)}`); for (const s of queries) { const d = dynamic.index.search(s).results.length, e = explicit.index.search(s).results.length; console.log(s.padEnd(18) + String(d).padStart(9) + String(e).padStart(9) + String(d - e).padStart(7)); } ``` ``` seed 20250317; 600 catalog records, three extra sources (donation, periodical, digital copy) field count growing with document count (dynamic mapping) docs processed fields metadata bytes 50 49 1684 150 77 2717 300 85 3013 600 91 3235 mapping fields metadata bytes terms index bytes accepted rejected unindexed fields dynamic 91 3235 660 161329 489 111 0 explicit 7 211 437 171862 600 0 84 type conflict: 111 documents rejected; first three #2 issue (text -> number), #12 issue (text -> number), #22 issue (text -> number) query dynamic explicit diff subjects:roman 70 82 -12 language:türkçe 253 310 -57 donor:kaya 18 0 18 periodicity:monthly 40 0 40 ``` ## Field Count Grows With Documents The first table shows the shape of mapping explosion. Once the first 50 documents are processed, the mapping already has **49 fields**: nearly one new field per record. Because field names are derived from the data — `donation_1994`, `note_kaya`, `census_2019` — every new donation year, every new donor, and every new census year adds a row to the mapping. Growth then slows (77 at 150 documents, 91 at 600) because the pool of years and last names runs out; the pool being limited is a property of the corpus, a real import has no pool that runs out. Metadata tracks this growth directly: from 1,684 bytes to 3,235 bytes. The number can look small, but it must be read at scale — metadata does not grow with document count, it grows with **field count**, and it is held again in every part of the index. In the explicit mapping the same line item is 211 bytes: about fifteen times smaller, and fixed regardless of document count. ## A Type Conflict Drops the Document The second table's harshest number is the `rejected` column. Dynamic mapping accepted **489** of the 600 records and rejected **111**. The reason is a single field: `issue`. In the first periodical record this field arrived as text in the form `3-7`, so the mapping fixed it as text; when the same field later arrived as a number, the document was rejected in its entirety. The loss is not limited to that field — the document's `title`, `summary`, and `subjects` fields never enter the index either. The result shows up in the query table. The `subjects:roman` query returns 70 documents in dynamic mapping and 82 in explicit mapping; for `language:türkçe` the gap is 253 against 310. So because of the `issue` field's type, 57 Turkish books are **unsearchable** in the catalog, and there is no sign of this in the query result at all: the missing documents are silently absent. The reverse direction is also measured. The `donor:kaya` query returns 18 documents in dynamic mapping and 0 in explicit mapping; for `periodicity:monthly` it is 40 against 0. Because explicit mapping does not index 84 separate fields, these queries go unanswered. Explicit mapping's control is not free: every new question the import brings stays unanswered until a row is added to the mapping by hand. Index size sits in a misleading order in this table: dynamic mapping holds 660 terms in 161,329 bytes, explicit mapping holds 437 terms in 171,862 bytes. Explicit mapping is not larger because of its field count, it is larger because **it indexes 111 more documents**. The same number needs to be read per accepted document: 330 bytes per document in dynamic mapping, 286 bytes in explicit mapping. ## Summary - Dynamic mapping adds every incoming field to the mapping and infers its type from the first value; explicit mapping fixes the field list in advance and does not index anything not on the list. - When field names are derived from data, the mapping grows with the documents: 49 fields at the first 50 documents, 91 fields at 600; metadata rises from 1,684 bytes to 3,235 bytes. In explicit mapping the same line item is 211 bytes and stays fixed. - A type conflict drops the document, not the field: because the `issue` field arrives as text in one record and as a number in another, 111 of 600 records are rejected and never enter the index. - The loss is invisible in the query result but measurable: `subjects:roman` returns 70 instead of 82, `language:türkçe` returns 253 instead of 310. Nothing in the result signals that documents are missing. - Explicit mapping's cost is unanswered queries: because 84 fields are not indexed, `donor:kaya` and `periodicity:monthly` return 0 documents. ## Next Step This lesson's two mappings both took in documents once and let them go: a record arrived, and it either entered the index or it did not. The catalog, however, does not stand still — a book's summary gets corrected, a subject tag changes, a lost book is removed from the catalog. The next lesson measures what these three operations do to the index: why an update happens as a delete plus an add, how much room a deleted document keeps in the posting lists, the effect of the delete marker on the query result and the scanned entries, and how bloated the index gets before a cleanup pass.