Dynamic and Explicit Mapping

In the previous lesson, which field would be indexed with which type was known in advance. In a real catalog import, this information is often missing: records arrive from donation lists, periodical registries, and digital copy records, each source carries its own fields, and none of them are declared in advance.

In this situation there are two routes. Dynamic mapping adds every new field name to the mapping the moment it sees it, and infers its type from that field’s first value. Explicit mapping fixes the field list in advance; a field not on the list does not enter the index. Both take in the same records, but one chooses flexibility, the other control. This lesson measures the difference between them with four numbers: field count, metadata bytes, accepted documents, and returned set.

The Import’s Fields

For the measurement, the same 600 records are enriched as if they arrived from three additional sources. Donation records bring, alongside the donor field, fields named after the donation year and the donor’s last name: donation_1994, note_kaya. Periodical records carry volume, issue, and periodicity; issue is an integer in most records, and a text like 3-7 in combined issues. Digital copy records add file_format, size_bytes, and page. Every record also gets a field named after a census year: census_2019.

IA1: the corpus is unchanged — 600 records, seed 20250317; the extra fields are derived from the same seed. IA2: mapping metadata per field costs the field name’s bytes plus 24 bytes (type, analyzer pointer, and options). IA3: the explicit mapping consists of seven fields, and a field not on the list is not indexed. IA4: in dynamic mapping, a field’s type is determined by its first value; a conflicting type arriving later rejects the entire document. IA5: terms are qualified by field name, meaning subjects:roman is a single term.

// search/catalog.mjs — corpus shared by three topics: generated from a library catalog,
// 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf.
export const SEED = 20250317, DOC_COUNT = 600;
let state = SEED;
const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648;
const pick = (d) => d[Math.floor(random() * d.length)];
const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)];   // frequency is skewed in real text
const splitBar = (s) => s.split("|");

// stem forms: base, genitive, plural, dative, ablative, locative
export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \
kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \
şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \
bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\
gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\
okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" "));
const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\
1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \
yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \
belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \
mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]);
const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \
örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır");
const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \
anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \
kapanır|adacıklardaki kuş türlerini sayar");
const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya");
const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü");
const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi");
const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri");
const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe");
const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı");
const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem");
const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca");

const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1);
function generateSummary() {
  const parts = [];
  for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); }
  parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS)));
  if (random() < 0.45) parts.push(pick(EXTRA));
  return capitalize(parts.join(", ")) + ".";
}
function generateTitle() {
  const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random();
  if (t < 0.25) return `${o} ${capitalize(k[2])}`;
  if (t < 0.5) return `${capitalize(k[0])} ${s}`;
  if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`;
  return `${capitalize(k[1])} ${s}`;
}
export const corpus = [];
for (let i = 1; i <= DOC_COUNT; i += 1) {
  const subjects = [pick(SUBJECTS)];
  if (random() < 0.55) subjects.push(pick(SUBJECTS));
  if (random() < 0.2) subjects.push(pick(SUBJECTS));
  corpus.push({
    id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`,
    year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES),
    shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`,
  });
}

// search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term
// frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it.
export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean);
export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8;   // IA2: byte constants

export class InvertedIndex {
  dictionary = new Map();                // term -> posting list
  documentCount = 0;
  constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); }

  add(id, text) {
    const local = new Map();
    this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i));
    for (const [t, k] of local) {
      if (!this.dictionary.has(t)) this.dictionary.set(t, []);
      this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] });
    }
    this.documentCount += 1;
  }

  postings(t) { return this.dictionary.get(t) ?? []; }

  search(...terms) {                     // intersection: lists are id-sorted, so one pass suffices
    const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = [];
    let comparisons = 0;
    while (l.every((x, i) => p[i] < x.length)) {
      const maxId = Math.max(...l.map((x, i) => x[p[i]].id));
      let matched = true;
      for (let i = 0; i < l.length; i += 1) {
        while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; }
        comparisons += 1;
        if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; }
      }
      if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); }
    }
    return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons };
  }

  bytes() {                              // dictionary + postings + positions
    let s = 0, g = 0, k = 0, postings = 0, positions = 0;
    for (const [t, list] of this.dictionary) {
      s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD;
      for (const gr of list) {
        g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0);
        k += gr.position.length * POSITION_BYTES;
        postings += 1; positions += gr.position.length;
      }
    }
    return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k };
  }
}

// search/mapping.mjs — the same records taken in under two mappings: dynamic (every incoming field
// enters the index) and explicit (field list fixed in advance). Field count, metadata bytes, rejected documents, set difference.
import { corpus, DOC_COUNT, SEED } from "./catalog.mjs";
import { InvertedIndex } from "./index.mjs";

const FIELD_OVERHEAD = 24;                                     // IA2: mapping record per field
let c = SEED;
const random = () => (c = (c * 1103515245 + 12345) % 2147483648) / 2147483648;
const LAST_NAMES = ["Yılmaz", "Kaya", "Demir", "Şahin", "Çelik", "Aydın", "Doğan", "Arslan", "Koç", "Ertem"];

// catalog import: each source brings its own fields (donation, periodical, digital copy)
const records = corpus.map((b) => {
  const k = { title: b.title, summary: b.summary, subjects: b.subjects.join(" "), author: b.author, year: b.year, language: b.language, shelf: b.shelf };
  const t = random();
  if (t < 0.35) {
    const year = 1968 + Math.floor(random() * 57), s = LAST_NAMES[Math.floor(random() * 10)];
    Object.assign(k, { donor: `${s} family`, [`donation_${year}`]: "accepted", [`note_${s.toLocaleLowerCase("tr")}`]: "archived" });
  } else if (t < 0.6) {
    k.volume = 1 + Math.floor(random() * 12);
    k.issue = random() < 0.25 ? `${1 + Math.floor(random() * 4)}-${5 + Math.floor(random() * 4)}` : 1 + Math.floor(random() * 12);
    k.periodicity = "monthly";
  } else if (t < 0.8) {
    Object.assign(k, { file_format: "scan", size_bytes: 1 << (18 + Math.floor(random() * 4)), page: 40 + Math.floor(random() * 400) });
  }
  k[`census_${2015 + Math.floor(random() * 10)}`] = 1 + Math.floor(random() * 3);
  return k;
});

const EXPLICIT = new Map([["title", "text"], ["summary", "text"], ["subjects", "text"], ["author", "text"],
  ["year", "number"], ["language", "text"], ["shelf", "text"]]);      // IA3: explicit mapping is seven fields
const typeOf = (v) => (typeof v === "number" ? "number" : "text");

// term qualified by field name: "subjects:roman" is a single term
const termOf = (a, v) => String(v).toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u)
  .filter(Boolean).map((t) => `${a}:${t}`).join(" ");

function take(records, explicit) {                             // build the mapping, accept or reject the document
  const field = explicit ? new Map(explicit) : new Map();
  const index = new InvertedIndex({ analyze: (s) => s.split(" ").filter(Boolean), position: false });
  const rejected = [], skipped = new Map();
  let accepted = 0;
  for (const [i, k] of records.entries()) {
    const conflict = [];
    for (const [a, v] of Object.entries(k)) {
      const type = typeOf(v);
      if (explicit && !field.has(a)) { skipped.set(a, (skipped.get(a) ?? 0) + 1); continue; }
      if (!field.has(a)) field.set(a, type);
      else if (field.get(a) !== type) conflict.push(`${a} (${field.get(a)} -> ${type})`);
    }
    if (conflict.length) { rejected.push([i + 1, conflict[0]]); continue; }
    index.add(i + 1, Object.entries(k).filter(([a]) => field.has(a)).map(([a, v]) => termOf(a, v)).join(" "));
    accepted += 1;
  }
  const metadata = [...field.keys()].reduce((t, a) => t + Buffer.byteLength(a) + FIELD_OVERHEAD, 0);
  return { field, index, rejected, skipped, accepted, metadata };
}

const dynamic = take(records, null), explicit = take(records, EXPLICIT);
console.log(`seed ${SEED}; ${DOC_COUNT} catalog records, three extra sources (donation, periodical, digital copy)`);

console.log(`\nfield count growing with document count (dynamic mapping)`);
console.log(`${"docs processed".padStart(15)}${"fields".padStart(8)}${"metadata bytes".padStart(16)}`);
for (const n of [50, 150, 300, 600]) {
  const d = take(records.slice(0, n), null);
  console.log(String(n).padStart(15) + String(d.field.size).padStart(8) + String(d.metadata).padStart(16));
}

console.log(`\n${"mapping".padEnd(9)}${"fields".padStart(7)}${"metadata bytes".padStart(16)}${"terms".padStart(7)}` +
  `${"index bytes".padStart(13)}${"accepted".padStart(9)}${"rejected".padStart(9)}${"unindexed fields".padStart(19)}`);
for (const [label, r] of [["dynamic", dynamic], ["explicit", explicit]])
  console.log(label.padEnd(9) + String(r.field.size).padStart(7) + String(r.metadata).padStart(16) +
    String(r.index.bytes().terms).padStart(7) + String(r.index.bytes().total).padStart(13) +
    String(r.accepted).padStart(9) + String(r.rejected.length).padStart(9) + String(r.skipped.size).padStart(19));

console.log(`\ntype conflict: ${dynamic.rejected.length} documents rejected; first three ` +
  `${dynamic.rejected.slice(0, 3).map(([i, a]) => `#${i} ${a}`).join(", ")}`);
const queries = ["subjects:roman", "language:türkçe", "donor:kaya", "periodicity:monthly"];
console.log(`\n${"query".padEnd(18)}${"dynamic".padStart(9)}${"explicit".padStart(9)}${"diff".padStart(7)}`);
for (const s of queries) {
  const d = dynamic.index.search(s).results.length, e = explicit.index.search(s).results.length;
  console.log(s.padEnd(18) + String(d).padStart(9) + String(e).padStart(9) + String(d - e).padStart(7));
}

seed 20250317; 600 catalog records, three extra sources (donation, periodical, digital copy)

field count growing with document count (dynamic mapping)
 docs processed  fields  metadata bytes
             50      49            1684
            150      77            2717
            300      85            3013
            600      91            3235

mapping   fields  metadata bytes  terms  index bytes accepted rejected   unindexed fields
dynamic       91            3235    660       161329      489      111                  0
explicit       7             211    437       171862      600        0                 84

type conflict: 111 documents rejected; first three #2 issue (text -> number), #12 issue (text -> number), #22 issue (text -> number)

query               dynamic explicit   diff
subjects:roman           70       82    -12
language:türkçe         253      310    -57
donor:kaya               18        0     18
periodicity:monthly       40        0     40

Field Count Grows With Documents

The first table shows the shape of mapping explosion. Once the first 50 documents are processed, the mapping already has 49 fields: nearly one new field per record. Because field names are derived from the data — donation_1994, note_kaya, census_2019 — every new donation year, every new donor, and every new census year adds a row to the mapping. Growth then slows (77 at 150 documents, 91 at 600) because the pool of years and last names runs out; the pool being limited is a property of the corpus, a real import has no pool that runs out.

Metadata tracks this growth directly: from 1,684 bytes to 3,235 bytes. The number can look small, but it must be read at scale — metadata does not grow with document count, it grows with field count, and it is held again in every part of the index. In the explicit mapping the same line item is 211 bytes: about fifteen times smaller, and fixed regardless of document count.

A Type Conflict Drops the Document

The second table’s harshest number is the rejected column. Dynamic mapping accepted 489 of the 600 records and rejected 111. The reason is a single field: issue. In the first periodical record this field arrived as text in the form 3-7, so the mapping fixed it as text; when the same field later arrived as a number, the document was rejected in its entirety. The loss is not limited to that field — the document’s title, summary, and subjects fields never enter the index either.

The result shows up in the query table. The subjects:roman query returns 70 documents in dynamic mapping and 82 in explicit mapping; for language:türkçe the gap is 253 against 310. So because of the issue field’s type, 57 Turkish books are unsearchable in the catalog, and there is no sign of this in the query result at all: the missing documents are silently absent.

The reverse direction is also measured. The donor:kaya query returns 18 documents in dynamic mapping and 0 in explicit mapping; for periodicity:monthly it is 40 against 0. Because explicit mapping does not index 84 separate fields, these queries go unanswered. Explicit mapping’s control is not free: every new question the import brings stays unanswered until a row is added to the mapping by hand.

Index size sits in a misleading order in this table: dynamic mapping holds 660 terms in 161,329 bytes, explicit mapping holds 437 terms in 171,862 bytes. Explicit mapping is not larger because of its field count, it is larger because it indexes 111 more documents. The same number needs to be read per accepted document: 330 bytes per document in dynamic mapping, 286 bytes in explicit mapping.

Summary

• Dynamic mapping adds every incoming field to the mapping and infers its type from the first value; explicit mapping fixes the field list in advance and does not index anything not on the list.
• When field names are derived from data, the mapping grows with the documents: 49 fields at the first 50 documents, 91 fields at 600; metadata rises from 1,684 bytes to 3,235 bytes. In explicit mapping the same line item is 211 bytes and stays fixed.
• A type conflict drops the document, not the field: because the issue field arrives as text in one record and as a number in another, 111 of 600 records are rejected and never enter the index.
• The loss is invisible in the query result but measurable: subjects:roman returns 70 instead of 82, language:türkçe returns 253 instead of 310. Nothing in the result signals that documents are missing.
• Explicit mapping’s cost is unanswered queries: because 84 fields are not indexed, donor:kaya and periodicity:monthly return 0 documents.

Next Step

This lesson’s two mappings both took in documents once and let them go: a record arrived, and it either entered the index or it did not. The catalog, however, does not stand still — a book’s summary gets corrected, a subject tag changes, a lost book is removed from the catalog. The next lesson measures what these three operations do to the index: why an update happens as a delete plus an add, how much room a deleted document keeps in the posting lists, the effect of the delete marker on the query result and the scanned entries, and how bloated the index gets before a cleanup pass.