--- title: 'Mappings and Field Types' source: 'https://academia.sh/en/courses/search-engines/mappings-and-field-types' course: 'Search Engines and Text Retrieval' language: en updated: '2026-08-23T07:00:49+00:00' license: 'CC BY-SA 4.0' --- # Mappings and Field Types A field's type deciding the indexing rule: separating the text field from the keyword field, indexing the same field with both types and comparing term count, posting entries, and index bytes, the effect of exact matching versus partial matching on the returned set, the cross-match a multi-valued field produces, whether a sort key is defined, and how aggregation buckets diverge between the two types. The previous lesson's three chains all applied the same rule to the entire text. But not every field in the catalog is text. The shelf code `TR-866.3` marks a location, and splitting it makes no sense; the language `Türkçe` is a value; the subject tag `çocuk edebiyatı` (children's literature) consists of two words but is a single category. The summary, however, really is text, and every word inside it is a separate search entry. The declaration that states which rule a field is indexed by is called a **mapping**; the decision the mapping carries is called the **field type**. This lesson compares two types. A **text field** passes the value through the analyzer chain; as a result the field spreads across multiple terms. A **keyword field** does not split or transform the value: the entire value is a single term. ## Two Types, the Same Field The measurement runs over three fields: the multi-valued `subjects`, and the single-valued `shelf` and `language`. Each field is indexed twice, and four things are counted: index size, the set an exact-value query returns, whether sorting is defined, and the number of buckets aggregation produces. **IA1:** the corpus is unchanged — 600 records, seed 20250317. **IA2:** the byte constants are the same constants as the previous lessons; position is not kept in this lesson. **IA3:** in a multi-valued field, the keyword type makes each value a separate term, the text type parses all the values as a single piece of text. **IA4:** because a text field can carry more than one term per document, a sort key must be chosen; here the "smallest term" rule is chosen. ```js // search/catalog.mjs — corpus shared by three topics: generated from a library catalog, // 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf. export const SEED = 20250317, DOC_COUNT = 600; let state = SEED; const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648; const pick = (d) => d[Math.floor(random() * d.length)]; const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)]; // frequency is skewed in real text const splitBar = (s) => s.split("|"); // stem forms: base, genitive, plural, dative, ablative, locative export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \ kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \ şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \ bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\ gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\ okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" ")); const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\ 1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \ yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \ belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \ mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]); const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \ örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır"); const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \ anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \ kapanır|adacıklardaki kuş türlerini sayar"); const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya"); const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü"); const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi"); const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri"); const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe"); const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı"); const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem"); const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca"); const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1); function generateSummary() { const parts = []; for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); } parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS))); if (random() < 0.45) parts.push(pick(EXTRA)); return capitalize(parts.join(", ")) + "."; } function generateTitle() { const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random(); if (t < 0.25) return `${o} ${capitalize(k[2])}`; if (t < 0.5) return `${capitalize(k[0])} ${s}`; if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`; return `${capitalize(k[1])} ${s}`; } export const corpus = []; for (let i = 1; i <= DOC_COUNT; i += 1) { const subjects = [pick(SUBJECTS)]; if (random() < 0.55) subjects.push(pick(SUBJECTS)); if (random() < 0.2) subjects.push(pick(SUBJECTS)); corpus.push({ id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`, year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES), shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`, }); } ``` ```js // search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term // frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it. export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8; // IA2: byte constants export class InvertedIndex { dictionary = new Map(); // term -> posting list documentCount = 0; constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); } add(id, text) { const local = new Map(); this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i)); for (const [t, k] of local) { if (!this.dictionary.has(t)) this.dictionary.set(t, []); this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] }); } this.documentCount += 1; } postings(t) { return this.dictionary.get(t) ?? []; } search(...terms) { // intersection: lists are id-sorted, so one pass suffices const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = []; let comparisons = 0; while (l.every((x, i) => p[i] < x.length)) { const maxId = Math.max(...l.map((x, i) => x[p[i]].id)); let matched = true; for (let i = 0; i < l.length; i += 1) { while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; } comparisons += 1; if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; } } if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); } } return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons }; } bytes() { // dictionary + postings + positions let s = 0, g = 0, k = 0, postings = 0, positions = 0; for (const [t, list] of this.dictionary) { s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD; for (const gr of list) { g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0); k += gr.position.length * POSITION_BYTES; postings += 1; positions += gr.position.length; } } return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k }; } } ``` ```js // search/field-type.mjs — the same fields are indexed with two types: text field (goes through // the analyzer) and keyword field (value stays a single term). Bytes, exact match, sort, aggregation. import { corpus, DOC_COUNT, SEED } from "./catalog.mjs"; import { InvertedIndex } from "./index.mjs"; const SEPARATOR = "\u0001"; // value boundary const fieldValues = { subjects: (b) => b.subjects, shelf: (b) => [b.shelf], language: (b) => [b.language] }; const TEXT = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean); const KEYWORD = (s) => s.split(SEPARATOR).filter(Boolean); // value is not split const build = (field, analyze) => { const d = new InvertedIndex({ analyze, position: false }); for (const b of corpus) d.add(b.id, fieldValues[field](b).join(SEPARATOR)); return d; }; const index = new Map(); for (const field of Object.keys(fieldValues)) for (const [type, analyze] of [["text", TEXT], ["keyword", KEYWORD]]) index.set(`${field}/${type}`, build(field, analyze)); console.log(`seed ${SEED}; corpus ${DOC_COUNT} docs; fields: subjects (multi-valued), shelf, language`); console.log(`\n${"field".padEnd(9)}${"type".padEnd(9)}${"terms".padStart(7)}${"postings".padStart(9)}` + `${"index bytes".padStart(13)}${"sample terms".padStart(28)}`); for (const [key, d] of index) { const [field, type] = key.split("/"), b = d.bytes(); console.log(field.padEnd(9) + type.padEnd(9) + String(b.terms).padStart(7) + String(b.postings).padStart(9) + String(b.total).padStart(13) + [...d.dictionary.keys()].slice(0, 2).join(" | ").padStart(28)); } const commonShelf = [...corpus.reduce((m, b) => m.set(b.shelf, (m.get(b.shelf) ?? 0) + 1), new Map())] .sort((x, y) => y[1] - x[1])[0][0]; // the shelf code repeated most often in the corpus const queries = [["subjects", "çocuk edebiyatı", "exact tag"], ["subjects", "çocuk", "single word of tag"], ["subjects", "çocuk tarihi", "tag not present"], ["shelf", commonShelf, "exact shelf code"], ["shelf", "TR", "shelf prefix"], ["language", "Türkçe", "value as written"], ["language", "türkçe", "value lowercased"]]; console.log(`\n${"field".padEnd(9)}${"query".padEnd(18)}${"description".padEnd(20)}${"text".padStart(8)}${"keyword".padStart(10)}`); for (const [field, s, note] of queries) { const m = index.get(`${field}/text`).search(...TEXT(s)).results.length; const k = index.get(`${field}/keyword`).search(s).results.length; console.log(field.padEnd(9) + s.padEnd(18) + note.padEnd(20) + String(m).padStart(8) + String(k).padStart(10)); } // --- sorting: a keyword field has one term per document, a text field can have more than one --- const termsOf = (d, id) => [...d.dictionary].filter(([, l]) => l.some((g) => g.id === id)).map(([t]) => t); const textShelf = index.get("shelf/text"), keywordShelf = index.get("shelf/keyword"); const multiTerm = corpus.filter((b) => termsOf(textShelf, b.id).length > 1).length; const smallest = (d, id) => termsOf(d, id).sort()[0]; const sortBy = (d) => corpus.map((b) => [smallest(d, b.id), b.id]).sort((x, y) => (x[0] < y[0] ? -1 : x[0] > y[0] ? 1 : x[1] - y[1])); const keywordOrder = sortBy(keywordShelf), textOrder = sortBy(textShelf); const different = keywordOrder.slice(0, 10).filter(([, id], i) => textOrder[i][1] !== id).length; console.log(`\nsort order (by shelf field): in the text type, ${multiTerm} documents have more than one term in the field; ` + `${new Set(textOrder.map(([t]) => t)).size} distinct sort keys`); console.log(`in the keyword type, ${new Set(keywordOrder.map(([t]) => t)).size} distinct keys; ` + `${different} of the first ten documents differ between the two orders`); console.log(` keyword first three: ${keywordOrder.slice(0, 3).map(([t, i]) => `${t} (#${i})`).join(", ")}`); console.log(` text first three : ${textOrder.slice(0, 3).map(([t, i]) => `${t} (#${i})`).join(", ")}`); // --- aggregation: bucket count on the subjects field --- const buckets = (d) => [...d.dictionary].map(([t, l]) => [t, l.length]).sort((x, y) => y[1] - x[1] || (x[0] < y[0] ? -1 : 1)); for (const type of ["keyword", "text"]) { const result = buckets(index.get(`subjects/${type}`)); console.log(`\nsubjects aggregation (${type}): ${result.length} buckets, ${result.reduce((t, [, n]) => t + n, 0)} total document count`); console.log(` largest three: ${result.slice(0, 3).map(([t, n]) => `${t} ${n}`).join(", ")}`); } ``` ``` seed 20250317; corpus 600 docs; fields: subjects (multi-valued), shelf, language field type terms postings index bytes sample terms subjects text 17 1430 11686 biyografi | kısa subjects keyword 12 999 8203 biyografi | kısa öykü shelf text 112 1800 15610 en | 893 shelf keyword 549 600 13584 EN-893.5 | FR-881.4 language text 4 600 4866 almanca | fransızca language keyword 4 600 4867 Almanca | Fransızca field query description text keyword subjects çocuk edebiyatı exact tag 93 93 subjects çocuk single word of tag 93 0 subjects çocuk tarihi tag not present 9 0 shelf TR-866.3 exact shelf code 3 3 shelf TR shelf prefix 154 0 language Türkçe value as written 310 310 language türkçe value lowercased 310 0 sort order (by shelf field): in the text type, 600 documents have more than one term in the field; 9 distinct sort keys in the keyword type, 549 distinct keys; 10 of the first ten documents differ between the two orders keyword first three: DE-801.1 (#460), DE-801.8 (#94), DE-802.5 (#594) text first three : 0 (#19), 0 (#31), 0 (#42) subjects aggregation (keyword): 12 buckets, 999 total document count largest three: coğrafya 95, çocuk edebiyatı 93, biyografi 88 subjects aggregation (text): 17 buckets, 1430 total document count largest three: coğrafya 95, edebiyatı 93, çocuk 93 ``` ## The Byte Difference Depends on the Field Itself The first table does not give a single rule, because which type is cheaper depends on the field. In the `subjects` field the text type produces both more terms (17 against 12) and more posting entries (1,430 against 999), and at 11,686 bytes it comes in above the keyword type's 8,203 bytes: once two-word tags are split, each document leaves two entries in the field. In the `shelf` field the direction reverses. The text type shrinks the dictionary — 112 pieces instead of 549 separate codes, because the same number and prefix repeat across hundreds of codes — but triples the posting entries: 1,800 instead of 600. In total the text type is again more expensive (15,610 against 13,584), but this time the expense comes from the posting side. The `language` field is the borderline case where the two types differ by only one byte: because the value is already a single word, the type decision does not change the size, it only preserves the spelling. ## Exact Matching, Sorting, and Aggregation The second table shows the decision's effect on the set. When the exact tag is asked for, both types return the same 93 documents; the distinction is in the other rows. When the tag's **single word** is asked for, the text field gives 93 documents, the keyword field gives 0: the text field is open to partial matching, the keyword field plainly is not. The same behavior repeats in the shelf field with the `TR` prefix: 154 documents against 0. The third row is the real warning. A tag reading `çocuk tarihi` does not **exist** in the corpus; despite this, the text field returns 9 documents. These are documents that carry the tags `çocuk edebiyatı` and `deniz tarihi` together: because the field is multi-valued, words coming from two separate values behave as a single match. The keyword field does not bring back these documents, because for it every tag is an unsplit whole. The cost in the opposite direction shows up in the language row: when asked with the spelling `türkçe`, the keyword field returns 0 documents, because the value carries a capital letter. The sorting difference is sharper. In the keyword field every document has exactly one term, and the shelf code produces 549 separate sort keys; sorting is defined. In the text field **all 600 documents** have more than one term in the field, so the question "sorted by which term" has no answer. Once the chosen rule takes the smallest term, only 9 keys remain, and all ten of the first ten documents swap places: sorting becomes meaningless in practice. On the aggregation side, too, the two types pull two different tables out of the same data. The keyword field gives 12 buckets and a document count of 999 — this is the exact count of tags in the corpus. The text field gives 17 buckets and a count of 1,430; the `çocuk edebiyatı` tag has split into two buckets, `çocuk` and `edebiyatı`, each counting 93 and landing at the top of the list. In a report that wants to see the category distribution, these two tables answer the same question differently. ## Summary - A mapping declares which rule a field is indexed by; a text field passes the value through the analyzer and splits it into terms, a keyword field keeps the value as a single, unsplit term. - The direction of the byte difference depends on the field: in `subjects`, the text type is more expensive at 11,686 against 8,203 bytes; in `shelf`, it shrinks the dictionary from 549 to 112 terms but triples the posting entries from 600 to 1,800. - Exact matching is only exact in a keyword field: the `çocuk` query returns 93 documents in the text field and 0 in the keyword field; the `TR` prefix gives 154 against 0. - In a multi-valued field, the text type produces a cross-match: the tag `çocuk tarihi`, which does not exist in the corpus, returns 9 documents because the words come from two separate tags. - Sorting is undefined in the text field: all 600 documents carry more than one term, the chosen rule leaves 9 keys, and all ten of the first ten documents swap places. Aggregation produces 17 buckets instead of 12, and `çocuk edebiyatı` splits into two separate buckets. ## Next Step The mapping in this lesson was written by hand: which field would be indexed with which type was known in advance. In a real catalog import, this information is often missing — records arrive from outside, each source carries its own fields, and the index itself makes the type decision. The next lesson measures this decision: how the field count grows as document diversity increases, what happens when the same name arrives with two different types, and how many bytes each field costs in index metadata.