Mappings and Field Types

The previous lesson’s three chains all applied the same rule to the entire text. But not every field in the catalog is text. The shelf code TR-866.3 marks a location, and splitting it makes no sense; the language Türkçe is a value; the subject tag çocuk edebiyatı (children’s literature) consists of two words but is a single category. The summary, however, really is text, and every word inside it is a separate search entry.

The declaration that states which rule a field is indexed by is called a mapping; the decision the mapping carries is called the field type. This lesson compares two types. A text field passes the value through the analyzer chain; as a result the field spreads across multiple terms. A keyword field does not split or transform the value: the entire value is a single term.

Two Types, the Same Field

The measurement runs over three fields: the multi-valued subjects, and the single-valued shelf and language. Each field is indexed twice, and four things are counted: index size, the set an exact-value query returns, whether sorting is defined, and the number of buckets aggregation produces.

IA1: the corpus is unchanged — 600 records, seed 20250317. IA2: the byte constants are the same constants as the previous lessons; position is not kept in this lesson. IA3: in a multi-valued field, the keyword type makes each value a separate term, the text type parses all the values as a single piece of text. IA4: because a text field can carry more than one term per document, a sort key must be chosen; here the “smallest term” rule is chosen.

// search/catalog.mjs — corpus shared by three topics: generated from a library catalog,
// 600 book records, seed 20250317. Fields: title, summary, subjects, author, year, language, shelf.
export const SEED = 20250317, DOC_COUNT = 600;
let state = SEED;
const random = () => (state = (state * 1103515245 + 12345) % 2147483648) / 2147483648;
const pick = (d) => d[Math.floor(random() * d.length)];
const pickSkewed = (d) => d[Math.floor(random() ** 2 * d.length)];   // frequency is skewed in real text
const splitBar = (s) => s.split("|");

// stem forms: base, genitive, plural, dative, ablative, locative
export const STEMS = splitBar("deniz denizin denizler denize denizden denizde|kitap kitabın kitaplar kitaba \
kitaptan kitapta|çocuk çocuğun çocuklar çocuğa çocuktan çocukta|şehir şehrin şehirler şehre \
şehirden şehirde|yol yolun yollar yola yoldan yolda|ada adanın adalar adaya adadan adada|bahçe \
bahçenin bahçeler bahçeye bahçeden bahçede|mektup mektubun mektuplar mektuba mektuptan mektupta|\
gemi geminin gemiler gemiye gemiden gemide|köprü köprünün köprüler köprüye köprüden köprüde|\
okul okulun okullar okula okuldan okulda|kuş kuşun kuşlar kuşa kuştan kuşta").map((s) => s.split(" "));
const PATTERNS = splitBar("0 {} ve gündelik hayat üzerine notlar sunar|0 {} bu derlemenin ana izleğidir|\
1 {} tarihine geniş yer ayırır|1 {} çevresinde gelişen olayları anlatır|2 {} üzerine derlenmiş \
yazılar içerir|2 {} hakkında kısa öyküler toplar|3 {} açılan bir yolculuğu izler|4 {} toplanmış \
belgeleri sıralar|5 {} tutulan günlüklerden seçmeler verir|5 {} geçen bölümleri İstanbul'un eski \
mahallelerine bağlar").map((s) => [Number(s[0]), s.slice(2)]);
const PATTERNS2 = splitBar("{Y} kütüphanesinde tutulan {N} üzerine kuruludur|{N} arasından seçilmiş \
örnekler taşır|{Y} ve çevresindeki {N} listesini verir|{Y} basımı bir {N} derlemesine dayanır");
const EXTRA = splitBar("denizci gelenekleri üzerine bir ek bölüm bulunur|Karadeniz kıyısındaki kasabaları \
anlatır|çocukluk anılarına yer verir|kitapçı raflarındaki dağılımı tartışır|yolculuk notlarıyla \
kapanır|adacıklardaki kuş türlerini sayar");
const PLACES = splitBar("Ankara|İzmir|Trabzon|Kars|Bursa|Edirne|Sinop|Antakya");
const OBJECTS = splitBar("harita|fotoğraf|söyleşi|günlük|arşiv belgesi|liman kaydı|kasaba adı|el yazması|gazete kupürü|şarkı sözü");
const PREFIXES = splitBar("Uzak|Kayıp|Sessiz|Eski|Kısa|Büyük|Küçük|Unutulmuş|Beyaz|Yedi");
const SUFFIXES = splitBar("Günleri|Öyküleri|Üzerine Notlar|Anıları|Sözlüğü|Rehberi|Yılları|Defteri");
const SUBJECTS = splitBar("çocuk edebiyatı|roman|kısa öykü|şiir|deniz tarihi|coğrafya|biyografi|gezi yazısı|halk bilimi|mimarlık|müzik|felsefe");
const FIRST_NAMES = splitBar("Ahmet|Ayşe|Zeynep|Cemal|Nuran|Selim|Elif|Kerem|Hatice|Bedri|Sevgi|Nazlı");
const LAST_NAMES = splitBar("Yılmaz|Kaya|Demir|Şahin|Çelik|Aydın|Doğan|Arslan|Koç|Ertem");
const LANGUAGES = splitBar("Türkçe|Türkçe|Türkçe|İngilizce|Almanca|Fransızca");

const capitalize = (s) => s[0].toLocaleUpperCase("tr") + s.slice(1);
function generateSummary() {
  const parts = [];
  for (let i = 0; i < 3; i += 1) { const [d, k] = pickSkewed(PATTERNS); parts.push(k.replace("{}", pickSkewed(STEMS)[d])); }
  parts.push(pick(PATTERNS2).replace("{Y}", pick(PLACES)).replace("{N}", pick(OBJECTS)));
  if (random() < 0.45) parts.push(pick(EXTRA));
  return capitalize(parts.join(", ")) + ".";
}
function generateTitle() {
  const k = pickSkewed(STEMS), o = pick(PREFIXES), s = pick(SUFFIXES), t = random();
  if (t < 0.25) return `${o} ${capitalize(k[2])}`;
  if (t < 0.5) return `${capitalize(k[0])} ${s}`;
  if (t < 0.75) return `${o} ${capitalize(k[0])} ${s}`;
  return `${capitalize(k[1])} ${s}`;
}
export const corpus = [];
for (let i = 1; i <= DOC_COUNT; i += 1) {
  const subjects = [pick(SUBJECTS)];
  if (random() < 0.55) subjects.push(pick(SUBJECTS));
  if (random() < 0.2) subjects.push(pick(SUBJECTS));
  corpus.push({
    id: i, title: generateTitle(), summary: generateSummary(), subjects: [...new Set(subjects)], author: `${pick(FIRST_NAMES)} ${pick(LAST_NAMES)}`,
    year: 1968 + Math.floor(random() * 57), language: pick(LANGUAGES),
    shelf: `${pick(splitBar("TR|EN|DE|FR"))}-${800 + Math.floor(random() * 99)}.${Math.floor(random() * 9)}`,
  });
}

// search/index.mjs — a hand-written inverted index: dictionary, posting list (document id, term
// frequency, position) and byte accounting. The analyzer is supplied from outside; later lessons use it.
export const SIMPLE = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean);
export const DOC_ID_BYTES = 4, FREQ_BYTES = 4, POSITION_BYTES = 4, DICTIONARY_OVERHEAD = 8;   // IA2: byte constants

export class InvertedIndex {
  dictionary = new Map();                // term -> posting list
  documentCount = 0;
  constructor({ analyze = SIMPLE, frequency = true, position = true } = {}) { Object.assign(this, { analyze, frequency, position }); }

  add(id, text) {
    const local = new Map();
    this.analyze(text).forEach((t, i) => (local.get(t) ?? local.set(t, []).get(t)).push(i));
    for (const [t, k] of local) {
      if (!this.dictionary.has(t)) this.dictionary.set(t, []);
      this.dictionary.get(t).push({ id, tf: k.length, position: this.position ? k : [] });
    }
    this.documentCount += 1;
  }

  postings(t) { return this.dictionary.get(t) ?? []; }

  search(...terms) {                     // intersection: lists are id-sorted, so one pass suffices
    const l = terms.map((t) => this.postings(t)), p = l.map(() => 0), results = [];
    let comparisons = 0;
    while (l.every((x, i) => p[i] < x.length)) {
      const maxId = Math.max(...l.map((x, i) => x[p[i]].id));
      let matched = true;
      for (let i = 0; i < l.length; i += 1) {
        while (p[i] < l[i].length && l[i][p[i]].id < maxId) { p[i] += 1; comparisons += 1; }
        comparisons += 1;
        if (p[i] >= l[i].length || l[i][p[i]].id !== maxId) { matched = false; break; }
      }
      if (matched) { results.push(maxId); p.forEach((_, i) => (p[i] += 1)); }
    }
    return { results, postings: l.reduce((t, x) => t + x.length, 0), comparisons };
  }

  bytes() {                              // dictionary + postings + positions
    let s = 0, g = 0, k = 0, postings = 0, positions = 0;
    for (const [t, list] of this.dictionary) {
      s += Buffer.byteLength(t) + DICTIONARY_OVERHEAD;
      for (const gr of list) {
        g += DOC_ID_BYTES + (this.frequency ? FREQ_BYTES : 0);
        k += gr.position.length * POSITION_BYTES;
        postings += 1; positions += gr.position.length;
      }
    }
    return { terms: this.dictionary.size, postings, positions, dictionary: s, postingBytes: g, positionBytes: k, total: s + g + k };
  }
}

// search/field-type.mjs — the same fields are indexed with two types: text field (goes through
// the analyzer) and keyword field (value stays a single term). Bytes, exact match, sort, aggregation.
import { corpus, DOC_COUNT, SEED } from "./catalog.mjs";
import { InvertedIndex } from "./index.mjs";

const SEPARATOR = "\u0001";                                  // value boundary
const fieldValues = { subjects: (b) => b.subjects, shelf: (b) => [b.shelf], language: (b) => [b.language] };
const TEXT = (s) => s.toLocaleLowerCase("tr").split(/[^\p{L}\p{N}]+/u).filter(Boolean);
const KEYWORD = (s) => s.split(SEPARATOR).filter(Boolean);   // value is not split
const build = (field, analyze) => {
  const d = new InvertedIndex({ analyze, position: false });
  for (const b of corpus) d.add(b.id, fieldValues[field](b).join(SEPARATOR));
  return d;
};
const index = new Map();
for (const field of Object.keys(fieldValues))
  for (const [type, analyze] of [["text", TEXT], ["keyword", KEYWORD]])
    index.set(`${field}/${type}`, build(field, analyze));

console.log(`seed ${SEED}; corpus ${DOC_COUNT} docs; fields: subjects (multi-valued), shelf, language`);
console.log(`\n${"field".padEnd(9)}${"type".padEnd(9)}${"terms".padStart(7)}${"postings".padStart(9)}` +
  `${"index bytes".padStart(13)}${"sample terms".padStart(28)}`);
for (const [key, d] of index) {
  const [field, type] = key.split("/"), b = d.bytes();
  console.log(field.padEnd(9) + type.padEnd(9) + String(b.terms).padStart(7) + String(b.postings).padStart(9) +
    String(b.total).padStart(13) + [...d.dictionary.keys()].slice(0, 2).join(" | ").padStart(28));
}

const commonShelf = [...corpus.reduce((m, b) => m.set(b.shelf, (m.get(b.shelf) ?? 0) + 1), new Map())]
  .sort((x, y) => y[1] - x[1])[0][0];                        // the shelf code repeated most often in the corpus
const queries = [["subjects", "çocuk edebiyatı", "exact tag"], ["subjects", "çocuk", "single word of tag"],
  ["subjects", "çocuk tarihi", "tag not present"], ["shelf", commonShelf, "exact shelf code"], ["shelf", "TR", "shelf prefix"],
  ["language", "Türkçe", "value as written"], ["language", "türkçe", "value lowercased"]];
console.log(`\n${"field".padEnd(9)}${"query".padEnd(18)}${"description".padEnd(20)}${"text".padStart(8)}${"keyword".padStart(10)}`);
for (const [field, s, note] of queries) {
  const m = index.get(`${field}/text`).search(...TEXT(s)).results.length;
  const k = index.get(`${field}/keyword`).search(s).results.length;
  console.log(field.padEnd(9) + s.padEnd(18) + note.padEnd(20) + String(m).padStart(8) + String(k).padStart(10));
}

// --- sorting: a keyword field has one term per document, a text field can have more than one ---
const termsOf = (d, id) => [...d.dictionary].filter(([, l]) => l.some((g) => g.id === id)).map(([t]) => t);
const textShelf = index.get("shelf/text"), keywordShelf = index.get("shelf/keyword");
const multiTerm = corpus.filter((b) => termsOf(textShelf, b.id).length > 1).length;
const smallest = (d, id) => termsOf(d, id).sort()[0];
const sortBy = (d) => corpus.map((b) => [smallest(d, b.id), b.id]).sort((x, y) => (x[0] < y[0] ? -1 : x[0] > y[0] ? 1 : x[1] - y[1]));
const keywordOrder = sortBy(keywordShelf), textOrder = sortBy(textShelf);
const different = keywordOrder.slice(0, 10).filter(([, id], i) => textOrder[i][1] !== id).length;
console.log(`\nsort order (by shelf field): in the text type, ${multiTerm} documents have more than one term in the field; ` +
  `${new Set(textOrder.map(([t]) => t)).size} distinct sort keys`);
console.log(`in the keyword type, ${new Set(keywordOrder.map(([t]) => t)).size} distinct keys; ` +
  `${different} of the first ten documents differ between the two orders`);
console.log(`  keyword first three: ${keywordOrder.slice(0, 3).map(([t, i]) => `${t} (#${i})`).join(", ")}`);
console.log(`  text first three   : ${textOrder.slice(0, 3).map(([t, i]) => `${t} (#${i})`).join(", ")}`);

// --- aggregation: bucket count on the subjects field ---
const buckets = (d) => [...d.dictionary].map(([t, l]) => [t, l.length]).sort((x, y) => y[1] - x[1] || (x[0] < y[0] ? -1 : 1));
for (const type of ["keyword", "text"]) {
  const result = buckets(index.get(`subjects/${type}`));
  console.log(`\nsubjects aggregation (${type}): ${result.length} buckets, ${result.reduce((t, [, n]) => t + n, 0)} total document count`);
  console.log(`  largest three: ${result.slice(0, 3).map(([t, n]) => `${t} ${n}`).join(", ")}`);
}

seed 20250317; corpus 600 docs; fields: subjects (multi-valued), shelf, language

field    type       terms postings  index bytes                sample terms
subjects text          17     1430        11686            biyografi | kısa
subjects keyword       12      999         8203       biyografi | kısa öykü
shelf    text         112     1800        15610                    en | 893
shelf    keyword      549      600        13584         EN-893.5 | FR-881.4
language text           4      600         4866         almanca | fransızca
language keyword        4      600         4867         Almanca | Fransızca

field    query             description             text   keyword
subjects çocuk edebiyatı   exact tag                 93        93
subjects çocuk             single word of tag        93         0
subjects çocuk tarihi      tag not present            9         0
shelf    TR-866.3          exact shelf code           3         3
shelf    TR                shelf prefix             154         0
language Türkçe            value as written         310       310
language türkçe            value lowercased         310         0

sort order (by shelf field): in the text type, 600 documents have more than one term in the field; 9 distinct sort keys
in the keyword type, 549 distinct keys; 10 of the first ten documents differ between the two orders
  keyword first three: DE-801.1 (#460), DE-801.8 (#94), DE-802.5 (#594)
  text first three   : 0 (#19), 0 (#31), 0 (#42)

subjects aggregation (keyword): 12 buckets, 999 total document count
  largest three: coğrafya 95, çocuk edebiyatı 93, biyografi 88

subjects aggregation (text): 17 buckets, 1430 total document count
  largest three: coğrafya 95, edebiyatı 93, çocuk 93

The Byte Difference Depends on the Field Itself

The first table does not give a single rule, because which type is cheaper depends on the field. In the subjects field the text type produces both more terms (17 against 12) and more posting entries (1,430 against 999), and at 11,686 bytes it comes in above the keyword type’s 8,203 bytes: once two-word tags are split, each document leaves two entries in the field.

In the shelf field the direction reverses. The text type shrinks the dictionary — 112 pieces instead of 549 separate codes, because the same number and prefix repeat across hundreds of codes — but triples the posting entries: 1,800 instead of 600. In total the text type is again more expensive (15,610 against 13,584), but this time the expense comes from the posting side. The language field is the borderline case where the two types differ by only one byte: because the value is already a single word, the type decision does not change the size, it only preserves the spelling.

Exact Matching, Sorting, and Aggregation

The second table shows the decision’s effect on the set. When the exact tag is asked for, both types return the same 93 documents; the distinction is in the other rows. When the tag’s single word is asked for, the text field gives 93 documents, the keyword field gives 0: the text field is open to partial matching, the keyword field plainly is not. The same behavior repeats in the shelf field with the TR prefix: 154 documents against 0.

The third row is the real warning. A tag reading çocuk tarihi does not exist in the corpus; despite this, the text field returns 9 documents. These are documents that carry the tags çocuk edebiyatı and deniz tarihi together: because the field is multi-valued, words coming from two separate values behave as a single match. The keyword field does not bring back these documents, because for it every tag is an unsplit whole. The cost in the opposite direction shows up in the language row: when asked with the spelling türkçe, the keyword field returns 0 documents, because the value carries a capital letter.

The sorting difference is sharper. In the keyword field every document has exactly one term, and the shelf code produces 549 separate sort keys; sorting is defined. In the text field all 600 documents have more than one term in the field, so the question “sorted by which term” has no answer. Once the chosen rule takes the smallest term, only 9 keys remain, and all ten of the first ten documents swap places: sorting becomes meaningless in practice.

On the aggregation side, too, the two types pull two different tables out of the same data. The keyword field gives 12 buckets and a document count of 999 — this is the exact count of tags in the corpus. The text field gives 17 buckets and a count of 1,430; the çocuk edebiyatı tag has split into two buckets, çocuk and edebiyatı, each counting 93 and landing at the top of the list. In a report that wants to see the category distribution, these two tables answer the same question differently.

Summary

• A mapping declares which rule a field is indexed by; a text field passes the value through the analyzer and splits it into terms, a keyword field keeps the value as a single, unsplit term.
• The direction of the byte difference depends on the field: in subjects, the text type is more expensive at 11,686 against 8,203 bytes; in shelf, it shrinks the dictionary from 549 to 112 terms but triples the posting entries from 600 to 1,800.
• Exact matching is only exact in a keyword field: the çocuk query returns 93 documents in the text field and 0 in the keyword field; the TR prefix gives 154 against 0.
• In a multi-valued field, the text type produces a cross-match: the tag çocuk tarihi, which does not exist in the corpus, returns 9 documents because the words come from two separate tags.
• Sorting is undefined in the text field: all 600 documents carry more than one term, the chosen rule leaves 9 keys, and all ten of the first ten documents swap places. Aggregation produces 17 buckets instead of 12, and çocuk edebiyatı splits into two separate buckets.

Next Step

The mapping in this lesson was written by hand: which field would be indexed with which type was known in advance. In a real catalog import, this information is often missing — records arrive from outside, each source carries its own fields, and the index itself makes the type decision. The next lesson measures this decision: how the field count grows as document diversity increases, what happens when the same name arrives with two different types, and how many bytes each field costs in index metadata.