---
title: 'Bulk Indexing Optimization'
source: 'https://academia.sh/en/courses/search-engines/bulk-indexing-optimization'
course: 'Search Engines and Text Retrieval'
language: en
updated: '2026-08-23T07:00:50+00:00'
license: 'CC BY-SA 4.0'
---

# Bulk Indexing Optimization

Indexing the same four-thousand-record corpus one at a time and in bulk: comparing them by request count, segments produced, total index bytes, and dictionary lookups per query, showing that batch size and refresh interval are two independent axes, and measuring the trade-off between visibility lag and query cost as the refresh interval changes, counted on a six-hundred-record donation batch by matched documents, missing documents, and the order of the top five results.

The previous lesson counted the 600 records arriving during the split one by one, quietly
assuming two things: that each record is a separate request, and that it becomes searchable
the moment it is accepted. Neither was measured. A catalog batch arrives in bulk; sending
600 records as 600 requests is not the same as sending them as 12. A document entering the
index does not become searchable instantly either: it is first written to a buffer, and only
becomes visible to queries once a refresh operation turns the buffer into a segment. This
lesson measures two settings separately — batch size and refresh interval.

## Setup

A **segment** is an immutable inverted index built from one buffer's worth of documents; a
new document is never added to an existing segment, a new segment is born instead. A query
must look at every segment: it does one dictionary lookup per term in each segment and reads
the posting lists. Because scoring statistics are collected across every segment, segment
count changes cost, not order; the only effect on order comes from **visibility**.

Time is measured too, but raw time is not printed: time depends on the environment, and
because the in-process model carries no network, it cannot measure the per-request network
cost. What is printed is whether time sits above or below a threshold; the numbers carrying
the real weight are requests, segments, bytes, and dictionary lookups.

**CO7 — the 600-record donation batch enters the catalog in requests of 50, and the same
query runs right after every request.** The effect is linear: if request size changes, the
invisible-document count changes in the same direction. **CO8 — refresh runs in the
background every `interval` documents and turns the buffer into a single segment.** A
document waiting in the buffer is invisible to the query; that is the model's rule.

```js
// cluster/corpus.mjs — library catalog corpus and inverted index (same generator and seed as the
// earlier lessons). buildIndex() here builds A SINGLE SEGMENT: a segment is an immutable inverted
// index built from one buffer's worth of documents.
const COMMON = ["book", "author", "work", "text", "chapter", "edition", "page", "language", "volume", "publication"];
const SPECIAL = {
  child: ["tale", "illustrated", "school", "play", "animal", "cartoon"], story: ["short", "narrative", "collection", "diary", "loneliness", "village"],
  novel: ["hero", "borough", "generation", "house", "journey", "letter"], history: ["empire", "document", "archive", "war", "century", "chronicle"],
  travel: ["sea", "road", "map", "city", "harbor", "ship"], poetry: ["verse", "meter", "image", "sound", "silence", "rhyme"],
  essay: ["thought", "critique", "reading", "time", "note", "conversation"], science: ["measurement", "experiment", "theory", "data", "observation", "equation"],
};
const RARE = ["lighthouse", "well", "silk", "tower", "garden", "snow", "island", "grove", "stone", "sycamore", "swallow", "blacksmith", "compass", "amber"];
const TOPIC = Object.keys(SPECIAL);

export function corpus({ count = 4000, seed = 20260731 } = {}) {
  let s = seed % 2147483647;
  const r = () => (s = (s * 48271) % 2147483647) / 2147483647;
  const pick = (a) => a[Math.floor(r() * a.length)], docs = [];
  for (let i = 1; i <= count; i += 1) {
    const topic = pick(TOPIC), sp = SPECIAL[topic], terms = [pick(sp), pick(RARE)];
    if (r() < 0.5) terms.push(pick(COMMON));
    for (let j = 0, n = 10 + Math.floor(r() * 7); j < n; j += 1)
      terms.push(r() < 0.45 ? pick(COMMON) : r() < 0.85 ? pick(sp) : pick(RARE));
    docs.push({ id: i, topic, year: 1990 + Math.floor(r() * 36), text: [terms[0], terms[1], topic, ...terms.slice(2)].join(" ") });
  }
  return docs;
}

export function buildIndex(docs) {
  const postings = new Map(), lengths = new Map();
  let bytes = 0;
  for (const d of docs) {
    const t = d.text.split(" "), counts = new Map();
    for (const x of t) counts.set(x, (counts.get(x) ?? 0) + 1);
    lengths.set(d.id, t.length);
    for (const [x, n] of counts) {
      if (!postings.has(x)) { postings.set(x, []); bytes += x.length + 4; }   // dictionary entry
      postings.get(x).push([d.id, n]); bytes += 8;
    }
  }
  const avg = [...lengths.values()].reduce((a, b) => a + b, 0) / (docs.length || 1);
  return { postings, lengths, N: docs.length, avg, bytes };
}

// Scoring: term frequency, inverse document frequency, and document length. In LOCAL scoring df
// and average length are read only from this index; if global is given, the whole corpus's
// statistics are used instead.
export function search(ix, terms, k, global = null) {
  const N = global ? global.N : ix.N, avg = global ? global.avg : ix.avg, scores = new Map();
  let scanned = 0;
  for (const t of terms) {
    const g = ix.postings.get(t) ?? [];
    const df = global ? global.df.get(t) ?? 0 : g.length;
    const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5));
    for (const [id, tf] of g) {
      scanned += 1;
      const norm = tf + 1.2 * (0.25 + 0.75 * ix.lengths.get(id) / avg);
      scores.set(id, (scores.get(id) ?? 0) + idf * tf * 2.2 / norm);
    }
  }
  const sorted = [...scores].sort((a, b) => b[1] - a[1] || a[0] - b[0]).slice(0, k);
  return { candidates: sorted.map(([id, p]) => ({ id, p })), scanned, matched: scores.size };
}
```

```js
// cluster/bulk.mjs — measuring one-at-a-time versus bulk indexing, and the refresh interval. The
// index is modeled IN-PROCESS: documents are first written to a buffer, REFRESH turns the buffer
// into an immutable segment, and only then does the document become visible to queries. Scoring
// statistics are collected across every segment, so segment count changes only cost, not order.
import { corpus, buildIndex, search } from "./corpus.mjs";

const N = 4000, NEW = 600, REQUEST = 50, K = 10, MEASURE = 100, QUERY = ["short", "story", "loneliness"];
const pad = (x, n) => String(x).padStart(n);

const empty = () => ({ segments: [], buffer: [], requests: 0, lookups: 0 });
const add = (D, batch) => { D.requests += 1; D.buffer.push(...batch); };
const refresh = (D) => { if (D.buffer.length) { D.segments.push(buildIndex(D.buffer)); D.buffer = []; } };

function ask(D, terms, k) {
  const df = new Map();
  let n = 0, len = 0;
  for (const b of D.segments) {                    // first pass: cross-segment statistics
    n += b.N; len += b.avg * b.N;
    for (const t of terms) { D.lookups += 1; const g = b.postings.get(t); if (g) df.set(t, (df.get(t) ?? 0) + g.length); }
  }
  const glo = { df, N: n, avg: len / n }, scores = [];
  for (const b of D.segments) { D.lookups += terms.length; scores.push(...search(b, terms, Infinity, glo).candidates); }
  scores.sort((a, b) => b.p - a.p || a.id - b.id);
  return { top: scores.slice(0, k).map((x) => x.id), matched: scores.length, visible: n };
}

// Send N documents in batches of `batch`; a refresh runs every `interval` documents.
function build(docs, batch, interval) {
  const D = empty();
  for (let i = 0; i < docs.length; i += batch) {
    add(D, docs.slice(i, i + batch));
    if ((i + batch) % interval === 0 || i + batch >= docs.length) refresh(D);
  }
  return D;
}

// Time measurement: indexing + MEASURE queries. Raw time DEPENDS ON THE ENVIRONMENT, so below
// only whether it is above or below a threshold is printed. The smallest of three runs is kept.
function time(batch, interval) {
  let best = Infinity;
  for (let k = 0; k < 3; k += 1) {
    const t0 = process.hrtime.bigint();
    const D = build(base, batch, interval);
    for (let q = 0; q < MEASURE; q += 1) ask(D, QUERY, K);
    best = Math.min(best, Number(process.hrtime.bigint() - t0));
  }
  return best;
}

const base = corpus({ count: N });
console.log(`${N} documents, seed 20260731. Query "short story loneliness", final list ${K}.`);
console.log("Time DEPENDS ON THE ENVIRONMENT; only a thresholded ratio is printed instead of raw time.\n");
console.log("batch | refresh interval | requests | segments | index bytes | dictionary lookups per query");
console.log("------|------------------|----------|----------|-------------|------------------------------");
for (const [batch, interval] of [[1, 1], [1, 500], [500, 500], [1000, 1000], [500, N]]) {
  const D = build(base, batch, interval);
  D.lookups = 0;
  ask(D, QUERY, K);
  console.log(`${pad(batch, 5)} | ${pad(interval, 16)} | ${pad(D.requests, 8)} | ${pad(D.segments.length, 8)} | ` +
    `${pad(D.segments.reduce((t, b) => t + b.bytes, 0), 11)} | ${pad(D.lookups, 30)}`);
}
time(500, 500);                                    // warm-up run
const ratio = time(1, 1) / time(500, 500);
console.log(`\nindexing + ${MEASURE} query time, one-at-a-time (1/1) over bulk (500/500): ` +
  `${ratio > 2 ? "more than 2x" : "2x or under"} (raw time depends on the environment, not printed)`);

// Visibility: a 600-record donation batch enters the 4000-document index in batches of 50, and the
// same query runs right after each request. Refresh runs in the background every `interval`
// documents; unrefreshed documents wait in the buffer and are invisible to the query.
const donation = corpus({ count: 9000, seed: 20260801 }).filter((d) => d.topic === "story")
  .slice(0, NEW).map((d, i) => ({ ...d, id: N + i + 1 }));
const baseline = ask(build([...base, ...donation], 500, 500), QUERY, K);
console.log(`\nrefresh interval (a ${NEW}-record batch enters in requests of ${REQUEST}, one query after each request):`);
console.log(`baseline: matched ${baseline.matched} once the batch is fully visible, top 5: ${baseline.top.slice(0, 5).join(" ")}`);
console.log("interval | segments added | avg. invisible | last query matched | missing | top 10 shared | lookups | top 5 result");
for (const interval of [1, 75, 250, NEW * 2]) {
  const D = build(base, 500, 500), baseSegments = D.segments.length;
  let invisible = 0, visible = 0, last = null;
  for (let g = REQUEST; g <= NEW; g += REQUEST) {
    const target = Math.floor(g / interval) * interval;
    while (visible < target) {                     // every refresh creates one segment
      add(D, donation.slice(visible, visible + interval)); refresh(D); visible += interval;
    }
    D.lookups = 0;
    last = ask(D, QUERY, K);
    invisible += g - visible;
  }
  console.log(`${pad(interval > NEW ? "none" : interval, 8)} | ${pad(D.segments.length - baseSegments, 15)} | ` +
    `${pad((invisible / (NEW / REQUEST)).toFixed(1), 14)} | ${pad(last.matched, 19)} | ${pad(baseline.matched - last.matched, 7)} | ` +
    `${pad(last.top.filter((x) => baseline.top.includes(x)).length + "/10", 14)} | ${pad(D.lookups, 7)} | ${last.top.slice(0, 5).join(" ")}`);
}
console.log(`\nrun-independent: request count = documents / batch size, segment count = documents / refresh` +
  ` interval, dictionary lookups per query = segments x terms x 2.`);
```

```
4000 documents, seed 20260731. Query "short story loneliness", final list 10.
Time DEPENDS ON THE ENVIRONMENT; only a thresholded ratio is printed instead of raw time.

batch | refresh interval | requests | segments | index bytes | dictionary lookups per query
------|------------------|----------|----------|-------------|------------------------------
    1 |                1 |     4000 |     4000 |      869327 |                          24000
    1 |              500 |     4000 |        8 |      388600 |                             48
  500 |              500 |        8 |        8 |      388600 |                             48
 1000 |             1000 |        4 |        4 |      385304 |                             24
  500 |             4000 |        8 |        1 |      382832 |                              6

indexing + 100 query time, one-at-a-time (1/1) over bulk (500/500): more than 2x (raw time depends on the environment, not printed)

refresh interval (a 600-record batch enters in requests of 50, one query after each request):
baseline: matched 1115 once the batch is fully visible, top 5: 4457 4534 4544 3159 3544
interval | segments added | avg. invisible | last query matched | missing | top 10 shared | lookups | top 5 result
       1 |             600 |            0.0 |                1115 |       0 |          10/10 |    3648 | 4457 4534 4544 3159 3544
      75 |               8 |           25.0 |                1115 |       0 |          10/10 |      96 | 4457 4534 4544 3159 3544
     250 |               2 |           95.8 |                1015 |     100 |           8/10 |      60 | 4457 3159 3544 2689 2362
    none |               0 |          325.0 |                 515 |     600 |           6/10 |      48 | 3159 3544 2689 2362 3370

run-independent: request count = documents / batch size, segment count = documents / refresh interval, dictionary lookups per query = segments x terms x 2.
```

## Two Separate Axes

The first table's first row is the worst case: every document arrives as its own request and
a refresh runs after every request. The result is 4000 requests, 4000 segments, an
869,327-byte index, and 24,000 dictionary lookups for a single query. The fifth row indexes
the same corpus with 8 requests and a single segment: 382,832 bytes and 6 lookups per query.
Same 4000 documents, same order, four thousand times fewer dictionary lookups.

The byte count more than doubling is not caused by the posting lists but by the dictionary:
the corpus's 80 terms are rewritten 4000 times, once per segment. There is a fixed cost per
segment, and as segment count grows this cost accumulates.

The second row gives this lesson's real distinction. Batch size is 1 but refresh interval is
500: request count is still 4000, yet segment count is 8, bytes are 388,600, and lookups are
48. **Batch size determines request count, refresh interval determines segment count.** They
are two separate settings and cost two separate things. Bulk indexing reduces the
per-request network and parsing overhead; because the in-process model does not carry that
overhead, the time ratio printed here reflects only the segment cost — measured together,
indexing plus a hundred queries under one-at-a-time placement takes more than twice as long
as under bulk placement.

## Refresh Interval: Visibility and Throughput

The second table feeds a 600-record donation batch into the 4000-document index in requests
of 50 and runs the same query right after every request. The baseline is the response once
the whole batch is visible: 1115 matches, with 4457, 4534, 4544 in the top five.

At interval 1, no document waits: average invisible is 0.0 and the last query is identical
to the baseline. The cost is 600 new segments and 3648 dictionary lookups per query — 76
times the no-refresh case, because every query looks individually at six hundred
one-document segments.

Interval 75 is this lesson's best-balanced row: 8 segments, an average of 25 documents
waiting in the buffer, yet the last query still returns 1115 matches and 10/10. Visibility
lag exists without destroying result accuracy; query cost stays at 96 lookups.

At interval 250 the lag reaches the returned set. An average of 95.8 documents are invisible
and the last query returns 1015 documents: **100 records missing**. The top ten drops to
8/10 and the top five changes — 4534 and 4544 drop off the list, replaced by older records.
With refresh off the table reaches its extreme: an average of 325 documents are invisible,
the last query answers with 515 documents, **all 600 records are missing**, and not one book
from the donation batch appears in the top five. The query is at its cheapest, 48 lookups;
the returned list is the list of a catalog the batch never reached.

The trade-off does not run in one direction only. Shortening the interval lowers visibility
lag but produces segments; lengthening it cheapens the query but pulls the returned set
backward. Somewhere in between sits an interval where lag is measurable but does not reach
the result — in this corpus, 75.

## Summary

- Indexing the same 4000 documents one at a time produced 4000 requests, 4000 segments,
  869,327 bytes, and 24,000 dictionary lookups per query; with batches of 500 and a single
  refresh it produced 8 requests, 1 segment, 382,832 bytes, and 6 lookups.
- The byte increase comes not from the posting lists but from the dictionary: 80 terms are
  repeated across 4000 segments.
- Batch size determines request count, refresh interval determines segment count; combining
  a batch of 1 with an interval of 500 keeps requests at 4000 while segments drop to 8.
- Measured together, indexing plus a hundred queries took more than twice as long under
  one-at-a-time placement as under bulk placement; because raw time depends on the
  environment, only the threshold was printed.
- At a refresh interval of 1, invisible documents were 0 but lookups per query were 3648; at
  interval 75, an average of 25 documents waited and the result was still 10/10.
- At interval 250 the last query missed 100 records and 4534 and 4544 dropped from the top
  five; with refresh off, all 600 records stayed missing and the top ten fell to 6/10.

## Next Step

This lesson treated the segment as a cheap byproduct and only counted how many there were.
The tables show this cannot last: one-at-a-time indexing left 4000 segments behind, the
interval-1 placement added 600 segments, and every new segment brought one more fixed cost
per query. Because segments are immutable, a deleted document also stays in place — it is
only marked deleted — and as the index grows, unread postings accumulate. The only way to
reclaim this overhead is to **merge** segments. The next lesson measures the background cost
of turning small segments into large ones: bytes read and rewritten during a merge, the
space deleted documents give back, and how the number of segments per query changes while a
merge is running.