--- title: 'Indexing and Updating' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/indexing-and-updating' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the index as a snapshot and measures staleness as a degradation separate from the correctness of the retrieved chunk.' - 'Sweeps the refresh interval and prints the stale chunk rate, concept hit rate and up-to-date answer rate side by side.' - 'Compares the embedding cost of full reindexing and incremental refresh on the same change stream.' - 'Computes the gain from lengthening the interval as re-embedded chunks per up-to-date answer and compares it with its price.' --- # Indexing and Updating Measuring that the index is a snapshot and goes stale as the source changes: in a fictional twelve-period change stream, when the refresh interval goes from one to twelve the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875. The retrieval column shows none of this degradation: retrieval recall is 0.9500 and concept hit rate 0.9250 at all six intervals; the only column that falls is the share of answers whose copied validity record is up to date. The cost of refreshing reads differently under the two policies: full reindexing embeds 8316 chunks at interval one, incremental refresh embeds 881. Multiplying the interval by twelve lowers the re-embedded chunks per up-to-date answer under incremental refresh only from 1.98 to 1.77, a gain of eleven percent; the price of the same decision is 0.3125 in the up-to-date answer rate. The difference between the two runs is 0.0000 in every row. The previous three lessons took the corpus as **static**. The documents were prepared once, chunked once, embedded once; after the index was built it was never touched again. Field records do not behave that way. A procedure is republished, a threshold is updated, a record is withdrawn. When the source changes, the index does not; the index is a **snapshot**, and unless it is written down when it was taken, what it shows is not known. The degradation this lesson measures is of a different kind from the earlier ones. Retrieval does not bring the wrong chunk: the retrieved chunk is still from the right document and the right concept. What degrades is the chunk's **content**. That is why the retrieval column stays silent and the fault shows only in the generation column. - **RP21.** The code is a **simulator**; the corpus, the change stream and the queries are **fictional**, seed **20260218**. The embedding and the language model are the definitions of the previous lessons. - **RP22.** Each document carries a **validity record** (the string `validity record` in the data). When the document is republished the record goes up by one; when a chunk is served from the index it is given together with its record. The change stream is twelve periods, and in each period **0.10** of the documents are republished. - **RP23.** A republication changes the chunk's record, not its **body**. This is an assumption that simplifies the measurement: because the embedding does not move, the retrieval ranking stays the same at all intervals, and it is certain that the column that falls falls **only** from staleness. - **RP24.** One behavior is added to the simulator: the model **copies as is the validity record** in the first source chunk that carries the surface of the concept it chose. This is what a quoting model does, and it is the place where staleness passes into the answer. - **RP25.** The **stale chunk rate** is the rate at which the records of the served chunks disagree with the source's actual record in that period. The **up-to-date answer rate** is the rate of answers whose concept is correct **and** whose copied record equals the source's current record. - **RP26.** Two refresh policies are measured: **full reindexing** re-embeds all chunks at every refresh, **incremental refresh** only the chunks of documents that changed in that interval. A document that changed twice in one interval is embedded once. - **RP27.** The measurement is taken in **all** twelve periods; forty queries are asked in each period, that is, 480 requests for each interval. The stale chunk rate and the number of re-embedded chunks are independent of the run. ## Where Staleness Passes into the Answer The first block sets up the corpus, puts the change stream and the validity record on top of it, defines the simulator together with the copying behavior and sweeps the refresh interval. ```python # FICTIONAL. The corpus, concept axis, embedding, change stream and language model are a simulator; # no real endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) import math # ---- FICTIONAL change stream: twelve periods, a fraction of the documents is republished each period PERIODS = 12 CHANGES = [] for t in range(1, PERIODS + 1): r = rng(SEED + 6007 * t) CHANGES.append([i for i in range(len(DOCUMENTS)) if r() < 0.10]) def stamp(i, t): """The validity record of document i at the end of period t.""" return "Y%02d" % sum(i in CHANGES[u] for u in range(t)) CHUNKS = [{"doc": i, "code": d["code"], "label": d["label"], "body": p, "concepts": [k for k in KEYS if CONCEPTS[k][1] in p]} for i, d in enumerate(DOCUMENTS) for p in d["paras"]] VEC = [embed(x["body"]) for x in CHUNKS] # the stamp does not change the embedding TOP = [] for s in QUERIES: q = embed(s["text"]) TOP.append(sorted(range(len(CHUNKS)), key=lambda i: -cosine(q, VEC[i]))[:3]) # ---- FICTIONAL language model simulator: the previous lessons' simulator, one addition # copying the validity record it sees in the context WINDOW = 240 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 w = [math.exp(p[a] / temperature) for a in sorted(p)] y = sorted(p)[pick(r(), [x / sum(w) for x in w])] for x in prompt.split(" . "): # copies the record in the source as is if CONCEPTS[y][1] in x and "validity record" in x: y += " " + x.split("validity record ")[1].split()[0] break return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y} INSTRUCTION = ("Link the request to one of these concepts and write down the validity entry of the source: " + " , ".join(KEYS) + " . Reply with one concept and the entry . ") def run(interval, seed=SEED): COUNTER[0] = 0 stale = concept = current = recall = n = 0 embedded = full = 0 for t in range(1, PERIODS + 1): last = (t // interval) * interval # the last period the index saw if t % interval == 0: # a refresh occurs this period changed = {i for u in range(t - interval, t) for i in CHANGES[u]} embedded += sum(x["doc"] in changed for x in CHUNKS) full += len(CHUNKS) for s, top in zip(QUERIES, TOP): texts = {i: "%s validity record %s" % (CHUNKS[i]["body"], stamp(CHUNKS[i]["doc"], last)) for i in top} stale += sum(stamp(CHUNKS[i]["doc"], last) != stamp(CHUNKS[i]["doc"], t) for i in top) / 3 recall += any(s["concept"] in CHUNKS[i]["concepts"] for i in top) y = call(INSTRUCTION + "Source: " + " . ".join(texts[i] for i in top) + " . Request: " + s["text"], seed) kv, _, rv = y["response"].partition(" ") rec = next((i for i in top if CONCEPTS[kv][1] in texts[i]), None) if kv in KEYS \ else None concept += kv == s["concept"] current += (kv == s["concept"] and rec is not None and rv == stamp(CHUNKS[rec]["doc"], t)) n += 1 return stale / n, recall / n, concept / n, current / n, embedded, full print("periods %d, total republications %d, documents never changed %d" % (PERIODS, sum(len(x) for x in CHANGES), sum(1 for i in range(len(DOCUMENTS)) if all(i not in d for d in CHANGES)))) print("interval refreshes embedded(incremental) embedded(full) stale chunks recall concept current") for a in (1, 2, 3, 4, 6, 12): st, rc, cn, cu, em, fl = run(a) print("%8d %9d %21d %14d %12.4f %7.4f %8.4f %8.4f" % (a, PERIODS // a, em, fl, st, rc, cn, cu)) ``` ``` periods 12, total republications 254, documents never changed 51 interval refreshes embedded(incremental) embedded(full) stale chunks recall concept current 1 12 881 8316 0.0000 0.9500 0.9250 0.9250 2 6 839 4158 0.0347 0.9500 0.9250 0.8917 3 4 774 2772 0.0625 0.9500 0.9250 0.8771 4 3 741 2079 0.0937 0.9500 0.9250 0.8208 6 2 662 1386 0.1278 0.9500 0.9250 0.7708 12 1 519 693 0.2562 0.9500 0.9250 0.6125 ``` The first reading of the table is the clearest reason for keeping the two columns apart. **Retrieval recall is 0.9500 in all six rows, and the concept hit rate is 0.9250 in all six rows.** As the refresh interval goes from one period to twelve, **nothing** changes in the retrieval column, because the index keeps bringing the right documents. **0.2562** of the chunks it brings now show a text that is no longer in the source, and none of the retrieval measures says so. The only column that falls is the up-to-date answer rate: **from 0.9250 to 0.6125**. The 0.3125 between them is more than four times the threshold. The user-visible counterpart of this is the following: the model finds the right procedure, names the right concept and writes beside it a record that has been **withdrawn**. This answer is not a hallucination; it is faithful to the source. The source itself is stale. Grounding in the source does not guarantee the freshness of the source. The stale chunk rate and the wrong answer rate do not coincide one to one either. At interval twelve **0.2562** of the chunks are stale but **0.3875** of the answers are wrong. The difference comes from the concept hit rate already being 0.9250, and from the probability that a stale chunk is the chunk carrying the chosen concept carrying a distribution of its own. Staleness is an **input** measure, the wrong answer is an **output** measure; they are not the same number and cannot be written in place of each other. ## The Price of Refreshing The embedding cost behaves entirely differently under the two policies. **Full reindexing** re-embeds six hundred ninety-three chunks at every refresh: **8316** at interval one, **693** at interval twelve. Under this policy the interval is a lever that divides the cost directly. **Incremental refresh** embeds only the chunks of changed documents: **881** at interval one, that is, **9.4 times** smaller than full reindexing. The second block reads these two policies per correct piece of work and tests the stability of the measurement itself. ```python print("interval wrong answer rate incremental/current full/current 2nd seed current difference") for a in (1, 2, 4, 12): st, rc, cn, cu, em, fl = run(a) c2 = run(a, SEED + 1)[3] n = cu * PERIODS * len(QUERIES) print("%8d %17.4f %19.2f %12.2f %16.4f %10.4f" % (a, 1 - cu, em / n, fl / n, c2, abs(cu - c2))) ``` ``` interval wrong answer rate incremental/current full/current 2nd seed current difference 1 0.0750 1.98 18.73 0.9250 0.0000 2 0.1083 1.96 9.71 0.8917 0.0000 4 0.1792 1.88 5.28 0.8208 0.0000 12 0.3875 1.77 2.36 0.6125 0.0000 ``` The incremental column shows why the interval is the wrong lever. When the interval goes from one to twelve, the embedded chunks fall from 881 to 519, that is, only **forty-one percent**. The reason is that a document republished several times in one interval is embedded only once; as the interval lengthens this merging increases, but the total number of changed documents does not change. Read per up-to-date answer, the gain melts further: **from 1.98 to 1.77**, that is, eleven percent. The price of the same decision is **0.3125** in the up-to-date answer rate. Lengthening the interval under incremental refresh is selling a quarter of the accuracy for next to nothing. In the full reindexing column the table looks the other way round: per up-to-date answer, **from 18.73 to 2.36**. This row does not justify lengthening the interval; it justifies giving up full reindexing. Setting interval one with the incremental policy is both cheaper than setting interval twelve with full reindexing (881 against 693, the same order of magnitude) and 0.3125 more accurate. The last two columns are the stability of the measurement itself. Run with the second seed, the up-to-date answer rate comes out **exactly the same** in all four rows, a difference of **0.0000**. This is the continuation of the first lesson's finding: the output of a call whose source has been given does not depend on the seed. For that reason all the differences in the table are far above the threshold and have been **measured**. ## Summary - The index is a snapshot; when the source changes the index does not, and in the time that passes it serves the right chunk with stale content. - As the refresh interval goes from one to twelve, the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875. - The retrieval column shows none of this degradation: recall is 0.9500 at all six intervals and the concept hit rate 0.9250. Grounding in the source does not guarantee the freshness of the source. - The stale chunk rate is an input measure, the wrong answer rate an output measure; 0.2562 and 0.3875 are not two names for the same thing. - Full reindexing embeds 8316 chunks at interval one, incremental refresh 881; lengthening the interval reduces the embedded chunks under the incremental policy by only forty-one percent. - The chunks embedded per up-to-date answer fall from 1.98 to 1.77 under the incremental policy: for an eleven percent gain, 0.3125 of accuracy is sold. ## Next Step In four lessons the pipeline was built from the input side: the source was prepared, chunked, embedded, the index refreshed. All four of these decisions were on the **document's** side. The text at the other end of the pipeline, the query the subscriber wrote, was used as it was in all four lessons: short, incomplete, in a language other than that of the record. Yet it is one of the two texts that determine what retrieval finds. Expanding the query, rewriting it, or translating it into the language of the record with a model call can raise recall, and that call itself is added to the cost of the pipeline. The next lesson takes up the query and measures the recall that the transformation adds together with the call it adds.