Indexing and Updating

The previous three lessons took the corpus as static. The documents were prepared once, chunked once, embedded once; after the index was built it was never touched again. Field records do not behave that way. A procedure is republished, a threshold is updated, a record is withdrawn. When the source changes, the index does not; the index is a snapshot, and unless it is written down when it was taken, what it shows is not known.

The degradation this lesson measures is of a different kind from the earlier ones. Retrieval does not bring the wrong chunk: the retrieved chunk is still from the right document and the right concept. What degrades is the chunk’s content. That is why the retrieval column stays silent and the fault shows only in the generation column.

• RP21. The code is a simulator; the corpus, the change stream and the queries are fictional, seed 20260218. The embedding and the language model are the definitions of the previous lessons.
• RP22. Each document carries a validity record (the string validity record in the data). When the document is republished the record goes up by one; when a chunk is served from the index it is given together with its record. The change stream is twelve periods, and in each period 0.10 of the documents are republished.
• RP23. A republication changes the chunk’s record, not its body. This is an assumption that simplifies the measurement: because the embedding does not move, the retrieval ranking stays the same at all intervals, and it is certain that the column that falls falls only from staleness.
• RP24. One behavior is added to the simulator: the model copies as is the validity record in the first source chunk that carries the surface of the concept it chose. This is what a quoting model does, and it is the place where staleness passes into the answer.
• RP25. The stale chunk rate is the rate at which the records of the served chunks disagree with the source’s actual record in that period. The up-to-date answer rate is the rate of answers whose concept is correct and whose copied record equals the source’s current record.
• RP26. Two refresh policies are measured: full reindexing re-embeds all chunks at every refresh, incremental refresh only the chunks of documents that changed in that interval. A document that changed twice in one interval is embedded once.
• RP27. The measurement is taken in all twelve periods; forty queries are asked in each period, that is, 480 requests for each interval. The stale chunk rate and the number of re-embedded chunks are independent of the run.

Where Staleness Passes into the Answer

The first block sets up the corpus, puts the change stream and the validity record on top of it, defines the simulator together with the copying behavior and sweeps the refresh interval.

# FICTIONAL. The corpus, concept axis, embedding, change stream and language model are a simulator;
# no real endpoint is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

import math
# ---- FICTIONAL change stream: twelve periods, a fraction of the documents is republished each period
PERIODS = 12
CHANGES = []
for t in range(1, PERIODS + 1):
    r = rng(SEED + 6007 * t)
    CHANGES.append([i for i in range(len(DOCUMENTS)) if r() < 0.10])


def stamp(i, t):
    """The validity record of document i at the end of period t."""
    return "Y%02d" % sum(i in CHANGES[u] for u in range(t))


CHUNKS = [{"doc": i, "code": d["code"], "label": d["label"], "body": p,
           "concepts": [k for k in KEYS if CONCEPTS[k][1] in p]}
          for i, d in enumerate(DOCUMENTS) for p in d["paras"]]
VEC = [embed(x["body"]) for x in CHUNKS]          # the stamp does not change the embedding
TOP = []
for s in QUERIES:
    q = embed(s["text"])
    TOP.append(sorted(range(len(CHUNKS)), key=lambda i: -cosine(q, VEC[i]))[:3])

# ---- FICTIONAL language model simulator: the previous lessons' simulator, one addition
# copying the validity record it sees in the context
WINDOW = 240
PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"}
COUNTER = [0]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def call(prompt, seed=SEED, temperature=0.6):
    COUNTER[0] += 1
    b = tokenize(prompt)
    if len(b) > WINDOW:
        return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    dm = " " + prompt.lower() + " "
    p = {k: 0.15 for k in KEYS}
    for k in KEYS:
        p[k] += 2.0 * dm.count(CONCEPTS[k][1])
        if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm:
            p[k] += 2.0
    w = [math.exp(p[a] / temperature) for a in sorted(p)]
    y = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    for x in prompt.split(" . "):                # copies the record in the source as is
        if CONCEPTS[y][1] in x and "validity record" in x:
            y += " " + x.split("validity record ")[1].split()[0]
            break
    return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y}


INSTRUCTION = ("Link the request to one of these concepts and write down the validity entry of the source: " +
               " , ".join(KEYS) + " . Reply with one concept and the entry . ")


def run(interval, seed=SEED):
    COUNTER[0] = 0
    stale = concept = current = recall = n = 0
    embedded = full = 0
    for t in range(1, PERIODS + 1):
        last = (t // interval) * interval            # the last period the index saw
        if t % interval == 0:                        # a refresh occurs this period
            changed = {i for u in range(t - interval, t) for i in CHANGES[u]}
            embedded += sum(x["doc"] in changed for x in CHUNKS)
            full += len(CHUNKS)
        for s, top in zip(QUERIES, TOP):
            texts = {i: "%s validity record %s" % (CHUNKS[i]["body"],
                                                  stamp(CHUNKS[i]["doc"], last)) for i in top}
            stale += sum(stamp(CHUNKS[i]["doc"], last) != stamp(CHUNKS[i]["doc"], t)
                         for i in top) / 3
            recall += any(s["concept"] in CHUNKS[i]["concepts"] for i in top)
            y = call(INSTRUCTION + "Source: " + " . ".join(texts[i] for i in top) +
                     " . Request: " + s["text"], seed)
            kv, _, rv = y["response"].partition(" ")
            rec = next((i for i in top if CONCEPTS[kv][1] in texts[i]), None) if kv in KEYS \
                else None
            concept += kv == s["concept"]
            current += (kv == s["concept"] and rec is not None
                        and rv == stamp(CHUNKS[rec]["doc"], t))
            n += 1
    return stale / n, recall / n, concept / n, current / n, embedded, full


print("periods %d, total republications %d, documents never changed %d"
      % (PERIODS, sum(len(x) for x in CHANGES),
         sum(1 for i in range(len(DOCUMENTS)) if all(i not in d for d in CHANGES))))
print("interval refreshes embedded(incremental) embedded(full) stale chunks  recall  concept  current")
for a in (1, 2, 3, 4, 6, 12):
    st, rc, cn, cu, em, fl = run(a)
    print("%8d %9d %21d %14d %12.4f %7.4f %8.4f %8.4f"
          % (a, PERIODS // a, em, fl, st, rc, cn, cu))

periods 12, total republications 254, documents never changed 51
interval refreshes embedded(incremental) embedded(full) stale chunks  recall  concept  current
       1        12                   881           8316       0.0000  0.9500   0.9250   0.9250
       2         6                   839           4158       0.0347  0.9500   0.9250   0.8917
       3         4                   774           2772       0.0625  0.9500   0.9250   0.8771
       4         3                   741           2079       0.0937  0.9500   0.9250   0.8208
       6         2                   662           1386       0.1278  0.9500   0.9250   0.7708
      12         1                   519            693       0.2562  0.9500   0.9250   0.6125

The first reading of the table is the clearest reason for keeping the two columns apart. Retrieval recall is 0.9500 in all six rows, and the concept hit rate is 0.9250 in all six rows. As the refresh interval goes from one period to twelve, nothing changes in the retrieval column, because the index keeps bringing the right documents. 0.2562 of the chunks it brings now show a text that is no longer in the source, and none of the retrieval measures says so.

The only column that falls is the up-to-date answer rate: from 0.9250 to 0.6125. The 0.3125 between them is more than four times the threshold. The user-visible counterpart of this is the following: the model finds the right procedure, names the right concept and writes beside it a record that has been withdrawn. This answer is not a hallucination; it is faithful to the source. The source itself is stale. Grounding in the source does not guarantee the freshness of the source.

The stale chunk rate and the wrong answer rate do not coincide one to one either. At interval twelve 0.2562 of the chunks are stale but 0.3875 of the answers are wrong. The difference comes from the concept hit rate already being 0.9250, and from the probability that a stale chunk is the chunk carrying the chosen concept carrying a distribution of its own. Staleness is an input measure, the wrong answer is an output measure; they are not the same number and cannot be written in place of each other.

The Price of Refreshing

The embedding cost behaves entirely differently under the two policies. Full reindexing re-embeds six hundred ninety-three chunks at every refresh: 8316 at interval one, 693 at interval twelve. Under this policy the interval is a lever that divides the cost directly. Incremental refresh embeds only the chunks of changed documents: 881 at interval one, that is, 9.4 times smaller than full reindexing.

The second block reads these two policies per correct piece of work and tests the stability of the measurement itself.

print("interval wrong answer rate incremental/current full/current 2nd seed current difference")
for a in (1, 2, 4, 12):
    st, rc, cn, cu, em, fl = run(a)
    c2 = run(a, SEED + 1)[3]
    n = cu * PERIODS * len(QUERIES)
    print("%8d %17.4f %19.2f %12.2f %16.4f %10.4f"
          % (a, 1 - cu, em / n, fl / n, c2, abs(cu - c2)))

interval wrong answer rate incremental/current full/current 2nd seed current difference
       1            0.0750                1.98        18.73           0.9250     0.0000
       2            0.1083                1.96         9.71           0.8917     0.0000
       4            0.1792                1.88         5.28           0.8208     0.0000
      12            0.3875                1.77         2.36           0.6125     0.0000

The incremental column shows why the interval is the wrong lever. When the interval goes from one to twelve, the embedded chunks fall from 881 to 519, that is, only forty-one percent. The reason is that a document republished several times in one interval is embedded only once; as the interval lengthens this merging increases, but the total number of changed documents does not change. Read per up-to-date answer, the gain melts further: from 1.98 to 1.77, that is, eleven percent. The price of the same decision is 0.3125 in the up-to-date answer rate. Lengthening the interval under incremental refresh is selling a quarter of the accuracy for next to nothing.

In the full reindexing column the table looks the other way round: per up-to-date answer, from 18.73 to 2.36. This row does not justify lengthening the interval; it justifies giving up full reindexing. Setting interval one with the incremental policy is both cheaper than setting interval twelve with full reindexing (881 against 693, the same order of magnitude) and 0.3125 more accurate.

The last two columns are the stability of the measurement itself. Run with the second seed, the up-to-date answer rate comes out exactly the same in all four rows, a difference of 0.0000. This is the continuation of the first lesson’s finding: the output of a call whose source has been given does not depend on the seed. For that reason all the differences in the table are far above the threshold and have been measured.

Summary

• The index is a snapshot; when the source changes the index does not, and in the time that passes it serves the right chunk with stale content.
• As the refresh interval goes from one to twelve, the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875.
• The retrieval column shows none of this degradation: recall is 0.9500 at all six intervals and the concept hit rate 0.9250. Grounding in the source does not guarantee the freshness of the source.
• The stale chunk rate is an input measure, the wrong answer rate an output measure; 0.2562 and 0.3875 are not two names for the same thing.
• Full reindexing embeds 8316 chunks at interval one, incremental refresh 881; lengthening the interval reduces the embedded chunks under the incremental policy by only forty-one percent.
• The chunks embedded per up-to-date answer fall from 1.98 to 1.77 under the incremental policy: for an eleven percent gain, 0.3125 of accuracy is sold.

Next Step

In four lessons the pipeline was built from the input side: the source was prepared, chunked, embedded, the index refreshed. All four of these decisions were on the document’s side. The text at the other end of the pipeline, the query the subscriber wrote, was used as it was in all four lessons: short, incomplete, in a language other than that of the record. Yet it is one of the two texts that determine what retrieval finds. Expanding the query, rewriting it, or translating it into the language of the record with a model call can raise recall, and that call itself is added to the cost of the pipeline. The next lesson takes up the query and measures the recall that the transformation adds together with the call it adds.