Query Transformation

The previous four lessons built the document side of the pipeline: the corpus was prepared, split into chunks, indexed and refreshed. All four stayed silent about one assumption: the query goes to the index in the form the user typed it.

The subscriber writes “which band”, and the corpus says “tier limit”. One way to close the gap is to give the query to a model before sending it to the index and have it translated into the document language; the cost is one call per query. The transformation is one step; seeing the result and querying again is the subject of the AI Agents course.

• RP41. The code is a simulator; no real embedding or model endpoint is called. The corpus, the concepts, the queries and the user phrasings are fictional, seed 20260218.
• RP42. The target has two parts. The target of a query is both a concept and a record type; a chunk is relevant only if it carries both. Recall@k is the share of queries with a relevant chunk in the top k, and precision@k is the share of the top k that is relevant.
• RP43. The pipeline uses the hybrid search measured in embeddings/06: token overlap is added next to cosine similarity. Its procedure is not repeated, and the weight is fixed. The baseline of this lesson is retrieval without transformation; the baseline of the no-retrieval call was set up in retrieval-pipeline/01.
• RP44. The generation simulator picks one of the chunks in the context and cites its code; the chunk’s position changes the weight. That weight is the subject of retrieval-pipeline/07 and is fixed here.
• RP45. Two phrasings of the same forty queries are measured: the short phrasing is the abbreviated form the subscriber types, and the long phrasing is the same question extended with context sentences. The target is the same in both.
• RP46. If the transformation hits the rate limit, the pipeline falls back to the raw query; the tokens of the failed call are still counted.

The Pipeline and the Target

The pipeline works at chunk level. The block below sets up the corpus, the embedding simulator, hybrid retrieval and the generation simulator; then it prints where the pipeline stands when the query is written in the form the index recognizes: the upper bound that transformation can reach.

# SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus,
# concepts and queries are fictional, seed 20260218.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


def text_of(d):
    return d["title"] + " " + " ".join(d["paras"])


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# ---- pipeline: chunks, hybrid retrieval, target definition and generation
CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]]
for p in CHUNKS:
    p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"]))
for s in QUERIES:
    s["record"] = s["text"].split()[-1]


def retrieve(q, k=5, w=0.5):
    qv, qb = embed(q), set(tokenize(q))
    s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i)
                for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1]))
    return [CHUNKS[i] for _, i in s[:k]]


def relevant(s, p):
    return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


WINDOW = 240
COUNTER = [0]
ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . "


def answer(query, chunks, seed=SEED, temperature=0.12):
    """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight."""
    COUNTER[0] += 1
    prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query
    toks = tokenize(prompt)
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None}
    r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks)
    score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb))
             for i, x in enumerate(chunks)]
    w = [math.exp(a / temperature) for a in score]
    chosen = chunks[pick(r(), [a / sum(w) for a in w])]
    return {"status": "ok", "input": len(toks), "chunk": chosen,
            "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))}


rec = gen = 0
for s in QUERIES:
    g = retrieve(s["text"], 5)
    rec += any(relevant(s, p) for p in g)
    gen += relevant(s, answer(s["text"], g)["chunk"])
print("corpus %d documents %d chunks ; average chunk %.1f tokens ; example query %r"
      % (len(DOCUMENTS), len(CHUNKS), sum(len(p["bag"]) for p in CHUNKS) / len(CHUNKS),
         QUERIES[0]["text"]))
print("target chunk set average %.1f chunks"
      % (sum(len([p for p in CHUNKS if relevant(s, p)]) for s in QUERIES) / 40))
print("query written in the index language  k=5  recall %.4f  generation %.4f" % (rec / 40, gen / 40))

corpus 200 documents 693 chunks ; average chunk 12.0 tokens ; example query 'how it gets written down form'
target chunk set average 8.9 chunks
query written in the index language  k=5  recall 0.8000  generation 0.6750

The target set averages 8.9 chunks, a little above one percent of six hundred ninety-three. When the query is written in the index language, the top five chunks bring one from this set with a share of 0.8000, and generation turns 0.6750 of them into an answer.

Two Phrasings and the Transformer

The subscriber does not write these sentences. “how it gets written down form” is an index query; the subscriber writes “is it written form” or describes the problem at length. The block below sets up the two phrasings of the same forty targets and runs three setups on both: the baseline (the raw query), expansion (a term in the document language is appended to it) and rewriting (the raw query is discarded and a sentence in the document language takes its place).

# FICTIONAL: the short form the user types -- it does not carry the surface the index recognizes
SHORT = {"tier": "which band", "unit": "costs per ton", "account": "home or office",
         "switch": "moving plans", "faultcode": "error code", "leak": "might be escaping",
         "pressure": "flow weak", "crew": "technician never came", "index": "dial digits",
         "read": "is it written", "swap": "hardware changed", "verify": "typed wrong",
         "planned": "announced ahead", "notice": "where to look", "duration": "hours it will last",
         "restore": "will it show up again", "periodic": "regular upkeep", "valve": "tap shutoff",
         "filter": "cleaning filter", "calib": "accuracy calibration"}
PREFIX = ["previous week", "for a few days", "since this morning", "yesterday evening"]
CONNECTOR = ["one more thing i wanted to ask", "also i wrote to the relevant department", "by the way i was curious",
             "i had also called before"]
SHORT_QUERIES, LONG_QUERIES = [], []
for i, s in enumerate(QUERIES):
    r = rng(SEED + 31337 * i)
    y, o2 = KEYS[int(r() * 20)], COMMON[int(r() * 7)]
    z = KEYS[int(r() * 20)]
    SHORT_QUERIES.append(dict(s, text=f"{SHORT[s['concept']]} {s['record']}"))
    LONG_QUERIES.append(dict(s, text=f"{PREFIX[int(r() * 4)]} {CONCEPTS[y][2]} {o2} "
                                     f"{CONNECTOR[int(r() * 4)]} {s['text']} {CONCEPTS[z][2]}"))

QUERY_TRACE = {}                    # query language -> concept
for k in KEYS:
    for t in tokenize(CONCEPTS[k][2]):
        if len(t) > 3:
            QUERY_TRACE.setdefault(t, k)
TRANSFORM_INSTRUCTION = "Translate the query into the document language . Query: "


def transform(q, mode, seed=SEED, temperature=0.5):
    """FICTIONAL transformer: a one-step model call, no loop and no planning."""
    COUNTER[0] += 1
    toks = tokenize(TRANSFORM_INSTRUCTION + q)
    if COUNTER[0] % 37 == 0:                     # rate limit: the raw query
        return {"status": "rate_limit", "input": len(toks), "output": 0, "text": q, "concept": None}
    r, score = rng(seed ^ trace_id(q)), {}
    for t in toks:
        if t in QUERY_TRACE:
            score[QUERY_TRACE[t]] = score.get(QUERY_TRACE[t], 0.0) + 1.0
    if not score:
        return {"status": "empty", "input": len(toks), "output": 0, "text": q, "concept": None}
    keys = sorted(score)
    w = [math.exp(score[x] / temperature) for x in keys]
    key = keys[pick(r(), [x / sum(w) for x in w])]
    common = next((t for t in toks if t in COMMON), "")
    out = f"{CONCEPTS[key][1]} for {common} goes" if mode == "rewriting" else f"{q} {CONCEPTS[key][1]}"
    return {"status": "ok", "input": len(toks), "output": len(tokenize(out)),
            "text": out, "concept": key}


def measure(qset, mode, seed=SEED, k=5):
    COUNTER[0] = 0
    rec = gen = calls = toks = failed = right = 0
    for s in qset:
        q = s["text"]
        if mode != "baseline":
            d = transform(q, mode, seed)
            q, calls, toks = d["text"], calls + 1, toks + d["input"] + d["output"]
            failed, right = failed + (d["status"] != "ok"), right + (d["concept"] == s["concept"])
        g = retrieve(q, k)
        rec += any(relevant(s, p) for p in g)
        y = answer(s["text"], g, seed)
        calls, toks = calls + 1, toks + y["input"] + y["output"]
        gen += y["chunk"] is not None and relevant(s, y["chunk"])
    return rec / 40, gen / 40, calls, toks, failed, right / 40


print("average query: short %.1f  long %.1f tokens"
      % (sum(len(tokenize(s["text"])) for s in SHORT_QUERIES) / 40,
         sum(len(tokenize(s["text"])) for s in LONG_QUERIES) / 40))
print("query setup          recall@5 generation calls   tokens correct concept failed")
gap = [0.0, 0.0]
for name, qset in (("short", SHORT_QUERIES), ("long ", LONG_QUERIES)):
    for mode in ("baseline", "expansion", "rewriting"):
        a, u, c, b, failed, right = measure(qset, mode)
        a2, u2 = measure(qset, mode, SEED + 1)[:2]
        gap = [max(gap[0], abs(a - a2)), max(gap[1], abs(u - u2))]
        print("%s %-14s %8.4f %10.4f %5d %8d %15s %6d"
              % (name, mode, a, u, c, b, "-" if mode == "baseline" else "%.4f" % right, failed))
print("run-to-run difference (seed 20260219): recall at most %.4f  generation at most %.4f"
      % (gap[0], gap[1]))

average query: short 4.5  long 26.5 tokens
query setup          recall@5 generation calls   tokens correct concept failed
short baseline         0.3250     0.1500    40     4609               -      0
short expansion        0.7500     0.7000    80     5804          0.9500      1
short rewriting        0.7500     0.7000    80     5749          0.9500      1
long  baseline         0.5500     0.2250    40     5827               -      0
long  expansion        0.5000     0.1250    80     8553          0.3500      1
long  rewriting        0.2750     0.1000    80     7647          0.3500      1
run-to-run difference (seed 20260219): recall at most 0.0500  generation at most 0.0500

The Gain on the Short Phrasing

The baseline of the short phrasing is 0.3250 recall and 0.1500 generation. The gap to the upper bound is not a defect of the pipeline but the query not touching the index: “which band” appears in no chunk, and all that is left is the overlap of the record type.

Transformation closes the gap. Expansion raises recall to 0.7500 and generation to 0.7000; both stand above the run-to-run difference (0.0500 and 0.0500), and generation reaches the upper bound within that difference. The reason is written in the table: the transformer finds the correct concept with a share of 0.9500.

That expansion and rewriting give the same result here is not a coincidence: keeping the raw query has neither a cost nor a gain, because there is no signal to be thrown away.

The Loss on the Long Phrasing

The baseline of the long phrasing is 0.5500, higher than that of the short phrasing: while lengthening the problem, the subscriber also writes the phrase the index recognizes. Transformation lowers the baseline, to 0.5000 with expansion and to 0.2750 with rewriting. Rewriting stands far above the run-to-run difference, so it is a measured loss. Expansion is a smaller story: its recall (0.5000) sits at the edge of the run-to-run difference, and only its generation, which goes from 0.2250 to 0.1250, clearly leaves it behind.

The cause is again in the correct concept column: the hit rate falls to 0.3500. The long query has three concept surfaces and two record types; a one-step call has to choose which one is the real question, and it is wrong most of the time. When it is wrong, expansion appends a wrong term but the raw query stays in place; rewriting deletes the one thing that was right, the phrase the subscriber wrote, and generation falls to 0.1000. This is the instance, in this lesson, of the course’s second claim: the added machinery made the job worse.

The Bill for Transformation

Calls go from one to two per query, from forty to eighty. Tokens go from 4609 to 5804 on the short phrasing with expansion and from 5827 to 8553 on the long one; having the model read the long query as well is the most expensive line of the pipeline, and one of the forty calls hits the rate limit. Latency is counted as a class: the pipeline turns from a single-call step into a two-stage chain. While the extra tokens on the short phrasing raise recall from 0.3250 to 0.7500, the larger extra tokens on the long phrasing lower it from 0.5500 to 0.5000 with expansion and to 0.2750 with rewriting.

Summary

• The target of a query is both a concept and a record type; the share with which a chunk carrying both is found in the top five is retrieval, and the share with which that chunk turns into the answer is the generation measure.
• Because the short form the user typed does not touch the index, the baseline is 0.3250 / 0.1500; the same query in the index language gives 0.8000 / 0.6750.
• Expansion raises recall to 0.7500 and generation to 0.7000 on the short phrasing; the share of finding the correct concept is 0.9500.
• On the long phrasing the same machinery drops the baseline recall from 0.5500 to 0.5000, and rewriting to 0.2750; the correct concept share falls to 0.3500, and rewriting also deletes the phrase the subscriber wrote.
• The cost is one call per query and more tokens (5804 and 8553 against 4609 and 5827); the pipeline turns into a chain.

Next Step

The measurements of this lesson count the top five chunks together: if a relevant chunk is among them, recall is written, and the order is not asked. Yet precision@5 says most of the top five are irrelevant. The next lesson splits retrieval in two: a cheap stage that gathers a wide candidate set and an expensive stage that reads every candidate together with the query.