--- title: 'Query Transformation' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/query-transformation' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds query transformation as a one-step model call and counts the call and the tokens it adds separately.' - 'Compares expansion with rewriting on the same query set and shows the case in which the two diverge.' - 'Measures that transformation gains on the short phrasing and, on the long phrasing, drops retrieval recall below the baseline.' - 'Judges the gain and loss claims by whether they stand above the run-to-run difference.' --- # Query Transformation Translating the query the user typed into the document language with a single model call before it reaches the index, and counting what this step gains and what it loses separately: on a fictional set of forty queries the short phrasing has a baseline of 0.3250 recall and 0.1500 generation, while expansion raises the two to 0.7500 and 0.7000, which is where a query written in the index language leaves off. The same setup drops the long phrasing from a baseline of 0.5500 recall to 0.5000 with expansion and to 0.2750 with rewriting, because the share with which the transformer finds the correct concept falls from 0.9500 to 0.3500. The cost on both sets is one call per query and more tokens: from 4609 to 5804 on the short set and from 5827 to 8553 on the long set; the run-to-run difference was measured at 0.0500 in recall and 0.0500 in generation. The previous four lessons built the document side of the pipeline: the corpus was prepared, split into chunks, indexed and refreshed. All four stayed silent about one assumption: **the query goes to the index in the form the user typed it**. The subscriber writes "which band", and the corpus says "tier limit". One way to close the gap is to give the query to a model before sending it to the index and have it **translated into the document language**; the cost is one call per query. The transformation is **one step**; seeing the result and querying again is the subject of the AI Agents course. - **RP41.** The code is a **simulator**; no real embedding or model endpoint is called. The corpus, the concepts, the queries and the user phrasings are **fictional**, seed **20260218**. - **RP42. The target has two parts.** The target of a query is both a **concept** and a **record type**; a chunk is relevant only if it carries both. **Recall@k** is the share of queries with a relevant chunk in the top k, and **precision@k** is the share of the top k that is relevant. - **RP43.** The pipeline uses the **hybrid search** measured in `embeddings/06`: token overlap is added next to cosine similarity. Its procedure is not repeated, and the weight is fixed. The baseline of this lesson is **retrieval without transformation**; the baseline of the no-retrieval call was set up in `retrieval-pipeline/01`. - **RP44.** The generation simulator picks one of the chunks in the context and cites its code; the chunk's **position** changes the weight. That weight is the subject of `retrieval-pipeline/07` and is fixed here. - **RP45.** Two phrasings of the same forty queries are measured: the **short phrasing** is the abbreviated form the subscriber types, and the **long phrasing** is the same question extended with context sentences. The target is the same in both. - **RP46.** If the transformation hits the rate limit, the pipeline **falls back to the raw query**; the tokens of the failed call are still counted. ## The Pipeline and the Target The pipeline works at chunk level. The block below sets up the corpus, the embedding simulator, hybrid retrieval and the generation simulator; then it prints where the pipeline stands when the query is written **in the form the index recognizes**: the upper bound that transformation can reach. ```python # SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus, # concepts and queries are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def text_of(d): return d["title"] + " " + " ".join(d["paras"]) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- pipeline: chunks, hybrid retrieval, target definition and generation CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]] for p in CHUNKS: p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"])) for s in QUERIES: s["record"] = s["text"].split()[-1] def retrieve(q, k=5, w=0.5): qv, qb = embed(q), set(tokenize(q)) s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i) for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1])) return [CHUNKS[i] for _, i in s[:k]] def relevant(s, p): return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h WINDOW = 240 COUNTER = [0] ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . " def answer(query, chunks, seed=SEED, temperature=0.12): """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight.""" COUNTER[0] += 1 prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None} r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks) score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb)) for i, x in enumerate(chunks)] w = [math.exp(a / temperature) for a in score] chosen = chunks[pick(r(), [a / sum(w) for a in w])] return {"status": "ok", "input": len(toks), "chunk": chosen, "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))} rec = gen = 0 for s in QUERIES: g = retrieve(s["text"], 5) rec += any(relevant(s, p) for p in g) gen += relevant(s, answer(s["text"], g)["chunk"]) print("corpus %d documents %d chunks ; average chunk %.1f tokens ; example query %r" % (len(DOCUMENTS), len(CHUNKS), sum(len(p["bag"]) for p in CHUNKS) / len(CHUNKS), QUERIES[0]["text"])) print("target chunk set average %.1f chunks" % (sum(len([p for p in CHUNKS if relevant(s, p)]) for s in QUERIES) / 40)) print("query written in the index language k=5 recall %.4f generation %.4f" % (rec / 40, gen / 40)) ``` ``` corpus 200 documents 693 chunks ; average chunk 12.0 tokens ; example query 'how it gets written down form' target chunk set average 8.9 chunks query written in the index language k=5 recall 0.8000 generation 0.6750 ``` The target set averages **8.9** chunks, a little above one percent of six hundred ninety-three. When the query is written in the index language, the top five chunks bring one from this set with a share of **0.8000**, and generation turns **0.6750** of them into an answer. ## Two Phrasings and the Transformer The subscriber does not write these sentences. "how it gets written down form" is an index query; the subscriber writes "is it written form" or describes the problem at length. The block below sets up the two phrasings of the same forty targets and runs three setups on both: the **baseline** (the raw query), **expansion** (a term in the document language is appended to it) and **rewriting** (the raw query is discarded and a sentence in the document language takes its place). ```python # FICTIONAL: the short form the user types -- it does not carry the surface the index recognizes SHORT = {"tier": "which band", "unit": "costs per ton", "account": "home or office", "switch": "moving plans", "faultcode": "error code", "leak": "might be escaping", "pressure": "flow weak", "crew": "technician never came", "index": "dial digits", "read": "is it written", "swap": "hardware changed", "verify": "typed wrong", "planned": "announced ahead", "notice": "where to look", "duration": "hours it will last", "restore": "will it show up again", "periodic": "regular upkeep", "valve": "tap shutoff", "filter": "cleaning filter", "calib": "accuracy calibration"} PREFIX = ["previous week", "for a few days", "since this morning", "yesterday evening"] CONNECTOR = ["one more thing i wanted to ask", "also i wrote to the relevant department", "by the way i was curious", "i had also called before"] SHORT_QUERIES, LONG_QUERIES = [], [] for i, s in enumerate(QUERIES): r = rng(SEED + 31337 * i) y, o2 = KEYS[int(r() * 20)], COMMON[int(r() * 7)] z = KEYS[int(r() * 20)] SHORT_QUERIES.append(dict(s, text=f"{SHORT[s['concept']]} {s['record']}")) LONG_QUERIES.append(dict(s, text=f"{PREFIX[int(r() * 4)]} {CONCEPTS[y][2]} {o2} " f"{CONNECTOR[int(r() * 4)]} {s['text']} {CONCEPTS[z][2]}")) QUERY_TRACE = {} # query language -> concept for k in KEYS: for t in tokenize(CONCEPTS[k][2]): if len(t) > 3: QUERY_TRACE.setdefault(t, k) TRANSFORM_INSTRUCTION = "Translate the query into the document language . Query: " def transform(q, mode, seed=SEED, temperature=0.5): """FICTIONAL transformer: a one-step model call, no loop and no planning.""" COUNTER[0] += 1 toks = tokenize(TRANSFORM_INSTRUCTION + q) if COUNTER[0] % 37 == 0: # rate limit: the raw query return {"status": "rate_limit", "input": len(toks), "output": 0, "text": q, "concept": None} r, score = rng(seed ^ trace_id(q)), {} for t in toks: if t in QUERY_TRACE: score[QUERY_TRACE[t]] = score.get(QUERY_TRACE[t], 0.0) + 1.0 if not score: return {"status": "empty", "input": len(toks), "output": 0, "text": q, "concept": None} keys = sorted(score) w = [math.exp(score[x] / temperature) for x in keys] key = keys[pick(r(), [x / sum(w) for x in w])] common = next((t for t in toks if t in COMMON), "") out = f"{CONCEPTS[key][1]} for {common} goes" if mode == "rewriting" else f"{q} {CONCEPTS[key][1]}" return {"status": "ok", "input": len(toks), "output": len(tokenize(out)), "text": out, "concept": key} def measure(qset, mode, seed=SEED, k=5): COUNTER[0] = 0 rec = gen = calls = toks = failed = right = 0 for s in qset: q = s["text"] if mode != "baseline": d = transform(q, mode, seed) q, calls, toks = d["text"], calls + 1, toks + d["input"] + d["output"] failed, right = failed + (d["status"] != "ok"), right + (d["concept"] == s["concept"]) g = retrieve(q, k) rec += any(relevant(s, p) for p in g) y = answer(s["text"], g, seed) calls, toks = calls + 1, toks + y["input"] + y["output"] gen += y["chunk"] is not None and relevant(s, y["chunk"]) return rec / 40, gen / 40, calls, toks, failed, right / 40 print("average query: short %.1f long %.1f tokens" % (sum(len(tokenize(s["text"])) for s in SHORT_QUERIES) / 40, sum(len(tokenize(s["text"])) for s in LONG_QUERIES) / 40)) print("query setup recall@5 generation calls tokens correct concept failed") gap = [0.0, 0.0] for name, qset in (("short", SHORT_QUERIES), ("long ", LONG_QUERIES)): for mode in ("baseline", "expansion", "rewriting"): a, u, c, b, failed, right = measure(qset, mode) a2, u2 = measure(qset, mode, SEED + 1)[:2] gap = [max(gap[0], abs(a - a2)), max(gap[1], abs(u - u2))] print("%s %-14s %8.4f %10.4f %5d %8d %15s %6d" % (name, mode, a, u, c, b, "-" if mode == "baseline" else "%.4f" % right, failed)) print("run-to-run difference (seed 20260219): recall at most %.4f generation at most %.4f" % (gap[0], gap[1])) ``` ``` average query: short 4.5 long 26.5 tokens query setup recall@5 generation calls tokens correct concept failed short baseline 0.3250 0.1500 40 4609 - 0 short expansion 0.7500 0.7000 80 5804 0.9500 1 short rewriting 0.7500 0.7000 80 5749 0.9500 1 long baseline 0.5500 0.2250 40 5827 - 0 long expansion 0.5000 0.1250 80 8553 0.3500 1 long rewriting 0.2750 0.1000 80 7647 0.3500 1 run-to-run difference (seed 20260219): recall at most 0.0500 generation at most 0.0500 ``` ## The Gain on the Short Phrasing The baseline of the short phrasing is **0.3250** recall and **0.1500** generation. The gap to the upper bound is not a defect of the pipeline but **the query not touching the index**: "which band" appears in no chunk, and all that is left is the overlap of the record type. Transformation closes the gap. Expansion raises recall to **0.7500** and generation to **0.7000**; both stand above the run-to-run difference (0.0500 and 0.0500), and generation reaches the upper bound within that difference. The reason is written in the table: the transformer finds the correct concept with a share of **0.9500**. That expansion and rewriting give the **same** result here is not a coincidence: keeping the raw query has neither a cost nor a gain, because there is no signal to be thrown away. ## The Loss on the Long Phrasing The baseline of the long phrasing is **0.5500**, higher than that of the short phrasing: while lengthening the problem, the subscriber also writes the phrase the index recognizes. Transformation **lowers** the baseline, to **0.5000** with expansion and to **0.2750** with rewriting. Rewriting stands far above the run-to-run difference, so it is a measured loss. Expansion is a smaller story: its recall (0.5000) sits at the edge of the run-to-run difference, and only its generation, which goes from 0.2250 to 0.1250, clearly leaves it behind. The cause is again in the correct concept column: the hit rate falls to **0.3500**. The long query has three concept surfaces and two record types; a one-step call has to choose which one is the real question, and it is wrong most of the time. When it is wrong, **expansion** appends a wrong term but the raw query stays in place; **rewriting** deletes the one thing that was right, the phrase the subscriber wrote, and generation falls to **0.1000**. This is the instance, in this lesson, of the course's second claim: the added machinery made the job worse. ## The Bill for Transformation **Calls** go from one to two per query, from forty to eighty. **Tokens** go from 4609 to 5804 on the short phrasing with expansion and from 5827 to 8553 on the long one; having the model read the long query as well is the most expensive line of the pipeline, and one of the forty calls hits the rate limit. Latency is counted as a **class**: the pipeline turns from a single-call step into a **two-stage chain**. While the extra tokens on the short phrasing raise recall from 0.3250 to 0.7500, the larger extra tokens on the long phrasing lower it from 0.5500 to 0.5000 with expansion and to 0.2750 with rewriting. ## Summary - The target of a query is both a concept and a record type; the share with which a chunk carrying both is found in the top five is retrieval, and the share with which that chunk turns into the answer is the generation measure. - Because the short form the user typed does not touch the index, the baseline is 0.3250 / 0.1500; the same query in the index language gives 0.8000 / 0.6750. - Expansion raises recall to 0.7500 and generation to 0.7000 on the short phrasing; the share of finding the correct concept is 0.9500. - On the long phrasing the same machinery drops the baseline recall from 0.5500 to 0.5000, and rewriting to 0.2750; the correct concept share falls to 0.3500, and rewriting also deletes the phrase the subscriber wrote. - The cost is one call per query and more tokens (5804 and 8553 against 4609 and 5827); the pipeline turns into a chain. ## Next Step The measurements of this lesson count the top five chunks **together**: if a relevant chunk is among them, recall is written, and the order is not asked. Yet precision@5 says most of the top five are irrelevant. The next lesson splits retrieval in two: a cheap stage that gathers a wide candidate set and an expensive stage that reads every candidate together with the query.