--- title: 'What Is an Embedding' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/what-is-an-embedding' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - "Separates a concept's document-side surface form from its query-side surface form and shows why keyword search collapses on this separation." - 'Measures the retrieval precision and retrieval recall of keyword search and embedding similarity side by side on the same query set.' - 'Prints concept hit rate and class hit rate as two separate columns and writes what reducing them to a single number hides.' - 'Counts the cost of the retrieval scaffolding as retrieved chunks, comparison count, embedding calls and stored value count.' --- # What Is an Embedding Deciding which document in the corpus a query resembles by meaning rather than by words, and counting that decision with two separate measures. Forty queries are run on a fictional corpus of two hundred documents, 693 paragraphs and twenty concepts: keyword matching leaves a retrieval precision of 0.0800 and a retrieval recall of 0.0363, while embedding similarity gives 0.4700 and 0.2258; the first result's concept hit rate rises from 0.0250 to 0.5000 and its class hit rate from 0.1000 to 1.0000. The ceiling on recall for five chunks is 0.4904, and reading the class hit rate alone hides the half share in the concept hit rate. The cost is 8000 comparisons, 240 embedding calls and a stored value count of 6400. The Building Applications with Model APIs course treated the call as a network operation across twenty-six lessons: token budget, streaming, format validation, cache, fallback model and the cost of each scaffolding were written down. The question left standing at the close was this: the **knowledge** given to the model was always put into the prompt **by hand**. Which **three** of a thousand documents in a corpus go into that prompt, **against which baseline** that choice is measured, and what a wrong choice costs were never asked. This lesson builds the first tool for that choice. The question is this: what does one look at when deciding which document in the corpus a query resembles. The most direct answer is words. The **embedding** is the second answer: the text is turned into a vector whose neighborhood reflects meaning, and similarity is computed between vectors. The definition of an embedding, cosine similarity and word neighborhood were built in the Natural Language Processing and Computer Vision course; they are not repeated here. In this lesson an embedding is a **search key** and has a single measure: the correctness of the retrieved document. - **EV1.** All the code is a **simulator**. The corpus, concept axis, embedding procedure and query set are **fictional**; none is a real institution, product or document. The seed is **20260218**. - **EV2.** What the simulator models is this: each concept has **two surface forms**. The form that appears in documents (`tier threshold`) and the form that appears in queries (`which band am i in`) **share no token**. Keyword search sees only the surface form; the embedding first turns the surface form into a **concept**. - **EV3.** The embedding procedure has two steps: a vector of twenty concepts is built, then this vector is **projected** to `DIM` dimensions. The projection is a compression; in this lesson the dimension is held fixed at thirty-two. - **EV4.** The embedding is **deterministic**: the same text gives the same vector on every call. This is the lesson's run-independent measure and it is shown in the block. - **EV5.** This lesson's **baseline is keyword search**: the number of a query's distinct tokens that appear in a document is counted, and document order decides ties. - **EV6.** For a query, the **relevant document** is a document whose main concept equals the query's target concept. The number of retrieved chunks is **five**. - **EV7. Retrieval precision** is the share of the five retrieved documents that are relevant; **retrieval recall** is the share of the relevant documents that enter these five. The pair of precision and recall was measured in the Search Engines and Text Retrieval course; the procedure is not repeated. - **EV8.** **Class hit rate is the easy measure, concept hit rate is the hard one.** The two are printed as **two separate columns**. - **EV9.** In this topic the corpus is used **at document level**; chunking is left to the retrieval pipeline. - **EV10.** The cost of retrieval is written as **the number of retrieved chunks, comparison count, embedding calls and stored value count**; raw time is not written. ## Fictional Corpus and Concept Axis The block below builds the corpus, the concept axis, the embedding simulator and the query set. The corpus is two hundred documents; each document carries a title and two to five paragraphs, and is spread across five classes and twenty concepts. The concepts are fictional procedures kept for the field crew of a measurement network: tariff tier, fault code, meter index, outage duration, valve check. The axis rests on a single design decision. The procedure the field crew writes says `tier threshold`; the question coming from the subscriber says `which band am i in`. Both point to the same concept and share not even a single token. In a real corpus this separation is partial; the simulator **carries it to the extreme** so that the difference between the two searches does not get lost inside a noise margin. ```python # SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus, # concept axis, embedding procedure and query set are FICTIONAL; none is a real # institution, product or document. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) # ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions def _to_concepts(m, in_domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2] if doc_form in dm: v[i] += 1.0 if query_form in dm: v[i] += 1.0 if in_domain else 0.0 # an out-of-domain embedding does not recognize the query-side form if not in_domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, in_domain=True): k = _to_concepts(m, in_domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query-side form is separate from the document-side form QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) print(f"corpus: {len(DOCUMENTS)} documents , {sum(len(d['paragraphs']) for d in DOCUMENTS)} paragraphs ," f" {len(KEYS)} concepts , {len(CLASSES)} classes") print("class distribution:", " , ".join( f"{s} {sum(1 for d in DOCUMENTS if d['label'] == s)}" for s in CLASSES)) shared = sum(len(set(tokenize(CONCEPTS[k][1])) & set(tokenize(CONCEPTS[k][2]))) for k in KEYS) print(f"shared tokens between document-side and query-side form , 20 concepts : {shared}") d = DOCUMENTS[0] print(f"\n{d['code']} ({d['label']}) title : {d['title']}") print(f" paragraph 1 : {d['paragraphs'][1]}") q = QUERIES[0] print(f"query : {q['text']} -> concept {q['concept']} , class {q['label']}") print(f" {q['concept']} in document-side form '{CONCEPTS[q['concept']][1]}' ," f" in query-side form '{CONCEPTS[q['concept']][2]}'") u, v = embed(q["text"]), embed(q["text"]) print(f"\nembedding {len(u)} dimensions , second call same vector : {u == v}") print("first six coordinates :", " ".join(f"{x:+.3f}" for x in u[:6])) ``` ``` corpus: 200 documents , 693 paragraphs , 20 concepts , 5 classes class distribution: tariff 52 , fault 46 , meter 39 , outage 32 , maintenance 31 shared tokens between document-side and query-side form , 20 concepts : 0 BK001 (maintenance) title : filter swap procedure paragraph 1 : calibration record for entry is kept and written together with filter swap query : how it gets saved form -> concept reading , class meter reading in document-side form 'reading procedure' , in query-side form 'how it gets saved' embedding 32 dimensions , second call same vector : True first six coordinates : -0.018 -0.155 -0.145 -0.101 +0.243 +0.028 ``` The first three lines of the output give the size of the corpus: two hundred documents, six hundred ninety-three paragraphs, twenty concepts, five classes. The classes are spread between fifty-two and thirty-one, so no class alone is a majority. The third line is the lesson's mechanism: across all twenty concepts, the number of shared tokens between the document-side and the query-side form is **zero**. The sample lines make this concrete. `BK001` is a document from the maintenance class, its paragraphs written in the document-side form; the first query is `how it gets saved form` and its target concept is `reading`. The reading concept appears in documents as `reading procedure` and in the query as `how it gets saved`. Between the query and the document there is **no word bridge**; a bridge can be built only through the concept. The remaining two lines close the run question: two calls with the same text give the same vector. In a call that generates text, the same prompt could give two different responses in two runs; on the embedding surface it cannot. ## The Numbers for Two Searches on the Same Query Set Each of the forty queries is run in two setups. The first is **keyword search**: the number of a query's distinct tokens that appear in a document is counted, the five highest-scoring documents are retrieved, and document order decides ties. The second is **embedding search**: the query and the documents are embedded with the same simulator, and the five documents with the highest cosine similarity are retrieved. Each setup produces four numbers; precision and recall are read over the whole retrieved set, concept and class hit rate over the first result only. ```python K = 5 TOKENS = [set(tokenize(doc_text(d))) for d in DOCUMENTS] VEC = [embed(doc_text(d)) for d in DOCUMENTS] RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS} def top_k(score): return sorted(range(len(DOCUMENTS)), key=lambda i: (-score[i], i))[:K] def keyword_search(q): qb = set(tokenize(q["text"])) return top_k([len(qb & TOKENS[i]) for i in range(len(DOCUMENTS))]) def embedding_search(q): qv = embed(q["text"]) return top_k([cosine(qv, VEC[i]) for i in range(len(DOCUMENTS))]) def measure(retrieve): precision = recall = concept = label = 0.0 for q in QUERIES: top = retrieve(q) d = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top) precision += d / K recall += d / RELEVANT[q["concept"]] concept += DOCUMENTS[top[0]]["main"] == q["concept"] label += DOCUMENTS[top[0]]["label"] == q["label"] n = len(QUERIES) return precision / n, recall / n, concept / n, label / n mean_relevant = sum(RELEVANT[q["concept"]] for q in QUERIES) / len(QUERIES) ceiling = sum(min(K, RELEVANT[q["concept"]]) / RELEVANT[q["concept"]] for q in QUERIES) / len(QUERIES) print(f"{len(QUERIES)} queries , k={K} , relevant documents per query {mean_relevant:.3f} ," f" recall ceiling {ceiling:.4f}") print(f"cost: {len(QUERIES) * K} retrieved chunks , {len(QUERIES) * len(DOCUMENTS)} comparisons ," f" {len(DOCUMENTS) + len(QUERIES)} embedding calls , {len(DOCUMENTS) * DIM} stored values") print("\nsetup precision recall concept class") for name, retrieve in (("keyword search", keyword_search), ("embedding", embedding_search)): p, a, kv, sn = measure(retrieve) print(f"{name:<16} {p:9.4f} {a:7.4f} {kv:7.4f} {sn:7.4f}") tied = 0 for q in QUERIES: qb = set(tokenize(q["text"])) score = [len(qb & TOKENS[i]) for i in range(len(DOCUMENTS))] tied += score.count(max(score)) print(f"\nmean number of documents sharing the highest keyword score :" f" {tied / len(QUERIES):.1f}") q = QUERIES[0] print(f"query : {q['text']} (concept {q['concept']})") for name, retrieve in (("keyword search", keyword_search), ("embedding", embedding_search)): print(f" {name:<14}", " ".join( f"{DOCUMENTS[i]['code']}/{DOCUMENTS[i]['main']}" for i in retrieve(q)[:3])) ``` ``` 40 queries , k=5 , relevant documents per query 10.825 , recall ceiling 0.4904 cost: 200 retrieved chunks , 8000 comparisons , 240 embedding calls , 6400 stored values setup precision recall concept class keyword search 0.0800 0.0363 0.0250 0.1000 embedding 0.4700 0.2258 0.5000 1.0000 mean number of documents sharing the highest keyword score : 79.2 query : how it gets saved form (concept reading) keyword search BK001/filter_swap AR002/pressure SY003/reading embedding SY025/replacement SY031/reading SY035/replacement ``` ## Reading the Table The keyword search row looks like a search result but is not. The precision is 0.0800; given that an average of 10.825 relevant documents per query stand among two hundred documents, the precision of picking five documents at random would be 0.0541. The last lines say where the difference comes from: an average of **79.2 documents share** the highest keyword score. What determines the first five is not relevance but the document order that breaks the tie. Recall stays at 0.0363. The embedding row gives 0.4700 precision and 0.2258 recall on the same queries; precision rises from 0.0800 to 0.4700. Recall has a ceiling of its own: with an average of 10.825 relevant documents per query, if five documents are retrieved, recall can be at most **0.4904**. The embedding gets close to half of this ceiling, and the remaining gap is the gap not of the scaffolding but of the **number of retrieved chunks**. The first three results of the sample query make the difference concrete. Keyword search retrieves `BK001`, `AR002` and `SY003`: three documents from three different classes, and the one document carrying the target concept has come to third place only because the tie was broken. Embedding search retrieves all three from the meter class and the second carries the target concept. ## Why the Two Columns Stay Separate The last two columns of the table are the first application of this course's rule. The embedding search's **class hit rate is 1.0000**: in forty of the forty queries the first result is from the right class. The same row's **concept hit rate is 0.5000**: in twenty queries the first result is a document of the wrong concept, from within the right class. What would have happened if it were reduced to a single measure. If the class hit rate were reported, the system would look flawless and there would be nothing left to improve. If the concept hit rate were reported, it would look as good as a coin toss, and the perfect hit at the class level would not be visible. Side by side the problem gets its name: **the scaffolding finds the class and cannot find the concept.** Since the wrongly retrieved document is from the same class, it is not irrelevant, only wrong; placed inside a response it does not look off-topic. On the keyword search row the same distinction reads the other way round: class hit rate 0.1000, concept hit rate 0.0250. The class column does not soften the failure here: with five classes, a hit rate of one in ten is below even the share of naming a single class every time. The cost row writes the price of the scaffolding: 8000 comparisons, 240 embedding calls, 6400 stored values, 200 retrieved chunks. Keyword search spends no embedding calls; the whole difference is what these 240 calls and 6400 values buy. As the corpus grows, the comparison count grows in direct proportion, because every query is compared with the whole corpus. ## Summary - A concept's document-side surface form and query-side surface form can differ; in this corpus the number of shared tokens is zero for all twenty concepts. - On the same forty queries, keyword search gives 0.0800 retrieval precision and 0.0363 retrieval recall, and embedding search gives 0.4700 and 0.2258. - In keyword search an average of 79.2 documents share the highest score; the order that breaks the tie, not relevance, determines the first five. - The recall ceiling for five chunks and an average of 10.825 relevant documents is 0.4904; the measured recall is read against this ceiling. - Class hit rate is 1.0000 while concept hit rate is 0.5000; two columns name the problem, a single number hides it. - The cost of retrieval is 8000 comparisons, 240 embedding calls, 6400 stored values and 200 retrieved chunks. ## Next Step This lesson used a single embedding, and that embedding recognizes the language of the corpus and the language of the query together. This is an assumption: if a ready-made embedding model was built with the texts of another domain, it may not recognize the query-side form. The next lesson builds two embeddings of the same dimension, one fitted to the language of this domain and the other not, and counts side by side the chunks the two retrieve on the same forty queries. The cost of the dimension decision was measured in the Building Applications with Model APIs course; what will be added on top of it is the share of domain fit.