What Is an Embedding

The Building Applications with Model APIs course treated the call as a network operation across twenty-six lessons: token budget, streaming, format validation, cache, fallback model and the cost of each scaffolding were written down. The question left standing at the close was this: the knowledge given to the model was always put into the prompt by hand. Which three of a thousand documents in a corpus go into that prompt, against which baseline that choice is measured, and what a wrong choice costs were never asked.

This lesson builds the first tool for that choice. The question is this: what does one look at when deciding which document in the corpus a query resembles. The most direct answer is words. The embedding is the second answer: the text is turned into a vector whose neighborhood reflects meaning, and similarity is computed between vectors. The definition of an embedding, cosine similarity and word neighborhood were built in the Natural Language Processing and Computer Vision course; they are not repeated here. In this lesson an embedding is a search key and has a single measure: the correctness of the retrieved document.

• EV1. All the code is a simulator. The corpus, concept axis, embedding procedure and query set are fictional; none is a real institution, product or document. The seed is 20260218.
• EV2. What the simulator models is this: each concept has two surface forms. The form that appears in documents (tier threshold) and the form that appears in queries (which band am i in) share no token. Keyword search sees only the surface form; the embedding first turns the surface form into a concept.
• EV3. The embedding procedure has two steps: a vector of twenty concepts is built, then this vector is projected to DIM dimensions. The projection is a compression; in this lesson the dimension is held fixed at thirty-two.
• EV4. The embedding is deterministic: the same text gives the same vector on every call. This is the lesson’s run-independent measure and it is shown in the block.
• EV5. This lesson’s baseline is keyword search: the number of a query’s distinct tokens that appear in a document is counted, and document order decides ties.
• EV6. For a query, the relevant document is a document whose main concept equals the query’s target concept. The number of retrieved chunks is five.
• EV7. Retrieval precision is the share of the five retrieved documents that are relevant; retrieval recall is the share of the relevant documents that enter these five. The pair of precision and recall was measured in the Search Engines and Text Retrieval course; the procedure is not repeated.
• EV8. Class hit rate is the easy measure, concept hit rate is the hard one. The two are printed as two separate columns.
• EV9. In this topic the corpus is used at document level; chunking is left to the retrieval pipeline.
• EV10. The cost of retrieval is written as the number of retrieved chunks, comparison count, embedding calls and stored value count; raw time is not written.

Fictional Corpus and Concept Axis

The block below builds the corpus, the concept axis, the embedding simulator and the query set. The corpus is two hundred documents; each document carries a title and two to five paragraphs, and is spread across five classes and twenty concepts. The concepts are fictional procedures kept for the field crew of a measurement network: tariff tier, fault code, meter index, outage duration, valve check.

The axis rests on a single design decision. The procedure the field crew writes says tier threshold; the question coming from the subscriber says which band am i in. Both point to the same concept and share not even a single token. In a real corpus this separation is partial; the simulator carries it to the extreme so that the difference between the two searches does not get lost inside a noise margin.

# SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus,
# concept axis, embedding procedure and query set are FICTIONAL; none is a real
# institution, product or document.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":             ("tariff", "tier threshold", "which band am i in"),
    "unit_rate":        ("tariff", "unit price", "fee per cubic metre"),
    "subscriber_type":  ("tariff", "subscriber type", "home or shop"),
    "switch":           ("tariff", "tariff switch", "changing plan"),
    "fault_code":       ("fault", "fault code", "error id"),
    "leak":             ("fault", "line leakage", "water escaping"),
    "pressure":         ("fault", "pressure drop", "weak flow"),
    "crew":             ("fault", "field crew", "who will be sent"),
    "index_value":      ("meter", "index value", "what the dial shows"),
    "reading":          ("meter", "reading procedure", "how it gets saved"),
    "replacement":      ("meter", "meter replacement", "new hardware"),
    "verify":           ("meter", "reading verification", "typed in wrong"),
    "planned":          ("outage", "planned outage", "announced stoppage"),
    "notice":           ("outage", "notice channel", "where i hear about it"),
    "duration":         ("outage", "outage duration", "how long it lasts"),
    "restoration":      ("outage", "service restoration", "when it comes back"),
    "periodic":         ("maintenance", "periodic maintenance", "regular checkup"),
    "valve":            ("maintenance", "valve inspection", "tap shutoff routine"),
    "filter_swap":      ("maintenance", "filter swap", "cleaning part"),
    "calibration":      ("maintenance", "calibration record", "gauge accuracy"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "region", "term", "step", "form", "approval", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paragraphs, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts,
                      "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs})


def doc_text(d):
    return d["title"] + " " + " ".join(d["paragraphs"])


# ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions
def _to_concepts(m, in_domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc_form in dm:
            v[i] += 1.0
        if query_form in dm:
            v[i] += 1.0 if in_domain else 0.0      # an out-of-domain embedding does not recognize the query-side form
    if not in_domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, in_domain=True):
    k = _to_concepts(m, in_domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query-side form is separate from the document-side form
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

print(f"corpus: {len(DOCUMENTS)} documents , {sum(len(d['paragraphs']) for d in DOCUMENTS)} paragraphs ,"
      f" {len(KEYS)} concepts , {len(CLASSES)} classes")
print("class distribution:", " , ".join(
    f"{s} {sum(1 for d in DOCUMENTS if d['label'] == s)}" for s in CLASSES))
shared = sum(len(set(tokenize(CONCEPTS[k][1])) & set(tokenize(CONCEPTS[k][2])))
             for k in KEYS)
print(f"shared tokens between document-side and query-side form , 20 concepts : {shared}")
d = DOCUMENTS[0]
print(f"\n{d['code']} ({d['label']}) title : {d['title']}")
print(f"  paragraph 1 : {d['paragraphs'][1]}")
q = QUERIES[0]
print(f"query       : {q['text']}  -> concept {q['concept']} , class {q['label']}")
print(f"  {q['concept']} in document-side form '{CONCEPTS[q['concept']][1]}' ,"
      f" in query-side form '{CONCEPTS[q['concept']][2]}'")
u, v = embed(q["text"]), embed(q["text"])
print(f"\nembedding {len(u)} dimensions , second call same vector : {u == v}")
print("first six coordinates :", " ".join(f"{x:+.3f}" for x in u[:6]))

corpus: 200 documents , 693 paragraphs , 20 concepts , 5 classes
class distribution: tariff 52 , fault 46 , meter 39 , outage 32 , maintenance 31
shared tokens between document-side and query-side form , 20 concepts : 0

BK001 (maintenance) title : filter swap procedure
  paragraph 1 : calibration record for entry is kept and written together with filter swap
query       : how it gets saved form  -> concept reading , class meter
  reading in document-side form 'reading procedure' , in query-side form 'how it gets saved'

embedding 32 dimensions , second call same vector : True
first six coordinates : -0.018 -0.155 -0.145 -0.101 +0.243 +0.028

The first three lines of the output give the size of the corpus: two hundred documents, six hundred ninety-three paragraphs, twenty concepts, five classes. The classes are spread between fifty-two and thirty-one, so no class alone is a majority. The third line is the lesson’s mechanism: across all twenty concepts, the number of shared tokens between the document-side and the query-side form is zero.

The sample lines make this concrete. BK001 is a document from the maintenance class, its paragraphs written in the document-side form; the first query is how it gets saved form and its target concept is reading. The reading concept appears in documents as reading procedure and in the query as how it gets saved. Between the query and the document there is no word bridge; a bridge can be built only through the concept. The remaining two lines close the run question: two calls with the same text give the same vector. In a call that generates text, the same prompt could give two different responses in two runs; on the embedding surface it cannot.

The Numbers for Two Searches on the Same Query Set

Each of the forty queries is run in two setups. The first is keyword search: the number of a query’s distinct tokens that appear in a document is counted, the five highest-scoring documents are retrieved, and document order decides ties. The second is embedding search: the query and the documents are embedded with the same simulator, and the five documents with the highest cosine similarity are retrieved. Each setup produces four numbers; precision and recall are read over the whole retrieved set, concept and class hit rate over the first result only.

K = 5
TOKENS = [set(tokenize(doc_text(d))) for d in DOCUMENTS]
VEC = [embed(doc_text(d)) for d in DOCUMENTS]
RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS}


def top_k(score):
    return sorted(range(len(DOCUMENTS)), key=lambda i: (-score[i], i))[:K]


def keyword_search(q):
    qb = set(tokenize(q["text"]))
    return top_k([len(qb & TOKENS[i]) for i in range(len(DOCUMENTS))])


def embedding_search(q):
    qv = embed(q["text"])
    return top_k([cosine(qv, VEC[i]) for i in range(len(DOCUMENTS))])


def measure(retrieve):
    precision = recall = concept = label = 0.0
    for q in QUERIES:
        top = retrieve(q)
        d = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top)
        precision += d / K
        recall += d / RELEVANT[q["concept"]]
        concept += DOCUMENTS[top[0]]["main"] == q["concept"]
        label += DOCUMENTS[top[0]]["label"] == q["label"]
    n = len(QUERIES)
    return precision / n, recall / n, concept / n, label / n


mean_relevant = sum(RELEVANT[q["concept"]] for q in QUERIES) / len(QUERIES)
ceiling = sum(min(K, RELEVANT[q["concept"]]) / RELEVANT[q["concept"]] for q in QUERIES) / len(QUERIES)
print(f"{len(QUERIES)} queries , k={K} , relevant documents per query {mean_relevant:.3f} ,"
      f" recall ceiling {ceiling:.4f}")
print(f"cost: {len(QUERIES) * K} retrieved chunks , {len(QUERIES) * len(DOCUMENTS)} comparisons ,"
      f" {len(DOCUMENTS) + len(QUERIES)} embedding calls , {len(DOCUMENTS) * DIM} stored values")
print("\nsetup            precision  recall concept   class")
for name, retrieve in (("keyword search", keyword_search), ("embedding", embedding_search)):
    p, a, kv, sn = measure(retrieve)
    print(f"{name:<16} {p:9.4f} {a:7.4f} {kv:7.4f} {sn:7.4f}")

tied = 0
for q in QUERIES:
    qb = set(tokenize(q["text"]))
    score = [len(qb & TOKENS[i]) for i in range(len(DOCUMENTS))]
    tied += score.count(max(score))
print(f"\nmean number of documents sharing the highest keyword score :"
      f" {tied / len(QUERIES):.1f}")
q = QUERIES[0]
print(f"query : {q['text']}  (concept {q['concept']})")
for name, retrieve in (("keyword search", keyword_search), ("embedding", embedding_search)):
    print(f"  {name:<14}", " ".join(
        f"{DOCUMENTS[i]['code']}/{DOCUMENTS[i]['main']}" for i in retrieve(q)[:3]))

40 queries , k=5 , relevant documents per query 10.825 , recall ceiling 0.4904
cost: 200 retrieved chunks , 8000 comparisons , 240 embedding calls , 6400 stored values

setup            precision  recall concept   class
keyword search      0.0800  0.0363  0.0250  0.1000
embedding           0.4700  0.2258  0.5000  1.0000

mean number of documents sharing the highest keyword score : 79.2
query : how it gets saved form  (concept reading)
  keyword search BK001/filter_swap AR002/pressure SY003/reading
  embedding      SY025/replacement SY031/reading SY035/replacement

Reading the Table

The keyword search row looks like a search result but is not. The precision is 0.0800; given that an average of 10.825 relevant documents per query stand among two hundred documents, the precision of picking five documents at random would be 0.0541. The last lines say where the difference comes from: an average of 79.2 documents share the highest keyword score. What determines the first five is not relevance but the document order that breaks the tie. Recall stays at 0.0363.

The embedding row gives 0.4700 precision and 0.2258 recall on the same queries; precision rises from 0.0800 to 0.4700. Recall has a ceiling of its own: with an average of 10.825 relevant documents per query, if five documents are retrieved, recall can be at most 0.4904. The embedding gets close to half of this ceiling, and the remaining gap is the gap not of the scaffolding but of the number of retrieved chunks.

The first three results of the sample query make the difference concrete. Keyword search retrieves BK001, AR002 and SY003: three documents from three different classes, and the one document carrying the target concept has come to third place only because the tie was broken. Embedding search retrieves all three from the meter class and the second carries the target concept.

Why the Two Columns Stay Separate

The last two columns of the table are the first application of this course’s rule. The embedding search’s class hit rate is 1.0000: in forty of the forty queries the first result is from the right class. The same row’s concept hit rate is 0.5000: in twenty queries the first result is a document of the wrong concept, from within the right class.

What would have happened if it were reduced to a single measure. If the class hit rate were reported, the system would look flawless and there would be nothing left to improve. If the concept hit rate were reported, it would look as good as a coin toss, and the perfect hit at the class level would not be visible. Side by side the problem gets its name: the scaffolding finds the class and cannot find the concept. Since the wrongly retrieved document is from the same class, it is not irrelevant, only wrong; placed inside a response it does not look off-topic.

On the keyword search row the same distinction reads the other way round: class hit rate 0.1000, concept hit rate 0.0250. The class column does not soften the failure here: with five classes, a hit rate of one in ten is below even the share of naming a single class every time.

The cost row writes the price of the scaffolding: 8000 comparisons, 240 embedding calls, 6400 stored values, 200 retrieved chunks. Keyword search spends no embedding calls; the whole difference is what these 240 calls and 6400 values buy. As the corpus grows, the comparison count grows in direct proportion, because every query is compared with the whole corpus.

Summary

• A concept’s document-side surface form and query-side surface form can differ; in this corpus the number of shared tokens is zero for all twenty concepts.
• On the same forty queries, keyword search gives 0.0800 retrieval precision and 0.0363 retrieval recall, and embedding search gives 0.4700 and 0.2258.
• In keyword search an average of 79.2 documents share the highest score; the order that breaks the tie, not relevance, determines the first five.
• The recall ceiling for five chunks and an average of 10.825 relevant documents is 0.4904; the measured recall is read against this ceiling.
• Class hit rate is 1.0000 while concept hit rate is 0.5000; two columns name the problem, a single number hides it.
• The cost of retrieval is 8000 comparisons, 240 embedding calls, 6400 stored values and 200 retrieved chunks.

Next Step

This lesson used a single embedding, and that embedding recognizes the language of the corpus and the language of the query together. This is an assumption: if a ready-made embedding model was built with the texts of another domain, it may not recognize the query-side form. The next lesson builds two embeddings of the same dimension, one fitted to the language of this domain and the other not, and counts side by side the chunks the two retrieve on the same forty queries. The cost of the dimension decision was measured in the Building Applications with Model APIs course; what will be added on top of it is the share of domain fit.