Embedding Models

The previous lesson used a single embedding, and that embedding recognized both the language of the corpus and the language of the query. Against the keyword search baseline, retrieval precision rose from 0.0800 to 0.4700 and concept hit rate from 0.0250 to 0.5000. That was not the number of an embedding model; it was the number of an embedding model that recognizes the query language. With a ready-made model this assumption does not come true by itself.

When an embedding model is chosen, three settings are discussed: dimension, domain fit and language coverage. The cost of the dimension decision was measured in the Building Applications with Model APIs course: the share of the three nearest neighbors from the same topic was 0.9259 at one hundred forty-four dimensions and 0.9028 at seventy-two dimensions, and the stored value count fell from 10368 to half. That procedure is not repeated; what this lesson adds is the share of domain fit at the same dimension. Language coverage looks like a third setting, but in this measure it is the same surface: the question in both cases is whether the model recognizes the query language.

• EV11. All the code is a simulator; the corpus, concept axis and embedding procedure are fictional. The seed is 20260218. No real embedding endpoint is called.
• EV12. Domain fit is modeled explicitly in the simulator. The in-domain embedding recognizes both surface forms of a concept: pressure drop in the document-side form and weak flow in the query-side form go to the same concept. The out-of-domain embedding recognizes only the document-side form; it does not see the query-side form, and the only signal left to it is the common words shared with the documents. This is the structural counterpart of an embedding built with the texts of another domain.
• EV13. The dimension is held fixed: thirty-two. The two setups store the same number of coordinates and perform the same number of multiplications; the difference between them comes only from domain fit.
• EV14. A second embedding model of the same family is built; only the projection seed is different (20260219). For a loss to count as measured, it must exceed the difference between the two models. This is the lesson’s run-independent baseline.
• EV15. The baseline is again keyword search; the measure definitions are the same as in the previous lesson: five retrieved chunks, and a relevant document is a document whose main concept is the same as the query’s target concept.
• EV16. The dimension sweep is read with two models together; if the two models do not give the same ranking, there is no measured order in that range.
• EV17. The storage cost is the number of documents times the dimension, that is, the stored value count.

What Domain Fit Models

The first block builds the corpus, the concept axis and the embedding simulator; at the end it shows domain fit on a single query. When the in-domain embedding sees the query-side form, it writes the concept’s weight in full; the out-of-domain embedding cannot see that concept, and what remains is a thin weight distributed by the common words.

# SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus,
# concept axis, embedding procedure and query set are FICTIONAL; none is a real
# institution, product or document.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":             ("tariff", "tier threshold", "which band am i in"),
    "unit_rate":        ("tariff", "unit price", "fee per cubic metre"),
    "subscriber_type":  ("tariff", "subscriber type", "home or shop"),
    "switch":           ("tariff", "tariff switch", "changing plan"),
    "fault_code":       ("fault", "fault code", "error id"),
    "leak":             ("fault", "line leakage", "water escaping"),
    "pressure":         ("fault", "pressure drop", "weak flow"),
    "crew":             ("fault", "field crew", "who will be sent"),
    "index_value":      ("meter", "index value", "what the dial shows"),
    "reading":          ("meter", "reading procedure", "how it gets saved"),
    "replacement":      ("meter", "meter replacement", "new hardware"),
    "verify":           ("meter", "reading verification", "typed in wrong"),
    "planned":          ("outage", "planned outage", "announced stoppage"),
    "notice":           ("outage", "notice channel", "where i hear about it"),
    "duration":         ("outage", "outage duration", "how long it lasts"),
    "restoration":      ("outage", "service restoration", "when it comes back"),
    "periodic":         ("maintenance", "periodic maintenance", "regular checkup"),
    "valve":            ("maintenance", "valve inspection", "tap shutoff routine"),
    "filter_swap":      ("maintenance", "filter swap", "cleaning part"),
    "calibration":      ("maintenance", "calibration record", "gauge accuracy"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "region", "term", "step", "form", "approval", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paragraphs, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts,
                      "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs})


def doc_text(d):
    return d["title"] + " " + " ".join(d["paragraphs"])


# ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions
def _to_concepts(m, in_domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc_form in dm:
            v[i] += 1.0
        if query_form in dm:
            v[i] += 1.0 if in_domain else 0.0      # an out-of-domain embedding does not recognize the query-side form
    if not in_domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, in_domain=True):
    k = _to_concepts(m, in_domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query-side form is separate from the document-side form
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

print(f"corpus: {len(DOCUMENTS)} documents , {len(KEYS)} concepts , {len(CLASSES)} classes ,"
      f" {len(QUERIES)} queries , embedding dimension {DIM}")
q = QUERIES[3]
k = q["concept"]
print(f"query : {q['text']}  -> concept {k} , class {q['label']}")
print(f"  {k} in document-side form '{CONCEPTS[k][1]}' , in query-side form '{CONCEPTS[k][2]}'")
print(f"  in-domain embedding weight of {k} in the concept vector :"
      f" {_to_concepts(q['text'])[KEYS.index(k)]:.2f}")
print(f"  out-of-domain embedding weight of {k} in the concept vector :"
      f" {_to_concepts(q['text'], in_domain=False)[KEYS.index(k)]:.2f}")

corpus: 200 documents , 20 concepts , 5 classes , 40 queries , embedding dimension 32
query : weak flow region  -> concept pressure , class fault
  pressure in document-side form 'pressure drop' , in query-side form 'weak flow'
  in-domain embedding weight of pressure in the concept vector : 1.00
  out-of-domain embedding weight of pressure in the concept vector : 0.05

The last two lines carry the whole measure. For the pressure concept the weight is 1.00 in the in-domain embedding and 0.05 in the out-of-domain embedding. The second is not a concept signal but the base noise that the common words distribute equally to all concepts: the out-of-domain embedding puts the query at a point equidistant from twenty concepts.

Two Embeddings at the Same Dimension

The block below runs five setups on the same forty queries: the keyword search baseline, the two models of the in-domain embedding and the two models of the out-of-domain embedding. All of them run at thirty-two dimensions and store the same number of coordinates. The last column counts how many distinct top-five sets the forty queries produce; whether a ranker changes from query to query is read from there. The last two lines write separately the difference between two models that stay in the same domain and the loss of falling out of domain.

K = 5
RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS}
TOKENS = [set(tokenize(doc_text(d))) for d in DOCUMENTS]

# SECOND embedding model of the same family: only the projection seed differs (20260219).
PROJECTION2 = []
for i in range(len(KEYS)):
    r = rng((SEED + 1) ^ (i + 1) * 2654435761)
    PROJECTION2.append([r() * 2 - 1 for _ in range(64)])


def embed2(m, dim=DIM, in_domain=True):
    k = _to_concepts(m, in_domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION2[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def measure(rank):
    """Four numbers and the number of DISTINCT top-five sets the forty queries produce."""
    precision = recall = concept = label = 0.0
    top_sets = set()
    for q in QUERIES:
        top = rank(q)[:K]
        top_sets.add(tuple(sorted(top)))
        t = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top)
        precision += t / K
        recall += t / RELEVANT[q["concept"]]
        concept += DOCUMENTS[top[0]]["main"] == q["concept"]
        label += DOCUMENTS[top[0]]["label"] == q["label"]
    n = len(QUERIES)
    return precision / n, recall / n, concept / n, label / n, len(top_sets)


def keyword_rank(q):
    qb = set(tokenize(q["text"]))
    return sorted(range(len(DOCUMENTS)), key=lambda i: (-len(qb & TOKENS[i]), i))


def embedding_rank(embedder, in_domain, dim=DIM):
    vecs = [embedder(doc_text(d), dim, in_domain) for d in DOCUMENTS]

    def rank(q):
        qv = embedder(q["text"], dim, in_domain)
        return sorted(range(len(DOCUMENTS)), key=lambda i: (-cosine(qv, vecs[i]), i))
    return rank


print("setup                   precision  recall concept   class  sets")
results = {}
for name, rank in (("keyword search", keyword_rank),
                   ("in-domain     , model 1", embedding_rank(embed, True)),
                   ("in-domain     , model 2", embedding_rank(embed2, True)),
                   ("out-of-domain , model 1", embedding_rank(embed, False)),
                   ("out-of-domain , model 2", embedding_rank(embed2, False))):
    results[name] = measure(rank)
    print(f"{name:<23} {results[name][0]:9.4f} " + " ".join(f"{x:7.4f}" for x in results[name][1:4])
          + f" {results[name][4]:5d}")

in1, in2 = results["in-domain     , model 1"], results["in-domain     , model 2"]
out1 = results["out-of-domain , model 1"]
print("\ndifference between the two models (in-domain) :",
      " ".join(f"{abs(a - b):7.4f}" for a, b in zip(in1[:4], in2[:4])))
print("loss from falling out of domain (model 1)     :",
      " ".join(f"{a - b:7.4f}" for a, b in zip(in1[:4], out1[:4])))
print(f"stored values , same in every setup : {len(DOCUMENTS) * DIM}")

setup                   precision  recall concept   class  sets
keyword search             0.0800  0.0363  0.0250  0.1000     7
in-domain     , model 1    0.4700  0.2258  0.5000  1.0000    17
in-domain     , model 2    0.4050  0.2035  0.5250  1.0000    17
out-of-domain , model 1    0.0650  0.0316  0.0500  0.2500     1
out-of-domain , model 2    0.0450  0.0247  0.0250  0.2500     1

difference between the two models (in-domain) :  0.0650  0.0223  0.0250  0.0000
loss from falling out of domain (model 1)     :  0.4050  0.1942  0.4500  0.7500
stored values , same in every setup : 6400

The most striking part of the table is the concept and class columns of the out-of-domain embedding: 0.0500 and 0.2500. The keyword search baseline gives 0.0250 and 0.1000; the out-of-domain embedding rises only a step above it, and stays a full 0.4500 below the in-domain embedding on concept hit rate. It produces vectors, computes cosine similarity, ranks, and gets from the first result barely more than a word match does. Dimension, storage and comparison are the same; almost all of what the scaffolding was built for has been wiped out.

Looking at the retrieved set, the situation is worse. The out-of-domain embedding’s retrieval precision is 0.0650 and its recall 0.0316; the keyword search baseline has 0.0800 and 0.0363. That is, the five-document set is worse than when no embedding was used at all. The added scaffolding did not raise the number, it lowered it. The reason is written in the last column: the out-of-domain setup produces a single top-five set on forty queries. Since all queries fall on the same point, the same five documents go to all of them. The keyword search baseline produces seven distinct sets, the in-domain embedding seventeen.

The last two lines decide whether the loss counts as measured. The difference between the two models is 0.0650 in precision, 0.0223 in recall, 0.0250 in concept hit rate and 0.0000 in class hit rate. The loss from falling out of domain is 0.4050, 0.1942, 0.4500 and 0.7500; in concept hit rate, eighteen times the model difference. In the Building Applications with Model APIs course, the loss from halving the dimension came out the same size as the model difference and was counted as unmeasured. The loss here is far above that threshold: domain fit is not a setting to be put on the same scale as dimension.

A language list of an embedding model does not take the place of this measure. A language being on the list means the model has seen the document texts in that language; it does not mean it has seen the words with which the user asks about the same concept.

Where Dimension Dies

What happens if domain fit is fixed and the dimension is swept. The block below runs the in-domain setup at five dimensions with two models.

print("  dim stored values  model 1 concept / class model 2 concept / class")
for d in (64, 32, 16, 8, 4):
    a = measure(embedding_rank(embed, True, d))
    b = measure(embedding_rank(embed2, True, d))
    print(f"{d:>5} {len(DOCUMENTS) * d:>13} {a[2]:>16.4f} {a[3]:>7.4f}"
          f" {b[2]:>15.4f} {b[3]:>7.4f}")

  dim stored values  model 1 concept / class model 2 concept / class
   64         12800           0.4250  1.0000          0.4500  1.0000
   32          6400           0.5000  1.0000          0.5250  1.0000
   16          3200           0.4500  1.0000          0.4000  0.9500
    8          1600           0.6500  1.0000          0.4750  0.8250
    4           800           0.3250  0.6000          0.3500  0.7000

The concept columns are not monotonic between 64 and 8. The first model gives 0.4250, 0.5000, 0.4500, 0.6500 and gets its highest value at the smallest dimension; the second model gives 0.4500, 0.5250, 0.4000, 0.4750 and gets its highest value at thirty-two. The two models do not agree on which dimension is better, and the difference at eight dimensions is 0.1750, seven times the model difference at thirty-two. In this range the dimension ranking is unmeasured; going down to eight by looking at the sweep of a single model would be relying on a draw.

The class column is more stable, and it carries the real information. In the first model it does not go below 1.0000 down to eight dimensions; in the second it goes down to 0.9500 at sixteen and to 0.8250 at eight. At four dimensions both collapse together: 0.6000 and 0.7000. The cliff lies between eight and four dimensions, and its cause can be counted: when twenty concepts are squeezed into four coordinates, two concepts falling in the same direction is unavoidable.

The storage side completes the reading. Going from sixty-four dimensions to eight lowers the stored value count from 12800 to 1600 and leaves no measured loss in class hit rate in the first model; going down to four only halves the storage again while cutting the class hit rate to 0.6000. The shape of the dimension decision is in the same direction as what was seen in the Building Applications with Model APIs course: free over a long range, then suddenly expensive.

Summary

• Domain fit is whether an embedding model ties the query-side surface form to the same concept as the document-side surface form; language coverage is another name for the same question.
• At the same thirty-two dimensions and the same 6400 stored values, the in-domain embedding gives 0.4700 precision and 0.5000 concept hit rate, and the out-of-domain embedding gives 0.0650 and 0.0500.
• The out-of-domain embedding’s concept and class hit rate (0.0500 / 0.2500) are only a step above the keyword search baseline’s (0.0250 / 0.1000); on the retrieved set it falls below the baseline, because it gives a single top-five set to forty queries.
• The difference between the two models is 0.0250 in concept hit rate and the loss from falling out of domain is 0.4500; the first is an unmeasured difference, the second a measured one.
• In the dimension sweep the two models do not agree between 64 and 8; in that range there is no measured order.
• The cliff lies between 8 and 4 dimensions and the threshold comes from the number of concepts: twenty concepts do not fit in four.

Next Step

In these two lessons the similarity was computed with a single procedure: corresponding coordinates were multiplied and summed, and before that the vectors were brought to unit length. The second step is a silent decision. If normalization is not done, the same product produces a different ranking and long documents start to get higher scores. The next lesson puts cosine similarity and the dot product side by side on the same forty queries: in how many queries the first three results diverge, which measure is biased toward length, and in which direction this bias moves the correctness of the retrieved chunk.