Similarity Metrics

The previous two lessons always computed similarity with the same procedure: the vectors were brought to unit length, then the corresponding coordinates were multiplied and summed. The first step was a silent decision and was never discussed. If the vectors are not brought to unit length, the same product gives a different number and produces a different ranking.

The two measures are the names of these two cases. Cosine similarity measures the angle between two vectors and ignores their lengths. The dot product sums the coordinate products directly and carries the length inside it. The relation between them is a single line:

$$\text{dot product}(u, v) = \text{cosine similarity}(u, v) \cdot \lVert u \rVert \cdot \lVert v \rVert$$

When a query is fixed, $\lVert u \rVert$ is the same for all documents. The conclusion that follows is the axis of this lesson: the entire ranking difference between the two measures comes from the value $\lVert v \rVert$, that is, from the norm of the document vector.

• EV18. The code is a simulator, the corpus and the embedding procedure are fictional, the seed is 20260218. The dimension is thirty-two, the number of retrieved chunks is five, the definition of a relevant document is the same as in the previous two lessons, and the embedding is in-domain.
• EV19. One vector set, two measures. The shared embedding procedure brings the vector to unit length, the raw procedure does not; cosine similarity is the dot product of unit vectors.
• EV20. The relation above is checked with numbers in the block.
• EV21. Document length is measured in tokens; in the corpus it varies between thirty-two and one hundred seven.
• EV22. Divergence is counted over the set of the first three results: if the two measures retrieve the same three documents, they count as not diverged even if their order is different.
• EV23. The length bias is read against two baselines: the mean tokens of the retrieved set against the corpus mean, and the share of the longest quarter in the first five against that quarter’s share in the corpus.
• EV24. The run-independent measure is a second embedding model of the same family (projection seed 20260219). If the ranking does not come out in the same direction in the two models, it does not count as measured.

One Vector Set, Two Measures

The first block builds the corpus, adds the procedure that produces the unnormalized vector, and shows two things: how the norm of the document vector grows with length, and whether the relation holds.

# SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus,
# concept axis, embedding procedure and query set are FICTIONAL; none is a real
# institution, product or document.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":             ("tariff", "tier threshold", "which band am i in"),
    "unit_rate":        ("tariff", "unit price", "fee per cubic metre"),
    "subscriber_type":  ("tariff", "subscriber type", "home or shop"),
    "switch":           ("tariff", "tariff switch", "changing plan"),
    "fault_code":       ("fault", "fault code", "error id"),
    "leak":             ("fault", "line leakage", "water escaping"),
    "pressure":         ("fault", "pressure drop", "weak flow"),
    "crew":             ("fault", "field crew", "who will be sent"),
    "index_value":      ("meter", "index value", "what the dial shows"),
    "reading":          ("meter", "reading procedure", "how it gets saved"),
    "replacement":      ("meter", "meter replacement", "new hardware"),
    "verify":           ("meter", "reading verification", "typed in wrong"),
    "planned":          ("outage", "planned outage", "announced stoppage"),
    "notice":           ("outage", "notice channel", "where i hear about it"),
    "duration":         ("outage", "outage duration", "how long it lasts"),
    "restoration":      ("outage", "service restoration", "when it comes back"),
    "periodic":         ("maintenance", "periodic maintenance", "regular checkup"),
    "valve":            ("maintenance", "valve inspection", "tap shutoff routine"),
    "filter_swap":      ("maintenance", "filter swap", "cleaning part"),
    "calibration":      ("maintenance", "calibration record", "gauge accuracy"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "region", "term", "step", "form", "approval", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paragraphs, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts,
                      "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs})


def doc_text(d):
    return d["title"] + " " + " ".join(d["paragraphs"])


# ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions
def _to_concepts(m, in_domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc_form in dm:
            v[i] += 1.0
        if query_form in dm:
            v[i] += 1.0 if in_domain else 0.0      # an out-of-domain embedding does not recognize the query-side form
    if not in_domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, in_domain=True):
    k = _to_concepts(m, in_domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query-side form is separate from the document-side form
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# Two measures, one vector set: the shared embed() vector is brought to unit length,
# raw() is not. Cosine similarity is the dot product of unit vectors.
def raw(m, dim=DIM, in_domain=True):
    k = _to_concepts(m, in_domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    return v


def length(v):
    return sum(x * x for x in v) ** 0.5


TOKEN_COUNTS = [len(tokenize(doc_text(d))) for d in DOCUMENTS]
NORMS = [length(raw(doc_text(d))) for d in DOCUMENTS]
ordered = sorted(range(len(DOCUMENTS)), key=lambda i: TOKEN_COUNTS[i])
quarter = len(DOCUMENTS) // 4
shortest, longest = ordered[:quarter], ordered[-quarter:]
print(f"corpus: mean tokens {sum(TOKEN_COUNTS) / len(DOCUMENTS):.1f} ,"
      f" shortest {TOKEN_COUNTS[ordered[0]]} , longest {TOKEN_COUNTS[ordered[-1]]}")
print(f"  shortest quarter : mean {sum(TOKEN_COUNTS[i] for i in shortest) / quarter:.1f} tokens ,"
      f" raw vector length {sum(NORMS[i] for i in shortest) / quarter:.4f}")
print(f"  longest quarter  : mean {sum(TOKEN_COUNTS[i] for i in longest) / quarter:.1f} tokens ,"
      f" raw vector length {sum(NORMS[i] for i in longest) / quarter:.4f}")
q, d = QUERIES[0], DOCUMENTS[0]
raw_q, raw_d = raw(q["text"]), raw(doc_text(d))
print(f"\nidentity check , query 1 with {d['code']}:")
print(f"  dot product                    {cosine(raw_q, raw_d):.6f}")
print(f"  cosine similarity              {cosine(embed(q['text']), embed(doc_text(d))):.6f}")
print(f"  cosine * both vector lengths   "
      f"{cosine(embed(q['text']), embed(doc_text(d))) * length(raw_q) * length(raw_d):.6f}")

corpus: mean tokens 64.1 , shortest 32 , longest 107
  shortest quarter : mean 39.7 tokens , raw vector length 4.6737
  longest quarter  : mean 89.4 tokens , raw vector length 5.4708

identity check , query 1 with BK001:
  dot product                    -0.218248
  cosine similarity              -0.011168
  cosine * both vector lengths   -0.218248

The quarter lines give the mechanism. In the shortest quarter a document has a mean of 39.7 tokens and a raw vector norm of 4.6737; in the longest quarter 89.4 tokens and 5.4708. While the document grows to two and a quarter times, the norm grows only seventeen percent, but because the dot product is multiplied by that norm, this seventeen percent enters the score directly. The reason is written in the procedure: the more concepts a document mentions, the more weight falls on the concept vector.

The identity check shows that the relation holds: when the dot product is computed directly and when the cosine similarity is multiplied by the two norms, the same number comes out to six digits. The two measures are not separate procedures; one is the other multiplied by the norms.

How Much the Ranking Changes When the Measure Changes

The block below runs the same forty queries with two measures and repeats this in two embedding models. For each model it prints first the number of queries whose first three results diverge, then the four retrieval measures and the mean tokens of the retrieved set.

K = 5
RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS}

# SECOND embedding model of the same family: only the projection seed differs (20260219).
PROJECTION2 = []
for i in range(len(KEYS)):
    r = rng((SEED + 1) ^ (i + 1) * 2654435761)
    PROJECTION2.append([r() * 2 - 1 for _ in range(64)])


def raw2(m, dim=DIM):
    k = _to_concepts(m)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION2[i][j % 64]
    return v


def unit(v):
    n = length(v)
    return [x / n for x in v] if n else v


def rank(qv, vecs):
    return sorted(range(len(DOCUMENTS)), key=lambda i: (-cosine(qv, vecs[i]), i))


def measure(embedder):
    raw_vecs = [embedder(doc_text(d)) for d in DOCUMENTS]
    unit_vecs = [unit(v) for v in raw_vecs]
    counts = {"cosine": [0.0] * 5, "dot product": [0.0] * 5}
    diverged = 0
    for q in QUERIES:
        raw_q = embedder(q["text"])
        a, b = rank(unit(raw_q), unit_vecs), rank(raw_q, raw_vecs)
        diverged += set(a[:3]) != set(b[:3])
        for name, top in (("cosine", a[:K]), ("dot product", b[:K])):
            t = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top)
            row = counts[name]
            row[0] += t / K
            row[1] += t / RELEVANT[q["concept"]]
            row[2] += DOCUMENTS[top[0]]["main"] == q["concept"]
            row[3] += DOCUMENTS[top[0]]["label"] == q["label"]
            row[4] += sum(TOKEN_COUNTS[i] for i in top) / K
    n = len(QUERIES)
    return diverged, {name: [x / n for x in row] for name, row in counts.items()}


for model_name, embedder in (("model 1", raw), ("model 2", raw2)):
    diverged, results = measure(embedder)
    print(f"{model_name} : queries whose first three results diverge {diverged}/{len(QUERIES)}")
    print("  measure     precision  recall concept   class mean tokens")
    for name, row in results.items():
        print(f"  {name:<11} {row[0]:9.4f} " + " ".join(f"{x:7.4f}" for x in row[1:4]) + f" {row[4]:11.1f}")

model 1 : queries whose first three results diverge 30/40
  measure     precision  recall concept   class mean tokens
  cosine         0.4700  0.2258  0.5000  1.0000        57.7
  dot product    0.4100  0.1951  0.4250  1.0000        66.2
model 2 : queries whose first three results diverge 25/40
  measure     precision  recall concept   class mean tokens
  cosine         0.4050  0.2035  0.5250  1.0000        54.6
  dot product    0.3600  0.1751  0.3500  1.0000        58.3

The first line says the choice of measure is not a detail: in thirty of the forty queries the two measures retrieve a different top three. The same vectors, the same corpus, the same queries; the only thing that changes is whether the score is read from normalized vectors or from raw vectors.

Cosine similarity is ahead in three of the four measures and tied in the fourth: precision 0.4700 against 0.4100, recall 0.2258 against 0.1951, concept hit rate 0.5000 against 0.4250; class hit rate is 1.0000 for both. In the second embedding model the numbers change but the direction does not: 0.4050 against 0.3600 and 0.5250 against 0.3500. The ranking of the measures is not the draw of a single projection.

The last column shows where the difference comes from. The mean of the five documents retrieved by cosine similarity is 57.7 tokens, that of the dot product 66.2. The 8.5-token shift between them is not a byproduct; it is a direct consequence of the definition of the measure.

Looking at Length Where Length Carries No Relevance

For a shift to count as bias, it must be shown that length carries no relevance. The block below prints two baselines: the mean length of the corpus against the mean length of the relevant documents, and the share of the longest quarter in the first five of the two measures.

quarter = len(DOCUMENTS) // 4
longest = set(sorted(range(len(DOCUMENTS)), key=lambda i: -TOKEN_COUNTS[i])[:quarter])
relevant_docs = {k: [i for i, d in enumerate(DOCUMENTS) if d["main"] == k] for k in KEYS}
print(f"corpus mean {sum(TOKEN_COUNTS) / len(DOCUMENTS):.1f} tokens ,"
      f" longest quarter share {quarter / len(DOCUMENTS):.4f}")
print("mean of relevant documents %.1f tokens"
      % (sum(sum(TOKEN_COUNTS[i] for i in relevant_docs[q["concept"]]) / len(relevant_docs[q["concept"]])
             for q in QUERIES) / len(QUERIES)))
raw_vecs = [raw(doc_text(d)) for d in DOCUMENTS]
unit_vecs = [unit(v) for v in raw_vecs]
print("\nmeasure     longest-quarter share in first five")
for name, vecs, query_vec in (("cosine", unit_vecs, lambda q: unit(raw(q["text"]))),
                              ("dot product", raw_vecs, lambda q: raw(q["text"]))):
    p = sum(sum(1 for i in rank(query_vec(q), vecs)[:K] if i in longest) / K
            for q in QUERIES) / len(QUERIES)
    print(f"{name:<11} {p:35.4f}")

corpus mean 64.1 tokens , longest quarter share 0.2500
mean of relevant documents 64.7 tokens

measure     longest-quarter share in first five
cosine                                   0.1500
dot product                              0.2950

The first two lines give the verdict. The corpus mean is 64.1 tokens; the mean of the relevant documents is also 64.7. There is no link between length and relevance in this corpus; that is, a long document has no greater chance of answering the query than a short one.

The lower table writes how this shows up in the ranking. The longest quarter makes up 0.2500 of the corpus. In the dot product’s first five the share of this quarter is 0.2950, above the baseline rate; in the cosine similarity’s first five it is 0.1500, below the baseline rate. The two measures stand in two different places against length: the dot product keeps long documents the dot product keeps long documents above their share in the corpus, cosine similarity holds them below it.

Why the suppression works can be read from the definition of the norm. A long document mentions more concepts; its raw vector is longer but more scattered. Since cosine similarity throws the norm away by dividing, the remaining question is not “how many concepts are mentioned” but “which concept is given what share”; since the query targets a single concept, the winner is the concentrated document.

The conclusion that follows is independent of the names of the measures. If all vectors are of unit length, the two measures give the same ranking; when the norms equal one, the product changes nothing. The real decision is not “which measure” but whether the vectors are normalized when they are stored. If normalization is done, the dot product saves a division and the result is the same; if it is not, the choice of measure changes the retrieved chunk in thirty of the forty queries.

Summary

• The dot product is the cosine similarity multiplied by the norms of the two vectors; for a fixed query the entire ranking difference comes from the norm of the document vector.
• The norm of the document vector grows with length: 4.6737 in the shortest quarter, 5.4708 in the longest.
• In thirty of the forty queries the two measures retrieve a different top three.
• Cosine similarity is ahead in three measures and tied in class hit rate (0.4700 / 0.2258 / 0.5000 / 1.0000 against 0.4100 / 0.1951 / 0.4250 / 1.0000) and the direction stays the same in the second embedding model.
• The length bias is in the dot product: the mean of the retrieved set is 66.2 tokens, 57.7 under cosine similarity, 64.1 in the corpus; the share of the longest quarter in the first five is 0.2950 against 0.1500.
• The mean of the relevant documents is also 64.7 tokens, so length carries no relevance; if the vectors are stored at unit length, the two measures give the same ranking.

Next Step

In three lessons three decisions of retrieval were measured: words or meaning, which embedding model, which measure. There was one thing common to all three that was never asked: every query was compared with the whole corpus, 8000 comparisons were made for forty queries and two hundred documents, and the full ranking was built from scratch each time. With two hundred documents this is not a problem; when the corpus grows to hundreds of thousands of chunks, touching all the vectors for every query is not sustainable. The next lesson puts the vectors into an index and asks, taking the exhaustive search as the baseline: how much does the set retrieved without touching all the documents deviate from the set retrieved by touching them, and what recall is given in return for the comparison count.