Embedding Interfaces

The previous lesson measured the format of the input and wrote the bill of each format in tokens. The question left at the end was this: if we had to find which of the sixty records resembles the current request, since putting all of them into the prompt is not possible, what is similarity computed over?

The interface offers a separate surface for this. An embedding API takes a text and returns, as the response, not text but a fixed-length sequence of numbers. This surface has a single setting worth measuring, and this lesson measures it: how many dimensions the vector will have. As the dimension falls, the stored numbers and the comparison cost fall, and so does the neighborhood accuracy. The point at which the two part ways is counted.

• AU49. The code is a simulator; no real embedding endpoint is called. The records, the piece vocabulary and the embedding routine itself are fictional, seed 20260218.
• AU50. The simulator produces coordinates from the trace of the token and places the information weighted toward the first coordinates. The dimension parameter means returning that many coordinates from the start. This is the structural counterpart of an endpoint that offers a dimension parameter.
• AU51. The embedding API is deterministic: the same text gives the same vector on every call. This is the run-independent measure of this lesson and is shown in the block.
• AU52. That is why the second run is read from a separate baseline: a second embedding model of the same family (a simulator built with seed 20260219). For a dimension decision to be considered portable, the loss has to exceed the difference between the two models.
• AU53. Neighborhood accuracy is the share of the three nearest neighbors of each record being from the same topic. The measure is taken over three neighbors instead of one, because a single neighbor changes with the movement of a single pair.
• AU54. The retrieval pipeline is not built in this lesson. Which document enters the prompt, how many documents are retrieved and the cost of retrieving wrongly are left to the Embeddings and Retrieval-Augmented Generation course. Here only the standing of the vectors relative to each other is measured.
• AU55. Embedding, angle similarity and dimension reduction were measured in the Unsupervised Learning and the Natural Language Processing and Computer Vision courses of the Machine Learning and Data Science curriculum; those procedures are not repeated here. The only thing added is the dimension decision at the interface level.
• AU56. The comparison cost is written as the number of multiplications needed for all pairs of the seventy-two records. The storage cost is the count of numbers kept.

The Endpoint That Returns a Vector

The block below produces seventy-two fictional records, builds the embedding API and shows two things. The first is that the endpoint is deterministic: the same text gives the same vector in two calls, which means none of the numbers of this lesson carries run-to-run variation. The second is the mechanism: what happens, as the dimension falls, to the gap between the similarity to a record from the same topic and the similarity to a record from another topic.

# SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The records,
# the piece vocabulary and the embedding routine are fictional.
SEED, M32 = 20260218, 0xFFFFFFFF
DIMENSION = 144                         # the largest embedding dimension of the endpoint


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per piece
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PHRASES = {
    "fatura": ["fatura tutari cok yuksek geldi", "odemeyi yaptim borc gorunuyor",
               "fatura ayrintisini goremiyorum", "son odeme tarihi gecmis",
               "gecmis donem dokumu lazim", "tutar gecen aya gore artmis"],
    "ariza": ["evde su akmiyor bir ariza var", "baglanti bozuk hic calismiyor",
              "ariza kaydi actim ekip gelmedi", "vana kapali kalmis",
              "boru patlamis sokak su icinde", "servis ekibi randevuyu kacirdi"],
    "sayac": ["sayac okumasi yanlis girilmis", "endeks tutmuyor gibi",
              "sayac degisimi randevusu istiyorum", "sayac cami buharlanmis",
              "muhur kopmus gorunuyor", "endeks fotografini gonderdim"],
    "tarife": ["hangi tarife basamagindayim", "tarife degisikligi nasil olur",
               "birim ucret abone tipine gore degisiyor", "indirimli tarife sarti nedir",
               "kademeli hesap anlasilmiyor", "abone tipim yanlis kayitli"],
    "kesinti": ["mahallede kesinti var", "planli kesinti duyurusu nerede",
                "aksamdan beri kesinti yasiyoruz", "kesinti suresi uzadi",
                "bakim calismasi ne zaman biter", "sokak boyunca su yok"]}
TAILS = ["abone numaram kayitli", "bolge merkez ilce", "daha once de yazdim",
         "acil degil ama takipteyim", "telefonla ulasilamiyor", "ek belge gonderdim"]
RECORDS = []
for i in range(72):                     # 72 FICTIONAL records: 2-4 of its own phrases, sometimes a foreign one
    r = rng(SEED + 400 + i)
    k = TOPICS[int(r() * 5)]
    pool = list(PHRASES[k])
    chosen = [pool.pop(int(r() * len(pool))) for _ in range(2 + int(r() * 3))]
    if r() < 0.35:
        y = TOPICS[(TOPICS.index(k) + 1 + int(r() * 4)) % 5]
        chosen.append(PHRASES[y][int(r() * 6)])
    chosen.append(TAILS[int(r() * 6)])
    RECORDS.append({"number": 5001 + i, "topic": k, "text": " ".join(chosen)})
CALL_COUNT = [0]


def embed(text, dimension=DIMENSION, seed=SEED):
    """FICTIONAL embedding endpoint. The information is placed weighted toward the first
    coordinates; the dimension parameter means returning that many coordinates from the start."""
    CALL_COUNT[0] += 1
    toks = tokenize(text)
    v = [0.0] * DIMENSION
    for t in toks:
        r = rng(hash_text(t) ^ seed)
        for _ in range(10):             # each token touches ten coordinates
            i = int(r() ** 2 * DIMENSION)   # squaring gathers the weight into the first coordinates
            v[i] += 1.0 if r() < 0.5 else -1.0
    v = v[:dimension]
    n = sum(x * x for x in v) ** 0.5 or 1.0
    return [x / n for x in v], len(toks)


def fold(v, d):
    """Dimension reduction on the client side: coordinates are folded by summing."""
    k = [0.0] * d
    for i, x in enumerate(v):
        k[i % d] += x
    n = sum(x * x for x in k) ** 0.5 or 1.0
    return [x / n for x in k]


def similarity(u, v):
    return sum(a * b for a, b in zip(u, v))


x = RECORDS[0]
same = next(y for y in RECORDS[1:] if y["topic"] == x["topic"])
other = next(y for y in RECORDS[1:] if y["topic"] != x["topic"])
full, n_tokens = embed(x["text"])
print(f"record {x['number']} ({x['topic']}) : {x['text'][:56]}")
print(f"input tokens {n_tokens} , returned numbers {len(full)} , second call same vector:"
      f" {embed(x['text'])[0] == full}")
print("first eight coordinates:", " ".join(f"{c:+.3f}" for c in full[:8]))
print("last eight coordinates:", " ".join(f"{c:+.3f}" for c in full[-8:]))
print(f"\ndimension  {x['number']}-{same['number']} ({same['topic']})  "
      f"{x['number']}-{other['number']} ({other['topic']})  gap")
for d in (144, 36, 9):
    a, b = embed(x["text"], d)[0], embed(same["text"], d)[0]
    c = embed(other["text"], d)[0]
    ab, ac = similarity(a, b), similarity(a, c)
    print(f"{d:>9} {ab:>17.4f} {ac:>19.4f} {ab - ac:>8.4f}")

record 5001 (sayac) : muhur kopmus gorunuyor sayac cami buharlanmis endeks fot
input tokens 37 , returned numbers 144 , second call same vector: True
first eight coordinates: -0.329 +0.094 +0.094 +0.235 +0.047 +0.047 +0.141 -0.094
last eight coordinates: +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.047

dimension  5001-5002 (sayac)  5001-5003 (tarife)  gap
      144            0.5074              0.1790   0.3284
       36            0.3991              0.3544   0.0447
        9            0.4423              0.3922   0.0501

The first output line shows how this surface differs from the surface that returns text: an input of thirty-seven tokens turns into an output of a hundred and forty-four numbers, and the second call gives the same vector. In a call that generates text, the same prompt could give two different responses in two runs; here it cannot. All the differences of this lesson are therefore real differences, not run-to-run movement.

The coordinate lines make the simulator’s assumption visible. The first eight coordinates are full; seven of the last eight coordinates are zero. The information has been gathered in the leading coordinates; this is why the dimension parameter means cutting from the start.

The gap table is the real mechanism. At a hundred and forty-four dimensions the similarity to a neighbor from the same topic is 0.5074, to a neighbor from another topic 0.1790, and the gap is 0.3284. At thirty-six dimensions the two similarities move toward each other and the gap falls to 0.0447. As the dimension falls the vectors do not move away from each other, they move closer to each other; what makes telling them apart harder is not the distance itself but the closing of the gap between the distances. The 0.0501 gap at nine dimensions does not show that the gap has opened again; a single pair says no more than a single draw, and the decision is read from the table.

Cost and Accuracy Table of Five Dimensions

The seventy-two records are embedded separately at five dimensions. Each row prints the count of stored numbers, the number of multiplications needed to compare all pairs, the neighborhood accuracy, the accuracy of the second embedding model at the same dimension and the accuracy obtained by folding the full vector on the client side.

N, K = len(RECORDS), 3
CALL_COUNT[0] = 0
FULL = [embed(y["text"])[0] for y in RECORDS]
SECOND = [embed(y["text"], DIMENSION, SEED + 1)[0] for y in RECORDS]
INPUT_TOKENS = sum(embed(y["text"])[1] for y in RECORDS)


def truncate(v, d):
    u = v[:d]
    n = sum(a * a for a in u) ** 0.5 or 1.0
    return [a / n for a in u]


def neighbor_accuracy(vectors):
    """Share of the three nearest neighbors being from the same topic. No retrieval pipeline is built."""
    nearest = [[] for _ in range(N)]
    for i in range(N):
        for j in range(i + 1, N):
            s = similarity(vectors[i], vectors[j])
            nearest[i].append((s, j))
            nearest[j].append((s, i))
    d = 0
    for i in range(N):
        for _, j in sorted(nearest[i], reverse=True)[:K]:
            d += RECORDS[j]["topic"] == RECORDS[i]["topic"]
    return d / (N * K)


print(f"{N} records , {CALL_COUNT[0]} embedding calls , total input {INPUT_TOKENS} tokens"
      f" -- this number is independent of the dimension")
print("dimension    stored     comparison  endpoint  second model     client folding")
for d in (144, 72, 36, 18, 9):
    ep = neighbor_accuracy([truncate(v, d) for v in FULL])
    sm = neighbor_accuracy([truncate(v, d) for v in SECOND])
    fd = neighbor_accuracy([fold(v, d) for v in FULL])
    print(f"{d:>9} {N * d:>9} {N * (N - 1) // 2 * d:>14} {ep:>9.4f}"
          f" {sm:>13.4f} {fd:>18.4f}")

72 records , 216 embedding calls , total input 1814 tokens -- this number is independent of the dimension
dimension    stored     comparison  endpoint  second model     client folding
      144     10368         368064    0.9259        0.9491             0.9259
       72      5184         184032    0.9028        0.9259             0.9120
       36      2592          92016    0.8102        0.9120             0.9120
       18      1296          46008    0.7685        0.8611             0.8148
        9       648          23004    0.6389        0.7176             0.7315

The header line carries the number of this table that is easiest to overlook. The total input is 1814 tokens and this number is the same at all five dimensions. The dimension decision does not change the cost of the call: the same texts, the same tokens, the same number of calls. What changes starts after the call. The stored numbers fall from 10368 to 648, the comparison multiplications from 368064 to 23004. The bill of an endpoint and the cost of a system do not stand in the same place, and the dimension decision belongs to the second.

Which Baseline the Loss Is Read Against

If the endpoint column is read alone, the decision looks easy: as the dimension falls the accuracy falls, so the largest dimension is taken. The second model column upsets this reading. At a hundred and forty-four dimensions the two models give 0.9259 and 0.9491; the difference between them is 0.0232. The cost of halving the dimension to seventy-two is from 0.9259 to 0.9028, that is, 0.0231. The loss is of the same size as the difference between two embedding models at the same dimension. This decision has no measured loss, and in return it cuts storage and comparison exactly in half. This is the clearest gain in the table and it runs against intuition.

When we go down to thirty-six dimensions the situation changes. The endpoint gives 0.8102; the loss against a hundred and forty-four is 0.1157, and this is five times the difference between the two models. At nine dimensions the loss rises to 0.2870. The break lies between seventy-two and thirty-six: until three quarters of the storage is won, the cost stays at an unmeasurable level, then it is paid quickly.

The second model column says something in itself, too. The difference between the two models is 0.0232 at a hundred and forty-four and rises to 0.1018 at thirty-six. As the dimension falls, which model you use matters more: at a high dimension the two models build almost the same neighborhoods, at a low dimension they part ways. The cost of changing models in a low-dimensional setup is more than four times that in a high-dimensional one.

The last column asks who reduces the dimension. Taking the full vector and folding the coordinates gives 0.9120 at thirty-six dimensions; this is 0.1018 higher than the 0.8102 of requesting thirty-six dimensions from the endpoint. But this gain is exactly equal to the difference between the two models, and at eighteen dimensions it falls to 0.0463, that is, half of the model difference of that row. Against this baseline it is not a measured superiority. Moreover it has a cost: folding requires the full vector, that is, receiving and carrying a hundred and forty-four numbers on every call. The storage gain is kept, the transfer gain is lost. On which side the abstraction stops is decisive here too: the dimension parameter of the endpoint makes a decision in your place, and you can see the cost of that decision only when you put the two baselines side by side.

Summary

• The embedding API returns a fixed-length sequence of numbers instead of text and is deterministic: the same text gives the same vector on every call, so the differences of this lesson are not run-to-run movement.
• The dimension decision does not change the cost of the call; the total input tokens of the seventy-two records are 1814 at all five dimensions. What changes is the stored numbers (from 10368 to 648) and the comparison multiplications (from 368064 to 23004).
• As the dimension falls, the similarity gap between the same topic and another topic closes: on a single pair it goes from 0.3284 to 0.0447, and the neighborhood accuracy falls from 0.9259 to 0.6389.
• The loss from halving the dimension is 0.0231 and the difference from the second model at the same dimension is 0.0232: while storage and comparison are halved there is no measured loss. The break lies between seventy-two and thirty-six.
• The difference between the two models grows from 0.0232 to 0.1018 as the dimension falls; at a low dimension the choice of model weighs more.
• Folding the full vector on the client gains 0.1018 at thirty-six dimensions, but this gain is at the level of the model difference and requires carrying a hundred and forty-four numbers on every call.

Next Step

The embedding endpoint in this lesson and the text endpoint in the earlier lessons do not have to come from the same provider. The name of the dimension parameter, the scale of the distribution temperature, the shape of the error field and the key under which the response returns change from provider to provider. An application wants to handle these differences in a single place instead of at every call, and writes a common interface. The next lesson builds two providers at the simulator level and counts what that common interface does: how many fields are carried, how many fields are lost, and which of the lost fields makes a measure invisible.