Compatible Interface Layers

The embedding endpoint in the previous lesson and the text endpoint in the lessons before it do not have to come from the same provider. As an application grows, a second provider is added, a task is moved to the cheaper one, another is kept at hand. At that point nobody wants to deal with field names written separately at every call site, and a common interface is written: a single function, a single set of names, and underneath a mapping that varies by provider.

This lesson counts what that layer does. Two providers are built at simulator level; both are fictional and both call the same engine underneath. What is measured is the fields the abstraction carries and the fields it loses: how many fields pass, how many fields drop, and which of the dropped fields makes a measure invisible. The rule built in the Data Analysis with Python course of the Data Analytics curriculum holds here as it is: an abstraction hides the decision beneath it.

• AU57. The code is a simulator. Both providers are fictional and are built at simulator level; no real endpoint is called. Seed 20260218, second run 20260219.
• AU58. The two providers share a single engine. So all the differences between them are interface differences, not model differences.
• AU59. Provider A offers twelve fields, provider B ten fields; the names are listed in the block. B has no stop sequence field and B’s variety setting is an integer between zero and four, that is, it fits its own scale.
• AU60. The common interface is the intersection of the two providers: only a field that has a counterpart in both can be carried. A field that exists on a single provider has no place in the common interface.
• AU61. A stop sequence is a field that cuts generation at a given token; in this lesson it takes effect at a word common to all three response patterns.
• AU62. The simulator writes the actual finish reason of each call to its own record; an application using the common interface would have no such record.

Two Providers and the Intersection Between Them

The block below builds the engine, the two providers, the field inventory and the common interface. The inventory is not a claim in prose but a list kept inside the code; the carried and lost fields are counted with set operations.

# SIMULATOR -- FICTIONAL. Both providers are fictional and are built at simulator
# level. No real model endpoint is called.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW = 240


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per piece
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
REQUESTS = []
for i in range(120):                    # 120 FICTIONAL support requests
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    REQUESTS.append({"id": 4001 + i, "topic": k,
                     "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]})
CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza",
         "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac",
         "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife",
         "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti",
         "mahal": "kesinti"}
CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
REPLIES = [                             # the third pattern carries no label
    "talebiniz {} konusunda kaydedildi ayrica ekibimiz sizinle iletisime gececek"
    " ve sonuc size bildirilecektir",
    "{} kaydiniz acilmistir ayrica islem numaraniz kisa mesajla tarafiniza"
    " gonderilecek ve surec takip edilebilecektir",
    "kaydiniz alinmistir ayrica ilgili birim inceleyip size donus yapacak"
    " ve gerekirse ek belge istenecektir"]
CALL_COUNT, ACTUAL = [0], []     # ACTUAL: the simulator's own record


def engine(prompt, max_length, temperature, stop, seed):
    """The single FICTIONAL engine that runs underneath both providers."""
    CALL_COUNT[0] += 1
    toks = tokenize(prompt)
    empty = {"text": "", "state": "error", "input": len(toks), "output": 0}
    if len(toks) > WINDOW or CALL_COUNT[0] % 37 == 0:
        name = "context_overflow" if len(toks) > WINDOW else "rate_limit"
        ACTUAL.append(name)
        return dict(empty, finish=name)
    r = rng(seed ^ hash_text(prompt))
    candidates = [k for k in CONCEPTS if k[:5] in toks]
    if len(candidates) < 3:
        candidates = list(CONCEPTS)
    p = {k: 0.15 for k in candidates}
    for t in toks:
        if CLUES.get(t) in p:
            p[CLUES[t]] += 1.0
    if temperature <= 0.0:
        choice = max(sorted(p), key=lambda a: p[a])
    else:
        names = sorted(p)
        w = [math.exp(p[a] / temperature) for a in names]
        choice = names[pick(r(), [q / sum(w) for q in w])]
    pattern = REPLIES[pick(r(), [0.40, 0.36, 0.24])]
    reply, finish = tokenize(pattern.format(choice)), "natural"
    for i, t in enumerate(reply):
        if t in stop:                   # stop sequence: generation is cut here
            reply, finish = reply[:i], "stopped"
            break
    if len(reply) > max_length:
        reply, finish = reply[:max_length], "cut"
    ACTUAL.append(finish)
    full_text = pattern.format(choice)
    return {"text": full_text if finish == "natural" else " ".join(reply), "state": "done",
            "input": len(toks), "output": len(reply), "finish": finish}


def provider_a(prompt, at_most=14, distribution=0.6, stop=(), mode="free", seed=SEED):
    """FICTIONAL provider A. The field names and the response format are its own."""
    s = engine(prompt, at_most, distribution, stop, seed)
    return {"text": s["text"], "state": s["state"], "input_tokens": s["input"],
            "output_tokens": s["output"], "finish": s["finish"],
            "truncated": s["finish"] == "cut"}


def provider_b(message, limit=14, variety=2, schema=None, key=SEED):
    """FICTIONAL provider B. variety is an integer between 0 and 4 and fits its own scale."""
    s = engine(message, limit, variety / 4 * 1.6, (), key)
    return {"result": s["text"], "status": s["state"], "reason": s["finish"],
            "usage": {"i": s["input"], "o": s["output"]}}


FIELDS = {"A": ["prompt", "at_most", "distribution", "stop", "mode", "seed",
                "text", "state", "input_tokens", "output_tokens", "finish", "truncated"],
          "B": ["message", "limit", "variety", "schema", "key",
                "result", "status", "usage.i", "usage.o", "reason"]}
MAPPING = {"prompt": ("prompt", "message"), "max_length": ("at_most", "limit"),
           "temperature": ("distribution", "variety"), "seed": ("seed", "key"),
           "response": ("text", "result"), "status": ("state", "status"),
           "input": ("input_tokens", "usage.i"), "output": ("output_tokens", "usage.o")}


def common(provider, prompt, max_length=14, temperature=0.6, seed=SEED):
    """Common interface: only the eight fields in MAPPING are carried."""
    if provider == "A":
        y = provider_a(prompt, at_most=max_length, distribution=temperature, seed=seed)
        return {"response": y["text"], "status": y["state"],
                "input": y["input_tokens"], "output": y["output_tokens"]}
    y = provider_b(prompt, limit=max_length, variety=round(temperature * 4), key=seed)
    return {"response": y["result"], "status": y["status"],
            "input": y["usage"]["i"], "output": y["usage"]["o"]}


PREFIX5 = {k[:5]: k for k in TOPICS}    # in a cut response the label may be split


def read_label(y):
    return next((PREFIX5[s[:5]] for s in y.split() if s[:5] in PREFIX5), None)


DIRECTIVE = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife ,"
             " kesinti . Talep: ")


print("fields of the common interface :", " , ".join(MAPPING))
for i, name in enumerate(FIELDS):
    carried = [e[i] for e in MAPPING.values()]
    lost = [a for a in FIELDS[name] if a not in carried]
    print(f"provider {name}: {len(FIELDS[name])} fields , carried {len(carried)} ,"
          f" lost {len(lost)} -> {' , '.join(lost)}")
prompt = DIRECTIVE + REQUESTS[2]["text"]
print("\nsame prompt , four paths -- fields other than the text:")
for name, y in (("A native + stop", provider_a(prompt, stop=("ayric",))),
                ("B native", provider_b(prompt)),
                ("common -> A", common("A", prompt)),
                ("common -> B", common("B", prompt))):
    fields = " , ".join(f"{k}={v}" for k, v in y.items()
                        if k not in ("text", "result", "response"))
    print(f" {name:<15} {fields}")

fields of the common interface : prompt , max_length , temperature , seed , response , status , input , output
provider A: 12 fields , carried 8 , lost 4 -> stop , mode , finish , truncated
provider B: 10 fields , carried 8 , lost 2 -> schema , reason

same prompt , four paths -- fields other than the text:
 A native + stop state=done , input_tokens=33 , output_tokens=5 , finish=stopped , truncated=False
 B native        status=done , reason=cut , usage={'i': 33, 'o': 14}
 common -> A     status=done , input=33 , output=14
 common -> B     status=done , input=33 , output=14

The inventory lines give the arithmetic of the abstraction. Provider A offers twelve fields, eight are carried, four are lost. Provider B offers ten fields, eight are carried, two are lost. The reason for the loss is not that the common interface is badly designed: stop and mode exist only in A, schema only in B. The intersection can carry only what has a counterpart on both sides, and the more fields a provider offers, the more of its fields it loses in the common interface.

The four response formats stand side by side on the same prompt. When A is called with its own interface, the stop sequence does its job: the output is five tokens and the finish field says stopped. When B is called with its own interface, the output is fourteen tokens and the reason field says cut, that is, the response was cut. The two rows looking from the common interface are indistinguishable: both say status: done and output: 14. Whether the response was completed or not is not written in these two rows.

The Cost of the Lost Fields

A hundred and twenty fictional requests are passed through five paths. The first two are A’s own interface (without and with the stop sequence), the third is B’s own interface, the last two are the common interface. Each row prints the input and output tokens, the label accuracy, the second run accuracy, the actual number of cuts the simulator recorded and the number of cuts readable from that path.

PATHS = ("A native", "A native + stop", "B native", "common -> A", "common -> B")


def run(path, temperature=0.6, seed=SEED):
    CALL_COUNT[0], ACTUAL[:] = 0, []
    tok_in = tok_out = hits = readable = 0
    for x in REQUESTS:
        prompt = DIRECTIVE + x["text"]
        if path.startswith("A native"):
            y = provider_a(prompt, distribution=temperature, seed=seed,
                           stop=("ayric",) if "stop" in path else ())
            m, t_in, t_out = y["text"], y["input_tokens"], y["output_tokens"]
            readable += y["truncated"]              # the finish field is readable
        elif path == "B native":
            y = provider_b(prompt, variety=2, key=seed)
            m, t_in, t_out = y["result"], y["usage"]["i"], y["usage"]["o"]
            readable += y["reason"] == "cut"        # the reason field is readable
        else:
            y = common(path[-1], prompt, temperature=temperature, seed=seed)
            m, t_in, t_out = y["response"], y["input"], y["output"]   # no finish reason
        tok_in, tok_out = tok_in + t_in, tok_out + t_out
        hits += read_label(m) == x["topic"]
    return tok_in, tok_out, hits / 120, ACTUAL.count("cut"), readable


print("path             input output accuracy    second run     actual cuts    read")
for path in PATHS:
    a, b = run(path), run(path, seed=SEED + 1)
    print(f"{path:<15} {a[0]:>6} {a[1]:>6} {a[2]:>8.4f} {b[2]:>13.4f}"
          f" {a[3]:>15} {a[4]:>7}")
print("\nsame common call , two providers:")
print("temperature variety sent to B common -> A common -> B difference")
for s in (0.0, 0.2, 0.4, 0.6, 0.8, 1.0):
    a, b = run("common -> A", s)[2], run("common -> B", s)[2]
    print(f"{s:>11.1f} {round(s * 4):>17} {a:>11.4f} {b:>11.4f} {b - a:>+10.4f}")

path             input output accuracy    second run     actual cuts    read
A native          4054   1638   0.5500        0.5000             117     117
A native + stop   4054    773   0.5500        0.5000               0       0
B native          4054   1638   0.5000        0.3917             117     117
common -> A       4054   1638   0.5500        0.5000             117       0
common -> B       4054   1638   0.5000        0.3917             117       0

same common call , two providers:
temperature variety sent to B common -> A common -> B difference
        0.0                 0      0.8417      0.8417    +0.0000
        0.2                 1      0.7583      0.7583    +0.0000
        0.4                 2      0.7583      0.5000    -0.2583
        0.6                 2      0.5500      0.5000    -0.0500
        0.8                 3      0.5000      0.3167    -0.1833
        1.0                 4      0.4500      0.3167    -0.1333

The second row is the cost of the lost stop field. When the stop sequence is used the output tokens are 773, when it is not, 1638: more than double. The accuracy is 0.5500 in both rows, that is, none of these tokens is useful; the polite closing sentence that comes after the body of the pattern is paid for and thrown away. The common interface cannot offer this field, because it has no counterpart in B. The cost of the intersection is written here directly in tokens.

The last two columns are the cost of the lost finish and reason fields, and this is the harshest row of the lesson. Of the hundred and twenty calls, 117 are cut at the length limit. A reports this with the truncated field in its own interface, B with the reason field in its own interface: both providers are honest. The number of cuts readable from the common interface is 0. The cutting did not disappear, it became uncountable. The rule of the course fits exactly here: a call whose failure is not written down counts as unmeasured. An application using the common interface writes a hundred and seventeen half-finished responses to its record system and not a single counter of it moves.

The only thing that can take the place of the lost field is an inference: if output equals the limit, the response is counted as cut. This inference also counts as cut the responses that end naturally by landing exactly on the limit, that is, a new class of error takes the place of a measurement field.

The Same Knob, Two Different Meanings

The number of fields is only one face of the abstraction. The assumption that the eight carried fields express the same thing is the second and more insidious face. The temperature field of the common interface goes to A as it is; on its way to B it is rounded to an integer between zero and four, because that is B’s scale.

The lower table shows two separate consequences of this. The first: the same setting means two different behaviors. At temperature 0.4, A gives 0.7583 and B gives 0.5000; the difference is 0.2583. The two numbers in the same row were produced with the same argument of the same function. The second: two different settings mean the same behavior. Temperatures 0.4 and 0.6 are rounded to the same variety value on B and both give 0.5000, while on A the two settings give 0.7583 and 0.5500, a separation of 0.2083. The continuous-looking knob of the common interface has five positions on B, and the positions in between are silently swallowed.

Which of these readings counts as measured comes from the second run column. The difference between the two runs is 0.0500 for A and 0.1083 for B. The provider difference of 0.0500 in the temperature 0.6 row is exactly the run difference and counts as unmeasured; at that value no choice can be made between the two providers. The 0.2583 in the temperature 0.4 row, on the other hand, is five times the run difference and is real. When an application using the common interface changes providers, it sees this difference nowhere: the field names stay the same, the call stays the same, only the number changes.

Summary

• The common interface is the intersection of two providers: of A’s twelve fields eight are carried and four drop, of B’s ten fields eight are carried and two drop.
• A field that exists on a single provider cannot enter the intersection; stop and mode stay in A, schema in B.
• The loss of the stop sequence is paid for in tokens: the output rises from 773 to 1638 while the accuracy stays fixed at 0.5500, that is, the added tokens are of no use.
• Of the hundred and twenty calls, 117 are cut at the length limit; both providers report this in their own field, and the number of cuts readable from the common interface is 0.
• A carried field may not mean the same thing either: at temperature 0.4 the two providers separate by 0.2583, and 0.4 and 0.6 collapse to a single value because they are rounded to the same setting on B.
• Counting a provider difference as measured depends on its exceeding the run difference: the 0.0500 in the 0.6 row is the run difference itself, the 0.2583 in the 0.4 row is five times it.

Next Step

Up to this lesson every measure was written under the assumption of one call per request: a hundred and twenty requests, a hundred and twenty calls. Yet the directive is sent again on every call, and this is why the input tokens stay fixed at 4054 in the table. If the requests had been accumulated and sent in a single call, this repetition would not have been paid for. The next lesson measures the batch call and ties it to a single question: as the batch size grows, the number of calls and tokens fall, but what happens to the number of requests affected by a single failure.