Nondeterminism and Reproducibility

In the previous two lessons one threshold was used again and again: the difference between two runs. In the cost ledger it came out as 0.0200, in the outage table again 0.0200, and every gain below this number was counted as unmeasured. The threshold itself was never asked about. Why do the same code, the same workload and the same prompt give two different numbers in two runs.

Part of the question was closed earlier. The Visualization, Reporting and Analysis at Scale course wrote the reproduction of a report as fifty-nine items: the thirty-nine decisions of the preparation ledger, the nine items of the runtime environment and the eleven items of the report itself. The Deep Learning course did the same count for a training run, added the eight items specific to training and raised it to sixty-seven. That count is not repeated here. The added question is this: what are the items of a model call, and how many of them are in our hands.

• PQ45. The code is a simulator; no real endpoint is called. The tickets, the conversations and the documents are fictional, the seed is 20260218.
• PQ46. The workload is the same as in the previous two lessons: 400 requests from 154 fictional conversations.
• PQ47. An item is every decision that could determine the output of the call and that has to be carried from run to run. Each item is moved to an equally defensible alternative; not to a wrong value.
• PQ48. Two measures are printed: the changed-response share (how many of the 400 responses differ from those of the base run) and accuracy. The two are different things.
• PQ49. The items are split into two sides. Nine are on the client, three on the provider. The provider items can be pinned in the simulator but cannot be pinned on a real endpoint; in this lesson they count as unpinnable.
• PQ50. Inherited items: the 39 decisions of the preparation ledger and the 9 items of the runtime environment. The eight items specific to training are not inherited, because the model is given and is not trained.
• PQ51. The format word item of the instruction is the extreme of the list: the simulator ties the format of the response to the word one in the prompt. This is the modeled form of format compliance depending on the surface word.
• PQ52. The version change of the provider is modeled by adding two entries to the cue table.

The Items of a Call

# SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations
# and documents are fictional, seed 20260218.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW = 240


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per token
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                        "faturami odedim ama sistemde borc gorunuyor",
                        "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                         "planli kesinti duyurusu nereden takip edilir",
                         "aksamdan beri kesinti yasiyoruz"]}
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme",
        "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza",
        "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"}
FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi",
             "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek",
             "kaydiniz alindi ilgili birim sizinle iletisime gececek"]
COUNTER = [0]


def read_label(y):
    return next((s for s in y.split() if s in TOPICS), None)


DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir",
             "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir",
             "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir",
             "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir",
             "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"]
CONVERSATIONS, i = [], 0
while sum(len(c["turns"]) for c in CONVERSATIONS) < 400:    # FICTIONAL workload: conversations
    r = rng(SEED + 900 + i)
    i += 1
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])]
    CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]),
                          "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]}
                                    for t in range(u)]})
REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]}
            for c in CONVERSATIONS for t in range(len(c["turns"]))][:400]
DEFAULTS = dict(seed=SEED, temperature=0.6, max_output=24, word="tek", labels="listed",
                spacing="single", state="full", piece="document last", normalize="raw",
                tries=3, cues="version a", threshold=37)
CUES_B = dict(CUES, gecen="fatura", ekip="kesinti")     # the version change of the provider


def call_config(prompt, a):
    """Call with a settings dictionary: every item is a separate key."""
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "reply": ""}
    if COUNTER[0] % a["threshold"] == 0:
        return {"status": "rate_limit", "reply": ""}
    r = rng(a["seed"] ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "reply": ""}
    cues = CUES if a["cues"] == "version a" else CUES_B
    candidates = [k for k in CONCEPTS if k[:5] in tok]
    p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)}
    for t in tok:
        if cues.get(t) in p:
            p[cues[t]] += 1.0
    w = [math.exp(p[q] / a["temperature"]) for q in sorted(p)]
    k = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k)
    out_tok = tokenize(y)
    return {"status": "ok",
            "reply": " ".join(out_tok[:a["max_output"]]) if len(out_tok) > a["max_output"] else y}


def build_prompt(x, a):
    labels = TOPICS if a["labels"] == "listed" else sorted(TOPICS)
    instruction = ("Talebi su etiketlerden birine ata: " + " , ".join(labels)
                   + " . Yanit olarak " + ("tek" if a["word"] == "tek" else "yalnizca")
                   + " etiket yaz . Talep: ")
    past = x["conv"]["turns"][:x["pos"]]
    history = "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . "
                      for o in (past[-2:] if a["state"] == "two turns" else past))
    docs = "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . " for j in range(x["conv"]["doc"]))
    s = instruction + (docs + history if a["piece"] == "document first" else history + docs) \
        + x["ticket"]["text"]
    if a["spacing"] == "double":
        s = s.replace(" ", "  ")
    if a["normalize"] == "simplified":
        s = " ".join(s.lower().split())
    return s


def run(**d):
    a = dict(DEFAULTS, **d)
    COUNTER[0] = 0
    outputs = []
    for x in REQUESTS:
        prompt = build_prompt(x, a)
        for i in range(a["tries"]):
            s = call_config(prompt, dict(a, seed=a["seed"] + i))
            if s["status"] == "ok":
                break
        outputs.append(s["reply"])
    return outputs


def accuracy(c):
    return sum(read_label(y) == x["ticket"]["topic"] for y, x in zip(c, REQUESTS)) / 400


def changed(c, ref):
    return sum(a != b for a, b in zip(ref, c)) / 400


ITEMS = [("provider", "sampling state", dict(seed=SEED + 1)),
         ("provider", "model version", dict(cues="version b")),
         ("provider", "rate limit threshold", dict(threshold=31)),
         ("client", "distribution temperature", dict(temperature=0.8)),
         ("client", "maximum output length", dict(max_output=10)),
         ("client", "instruction format word", dict(word="yalnizca")),
         ("client", "label set order", dict(labels="alphabetical")),
         ("client", "whitespace layout", dict(spacing="double")),
         ("client", "conversation state", dict(state="two turns")),
         ("client", "context piece order", dict(piece="document first")),
         ("client", "client normalization", dict(normalize="simplified")),
         ("client", "retry count", dict(tries=1))]
BASE = run()
print("base accuracy %.4f , item count %d" % (accuracy(BASE), len(ITEMS)))
print("side       item                      changed accuracy     diff")
for side, name, d in ITEMS:
    c = run(**d)
    print(f"{side:<10} {name:<25} {changed(c, BASE):>7.4f} {accuracy(c):>8.4f}"
          f" {accuracy(c) - accuracy(BASE):>+8.4f}")
print("items that move", sum(changed(run(**d), BASE) > 0 for _, _, d in ITEMS), "/", len(ITEMS))

base accuracy 0.9125 , item count 12
side       item                      changed accuracy     diff
provider   sampling state             0.1200   0.8925  -0.0200
provider   model version              0.0150   0.9175  +0.0050
provider   rate limit threshold       0.0050   0.9075  -0.0050
client     distribution temperature   0.0600   0.8575  -0.0550
client     maximum output length      0.0000   0.9125  +0.0000
client     instruction format word    1.0000   0.5525  -0.3600
client     label set order            0.1100   0.9100  -0.0025
client     whitespace layout          0.0700   0.9325  +0.0200
client     conversation state         0.0050   0.9125  +0.0000
client     context piece order        0.0025   0.9150  +0.0025
client     client normalization       0.1300   0.8775  -0.0350
client     retry count                0.0375   0.8750  -0.0375
items that move 11 / 12

Of the twelve items, eleven move the responses when released alone. The most instructive rows of the list are the ones that were not expected to move anything. Whitespace layout does not touch the meaning of the prompt at all, it sends the same words in the same order, and it changes 0.0700 of the responses. Client normalization, while keeping the meaning in the same way, changes 0.1300 of them. The order of the label set writes the same five labels in another order and moves 0.1100. The text of a prompt is an input not by its meaning but by its exact self.

A changed response and a broken response are not the same thing, and the table keeps them in separate columns. Whitespace layout changes seven percent of the responses but raises accuracy from 0.9125 to 0.9325; the gain is accidental and can reverse in the next run. The label order moves 0.1100 and lowers accuracy by only 0.0025. By contrast, the distribution temperature moves only 0.0600 but lowers accuracy by 0.0550: few responses change, but nearly all of those that change go straight to wrong. The movement share alone is not a measure of harm.

Two rows come out smaller than expected. Conversation state moves only 0.0050, although the prompt text changed in 113 of the 400 requests; context piece order moves 0.0025, although 60 requests had two pieces at once. Long conversations are the most stable requests, because when traces from the same topic accumulate, the choice gathers on a single concept and the draw cannot disturb it. Stability varies by request; a single movement share does not describe the whole workload.

The largest row, though, is separated from the others by scale. When the format word of the instruction is replaced with its synonym, all of the responses change and accuracy falls from 0.9125 to 0.5525. This item changes not the draw but the task: once the format constraint is lifted, the model starts writing sentences instead of a single label, and a part of the sentences carry no label at all. A synonymous word lowered the measure by 0.3600.

Items Released Together

both = run(max_output=10, word="yalnizca")
single = run(word="yalnizca")
print("only maximum output length     changed %.4f  accuracy %.4f"
      % (changed(run(max_output=10), BASE), accuracy(run(max_output=10))))
print("only instruction format word   changed %.4f  accuracy %.4f"
      % (changed(single, BASE), accuracy(single)))
print("both together                  changed %.4f  accuracy %.4f"
      % (changed(both, BASE), accuracy(both)))
three = run(seed=SEED + 1, cues="version b", threshold=31)
print("nine client items fixed, three provider items released:"
      " changed %.4f  accuracy %.4f" % (changed(three, BASE), accuracy(three)))
print("fixable items 39 + 9 + 9 =", 39 + 9 + 9,
      ", unfixable 3 , total", 39 + 9 + 12)

only maximum output length     changed 0.0000  accuracy 0.9125
only instruction format word   changed 1.0000  accuracy 0.5525
both together                  changed 1.0000  accuracy 0.2675
nine client items fixed, three provider items released: changed 0.1150  accuracy 0.8925
fixable items 39 + 9 + 9 = 57 , unfixable 3 , total 60

The only zero in the table was the maximum output length. When the limit is lowered from twenty-four to ten, no response changes, because single-label responses are no longer than two tokens. When preparing a pinning list, dropping this item looks plausible.

The second and third rows show why that is wrong. When the format word is released alone, accuracy falls to 0.5525; when both are released together, it falls to 0.2675. The 0.2850 in between is the account of the item that alone moves nothing: the responses that turned into sentences are cut at ten tokens, and the label cannot be read in the cut text. Two more courses had reached the same result: the Visualization, Reporting and Analysis at Scale course measured that an item that alone does not move the report took the ratio from 87.168 to 94.432 together with another item, and the Deep Learning course measured that a number of rounds that alone moves zero shifted two subscribers together with the batch size. An item that does not move alone is not an item that need not be pinned.

Three Unpinnable Items

The last two rows give the closing number of this topic. Reproducing a model call requires pinning 60 items: the 39 decisions of the preparation ledger, the 9 items of the runtime environment and the 12 items counted in this lesson. 57 of these are in our hands. The remaining three are the sampling state, the model version and the rate limit threshold; they are in the hands of the provider and cannot be pinned by writing them to any file.

When these three are released together while the nine client items are pinned, 0.1150 of the responses move and accuracy falls from 0.9125 to 0.8925. The drop is 0.0200. This is exactly the threshold of the previous two lessons: the 0.0200 in the cost ledger and the 0.0200 in the outage table are not noise, they are the name of these three items. The threshold is not an arbitrarily chosen number, it is the measured share of the unpinnable items.

The conclusion from this is structurally different from that of the previous two courses. A report can be pinned with 59 of its 59 items, a training run with 67 of its 67; both are fully reproducible. A model call cannot be pinned. This is why determinism is not a goal but a limit, and the best that can be reached is for the movement to be measured. Its counterpart in practice is a single rule: the run-to-run difference is written next to every measurement report, and no gain below that difference counts as a gain.

Summary

• Reproducing a model call requires 60 items: the 39 inherited preparation decisions, the 9 environment items and the 12 items specific to the call.
• Of the twelve items, 11 move the responses when released alone; the largest is the format word of the instruction and it drops accuracy from 0.9125 to 0.5525.
• Items that change no meaning also move responses: whitespace layout 0.0700, client normalization 0.1300, label set order 0.1100.
• A changed response and a broken response are counted separately: whitespace layout moves 0.0700 and raises accuracy, while distribution temperature moves 0.0600 and lowers it by 0.0550.
• The maximum output length, which alone moves 0.0000, drops accuracy from 0.5525 to 0.2675 when released together with the format word.
• Three items are on the provider and cannot be pinned; when released together they move 0.1150 of the responses and accuracy falls by 0.0200, that is, the threshold of the previous two lessons is measured.

Next Step

In fourteen lessons the call was handled inside a product. What the message-based interface offers, what streaming changes, how many of the tool calls are valid, what the rate limit costs, what the cache gains, which class the latency falls to, which items the bill consists of, how the outage silently lowers quality and why the same input gives two outputs were counted. In all of these measurements the one making the call was an application, and the person who wrote that application stayed outside the measurement. Yet the same model also works at the developer’s own desk: while completing code, while explaining an existing function, while converting a file, even while taking over a multi-step task. The next topic carries the measure there and asks the same question: in a fictional codebase, in which task class does the bare call collapse, how many tokens does each context cost, and how many of the accepted suggestions are actually correct.