--- title: 'Building Applications with Model APIs' source: 'https://academia.sh/en/courses/model-apis' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' lessons: 26 outcomes: - 'Uses model interfaces with streaming, tool calls and structured output and writes the cost of each' - 'Measures and improves latency and token cost and shows where the resilience scaffolding breaks' - 'Compares self-hosted and remotely called options along the axes of memory, accuracy and terms of use' --- # Building Applications with Model APIs Measuring the scaffolding under the assumption that a model call is not a function call but a network operation: the token cost of resending the conversation state with every request, the steps until the first token in a streamed response and the usability of a response cut off midway, the valid call rate and the fabricated tool name, what interface-level schema enforcement adds, the token cost of the input format, the trade-off between storage and accuracy as the embedding dimension drops, the field a compatible API layer loses, the number of requests a single failure affects as batch size grows, retrying also multiplying tokens, the stale response share that grows together with the cache hit rate, the trade-off between latency class and accuracy, the share of the most expensive one percent in the total, falling back to a backup model silently lowering quality, the items that pin the output, the acceptance rate of a code assistant by task class, the point where irrelevant context lowers accuracy, a checkpoint catching an error before it spreads, what three checks catch and miss, the share of empty tests among generated tests that always pass, the number of fields of the secret class that enter the context, the concurrent requests that the memory budget supports, the point where accuracy drops as the number of digits drops, the case where fine-tuning does not win, the share of updated parameters and accuracy, the gap that a distilled small model cannot close, and the deployment option that closes per condition class. ## API Usage Measuring what a message-based interface offers and what each capability costs in tokens: carrying the conversation state on the client and the resending that grows as it lengthens, with the three shortening decisions that bring it down; the steps until the first token and how much of a response cut off midway stays usable; the valid call rate and a fabricated tool name caught by validation; the accuracy and token difference between prompt-level and interface-level enforcement; the tokens the same information costs as text, grid and sequence; the number stored and the neighborhood accuracy as the embedding dimension drops; the number of fields a compatible API layer carries and loses; the number of calls that falls and the number of affected requests that grows as batch size grows. ### Message-Based Interfaces Source: https://academia.sh/en/courses/model-apis/message-based-interfaces The token cost of carrying conversation state on the client: in an eight-turn conversation the total input tokens sent is 3.96 times the last turn, because every turn resends the whole transcript. Across forty fictional conversations, five send policies are measured side by side: a client that sends no state gets 0.2000 accuracy with 8346 input tokens, the full transcript 0.8250 with 31160, the last three turns 0.4500 with 23102, the user turns only 0.6750 with 19738, an extracted state record 0.6500 with 10474. Reading tokens per correct label changes the ranking: the last three turns at 1413.8 cost more than a client that does nothing, and the extracted state record at 494.9 is the cheapest. The difference between the two runs is 0.1250, so the 0.0250 gap between the state record and user only counts as unmeasured. The Prompt Engineering course turned the prompt into a design object over twenty-two lessons and wrote the cost of every change in tokens. Every line of it carried a silent assumption: the call happened **alone, instantly and without error**. None of these assumptions holds inside an application. A call travels over a network, queues, gets rejected, gets cut off midway and each time spends **tokens instead of money**. This is the rule of this course: **a model call is not a function call but a network operation; a call whose failure is not written down counts as unmeasured.** The first lesson starts from the most basic property of the interface. In a message-based interface **conversation state is not kept on the server**; the transcript lives on the client and is resent with every request. This lesson counts what that resending costs and how far five send policies bring it down. - **AU1.** The code is a **simulator**; no real model endpoint is called. The conversations, request texts and label set are **fictional**, seed **20260218**. - **AU2. The simulator also carries network behavior:** rate limit rejection (every thirty-seventh call), context window overflow, timeout and **mid-response cut** of the response. The thresholds are fictional; what is measured is not the value of the threshold but the client's response to it. - **AU3. The interface recognizes three roles:** system, user and model. Role names also count as tokens. - **AU4. Conversation state is not kept on the server.** The transcript stays on the client; what to send with each request is a **client decision**, and this lesson measures five decisions. - **AU5. Latency is written not as raw time but as steps:** the first token step depends on the input, the total steps on the output. A conversation is eight sequential calls; the latency class is **the chain**. - **AU6. The fictional conversation has eight turns.** The **key turn** that determines the topic falls between the second and the sixth; the remaining turns are filler and three of them carry a misleading word. - **AU7. The metric is the label given in the last turn.** A call that is rejected, times out or whose label is not completed counts as **wrong**. - **AU8.** The cost is **input tokens, output tokens, steps and failure share**; the number of calls is **320** in all five modes. #### Message-Based Interface and a Simulator with Network Behavior Resource layout, versioning, the error response format and the contract were measured in the Web API Design course; that procedure is not repeated. The only thing a message-based interface adds is the shape of the body: the request is not a set of fields but **a list of role and content pairs**. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the network layer is simulated too: # rate limit rejection, mid-response cut, chunk-by-chunk generation. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_LENGTH, RATE_EVERY = 240, 24, 37 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def split_pieces(m): """FICTIONAL tokenizer; a piece has at most five characters.""" b = [] for s in m.lower().replace(",", " , ").split(): new = True while len(s) > 5: b.append((s[:5], new)) s, new = s[5:], False b.append((s, new)) return b def tokenize(m): return [p for p, _ in split_pieces(m)] def join_pieces(pl): """Binds the pieces back into words.""" s = "" for p, new in pl: s += (" " if new and s else "") + p return s def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def draw(prompt, seed, salt): return rng((seed + salt) ^ hash_text(prompt))() TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "odeme": "odeme", "sikay": "sikayet", "bilgi": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"} ACK = ["talebinizi aldim ve kaydi guncelledim", # when no label is requested "ilgili ekibe ilettim donus yapilacak", "kaydinizi kontrol ediyorum lutfen bekleyin"] CALL_COUNT = [0] def flatten(messages): """Role labels count as tokens too.""" return " ".join(f"{r} : {i}" for r, i in messages) def call(messages, seed=SEED, temperature=0.6, max_length=MAX_LENGTH): """Message-based interface. messages: [(role, content)], role sistem|kullanici|model.""" CALL_COUNT[0] += 1 prompt = flatten(messages) toks = tokenize(prompt) first = 1 + len(toks) // 40 # steps until the first token empty = {"input": len(toks), "output": 0, "response": "", "first": first, "steps": first} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % RATE_EVERY == 0: # rate limit by call count return dict(empty, status="rate_limit") r = rng(seed ^ hash_text(prompt)) if len(toks) > 90 and r() < 0.04: return dict(empty, status="timeout") candidates = [k for k in CONCEPTS if k[:5] in toks] if len(candidates) < 3: # the label set narrows the candidates candidates = list(CONCEPTS) p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CLUES, 1.0), (TRAPS, 0.6)): if table.get(t) in p: p[table[t]] += w names = sorted(p) if temperature <= 0.0: k = max(names, key=lambda a: p[a]) else: from math import exp e = [exp(p[a] / temperature) for a in names] k = names[pick(r(), [x / sum(e) for x in e])] c = draw(prompt, seed, 17) if "tek" in toks: # only the label was requested text = k else: text = ACK[pick(c, [0.40, 0.33, 0.27])] pl, status = split_pieces(text), "done" if len(pl) > max_length: # stop condition pl, status = pl[:max_length], "stopped" u = draw(prompt, seed, 91) # the network cuts the response midway if len(pl) > 3 and u < 0.08: pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut" return {"status": status, "input": len(toks), "output": len(pl), "response": join_pieces(pl), "first": first, "steps": first + len(pl)} def read_label(y): return next((s for s in y.split() if s in TOPICS), None) OPENERS = ["merhaba yardim gerekiyor", "iyi gunler destek istiyorum", "selam bir sorunum var", "merhaba destek almak istiyorum"] KEY_TURNS = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} FILLER = ["tesekkur ederim not aldim", "peki bunun icin ne kadar beklemem gerekir", "daha once de ayni durum olmustu", "yazili bir bildirim alabilir miyim", "bu konuda kayit acildi mi", "degisiklik yapilirsa haber verilir mi", "ekip ne zaman gelir", "hafta sonu da ulasabilir miyim"] SYSTEM = ("sistem", "destek yardimcisisin . kisa yanit ver .") QUESTION = ("gorusme {n} konusunu su etiketlerden birine ata : fatura , ariza , " "sayac , tarife , kesinti . tek etiket yaz") CONVERSATIONS = [] for i in range(40): # 40 FICTIONAL conversations, eight turns each r = rng(SEED + 7 * i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] key_text = KEY_TURNS[k][int(r() * 3)] key_pos = 1 + int(r() * 5) # key turn from two to six turns = [OPENERS[int(r() * 4)]] for t in range(1, 7): turns.append(key_text if t == key_pos else FILLER[int(r() * 8)]) turns.append(QUESTION.format(n=7301 + i)) CONVERSATIONS.append({"id": 7301 + i, "topic": k, "key_pos": key_pos + 1, "turns": turns}) g = CONVERSATIONS[0] print(f"conversation {g['id']} -- topic {g['topic']} , key turn {g['key_pos']}") history, sent = [SYSTEM], [] for t, text in enumerate(g["turns"], 1): history.append(("kullanici", text)) s = call(history) sent.append(s["input"]) if t in (1, g["key_pos"], 8): print(f" turn {t} sent {s['input']:>3} output {s['output']:>2}" f" first token {s['first']} {s['status']:<6} -> {s['response'][:34]}") history.append(("model", s["response"] or "-")) print(" input tokens sent per turn:", sent) print(f" total {sum(sent)} tokens , {sum(sent) / sent[-1]:.2f} times the last turn") print(" same last turn with seed 20260219 ->", call(history[:-1], SEED + 1)["response"]) ``` ``` conversation 7301 -- topic kesinti , key turn 5 turn 1 sent 21 output 8 first token 1 done -> talebinizi aldim ve kaydi guncelle turn 5 sent 99 output 10 first token 3 done -> kaydinizi kontrol ediyorum lutfen turn 8 sent 187 output 2 first token 5 done -> kesinti input tokens sent per turn: [21, 41, 53, 75, 99, 123, 142, 187] total 741 tokens , 3.96 times the last turn same last turn with seed 20260219 -> kesinti ``` The user wrote eight short sentences, but the total going to the interface is **741 tokens**, which is **3.96 times** what the last turn costs alone. The reason is plain: every turn carries all the turns before it again. If turn lengths are roughly equal, the total sent over an $n$-turn conversation grows as $n(n+1)/2$; when the number of turns doubles, the cost approaches fourfold. The first token step also rises from 1 to 5: a growing state is not only expensive but slow. In the last line the same last turn with the second seed again returned `outage`; in a single call the output stayed fixed, but the table below will show that this stability does not hold at the scale of forty conversations. #### Five Send Policies Since the transcript is on the client, **which part of it** to send with each request is a design decision. Five decisions run on the same forty conversations: sending no state, the whole transcript, the last three turns, the user turns only, and **an extracted state record** produced on the client instead of the transcript. The extractor of the last one recognizes seven fields and silently drops what it does not recognize. ```python FIELDS = {"fatur": "fatura", "borc": "borc", "ariza": "ariza", "sayac": "sayac", "endek": "endeks", "tarif": "tarife", "kesin": "kesinti"} def state_record(history): """State kept on the client: not the transcript, a list of fields.""" a = [] for role, content in history: if role == "kullanici": for t in tokenize(content): if t in FIELDS and FIELDS[t] not in a: a.append(FIELDS[t]) return " , ".join(a) if a else "yok" def send(mode, history, now): """Decision on which part of the transcript to send.""" current = [("kullanici", now)] if mode == "stateless": return [SYSTEM] + current if mode == "full state": return [SYSTEM] + history + current if mode == "last three turns": return [SYSTEM] + history[-6:] + current if mode == "user only": return [SYSTEM] + [m for m in history if m[0] == "kullanici"] + current return [SYSTEM, ("kullanici", "durum : " + state_record(history))] + current def measure(mode, seed=SEED): CALL_COUNT[0] = 0 tok_in = tok_out = steps = correct = failed = 0 for conv in CONVERSATIONS: history = [] for t, text in enumerate(conv["turns"], 1): s = call(send(mode, history, text), seed) tok_in, tok_out, steps = tok_in + s["input"], tok_out + s["output"], steps + s["steps"] failed += s["status"] != "done" history += [("kullanici", text), ("model", s["response"] or "-")] if t == 8: correct += read_label(s["response"]) == conv["topic"] return tok_in, tok_out, steps, correct, failed / 320 MODES = ["stateless", "full state", "last three turns", "user only", "state record"] print("mode input output steps accuracy A accuracy B failure" " per correct") for mode in MODES: a, b = measure(mode), measure(mode, SEED + 1) print(f"{mode:<17} {a[0]:>6} {a[1]:>6} {a[2]:>5} {a[3] / 40:>10.4f}" f" {b[3] / 40:>10.4f} {a[4]:>9.4f} {(a[0] + a[1]) / a[3]:>12.1f}") g0 = CONVERSATIONS[0] print("\nstate record , conversation", g0["id"], ":", state_record([("kullanici", m) for m in g0["turns"][:7]])) no_field = sum(state_record([("kullanici", m) for m in conv["turns"][:7]]) == "yok" for conv in CONVERSATIONS) print("conversations where the extractor found no field:", no_field, "/ 40") history, t = [], 0 while True: # where the full state overflows the window t += 1 text = g0["turns"][t - 1] if t <= 7 else FILLER[t % 8] s = call([SYSTEM] + history + [("kullanici", text)]) if s["status"] == "context_overflow": print(f"full state exceeds the window at turn {t}: {s['input']} > {WINDOW}") break history += [("kullanici", text), ("model", s["response"])] ``` ``` mode input output steps accuracy A accuracy B failure per correct stateless 8346 2350 2710 0.2000 0.2500 0.0406 1337.0 full state 31160 2341 3294 0.8250 0.7000 0.0625 1015.2 last three turns 23102 2347 3132 0.4500 0.4250 0.0844 1413.8 user only 19738 2297 2946 0.6750 0.7750 0.1000 816.1 state record 10474 2393 2753 0.6500 0.5500 0.0375 494.9 state record , conversation 7301 : kesinti conversations where the extractor found no field: 5 / 40 full state exceeds the window at turn 12: 246 > 240 ``` The first row is the client without scaffolding: when no state is carried, the last turn sees only the label set and the conversation number, and the accuracy of **0.2000** is the chance share among five labels. The second row does the same job with conversation state and raises the accuracy to **0.8250**. The cost is in the input column: **from 8346 to 31160**, that is **3.73 times**. Steps go from 2710 to 3294, because the growing prompt delays the first token; the failure share also rises from 0.0406 to 0.0625. #### Measured Gain, Unmeasured Gain and Silent Loss The three shortening decisions split the table down the middle. Sending the **last three turns** cuts the input by 26% but drops the accuracy to **0.4500**, because the key turn falls outside the window in most conversations. Sending **only the user turns** cuts the input by 37% and holds the accuracy at **0.6750**; when the model does not see its own earlier responses, it loses nothing. The **extracted state record** **cuts the input by 66%** and holds the accuracy at **0.6500**. The difference between the last two numbers is **0.0250**. Both run columns need to be consulted: the full state row gives 0.7000 on the second seed instead of 0.8250, so the run difference of this scaffolding can reach **0.1250**. 0.0250 is one fifth of that and **counts as unmeasured**; the 0.1750 between full state and the state record, on the other hand, exceeds the run difference and is measured. The last column changes the ranking. The tokens per correct label are **1413.8** in the last three turns mode, and this is **more expensive** than the **1337.0** of the client that sends no state at all: a shortening decision can fall below both the full transcript and the client without scaffolding. The state record gives the cheapest correct result at **494.9**. The price of the cheapness is the silence of the extractor: in **five** of the forty conversations no field is found and the client reports this not as an error but as `yok` (none). The last line states the limit: if the conversation continues, the full transcript exceeds the context window with 246 tokens at the **twelfth turn** and the call is never made. Window overflow was measured in the Introduction to AI Engineering course; the only thing added here is that **the client's own state decision** drives it to that limit. #### Summary - Conversation state is not kept on the server; because the transcript is resent with every request, the total sent over an eight-turn conversation is 3.96 times the last turn. - When no state is carried, accuracy stays at chance level with 0.2000; when the transcript is sent in full it rises to 0.8250 and input tokens grow 3.73 times. - Sending the last three turns cuts the input by 26% but drops the accuracy to 0.4500; the key turn falls outside in most conversations. - The extracted state record cuts the input by 66% and holds the accuracy at 0.6500; the 0.0250 gap with user turns only counts as unmeasured because the run difference of the two runs is 0.1250. - Tokens per correct label change the ranking: the last three turns at 1413.8 are more expensive than the client without scaffolding, and the state record at 494.9 is the cheapest. - The price of the cheap decision is silent loss: the extractor finds no field in five of forty conversations and does not report this as an error. #### Next Step In this lesson every response arrived as a single piece and the user waited for it to complete. The printed **first token step** entered no decision: while the growing prompt pushed the first token from 1 step to 5 steps, the expected time still stretched to the last chunk. Had the response reached our hands in the order it was produced instead of as a whole, which number changes and which stays the same. There is also the **cut** seen in the failure column: what is left when it happens in the middle of a response. The next lesson counts these three separately. ### Response Streaming Source: https://academia.sh/en/courses/model-apis/response-streaming Measuring receiving the response chunk by chunk: on the same 320-call workload all three setups spend 23338 input and 4049 output tokens and are cut midway in 33 calls, that is, streaming does not change a single token. What changes is the waiting step: the time to first token is 16.03 steps on average in the non-streaming client and 3.29 steps in the streaming client. But streaming alone does not change usability; when the label is at the end of the response, the time to actionable output is 16.03 in both setups and the number recovered from the 33 cut responses is 0. When the label is moved to the start of the response, the time to actionable output falls to 4.96, the number recovered from the cut ones rises to 31 and the usable share rises from 0.8625 to 0.9594. On average 0.5801 of the cut responses has reached us, and the field order decides whether this share is of any use. The previous lesson placed conversation state on the client and wrote the token cost of every send policy. The response of every call arrived there as a single piece: the client sent the request, waited and found a whole text in hand. The printed **first token step** column had entered no decision. A response is not actually produced as a single piece. The model builds the output step by step and the interface can send these chunks in the order they are produced. This lesson measures that option and separates three questions: whether streaming changes the **total tokens**, how much it lowers the **time to first token**, and what stays **usable** when the network cuts the response in the middle. The third question is tied directly to the rule of this course: a call whose failure is not written down counts as unmeasured, and a response left half finished is a failure. - **AU9.** The code is a **simulator**; no real endpoint is called. The conversations are **fictional**, seed **20260218**. The simulator carries the network layer set up in the previous lesson. - **AU10. The response is a list of chunks.** The simulator also returns the `chunks` field; the streaming client joins this list in the order it arrives, while the non-streaming client sees it only when the response is complete. - **AU11. The network cut depends on the content of the prompt, not on the field order.** Because the three setups send the same prompts, **the same calls are cut**; the comparison is made under the same network condition. - **AU12. Field order is a request field**, not a sentence written into the prompt. So the input tokens stay exactly the same in the three setups and only the order changes. - **AU13. Two separate waits are measured:** the **time to first token** (the moment the user sees something on screen) and the **time to actionable output** (the moment the client can read the label and start work). - **AU14. The usability criterion is that the label can be read.** In the non-streaming client the body of a response cut midway is never in hand; in the streaming client the chunks that arrived stay in hand. - **AU15.** The send policy is inherited from the previous lesson and is the same in all three setups: **user turns only**. The workload is the eight turns of forty conversations, **320 calls** in total. - **AU16.** Latency is written not as raw time but as **steps**; the latency class is again **the chain**. #### A Response Arriving Chunk by Chunk The job of a streaming client is two lines: accumulate the arriving chunk and check whether the accumulated text has become **decidable**. The `streaming_client` below does this and writes the step at which it first read the label. `whole_client` waits for the response to complete; if it does not complete, nothing is left in its hand. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the network layer is simulated too. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_LENGTH, RATE_EVERY = 240, 24, 37 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def split_pieces(m): """FICTIONAL tokenizer; a piece has at most five characters.""" b = [] for s in m.lower().replace(",", " , ").split(): new = True while len(s) > 5: b.append((s[:5], new)) s, new = s[5:], False b.append((s, new)) return b def tokenize(m): return [p for p, _ in split_pieces(m)] def join_pieces(pl): s = "" for p, new in pl: s += (" " if new and s else "") + p return s def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def draw(prompt, seed, salt): return rng((seed + salt) ^ hash_text(prompt))() TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "odeme": "odeme", "sikay": "sikayet", "bilgi": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"} BODY = "talebiniz incelendi ve ilgili ekibe iletildi" CALL_COUNT = [0] def flatten(messages): return " ".join(f"{r} : {i}" for r, i in messages) def call(messages, seed=SEED, temperature=0.6, max_length=MAX_LENGTH, position="only"): """position is a request field: whether the label is produced at the start or the end.""" CALL_COUNT[0] += 1 prompt = flatten(messages) toks = tokenize(prompt) first = 1 + len(toks) // 40 # steps until the first token empty = {"input": len(toks), "output": 0, "chunks": [], "first": first} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % RATE_EVERY == 0: return dict(empty, status="rate_limit") r = rng(seed ^ hash_text(prompt)) if len(toks) > 90 and r() < 0.04: return dict(empty, status="timeout") candidates = [k for k in CONCEPTS if k[:5] in toks] if len(candidates) < 3: candidates = list(CONCEPTS) p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CLUES, 1.0), (TRAPS, 0.6)): if table.get(t) in p: p[table[t]] += w names = sorted(p) if temperature <= 0.0: k = max(names, key=lambda a: p[a]) else: from math import exp e = [exp(p[a] / temperature) for a in names] k = names[pick(r(), [x / sum(e) for x in e])] if position == "start": text = f"konu {k} , {BODY}" elif position == "end": text = f"{BODY} , konu {k}" else: text = k pl, status = split_pieces(text), "done" if len(pl) > max_length: pl, status = pl[:max_length], "stopped" u = draw(prompt, seed, 91) # the network cuts the response midway if len(pl) > 3 and u < 0.08: pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut" return {"status": status, "input": len(toks), "output": len(pl), "chunks": pl, "first": first} def read_label(y): return next((s for s in y.split() if s in TOPICS), None) def whole_client(s): """If the response does not complete, the body is never in hand.""" if s["status"] != "done": return "", None text = join_pieces(s["chunks"]) return text, (s["first"] + s["output"]) if read_label(text) else None def streaming_client(s): """Chunks are joined as they arrive; the step is written as soon as the label is read.""" text, usable_step = "", None for i, (p, new) in enumerate(s["chunks"], 1): text += (" " if new and text else "") + p if usable_step is None and read_label(text): usable_step = s["first"] + i return text, usable_step OPENERS = ["merhaba yardim gerekiyor", "iyi gunler destek istiyorum", "selam bir sorunum var", "merhaba destek almak istiyorum"] KEY_TURNS = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} FILLER = ["tesekkur ederim not aldim", "peki bunun icin ne kadar beklemem gerekir", "daha once de ayni durum olmustu", "yazili bir bildirim alabilir miyim", "bu konuda kayit acildi mi", "degisiklik yapilirsa haber verilir mi", "ekip ne zaman gelir", "hafta sonu da ulasabilir miyim"] SYSTEM = ("sistem", "destek yardimcisisin . etiket kumesi : fatura , ariza , " "sayac , tarife , kesinti .") QUESTION = "gorusme {n} icin konu etiketi ver" CONVERSATIONS = [] for i in range(40): # 40 FICTIONAL conversations, eight turns each r = rng(SEED + 7 * i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] key_text = KEY_TURNS[k][int(r() * 3)] key_pos = 1 + int(r() * 5) turns = [OPENERS[int(r() * 4)]] for t in range(1, 7): turns.append(key_text if t == key_pos else FILLER[int(r() * 8)]) turns.append(QUESTION.format(n=7301 + i)) CONVERSATIONS.append({"id": 7301 + i, "topic": k, "key_pos": key_pos + 1, "turns": turns}) g = CONVERSATIONS[0] messages = [SYSTEM] + [("kullanici", m) for m in g["turns"]] print(f"conversation {g['id']} , topic {g['topic']} , input" f" {call(messages)['input']} tokens") for position in ("end", "start"): s = call(messages, position=position) text, usable_step = streaming_client(s) print(f" label {position:<5} output {s['output']:>2} first token step {s['first']}" f" time to actionable output in streaming client {usable_step}" f" whole response step {whole_client(s)[1]}") print(f" response : {text}") print(" same request with seed 20260219 ->", join_pieces(call(messages, SEED + 1, position="start")["chunks"])) ``` ``` conversation 7301 , topic kesinti , input 113 tokens label end output 14 first token step 3 time to actionable output in streaming client 17 whole response step 17 response : talebiniz incelendi ve ilgili ekibe iletildi , konu kesinti label start output 14 first token step 3 time to actionable output in streaming client 6 whole response step 17 response : konu kesinti , talebiniz incelendi ve ilgili ekibe iletildi same request with seed 20260219 -> konu kesinti , talebiniz incelendi ve ilgili ekibe iletildi ``` Same conversation, same prompt and same output length: in both setups the response is **14 chunks**. The only thing that changes is where the label stands. When the label is at the end, both the streaming client and the whole client can decide at **step 17**; streaming gains nothing. When the label is moved to the start, the streaming client decides at **step 6**, while the whole client again waits for step 17. The eleven steps in between are a wait that could be skipped without reading the body of the response. The last line follows from the third rule of the course: the same request with a second seed again gave the same label and the same order. #### Waiting and Usability Table of the Three Setups The three setups run on the same 320 calls. The first is non-streaming and asks for the label at the end; the second reads the same request by streaming; the third streams and asks for the label at the start. Because the calls are exactly the same, the network condition is also the same. ```python def measure(position, streaming, seed=SEED): """position: end | start. streaming: whether chunks are read as they arrive.""" CALL_COUNT[0] = 0 tok_in = tok_out = cut = cut_recovered = correct = 0 visible, usable = [], [] for conv in CONVERSATIONS: for t in range(1, 9): messages = [SYSTEM] + [("kullanici", m) for m in conv["turns"][:t]] s = call(messages, seed, position=position) tok_in, tok_out = tok_in + s["input"], tok_out + s["output"] if streaming: text, step = streaming_client(s) if s["chunks"]: visible.append(s["first"] + 1) else: text, step = whole_client(s) if step is not None: visible.append(step) if s["status"] == "cut": cut += 1 cut_recovered += step is not None if step is not None: usable.append(step) if t == 8 and read_label(text) == conv["topic"]: correct += 1 avg = sum(visible) / len(visible), sum(usable) / len(usable) return tok_in, tok_out, cut, cut_recovered, avg[0], avg[1], len(usable) / 320, correct / 40 SETUPS = [("non-streaming , label end", "end", False), ("streaming , label end", "end", True), ("streaming , label start", "start", True)] print("setup input output cut recovered time to first token" " time to actionable output usable share accuracy A accuracy B") for name, position, streaming in SETUPS: a, b = measure(position, streaming), measure(position, streaming, SEED + 1) print(f"{name:<25} {a[0]:>6} {a[1]:>6} {a[2]:>6} {a[3]:>9} {a[4]:>19.2f}" f" {a[5]:>25.2f} {a[6]:>14.4f} {a[7]:>10.4f} {b[7]:>10.4f}") CALL_COUNT[0] = 0 share = [] for conv in CONVERSATIONS: for t in range(1, 9): s = call([SYSTEM] + [("kullanici", m) for m in conv["turns"][:t]], position="start") if s["status"] == "cut": share.append(s["output"] / 14) print(f"\nchunk share arriving in {len(share)} cut responses : min {min(share):.4f} ," f" mean {sum(share) / len(share):.4f} , max {max(share):.4f}") ``` ``` setup input output cut recovered time to first token time to actionable output usable share accuracy A accuracy B non-streaming , label end 23338 4049 33 0 16.03 16.03 0.8625 0.7500 0.6500 streaming , label end 23338 4049 33 0 3.29 16.03 0.8625 0.7500 0.6500 streaming , label start 23338 4049 33 31 3.29 4.96 0.9594 0.7750 0.6750 chunk share arriving in 33 cut responses : min 0.0714 , mean 0.5801 , max 0.9286 ``` The first two columns are the most important result of this lesson. Input **23338**, output **4049** and the number of cut calls **33** are exactly the same in all three setups. Streaming **does not change a single token**; it neither lowers nor raises the cost. In a cost accounting table, a streaming and a non-streaming client cannot be told apart. What changes is the waiting column. In the non-streaming client the time to first token is **16.03 steps** on average, because the first thing visible is already the whole response. In the streaming client this drops to **3.29 steps**: about one fifth. The appearance of something on the user's screen is pulled this early, and no token is paid for it. #### What Streaming Did Not Gain and Field Order Did The **time to actionable output** value of the second row is 16.03, the same as the first row. Streaming brought something to the screen but **did not let the client decide**, because the field the decision rests on was at the very end of the response. The recovered column of the same row is also zero: none of the 33 cut responses stayed usable. Streaming alone is not a resilience scaffolding. The third row opens the difference with field order. The time to actionable output falls to **4.96**, **31** of the 33 cut responses are recovered, and the usable share rises **from 0.8625 to 0.9594**. The chunk share arriving in the cut responses is at least 0.0714, on average **0.5801**, at most 0.9286: on average more than half of a response cut midway reaches us. Whether that share is of any use depends entirely on **which field was produced first**. The last two columns carry a warning. The label accuracy in the three setups is 0.7500 / 0.7500 / 0.7750; on the second seed 0.6500 / 0.6500 / 0.6750. The **0.0250** that field order adds to accuracy, while the difference between the two runs of the same row is **0.1000**. This gain **counts as unmeasured** and is written that way. The measured gain of field order is not in accuracy but in the **usability and waiting** columns; these are also numbers that do not change from run to run. #### Summary - Streaming does not change the total tokens: all three setups spend 23338 input and 4049 output tokens and are cut in 33 calls. - Streaming lowers the time to first token from 16.03 to 3.29 and no token is paid for it. - Streaming alone does not change usability: when the label is at the end, the time to actionable output is 16.03 in both setups and the number recovered from the 33 cut responses is 0. - When the label is moved to the start of the response, the time to actionable output falls to 4.96, while the number recovered rises to 31 and the usable share to 0.9594. - On average 0.5801 of the cut responses reaches us; the field order decides whether this share is of any use. - The 0.0250 that field order adds to accuracy counts as unmeasured because the difference between the two runs is 0.1000; the measured gain is in the waiting and usability columns. #### Next Step In two lessons the response was always **a text to be read**. When the label came early the work started early, but what started that work was always a person or a simple routing rule. If the response is going to trigger **code execution** directly, the criterion changes: it is not enough for the format to be readable; the thing called must also **exist** and the values given must be **acceptable**. The next lesson ties the model's output to a call that will be executed and prints three numbers side by side: how many responses produce a call that conforms to the schema, how many fabricate a name that does not exist, and how many of these validation catches. ### Tool and Function Calling Source: https://academia.sh/en/courses/model-apis/tool-and-function-calling Measuring how a model output is tied to a call that will be executed: on 120 fictional requests, when the tool catalog is not written into the prompt the valid call rate is 0.0500 and 0.6750 of the responses fabricate a tool name that does not exist. When only the names are given, the fabricated name share falls to 0.1500 but the parameter error jumps from 0.0667 to 0.5583: the error does not disappear, it moves. When names and parameters are given together, the valid rate is 0.6917 and the rate of calling the correct tool is 0.5750, and the price is input growing from 3694 to 11614 tokens. Still, the tokens per correct tool fall from 750.5 to 178.2. Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong tool, and no schema catches this residue. The format error does not shrink with the catalog, because its source is the network, not the prompt. In the previous two lessons the response was a text, and it was read by a person or a simple routing rule. An early label pulled the work earlier, but what did the work was always something else. When a model output is tied directly to a **code execution**, the criterion changes. It is not enough for the response to be readable: the function called must **exist**, the parameter names must **match** and the values must be **acceptable**. Multi-step continuation, memory and planning are **not** here; they are left to the AI Agents course. In this lesson no tool is **executed**; only the validity of the call is counted. - **AU17.** The code is a **simulator** and **the tool registry is fictional too**: the measurement network's support line's own functions. The requests are fictional, seed **20260218**. No tool is run. - **AU18. The tool registry is five functions**, each corresponding to one topic; one of them has two parameters. A parameter rule is either a **number** or a closed set of values. - **AU19. The validator counts four error classes:** format (the call could not be parsed at all), **fabricated name** (a function not in the registry), parameter (missing, extra or invalid value) and no error. - **AU20. The catalog is given at three levels:** none, **names** only, or **names and parameters**. The simulator changes its tendency to use a registered name according to these three cases; the tendency is fictional, what is measured is the difference between the levels. - **AU21. The concept choice is read only from the last user message.** So between the three levels only the form of the call changes; which topic the model sees does not change. - **AU22. The network layer continues.** Rate limit, timeout and mid-response cut corrupt the call text; a corrupted call falls into the format or fabricated name class. This is why **part of the format error comes from the network, not the prompt** and does not shrink with the catalog. - **AU23. The product criterion is the correct tool rate:** the response that both passes validation and calls the function corresponding to the topic of the request. - **AU24.** The cost is written as input and output tokens; the number of calls is **120** at all three levels. #### Tool Registry, Parser and Validator A function call is the fitting of the text produced by the interface to a **registered signature**. The code below sets up the registry, the call parser and the validator, then asks the same request at three catalog levels. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the tools are fictional too and # none of them is executed, only the validity of the call is counted. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_LENGTH, RATE_EVERY = 240, 24, 37 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def split_pieces(m): """FICTIONAL tokenizer; a piece has at most five characters.""" b = [] for s in m.lower().replace(",", " , ").split(): new = True while len(s) > 5: b.append((s[:5], new)) s, new = s[5:], False b.append((s, new)) return b def tokenize(m): return [p for p, _ in split_pieces(m)] def join_pieces(pl): s = "" for p, new in pl: s += (" " if new and s else "") + p return s def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def draw(prompt, seed, salt): return rng((seed + salt) ^ hash_text(prompt))() TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"} CALL_COUNT = [0] # FICTIONAL tool registry: the measurement network's support line's own non-third-party functions TOOLS = {"fatura_getir": {"abone": "sayi"}, "ariza_kaydi_ac": {"abone": "sayi", "konu": TOPICS}, "sayac_endeks_sor": {"sayac": "sayi"}, "tarife_bilgisi": {"abone": "sayi"}, "kesinti_sorgula": {"bolge": "sayi"}} CORRECT = dict(zip(TOPICS, TOOLS)) FABRICATED = {"fatura": "fatura_sorgula", "ariza": "ariza_bildir", "sayac": "sayac_oku", "tarife": "tarife_getir", "kesinti": "kesinti_listele"} WRONG_KEYS = {"abone": "abone_no", "sayac": "sayac_no", "bolge": "bolge_kodu", "konu": "konu_adi"} def flatten(messages): return " ".join(f"{r} : {i}" for r, i in messages) def call(messages, seed=SEED, temperature=0.6, max_length=MAX_LENGTH): """The response is the text of a tool call; no tool is executed.""" CALL_COUNT[0] += 1 prompt = flatten(messages) toks = tokenize(prompt) first = 1 + len(toks) // 40 empty = {"input": len(toks), "output": 0, "response": "", "first": first} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % RATE_EVERY == 0: return dict(empty, status="rate_limit") if len(toks) > 90 and draw(prompt, seed, 3) < 0.04: return dict(empty, status="timeout") last = messages[-1][1] # the concept is read only from the last message p = {k: 0.15 for k in TOPICS} for t in tokenize(last): for table, w in ((CLUES, 1.0), (TRAPS, 0.6)): if table.get(t) in p: p[table[t]] += w names = sorted(p) from math import exp e = [exp(p[a] / temperature) for a in names] k = names[pick(draw(last, seed, 5), [x / sum(e) for x in e])] has_names, has_params = "arac" in toks, "param" in toks # is there a catalog in the prompt u4 = draw(prompt, seed, 67) if u4 < (0.04 if has_names else 0.12): # free text instead of a call text = f"talebiniz {k} konusunda kaydedildi" else: threshold = 0.15 + (0.73 if has_names else 0.0) + (0.06 if has_params else 0.0) name = CORRECT[k] if draw(prompt, seed, 23) < threshold else FABRICATED[k] number = next((x for x in prompt.split() if x.isdigit()), "0") right_keys = has_params or draw(prompt, seed, 41) < 0.35 bad_value = draw(prompt, seed, 59) < 0.10 fields = [] for a, rule in TOOLS[CORRECT[k]].items(): key = a if right_keys else WRONG_KEYS[a] if rule == "sayi": fields.append(f"{key}={'bilinmiyor' if bad_value else number}") else: fields.append(f"{key}={'acil' if bad_value else k}") text = "cagri " + name + " " + " ".join(fields) pl, status = split_pieces(text), "done" if len(pl) > max_length: pl, status = pl[:max_length], "stopped" u = draw(prompt, seed, 91) if len(pl) > 3 and u < 0.08: pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut" return {"status": status, "input": len(toks), "output": len(pl), "response": join_pieces(pl), "first": first} def parse_call(y): """Extracts the tool call from the response: (name, {parameter: value}) or None.""" parts = y.split() if len(parts) < 2 or parts[0] != "cagri": return None args = {} for x in parts[2:]: if "=" not in x: return None a, _, v = x.partition("=") args[a] = v return parts[1], args def validate(c): """Four error classes: format, fabricated name, missing/extra parameter, invalid value.""" if c is None: return ["format"] name, args = c if name not in TOOLS: return ["fabricated_name"] schema = TOOLS[name] h = [f"missing:{a}" for a in schema if a not in args] h += [f"extra:{a}" for a in args if a not in schema] for a, rule in schema.items(): if a not in args: continue if rule == "sayi" and not args[a].isdigit(): h.append(f"not_a_number:{a}") elif rule != "sayi" and args[a] not in rule: h.append(f"invalid:{a}") return h PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] REQUESTS = [] for i in range(120): # 120 FICTIONAL support requests r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] REQUESTS.append({"id": 4001 + i, "topic": k, "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]}) PLAIN = "destek talebini uygun bir islev cagrisiyla karsila ." NAMES = PLAIN + " arac adlari : " + " , ".join(TOOLS) + " ." FULL = NAMES + " parametreler : " + " , ".join( f"{a} ( {' '.join(s)} )" for a, s in TOOLS.items()) + " ." print("tool registry:", " , ".join(f"{a}({' '.join(s)})" for a, s in TOOLS.items())) for label, prefix in (("no catalog", PLAIN), ("names only", NAMES), ("names and parameters", FULL)): x = REQUESTS[0] s = call([("sistem", prefix), ("kullanici", f"Talep {x['id']} : {x['text']}")]) errors = validate(parse_call(s["response"])) print(f" {label:<20} input {s['input']:>3} {s['response']}") print(f" error {errors if errors else 'none'}") x = REQUESTS[0] print(" same request with seed 20260219 ->", call([("sistem", FULL), ("kullanici", f"Talep {x['id']} : {x['text']}")], SEED + 1)["response"]) ``` ``` tool registry: fatura_getir(abone) , ariza_kaydi_ac(abone konu) , sayac_endeks_sor(sayac) , tarife_bilgisi(abone) , kesinti_sorgula(bolge) no catalog input 32 cagri kesinti_listele bolge_kodu=4001 error ['fabricated_name'] names only input 57 cagri kesinti_sorgula bolge_kodu=4001 error ['missing:bolge', 'extra:bolge_kodu'] names and parameters input 98 cagri kesinti_sorgula bolge=4001 error none same request with seed 20260219 -> cagri sayac_endeks_sor sayac=4001 ``` Three lines show the same request, the same topic and three different errors. At the no-catalog level the format is flawless: the response looks like a call, it has a parameter, the value is correct. But there is **no** function called `kesinti_listele` (list outages). This is the error that validation catches most readily and the most dangerous one: looked at without the registry, it seems valid. In the second line the name is now registered, but the parameter name is written `bolge_kodu` (region code) and the registry expects `bolge` (region). The validator counts this as two errors: one missing key, one extra key. In the third line both the name and the parameter agree with the registry. The last line follows from the third rule of the course: the same request on a second seed called **a different tool**, and that call too would have passed validation. #### Numbers of the Three Catalog Levels The same 120 fictional requests are run at three levels and the error classes are counted separately. ```python def measure(prefix, seed=SEED): CALL_COUNT[0] = 0 tok_in = tok_out = network_failed = valid = fabricated = bad_format = 0 param_err = right_tool = 0 for x in REQUESTS: s = call([("sistem", prefix), ("kullanici", f"Talep {x['id']} : {x['text']}")], seed) tok_in, tok_out = tok_in + s["input"], tok_out + s["output"] network_failed += s["status"] != "done" parsed = parse_call(s["response"]) errors = validate(parsed) if not errors: valid += 1 right_tool += parsed[0] == CORRECT[x["topic"]] elif errors == ["fabricated_name"]: fabricated += 1 elif errors == ["format"]: bad_format += 1 else: param_err += 1 n = len(REQUESTS) return (tok_in, tok_out, network_failed, valid / n, fabricated / n, bad_format / n, param_err / n, right_tool / n) LEVELS = [("no catalog", PLAIN), ("names only", NAMES), ("names and parameters", FULL)] print("level input output network valid A valid B fabricated" " format param correct tool") for name, prefix in LEVELS: a, b = measure(prefix), measure(prefix, SEED + 1) print(f"{name:<20} {a[0]:>6} {a[1]:>6} {a[2]:>7} {a[3]:>9.4f} {b[3]:>9.4f}" f" {a[4]:>10.4f} {a[5]:>6.4f} {a[6]:>6.4f} {a[7]:>12.4f}") a = measure(FULL) print(f"\nvalidation caught {round((1 - a[3]) * 120)} calls ;" f" of the {round(a[3] * 120)} calls counted valid," f" {round((a[3] - a[7]) * 120)} call the wrong tool") print("tokens per correct tool :", end=" ") for name, prefix in LEVELS: z = measure(prefix) print(f"{name} {(z[0] + z[1]) / (z[7] * 120):.1f}", end=" ") print() CALL_COUNT[0] = 0 n = 0 for x in REQUESTS: s = call([("sistem", FULL), ("kullanici", f"Talep {x['id']} : {x['text']}")]) parsed = parse_call(s["response"]) if parsed and not validate(parsed) and parsed[0] != CORRECT[x["topic"]] and n < 3: n += 1 print(f" residue : topic {x['topic']:<8} -> {s['response']}") ``` ``` level input output network valid A valid B fabricated format param correct tool no catalog 3694 809 13 0.0500 0.0500 0.6750 0.2083 0.0667 0.0500 names only 6694 825 12 0.2250 0.2833 0.1500 0.0667 0.5583 0.1833 names and parameters 11614 684 25 0.6917 0.6917 0.0500 0.1583 0.1000 0.5750 validation caught 37 calls ; of the 83 calls counted valid, 14 call the wrong tool tokens per correct tool : no catalog 750.5 names only 341.8 names and parameters 178.2 residue : topic ariza -> cagri tarife_bilgisi abone=4004 residue : topic kesinti -> cagri ariza_kaydi_ac abone=4011 konu=ariza residue : topic tarife -> cagri fatura_getir abone=4012 ``` The first row is the client without scaffolding and is summarized by a single number: **0.7250** of the responses carry a function name that does not exist. The valid call rate is **0.0500**, that is, about one request in twenty produces an executable call. The model **fabricates** the name of an interface without seeing it, and the names it fabricates look plausible; a human eye cannot tell them apart at once. The second row is the most instructive row of this lesson. Giving the names lowers the fabricated name share **from 0.6750 to 0.1500**, but the valid rate rises only to **0.2250**. This is because the parameter error jumps **from 0.0667 to 0.5583**. The error did not disappear, it **moved**: the model now knocks on the right door but with the wrong key. A measurement that looks at a single number cannot see this shift. #### What Validation Caught and What It Did Not The third row gives both pieces of information at once: valid rate **0.6917**, fabricated name **0.0500**, parameter error **0.1000**. The two runs differ by 0.0583 in the second row, and the steps of 0.1750 and 0.4667 between the levels are many times larger than that; these gains are **measured**. The price stands in the input column: **from 3694 to 11614 tokens**, that is, **3.14 times**. In return, the tokens per correct tool fall **from 750.5 to 178.2**. The catalog is expensive and still cheap, because the tokens paid fall faster than the number of calls thrown away. The format column does not go down with the levels: 0.2083, then 0.0667, then 0.1583. The reason is in the neighboring column. The number of calls that failed in the network layer is 13, 12 and **25** respectively; the level that sends the longest prompt is the level cut most often, and a cut call text cannot be parsed. Part of the format error comes not from the prompt but from the **network**, and no catalog reduces it. The last two lines write the limit of validation. The validator caught **37 calls**; of the remaining 83 calls, **14** call a registered but **wrong** function. In the residue examples, tariff information is requested for a fault request and a fault record for an outage request. These calls pass the schema, are executed and return the **wrong result**. Validity is not correctness; a schema says what can be called, not what should be called. #### Summary - The tool registry, the parser and the validator with four error classes are written in the lesson with the standard library; no tool is executed. - When no catalog is given, 0.6750 of the responses fabricate a function name that does not exist and the valid call rate stays at 0.0500. - Giving only the names lowers the fabricated name to 0.1500 but raises the parameter error to 0.5583: the error does not disappear, it moves. - When names and parameters are given together, the valid rate is 0.6917 and the correct tool rate is 0.5750; while the input tokens grow from 3694 to 11614, the tokens per correct tool fall from 750.5 to 178.2. - The format error does not shrink with the catalog, because part of its source is the network: the number of failed calls rises to 25 at the longest prompt. - Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong function and no schema catches this residue. #### Next Step In this lesson validity was checked **afterwards**: the response arrived, was parsed, was validated and what did not pass was thrown away. The validator worked well as a counter, but for the 37 calls that did not pass the schema, the only thing it could do was count them. The Prompt Engineering course measured imposing the format **by writing it into the prompt** and raised the conformance rate to 0.8000. If the interface itself can constrain generation so that it **cannot step outside** the schema, then there are two differences between imposing with the prompt and imposing with the interface: the conformance rate and the tokens. The next lesson prints these two side by side and counts what enforcement at the interface level adds and what it does not. ### Structured Output Mode Source: https://academia.sh/en/courses/model-apis/structured-output-mode Measuring side by side writing the schema into the prompt and giving it to the interface: on a four-field schema, writing it into the prompt spends 7654 input tokens and gives 0.7583 conformance, while the interface field gives 0.9000 with 4294 tokens. When the schema grows to twelve fields, conformance with the prompt falls to 0.6083, with the interface it stays fixed at 0.9000 and the saving rises to 6360 tokens. When the network layer is left out, the conformance of interface enforcement is exactly 1.0000, and this is the number measured by the Prompt Engineering course; when the network is counted, the same number falls to 0.9000, that is, the remaining error share of schema enforcement comes not from the model but from the network. The tokens per correct record fall from 125.8 to 65.7 at four fields and from 287.1 to 123.0 at twelve fields. Adding fields grows the output tokens in both modes but does not raise the correct record rate. The previous lesson checked the validity of a call **afterwards**: the response arrived, was parsed, was validated and what did not pass was thrown away. The validator was a good counter, but for the calls that did not pass, the only thing it could do was count them. The Prompt Engineering course measured imposing the format **by writing it into the prompt**: the share of responses conforming to the schema came out 0.0000 in free text, 0.8000 with a format description and 1.0000 with schema enforcement that constrains generation. Those numbers are the **input** of this lesson and are not derived again. Two things are added here. First, when the schema can be carried as a request field, it becomes **independent of the length of the prompt**; as the schema grows, the cost of the two paths diverges. Second, that 1.0000 was measured in a **networkless** world; in a network operation the same enforcement does not give the same number. - **AU25.** The code is a **simulator**; the requests and the field set are **fictional**, seed **20260218**. The schema validation routine was measured in the Prompt Engineering course and is used here only as a **counter**. - **AU26. The field pool is twelve fields**; the schema is the first four, the first eight or all of them. A field rule is either a **number** or a closed set of values. - **AU27. Two enforcement paths are measured.** In prompt-level enforcement the **description** of the schema is written into the system message: the field names and the allowed values. In interface-level enforcement the schema is a **request field**; only the field names are carried and these tokens also enter the count. - **AU28. In interface-level enforcement generation cannot step outside the schema.** In prompt-level enforcement each field is produced correctly by an independent draw; in a failed draw the field is skipped, its name is corrupted or its value falls outside the closed set. - **AU29. The network layer continues.** Rate limit, timeout and mid-response cut corrupt the record; the **network-free conformance** column is read only over completed calls and corresponds to the number measured by the Prompt Engineering course. - **AU30. The concept is read only from the last user message**, so between the two modes only the enforcement path changes. - **AU31. The product criterion is the correct record:** the response that both conforms to the schema and whose topic field equals the topic of the request. - **AU32.** The context window and the maximum output length are widened in this lesson for the large records; the thresholds are fictional. The number of calls is **120** in all six setups. #### Two Ways of Carrying the Schema The same schema can travel by two roads: as a sentence inside the prompt, or as a field of the request. In the first, the schema is part of the text the model reads and costs a few tokens per field; in the second only the field names are carried and **generation is constrained**. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the network layer is in the simulator too. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_LENGTH, RATE_EVERY = 480, 60, 37 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def split_pieces(m): """FICTIONAL tokenizer; a piece has at most five characters.""" b = [] for s in m.lower().replace(",", " , ").split(): new = True while len(s) > 5: b.append((s[:5], new)) s, new = s[5:], False b.append((s, new)) return b def tokenize(m): return [p for p, _ in split_pieces(m)] def join_pieces(pl): s = "" for p, new in pl: s += (" " if new and s else "") + p return s def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def draw(prompt, seed, salt): return rng((seed + salt) ^ hash_text(prompt))() TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"} FIELDS = [("talep_no", "sayi"), ("konu", TOPICS), ("oncelik", ["dusuk", "orta", "yuksek"]), ("aksiyon", ["kayit", "yonlendirme", "bilgi"]), ("abone_no", "sayi"), ("kanal", ["telefon", "uygulama", "web"]), ("bolge_no", "sayi"), ("durum", ["acik", "beklemede", "kapali"]), ("sayac_no", "sayi"), ("ariza_kodu", "sayi"), ("adim_no", "sayi"), ("kayit_no", "sayi")] CALL_COUNT = [0] def flatten(messages): return " ".join(f"{r} : {i}" for r, i in messages) def call(messages, schema, at_interface, seed=SEED, temperature=0.6): """If the schema is enforced at the interface it is carried as a request field and generation cannot step outside it.""" CALL_COUNT[0] += 1 prompt = flatten(messages) toks = tokenize(prompt) if at_interface: toks = toks + tokenize(" ".join(schema)) # the schema is carried as a request field first = 1 + len(toks) // 40 empty = {"input": len(toks), "output": 0, "response": "", "first": first} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % RATE_EVERY == 0: return dict(empty, status="rate_limit") if len(toks) > 90 and draw(prompt, seed, 3) < 0.04: return dict(empty, status="timeout") last = messages[-1][1] # the concept is read only from the last message p = {k: 0.15 for k in TOPICS} for t in tokenize(last): for table, w in ((CLUES, 1.0), (TRAPS, 0.6)): if table.get(t) in p: p[table[t]] += w names = sorted(p) from math import exp e = [exp(p[a] / temperature) for a in names] k = names[pick(draw(last, seed, 5), [x / sum(e) for x in e])] number = int(next((x for x in prompt.split() if x.isdigit()), "0")) fields = [] for i, (a, rule) in enumerate(schema.items()): value = k if rule == TOPICS else (str(number + i) if rule == "sayi" else rule[(number + i) % len(rule)]) if at_interface or draw(prompt, seed, 101 + i) < 0.955: fields.append(f"{a}={value}") # cannot step outside the schema else: # three corruptions in prompt-level enforcement flaw = pick(draw(prompt, seed, 211 + i), [0.34, 0.33, 0.33]) if flaw == 1: fields.append(f"{a.replace('_', '')}={value}") elif flaw == 2: fields.append(f"{a}={'x' + value if rule == 'sayi' else 'acil'}") pl, status = split_pieces(" ; ".join(fields)), "done" if len(pl) > MAX_LENGTH: pl, status = pl[:MAX_LENGTH], "stopped" u = draw(prompt, seed, 91) # the network cuts the response midway if len(pl) > 3 and u < 0.08: pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut" return {"status": status, "input": len(toks), "output": len(pl), "response": join_pieces(pl), "first": first} def validate(y, schema): """The schema validation routine was measured in the Prompt Engineering course; here it is a counter.""" record = {} for part in y.split(" ; "): a, _, value = part.partition("=") record[a.strip()] = value.strip() if sorted(record) != sorted(schema): return False for a, rule in schema.items(): value = record[a] if (not value.isdigit()) if rule == "sayi" else (value not in rule): return False return True def read_topic(y, schema): for part in y.split(" ; "): a, _, value = part.partition("=") if a.strip() == "konu" and "konu" in schema: return value.strip() return None PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] REQUESTS = [] for i in range(120): # 120 FICTIONAL support requests r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] REQUESTS.append({"id": 4001 + i, "topic": k, "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]}) def describe_schema(schema): """The form of the schema written into the prompt: field names and allowed values.""" p = [f"{a}=<{'sayi' if k == 'sayi' else '|'.join(k)}>" for a, k in schema.items()] return "Yaniti su bicimde yaz : " + " ; ".join(p) + " ." def build_prompt(schema, x, at_interface): prefix = "" if at_interface else describe_schema(schema) return [("sistem", "destek kaydi cikaran bir yardimcisin . " + prefix), ("kullanici", f"Talep {x['id']} : {x['text']}")] SCHEMA4 = dict(FIELDS[:4]) for at_interface in (False, True): s = call(build_prompt(SCHEMA4, REQUESTS[0], at_interface), SCHEMA4, at_interface) path = "interface" if at_interface else "prompt" print(f"{path:<9} input {s['input']:>3} output {s['output']:>2} " f"conformance {validate(s['response'], SCHEMA4)} {s['response']}") print("description of the four-field schema:", len(tokenize(describe_schema(SCHEMA4))), "tokens , as a request field:", len(tokenize(" ".join(SCHEMA4)))) print("twelve-field:", len(tokenize(describe_schema(dict(FIELDS)))), "tokens ,", len(tokenize(" ".join(dict(FIELDS)))), "tokens") print("same request with seed 20260219 ->", call(build_prompt(SCHEMA4, REQUESTS[0], False), SCHEMA4, False, SEED + 1)["response"]) ``` ``` prompt input 65 output 15 conformance True talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi interface input 37 output 15 conformance True talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi description of the four-field schema: 35 tokens , as a request field: 7 twelve-field: 74 tokens , 21 tokens same request with seed 20260219 -> talep_no=4001 ; konu=sayac ; oncelik=orta ; aksiyon=bilgi ``` The two paths produced the same record on the same request and both passed validation; the difference is in the input column. The **description of the four-field schema is 35 tokens**, the same schema carried as a request field is **7 tokens**. The gap continues as the schema grows: 74 against 21 at twelve fields. The description has to carry the field names, the allowed values and an instruction sentence; the request field carries only the names and the interface already knows the rule. The last line follows from the third rule of the course: the same request on a second seed wrote a different value into the topic field, and that record too would have passed the schema. #### Conformance and Cost Table of the Six Setups The same 120 fictional requests are run on two enforcement paths and at three schema sizes. ```python def measure(n, at_interface, seed=SEED): CALL_COUNT[0] = 0 schema = dict(FIELDS[:n]) tok_in = tok_out = net_fail = conforming = done = done_ok = good = 0 for x in REQUESTS: s = call(build_prompt(schema, x, at_interface), schema, at_interface, seed) tok_in, tok_out = tok_in + s["input"], tok_out + s["output"] ok = validate(s["response"], schema) net_fail += s["status"] != "done" conforming += ok if s["status"] == "done": done, done_ok = done + 1, done_ok + ok good += ok and read_topic(s["response"], schema) == x["topic"] return tok_in, tok_out, net_fail, conforming / 120, done_ok / done, good / 120 print("mode fields input output network conformance A conformance B" " network-free conformance correct record") for at_interface in (False, True): for n in (4, 8, 12): a, b = measure(n, at_interface), measure(n, at_interface, SEED + 1) print(f"{'interface' if at_interface else 'prompt':<9} {n:>6} {a[0]:>6} {a[1]:>6}" f" {a[2]:>7} {a[3]:>13.4f} {b[3]:>13.4f} {a[4]:>24.4f} {a[5]:>14.4f}") for n in (4, 12): i, y = measure(n, False), measure(n, True) print(f"\n{n} fields : interface saves {i[0] - y[0]} input tokens ," f" conformance gap {y[3] - i[3]:.4f} ," f" tokens per correct record {(i[0] + i[1]) / (i[5] * 120):.1f}" f" -> {(y[0] + y[1]) / (y[5] * 120):.1f}") ``` ``` mode fields input output network conformance A conformance B network-free conformance correct record prompt 4 7654 1658 11 0.7583 0.8167 0.8349 0.6167 prompt 8 10294 3361 14 0.7250 0.7333 0.8208 0.6000 prompt 12 12334 4894 16 0.6083 0.5583 0.7019 0.5000 interface 4 4294 1688 12 0.9000 0.9167 1.0000 0.7583 interface 8 5014 3417 12 0.9000 0.9167 1.0000 0.7583 interface 12 5974 5219 12 0.9000 0.9167 1.0000 0.7583 4 fields : interface saves 3360 input tokens , conformance gap 0.1417 , tokens per correct record 125.8 -> 65.7 12 fields : interface saves 6360 input tokens , conformance gap 0.2917 , tokens per correct record 287.1 -> 123.0 ``` The prompt-level enforcement rows go down as the schema grows: conformance **0.7583**, then 0.7250, then **0.6083**. The cause is structural. If every field is tied to an independent draw, the probability that all the fields are correct at once falls exponentially with the number of fields; a small risk paid at four fields becomes an unaffordable one at twelve. The interface-level enforcement rows stay at **0.9000** in all three sizes. The schema grew, the conformance did not change. The input column shows the same divergence in cost. Prompt-level enforcement goes from 7654 to **12334** tokens while interface-level enforcement goes from 4294 to **5974**; the saving is 3360 tokens at four fields and **6360** tokens at twelve fields. The output column grows in both modes (from 1688 to 5219), because the record really gets longer. At twelve fields the output of interface-level enforcement is **more** than that of prompt-level enforcement: 5219 against 4894. The reason is that corrupted fields are **skipped**; a record with missing fields is shorter. #### Network-Free Conformance and the Share of the Network The most important column of the table is **network-free conformance**. In interface-level enforcement this column is exactly **1.0000** in all three rows. This is the number measured by the Prompt Engineering course and it comes out here unchanged: if generation cannot step outside the schema, every completed response conforms to the schema. The conformance column of the same rows is **0.9000**. The gap between it and 1.0000 is the share not of the model but of the **network**: 12 of the 120 calls are caught in a rate limit, a timeout or a mid-response cut, and a half record does not pass the schema. The rule of the course becomes directly visible here. Measured as a function call, the conformance of schema enforcement is 1.0000; measured as a **network operation**, it is 0.9000. A call whose failure is not written down counts as unmeasured, and the difference between the two is exactly that unwritten failure. The two runs differ by at most **0.0583** (prompt-level enforcement, four fields). The gain of the interface is 0.1417 at four fields and 0.2917 at twelve fields; both are above this difference and are **measured**. The tokens per correct record fall **from 125.8 to 65.7** at four fields and **from 287.1 to 123.0** at twelve fields. The last column carries a warning. In interface-level enforcement the correct record rate is fixed at **0.7583** in all three sizes; the gap between it and the conformance 0.9000 is records that pass the schema but have the **wrong topic**. Adding fields tripled the output tokens and did **not** change the correct record rate at all. A guarantee of the format is not a guarantee of the content; this distinction was measured in the Prompt Engineering course and here it stands once more, at the interface level, in the same place. #### Summary - The description of the same schema written into the prompt costs 35 tokens at four fields and 74 at twelve fields; its form carried as a request field is 7 and 21 tokens. - The conformance of prompt-level enforcement falls from 0.7583 to 0.6083 as the schema grows, because each field is an independent risk; interface-level enforcement stays at 0.9000 in all three sizes. - The input tokens go from 7654 to 12334 with the prompt and from 4294 to 5974 with the interface; the saving is 3360 tokens at four fields and 6360 at twelve fields. - The network-free conformance of interface-level enforcement is exactly 1.0000; when the network is counted it falls to 0.9000, and the gap between the two is the share of the network, not the model. - The tokens per correct record fall from 125.8 to 65.7 at four fields and from 287.1 to 123.0 at twelve fields; these gains are measured because the difference between the two runs is at most 0.0583. - Adding fields grows the output tokens from 1688 to 5219 but leaves the correct record rate fixed at 0.7583: a format guarantee is not a content guarantee. #### Next Step In four lessons the four abilities of the interface were measured and the cost of each was written in tokens: carrying the conversation state, receiving the response chunk by chunk, tying the output to a call and fitting the output to a schema. The four shared an assumption that was never questioned: **the input was always text**. The request was a sentence, the state was a transcript, the schema was a list of fields and all of them went through the same tokenizer. When a photograph of a meter, a voice message or a short video arrives next to a support request, none of these numbers stays as it is. If the same information can also be given as text, how many tokens does each cost and what happens to accuracy when the form changes? The next lesson makes the form of the input a variable and counts its cost on the same scale. ### Multimodal Input Source: https://academia.sh/en/courses/model-apis/multimodal-input Measuring the same field inspection record in six input formats: the text summary 21, the cropped fine grid 42, the coarse grid 59, the fine grid 183 tokens, and the sequence between 195 and 330 per record. The usable record rate is 0.0833 on the coarse grid, 0.8167 on the fine grid and 0.7000 on the cropped grid; the cropped format delivers about eighty-six percent of the fine grid's usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0. Although the sequence format has the highest visibility, it stays at 0.3667 because it exceeds the context window in thirty-four of sixty records, and it becomes the most expensive format. The audio mode reads the index at 0.8833 but, being blind to the body condition, produces no usable record. The previous four lessons measured what the interface offers: a message sequence made of roles and contents, a response that arrives piece by piece, a call tied to code execution and an output mode that constrains generation. In all four, what entered the model was **text**. Token counting was done on text, the context window filled up with text, truncation decisions were pruned from text. The inspection record in the field team's hands is not text. The face of the meter has an image, sometimes there is a voice note, sometimes a short capture of a few frames. A multimodal interface accepts these; accepting them is not free. This lesson sends the same inspection record in six different formats and prints two numbers side by side: **the token cost of the input** and **the accuracy of the reading**. What is measured is not the model's ability to see, but the bill for the choice of format. - **AU41.** The code is a **simulator**; no real multimodal endpoint is called. The inspection records, the meter face and all the mode visibilities are **fictional**, seed **20260218**. - **AU42.** The unit a multimodal endpoint bills is still the **token**. An image is split into grid cells, a sound into pieces, a sequence into frames, and **each cell counts as a token**. The whole cost column of this lesson is written in this unit. - **AU43.** The six formats are: a hand-written **text** summary, the **coarse grid** (4x10), the **fine grid** (8x20), the **cropped** fine grid (3x8), the **sequence** (four to seven frames per record, each frame a coarse grid) and **audio** (eighteen to thirty pieces). - **AU44.** Each mode carries two **visibility** shares: the share with which the index and the body condition are read correctly. These shares are fictional constants of the simulator; they rise as the resolution rises. - **AU45.** The audio mode's visibility of the body condition is **zero**. A mode can be **blind** to a field, and this is not an error but the limit of the information that mode carries. - **AU46.** A **usable record** is a record whose index and body condition are both read correctly; this is the product criterion of this lesson. A record with only one field read correctly is of no use to the field team. - **AU47.** The context window is **240 tokens** and the rate limit is **once in thirty-seven calls**. The surface built in the Introduction to AI Engineering course continues unchanged. - **AU48.** The Natural Language Processing and Computer Vision course of the Machine Learning and Data Science curriculum measured the **shared representation** of a multimodal model; that procedure is not repeated here. The only thing measured is the **cost of the input format**. #### Six Formats of One Record When an image passes through the interface it is turned into a sequence of numbers, and that sequence into tokens. To make the conversion visible, the simulator of this lesson writes the grid directly into the prompt text: each cell a mark, each mark a token. At a real endpoint the intermediate steps are different, but **the bill comes from the same place**: as the resolution rises, the token count grows together with the cell count. The block below produces sixty fictional inspection records, builds the six formats and defines the multimodal call. The visibility shares changing by mode is the simulator's only assumption. ```python # SIMULATOR -- FICTIONAL. No real multimodal endpoint is called. The inspection # records, the meter face and all the mode visibilities are fictional. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW = 240 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per piece b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CONDITIONS = ["saglam", "catlak", "muhursuz"] RECORDS = [] for i in range(60): # 60 FICTIONAL field inspection records r = rng(SEED + 900 + i) RECORDS.append({"number": 7001 + i, "index": "".join(str(int(r() * 10)) for _ in range(5)), "condition": CONDITIONS[pick(r(), [0.58, 0.27, 0.15])], "frames": 4 + int(r() * 4), "length": 18 + int(r() * 13)}) CELLS = ".:-=+*#" def grid(record, rows, cols): """FICTIONAL grid of the meter face: each cell is written as a separate token.""" r = rng(hash_text(record["index"] + record["condition"]) ^ (rows * 31 + cols)) out = [] for y in range(rows): out.append(f"s{y}") out += [CELLS[int(r() * 7)] for _ in range(cols)] return " ".join(out) def render(record, mode): """Six input formats of the same record. The information carried is the same, the length is not.""" if mode == "text": return f"endeks {record['index']} govde {record['condition']}" if mode == "coarse grid": return grid(record, 4, 10) if mode == "fine grid": return grid(record, 8, 20) if mode == "cropped grid": return grid(record, 3, 8) if mode == "audio": r = rng(hash_text(record["index"]) ^ 7717) return " ".join(f"d{int(r() * 9)}" for _ in range(record["length"])) return " ".join(f"k{j} " + grid(record, 4, 10) for j in range(record["frames"])) # sequence: frame by frame VISIBILITY = {"text": (1.00, 1.00), "coarse grid": (0.62, 0.16), "fine grid": (0.93, 0.85), "cropped grid": (0.91, 0.80), "sequence": (0.88, 0.95), "audio": (0.90, 0.00)} PREFIX = "Sayac yuzunu oku . Endeks ve govde durumunu yaz . Kayit: " CALL_COUNT = [0] def multimodal_call(record, mode, seed=SEED): """Multimodal simulator: each mode carries each field with its own visibility share.""" CALL_COUNT[0] += 1 toks = tokenize(PREFIX + render(record, mode)) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0} if CALL_COUNT[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0} r = rng(seed ^ hash_text(" ".join(toks))) vis_index, vis_condition = VISIBILITY[mode] e, d = record["index"], record["condition"] if r() >= vis_index: # an unreadable digit slips to its neighbor j = int(r() * 5) e = e[:j] + str((int(e[j]) + 1 + int(r() * 8)) % 10) + e[j + 1:] if r() >= vis_condition: d = CONDITIONS[(CONDITIONS.index(d) + 1 + int(r() * 2)) % 3] return {"status": "done", "input": len(toks), "index": e, "body": d, "output": len(tokenize(f"endeks {e} govde {d}"))} k = RECORDS[0] print(f"record {k['number']} : index {k['index']} , condition {k['condition']} ," f" {k['frames']} frames , {k['length']} audio pieces") for mode in VISIBILITY: print(f" {mode:<14} input tokens {len(tokenize(PREFIX + render(k, mode))):>4}") print("first two rows of the coarse grid:") print(" ", " ".join(grid(k, 4, 10).split()[:22])) long_record = next(x for x in RECORDS if x["frames"] == 7) for record, mode in ((k, "text"), (k, "coarse grid"), (k, "fine grid"), (long_record, "sequence")): s = multimodal_call(record, mode) info = (f"index {s['index']} condition {s['body']}" if s["status"] == "done" else f"{s['input']} tokens , window {WINDOW}") print(f"{record['number']} {mode:<14} {s['status']:<16} {info}") ``` ``` record 7001 : index 17328 , condition saglam , 5 frames , 21 audio pieces text input tokens 21 coarse grid input tokens 59 fine grid input tokens 183 cropped grid input tokens 42 sequence input tokens 240 audio input tokens 36 first two rows of the coarse grid: s0 . * : - = * - : : + s1 = . : + = + : * + . 7001 text done index 17328 condition saglam 7001 coarse grid done index 17328 condition catlak 7001 fine grid done index 17328 condition saglam 7005 sequence context_overflow 330 tokens , window 240 ``` The difference among the six formats of a single record ranges between **21 tokens and 240 tokens**. The information carried is the same in every row: a five-digit index and a three-valued body condition. The elevenfold gap between them comes not from the information but from **the container the information is wrapped in**. The last four rows show two separate failures. The coarse grid reads the index correctly but gives the body condition as `cracked` instead of `intact`: the resolution is not enough to tell the seal from the crack. For record 7005, which has seven frames, no reading takes place at all; the 330-token sequence does not fit the 240-token window and the call returns with `context_overflow`. **The richest input is the input that is no input.** #### Cost and Accuracy Table of the Six Formats All sixty records are run separately in the six formats. Each row prints the total input tokens, the total output tokens, the number of failed calls, the separate accuracies of the two fields, the usable record rate and the **tokens per usable record**. The last column is the second run: the usable record rate obtained with seed 20260219, which says whether a difference can be told from noise. ```python def measure(mode, seed=SEED): CALL_COUNT[0] = 0 tok_in = tok_out = failed = index_ok = cond_ok = usable = 0 for x in RECORDS: s = multimodal_call(x, mode, seed) tok_in, tok_out = tok_in + s["input"], tok_out + s["output"] if s["status"] != "done": failed += 1 continue e, d = s["index"] == x["index"], s["body"] == x["condition"] index_ok, cond_ok, usable = index_ok + e, cond_ok + d, usable + (e and d) return tok_in, tok_out, failed, index_ok / 60, cond_ok / 60, usable / 60 print("mode input output failed index condition usable record" " tokens/record second run") for mode in VISIBILITY: a, b = measure(mode), measure(mode, SEED + 1) tpr = f"{a[0] / (a[5] * 60):.1f}" if a[5] else "-" print(f"{mode:<14} {a[0]:>6} {a[1]:>6} {a[2]:>10} {a[3]:>7.4f} {a[4]:>9.4f}" f" {a[5]:>15.4f} {tpr:>13} {b[5]:>13.4f}") ``` ``` mode input output failed index condition usable record tokens/record second run text 1260 354 1 0.9833 0.9833 0.9833 21.4 0.9833 coarse grid 3540 354 1 0.6500 0.1333 0.0833 708.0 0.0833 fine grid 10980 354 1 0.9500 0.8500 0.8167 224.1 0.7833 cropped grid 2520 354 1 0.8833 0.7833 0.7000 60.0 0.6833 sequence 16155 150 35 0.3667 0.4167 0.3667 734.3 0.2833 audio 2300 354 1 0.8833 0.0000 0.0000 - 0.0000 ``` The output token column is the same in five rows: 354; in the sequence it is 150, because thirty-five calls produce no response at all. The response is a single record of six tokens in every format. What changes is only the input, and this is the plainest row showing **which side of the bill of a multimodal interface swells**. The total cost of the text summary is 1260 tokens; the cost of the fine grid is 10980, that is, eight and a half times as much. The same information, the same response. The first row is a boundary case and should be read honestly. The text summary is both the cheapest and the most accurate, but that summary must **already have been read by someone**. The reason the usable record rate stays at 0.9833 is not the reading but the rate limit rejection that arrives on the thirty-seventh call. This row is not a mode option but the form the job takes once it has been solved on the human side; its function in the table is to show **how much of it the other five rows are trying to win back**. #### Cropping, the Blind Mode and the Sequence That Does Not Fit the Window The difference between the coarse grid and the fine grid tells which field the resolution rescues. The index accuracy rises from 0.6500 to 0.9500, but the real leap is in the body condition: from 0.1333 to 0.8500. A crack is **a finer detail** than a five-digit number and is invisible on the coarse grid. The usable record behaves like the product of these two fields and rises from 0.0833 to 0.8167. The cost triples: from 3540 tokens to 10980 tokens. The cropped grid row is the real finding of this lesson. Cropping shrinks not the resolution but the **area**: the same fine level of detail is applied only to the index and seal region of the meter. The result is 2520 tokens and 0.7000 usable record. It delivers about eighty-six percent of the fine grid's usable records with **a quarter** of its tokens. The cost per usable record falls from 224.1 to 60.0. The two run columns support this reading: the fine grid gives 0.8167 and 0.7833, the cropped grid gives 0.7000 and 0.6833, so the 0.1167 difference between them is larger than the difference between runs and is real. What is real is the superiority of the fine grid; what is debatable is **whether that superiority is worth 164 tokens per record**. The sequence row is the format with the highest visibility and one of the worst rows of the table. Looking frame by frame makes the body condition visible with a 0.95 share, because motion gives the crack away. But thirty-four of the sixty records carry six or seven frames and do not fit the 240-token window. The number of failed calls is 35 (thirty-four context overflows and one rate limit), the usable record is 0.3667, and the total cost is the highest of the table at 16155 tokens. The mode with the highest visibility is, in the records it delivers, half of the cropped grid. This is the counterpart in multimodal input of the second claim of this curriculum: **added richness can lower the measure.** The audio row shows another limit. The index accuracy is 0.8833, close to the fine grid, and the body condition accuracy is **0.0000**. The audio recording carries a person reading out the index; it does not carry the crack. The usable record is therefore zero and the cost per record cannot be computed. A mode not carrying a field does not show up in an average accuracy number; it appears only when the fields are counted **separately**. Choosing a mode is not a resolution decision but a decision about **which fields will be carried**. #### Summary - The bill of a multimodal interface is still written in tokens: a grid cell, an audio piece and a sequence frame turn into input tokens, and the output tokens stay fixed at 354 in five formats. - The cost of the same record is 21 tokens in the text summary, 42 on the cropped grid, 59 on the coarse grid and 183 on the fine grid; the information carried does not change, the container does. - Raising the resolution lifts the body condition accuracy from 0.1333 to 0.8500; the fine detail is invisible on the coarse grid and the usable record stays at 0.0833. - Cropping shrinks the area, not the resolution: it delivers about eighty-six percent of the fine grid's usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0. - The sequence format has the highest visibility but stays at 0.3667 because thirty-four of the sixty records do not fit the window, and at 16155 tokens it is the most expensive row. - The audio mode reads the index at 0.8833, is blind to the body condition and produces no usable record; a field a mode does not carry becomes visible only when the fields are counted separately. #### Next Step In this lesson the input was always sent **about one record** and the model read it. The question left unasked is this: if we had to find which of the sixty records resembles the current request, since putting all of them into the prompt is not possible, what would similarity be computed over? The interface offers a separate surface for this: an endpoint that turns text into a sequence of numbers and returns not text but a **vector** as the response. The next lesson measures this surface and focuses on a single decision: how many dimensions the vector will have. As the dimensions fall, the stored numbers and the comparison cost fall, and so does the neighborhood accuracy; the point at which the two part ways is counted. ### Embedding Interfaces Source: https://academia.sh/en/courses/model-apis/embedding-interfaces Measuring an endpoint that returns a vector instead of text in the response, and a single decision: how many dimensions to request for the embedding. Over seventy-two fictional records the cost of the call is independent of the dimension and stays fixed at 1814 input tokens; what changes is the stored numbers (from 10368 to 648) and the comparison multiplications (from 368064 to 23004). The share of the three nearest neighbors being from the same topic is 0.9259 at 144 dimensions, 0.9028 at 72, 0.8102 at 36, 0.6389 at 9. The loss from halving the dimension is 0.0231, the difference from a second embedding model at the same dimension is 0.0232: halving cuts the storage in half at an unmeasurable loss. As the dimension falls, the difference between the two models grows from 0.0232 to 0.1018, and the gain of folding on the client side does not exceed this difference. The previous lesson measured the format of the input and wrote the bill of each format in tokens. The question left at the end was this: if we had to find which of the sixty records resembles the current request, since putting all of them into the prompt is not possible, what is similarity computed over? The interface offers a separate surface for this. An **embedding API** takes a text and returns, as the response, not text but a fixed-length **sequence of numbers**. This surface has a single setting worth measuring, and this lesson measures it: **how many dimensions the vector will have**. As the dimension falls, the stored numbers and the comparison cost fall, and so does the neighborhood accuracy. The point at which the two part ways is counted. - **AU49.** The code is a **simulator**; no real embedding endpoint is called. The records, the piece vocabulary and the embedding routine itself are **fictional**, seed **20260218**. - **AU50.** The simulator produces coordinates from the trace of the token and **places the information weighted toward the first coordinates**. The `dimension` parameter means returning that many coordinates from the start. This is the structural counterpart of an endpoint that offers a dimension parameter. - **AU51.** The embedding API is **deterministic**: the same text gives the same vector on every call. This is the run-independent measure of this lesson and is shown in the block. - **AU52.** That is why the second run is read from a separate baseline: **a second embedding model of the same family** (a simulator built with seed 20260219). For a dimension decision to be considered portable, the loss has to **exceed the difference between the two models**. - **AU53. Neighborhood accuracy** is the share of the **three nearest neighbors** of each record being from the same topic. The measure is taken over three neighbors instead of one, because a single neighbor changes with the movement of a single pair. - **AU54. The retrieval pipeline is not built in this lesson.** Which document enters the prompt, how many documents are retrieved and the cost of retrieving wrongly are left to the Embeddings and Retrieval-Augmented Generation course. Here only the standing of the vectors **relative to each other** is measured. - **AU55.** Embedding, angle similarity and dimension reduction were measured in the Unsupervised Learning and the Natural Language Processing and Computer Vision courses of the Machine Learning and Data Science curriculum; those procedures are not repeated here. The only thing added is **the dimension decision at the interface level**. - **AU56.** The comparison cost is written as the **number of multiplications** needed for all pairs of the seventy-two records. The storage cost is **the count of numbers kept**. #### The Endpoint That Returns a Vector The block below produces seventy-two fictional records, builds the embedding API and shows two things. The first is that the endpoint is deterministic: the same text gives the same vector in two calls, which means none of the numbers of this lesson carries run-to-run variation. The second is the mechanism: what happens, as the dimension falls, to the **gap** between the similarity to a record from the same topic and the similarity to a record from another topic. ```python # SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The records, # the piece vocabulary and the embedding routine are fictional. SEED, M32 = 20260218, 0xFFFFFFFF DIMENSION = 144 # the largest embedding dimension of the endpoint def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(m): # FICTIONAL tokenizer, at most five characters per piece b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PHRASES = { "fatura": ["fatura tutari cok yuksek geldi", "odemeyi yaptim borc gorunuyor", "fatura ayrintisini goremiyorum", "son odeme tarihi gecmis", "gecmis donem dokumu lazim", "tutar gecen aya gore artmis"], "ariza": ["evde su akmiyor bir ariza var", "baglanti bozuk hic calismiyor", "ariza kaydi actim ekip gelmedi", "vana kapali kalmis", "boru patlamis sokak su icinde", "servis ekibi randevuyu kacirdi"], "sayac": ["sayac okumasi yanlis girilmis", "endeks tutmuyor gibi", "sayac degisimi randevusu istiyorum", "sayac cami buharlanmis", "muhur kopmus gorunuyor", "endeks fotografini gonderdim"], "tarife": ["hangi tarife basamagindayim", "tarife degisikligi nasil olur", "birim ucret abone tipine gore degisiyor", "indirimli tarife sarti nedir", "kademeli hesap anlasilmiyor", "abone tipim yanlis kayitli"], "kesinti": ["mahallede kesinti var", "planli kesinti duyurusu nerede", "aksamdan beri kesinti yasiyoruz", "kesinti suresi uzadi", "bakim calismasi ne zaman biter", "sokak boyunca su yok"]} TAILS = ["abone numaram kayitli", "bolge merkez ilce", "daha once de yazdim", "acil degil ama takipteyim", "telefonla ulasilamiyor", "ek belge gonderdim"] RECORDS = [] for i in range(72): # 72 FICTIONAL records: 2-4 of its own phrases, sometimes a foreign one r = rng(SEED + 400 + i) k = TOPICS[int(r() * 5)] pool = list(PHRASES[k]) chosen = [pool.pop(int(r() * len(pool))) for _ in range(2 + int(r() * 3))] if r() < 0.35: y = TOPICS[(TOPICS.index(k) + 1 + int(r() * 4)) % 5] chosen.append(PHRASES[y][int(r() * 6)]) chosen.append(TAILS[int(r() * 6)]) RECORDS.append({"number": 5001 + i, "topic": k, "text": " ".join(chosen)}) CALL_COUNT = [0] def embed(text, dimension=DIMENSION, seed=SEED): """FICTIONAL embedding endpoint. The information is placed weighted toward the first coordinates; the dimension parameter means returning that many coordinates from the start.""" CALL_COUNT[0] += 1 toks = tokenize(text) v = [0.0] * DIMENSION for t in toks: r = rng(hash_text(t) ^ seed) for _ in range(10): # each token touches ten coordinates i = int(r() ** 2 * DIMENSION) # squaring gathers the weight into the first coordinates v[i] += 1.0 if r() < 0.5 else -1.0 v = v[:dimension] n = sum(x * x for x in v) ** 0.5 or 1.0 return [x / n for x in v], len(toks) def fold(v, d): """Dimension reduction on the client side: coordinates are folded by summing.""" k = [0.0] * d for i, x in enumerate(v): k[i % d] += x n = sum(x * x for x in k) ** 0.5 or 1.0 return [x / n for x in k] def similarity(u, v): return sum(a * b for a, b in zip(u, v)) x = RECORDS[0] same = next(y for y in RECORDS[1:] if y["topic"] == x["topic"]) other = next(y for y in RECORDS[1:] if y["topic"] != x["topic"]) full, n_tokens = embed(x["text"]) print(f"record {x['number']} ({x['topic']}) : {x['text'][:56]}") print(f"input tokens {n_tokens} , returned numbers {len(full)} , second call same vector:" f" {embed(x['text'])[0] == full}") print("first eight coordinates:", " ".join(f"{c:+.3f}" for c in full[:8])) print("last eight coordinates:", " ".join(f"{c:+.3f}" for c in full[-8:])) print(f"\ndimension {x['number']}-{same['number']} ({same['topic']}) " f"{x['number']}-{other['number']} ({other['topic']}) gap") for d in (144, 36, 9): a, b = embed(x["text"], d)[0], embed(same["text"], d)[0] c = embed(other["text"], d)[0] ab, ac = similarity(a, b), similarity(a, c) print(f"{d:>9} {ab:>17.4f} {ac:>19.4f} {ab - ac:>8.4f}") ``` ``` record 5001 (sayac) : muhur kopmus gorunuyor sayac cami buharlanmis endeks fot input tokens 37 , returned numbers 144 , second call same vector: True first eight coordinates: -0.329 +0.094 +0.094 +0.235 +0.047 +0.047 +0.141 -0.094 last eight coordinates: +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.047 dimension 5001-5002 (sayac) 5001-5003 (tarife) gap 144 0.5074 0.1790 0.3284 36 0.3991 0.3544 0.0447 9 0.4423 0.3922 0.0501 ``` The first output line shows how this surface differs from the surface that returns text: an input of thirty-seven tokens turns into an output of a hundred and forty-four numbers, and the second call gives **the same vector**. In a call that generates text, the same prompt could give two different responses in two runs; here it cannot. All the differences of this lesson are therefore real differences, not run-to-run movement. The coordinate lines make the simulator's assumption visible. The first eight coordinates are full; seven of the last eight coordinates are zero. The information has been gathered in the leading coordinates; this is why the dimension parameter means **cutting from the start**. The gap table is the real mechanism. At a hundred and forty-four dimensions the similarity to a neighbor from the same topic is 0.5074, to a neighbor from another topic 0.1790, and the gap is **0.3284**. At thirty-six dimensions the two similarities move toward each other and the gap falls to **0.0447**. As the dimension falls the vectors do not move away from each other, they **move closer to each other**; what makes telling them apart harder is not the distance itself but the closing of the gap between the distances. The 0.0501 gap at nine dimensions does not show that the gap has opened again; a single pair says no more than a single draw, and the decision is read from the table. #### Cost and Accuracy Table of Five Dimensions The seventy-two records are embedded separately at five dimensions. Each row prints the count of stored numbers, the number of multiplications needed to compare all pairs, the neighborhood accuracy, the accuracy of the second embedding model at the same dimension and the accuracy obtained by folding the full vector on the client side. ```python N, K = len(RECORDS), 3 CALL_COUNT[0] = 0 FULL = [embed(y["text"])[0] for y in RECORDS] SECOND = [embed(y["text"], DIMENSION, SEED + 1)[0] for y in RECORDS] INPUT_TOKENS = sum(embed(y["text"])[1] for y in RECORDS) def truncate(v, d): u = v[:d] n = sum(a * a for a in u) ** 0.5 or 1.0 return [a / n for a in u] def neighbor_accuracy(vectors): """Share of the three nearest neighbors being from the same topic. No retrieval pipeline is built.""" nearest = [[] for _ in range(N)] for i in range(N): for j in range(i + 1, N): s = similarity(vectors[i], vectors[j]) nearest[i].append((s, j)) nearest[j].append((s, i)) d = 0 for i in range(N): for _, j in sorted(nearest[i], reverse=True)[:K]: d += RECORDS[j]["topic"] == RECORDS[i]["topic"] return d / (N * K) print(f"{N} records , {CALL_COUNT[0]} embedding calls , total input {INPUT_TOKENS} tokens" f" -- this number is independent of the dimension") print("dimension stored comparison endpoint second model client folding") for d in (144, 72, 36, 18, 9): ep = neighbor_accuracy([truncate(v, d) for v in FULL]) sm = neighbor_accuracy([truncate(v, d) for v in SECOND]) fd = neighbor_accuracy([fold(v, d) for v in FULL]) print(f"{d:>9} {N * d:>9} {N * (N - 1) // 2 * d:>14} {ep:>9.4f}" f" {sm:>13.4f} {fd:>18.4f}") ``` ``` 72 records , 216 embedding calls , total input 1814 tokens -- this number is independent of the dimension dimension stored comparison endpoint second model client folding 144 10368 368064 0.9259 0.9491 0.9259 72 5184 184032 0.9028 0.9259 0.9120 36 2592 92016 0.8102 0.9120 0.9120 18 1296 46008 0.7685 0.8611 0.8148 9 648 23004 0.6389 0.7176 0.7315 ``` The header line carries the number of this table that is easiest to overlook. The total input is **1814 tokens** and this number is the same at all five dimensions. The dimension decision **does not change the cost of the call**: the same texts, the same tokens, the same number of calls. What changes starts after the call. The stored numbers fall from 10368 to 648, the comparison multiplications from 368064 to 23004. The bill of an endpoint and the cost of a system do not stand in the same place, and the dimension decision belongs to the second. #### Which Baseline the Loss Is Read Against If the endpoint column is read alone, the decision looks easy: as the dimension falls the accuracy falls, so the largest dimension is taken. The second model column upsets this reading. At a hundred and forty-four dimensions the two models give 0.9259 and 0.9491; the difference between them is **0.0232**. The cost of halving the dimension to seventy-two is from 0.9259 to 0.9028, that is, **0.0231**. The loss is of the same size as the difference between two embedding models at the same dimension. This decision has **no measured loss**, and in return it cuts storage and comparison exactly in half. This is the clearest gain in the table and it runs against intuition. When we go down to thirty-six dimensions the situation changes. The endpoint gives 0.8102; the loss against a hundred and forty-four is 0.1157, and this is five times the difference between the two models. At nine dimensions the loss rises to 0.2870. The break lies between seventy-two and thirty-six: until three quarters of the storage is won, the cost stays at an unmeasurable level, then it is paid quickly. The second model column says something in itself, too. The difference between the two models is 0.0232 at a hundred and forty-four and rises to **0.1018** at thirty-six. As the dimension falls, **which model you use matters more**: at a high dimension the two models build almost the same neighborhoods, at a low dimension they part ways. The cost of changing models in a low-dimensional setup is more than four times that in a high-dimensional one. The last column asks who reduces the dimension. Taking the full vector and folding the coordinates gives 0.9120 at thirty-six dimensions; this is 0.1018 higher than the 0.8102 of requesting thirty-six dimensions from the endpoint. But this gain is exactly equal to the difference between the two models, and at eighteen dimensions it falls to 0.0463, that is, half of the model difference of that row. Against this baseline it is **not a measured superiority**. Moreover it has a cost: folding requires the full vector, that is, receiving and carrying a hundred and forty-four numbers on every call. The storage gain is kept, the transfer gain is lost. On which side the abstraction stops is decisive here too: the dimension parameter of the endpoint makes a decision in your place, and you can see the cost of that decision only when you put the two baselines side by side. #### Summary - The embedding API returns a fixed-length sequence of numbers instead of text and is deterministic: the same text gives the same vector on every call, so the differences of this lesson are not run-to-run movement. - The dimension decision does not change the cost of the call; the total input tokens of the seventy-two records are 1814 at all five dimensions. What changes is the stored numbers (from 10368 to 648) and the comparison multiplications (from 368064 to 23004). - As the dimension falls, the similarity gap between the same topic and another topic closes: on a single pair it goes from 0.3284 to 0.0447, and the neighborhood accuracy falls from 0.9259 to 0.6389. - The loss from halving the dimension is 0.0231 and the difference from the second model at the same dimension is 0.0232: while storage and comparison are halved there is no measured loss. The break lies between seventy-two and thirty-six. - The difference between the two models grows from 0.0232 to 0.1018 as the dimension falls; at a low dimension the choice of model weighs more. - Folding the full vector on the client gains 0.1018 at thirty-six dimensions, but this gain is at the level of the model difference and requires carrying a hundred and forty-four numbers on every call. #### Next Step The embedding endpoint in this lesson and the text endpoint in the earlier lessons do not have to come from the same provider. The name of the dimension parameter, the scale of the distribution temperature, the shape of the error field and the key under which the response returns change from provider to provider. An application wants to handle these differences in a single place instead of at every call, and writes a common interface. The next lesson builds two providers at the simulator level and counts what that common interface does: how many fields are carried, how many fields are lost, and which of the lost fields makes a measure invisible. ### Compatible Interface Layers Source: https://academia.sh/en/courses/model-apis/compatible-interface-layers Counting, field by field, two fictional providers and the common interface between them: provider A offers twelve fields, eight are carried and four are lost; provider B offers ten fields, eight are carried and two are lost. Because the stop sequence field exists only in A, it finds no place in the intersection and the output tokens rise from 773 to 1638. Of the hundred and twenty calls, 117 are cut at the length limit; both providers report this in their own interface, and the number of cuts readable from the common interface is 0. With the same common call, temperature 0.6 gives 0.5500 accuracy on A and 0.5000 on B; because temperatures 0.4 and 0.6 are rounded to the same setting on B, both give 0.5000 there while A separates between 0.7583 and 0.5500. The embedding endpoint in the previous lesson and the text endpoint in the lessons before it do not have to come from the same provider. As an application grows, a second provider is added, a task is moved to the cheaper one, another is kept at hand. At that point nobody wants to deal with field names written separately at every call site, and a **common interface** is written: a single function, a single set of names, and underneath a mapping that varies by provider. This lesson counts what that layer does. Two providers are built **at simulator level**; both are fictional and both call the same engine underneath. What is measured is **the fields the abstraction carries and the fields it loses**: how many fields pass, how many fields drop, and which of the dropped fields makes a measure invisible. The rule built in the Data Analysis with Python course of the Data Analytics curriculum holds here as it is: **an abstraction hides the decision beneath it.** - **AU57.** The code is a **simulator**. **Both providers are fictional** and are built at simulator level; no real endpoint is called. Seed **20260218**, second run **20260219**. - **AU58.** The two providers share **a single engine**. So all the differences between them are **interface differences**, not model differences. - **AU59.** Provider A offers **twelve fields**, provider B **ten fields**; the names are listed in the block. B **has no stop sequence field** and B's `variety` setting is an **integer** between zero and four, that is, it fits its own scale. - **AU60.** The common interface is the **intersection** of the two providers: only a field that has a counterpart in both can be carried. A field that exists on a single provider has no place in the common interface. - **AU61.** A **stop sequence** is a field that cuts generation at a given token; in this lesson it takes effect at a word common to all three response patterns. - **AU62.** The simulator writes the **actual finish reason** of each call to its own record; an application using the common interface would have no such record. #### Two Providers and the Intersection Between Them The block below builds the engine, the two providers, the field inventory and the common interface. The inventory is not a claim in prose but a list kept inside the code; the carried and lost fields are counted with set operations. ```python # SIMULATOR -- FICTIONAL. Both providers are fictional and are built at simulator # level. No real model endpoint is called. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW = 240 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per piece b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] REQUESTS = [] for i in range(120): # 120 FICTIONAL support requests r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] REQUESTS.append({"id": 4001 + i, "topic": k, "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]}) CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] REPLIES = [ # the third pattern carries no label "talebiniz {} konusunda kaydedildi ayrica ekibimiz sizinle iletisime gececek" " ve sonuc size bildirilecektir", "{} kaydiniz acilmistir ayrica islem numaraniz kisa mesajla tarafiniza" " gonderilecek ve surec takip edilebilecektir", "kaydiniz alinmistir ayrica ilgili birim inceleyip size donus yapacak" " ve gerekirse ek belge istenecektir"] CALL_COUNT, ACTUAL = [0], [] # ACTUAL: the simulator's own record def engine(prompt, max_length, temperature, stop, seed): """The single FICTIONAL engine that runs underneath both providers.""" CALL_COUNT[0] += 1 toks = tokenize(prompt) empty = {"text": "", "state": "error", "input": len(toks), "output": 0} if len(toks) > WINDOW or CALL_COUNT[0] % 37 == 0: name = "context_overflow" if len(toks) > WINDOW else "rate_limit" ACTUAL.append(name) return dict(empty, finish=name) r = rng(seed ^ hash_text(prompt)) candidates = [k for k in CONCEPTS if k[:5] in toks] if len(candidates) < 3: candidates = list(CONCEPTS) p = {k: 0.15 for k in candidates} for t in toks: if CLUES.get(t) in p: p[CLUES[t]] += 1.0 if temperature <= 0.0: choice = max(sorted(p), key=lambda a: p[a]) else: names = sorted(p) w = [math.exp(p[a] / temperature) for a in names] choice = names[pick(r(), [q / sum(w) for q in w])] pattern = REPLIES[pick(r(), [0.40, 0.36, 0.24])] reply, finish = tokenize(pattern.format(choice)), "natural" for i, t in enumerate(reply): if t in stop: # stop sequence: generation is cut here reply, finish = reply[:i], "stopped" break if len(reply) > max_length: reply, finish = reply[:max_length], "cut" ACTUAL.append(finish) full_text = pattern.format(choice) return {"text": full_text if finish == "natural" else " ".join(reply), "state": "done", "input": len(toks), "output": len(reply), "finish": finish} def provider_a(prompt, at_most=14, distribution=0.6, stop=(), mode="free", seed=SEED): """FICTIONAL provider A. The field names and the response format are its own.""" s = engine(prompt, at_most, distribution, stop, seed) return {"text": s["text"], "state": s["state"], "input_tokens": s["input"], "output_tokens": s["output"], "finish": s["finish"], "truncated": s["finish"] == "cut"} def provider_b(message, limit=14, variety=2, schema=None, key=SEED): """FICTIONAL provider B. variety is an integer between 0 and 4 and fits its own scale.""" s = engine(message, limit, variety / 4 * 1.6, (), key) return {"result": s["text"], "status": s["state"], "reason": s["finish"], "usage": {"i": s["input"], "o": s["output"]}} FIELDS = {"A": ["prompt", "at_most", "distribution", "stop", "mode", "seed", "text", "state", "input_tokens", "output_tokens", "finish", "truncated"], "B": ["message", "limit", "variety", "schema", "key", "result", "status", "usage.i", "usage.o", "reason"]} MAPPING = {"prompt": ("prompt", "message"), "max_length": ("at_most", "limit"), "temperature": ("distribution", "variety"), "seed": ("seed", "key"), "response": ("text", "result"), "status": ("state", "status"), "input": ("input_tokens", "usage.i"), "output": ("output_tokens", "usage.o")} def common(provider, prompt, max_length=14, temperature=0.6, seed=SEED): """Common interface: only the eight fields in MAPPING are carried.""" if provider == "A": y = provider_a(prompt, at_most=max_length, distribution=temperature, seed=seed) return {"response": y["text"], "status": y["state"], "input": y["input_tokens"], "output": y["output_tokens"]} y = provider_b(prompt, limit=max_length, variety=round(temperature * 4), key=seed) return {"response": y["result"], "status": y["status"], "input": y["usage"]["i"], "output": y["usage"]["o"]} PREFIX5 = {k[:5]: k for k in TOPICS} # in a cut response the label may be split def read_label(y): return next((PREFIX5[s[:5]] for s in y.split() if s[:5] in PREFIX5), None) DIRECTIVE = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife ," " kesinti . Talep: ") print("fields of the common interface :", " , ".join(MAPPING)) for i, name in enumerate(FIELDS): carried = [e[i] for e in MAPPING.values()] lost = [a for a in FIELDS[name] if a not in carried] print(f"provider {name}: {len(FIELDS[name])} fields , carried {len(carried)} ," f" lost {len(lost)} -> {' , '.join(lost)}") prompt = DIRECTIVE + REQUESTS[2]["text"] print("\nsame prompt , four paths -- fields other than the text:") for name, y in (("A native + stop", provider_a(prompt, stop=("ayric",))), ("B native", provider_b(prompt)), ("common -> A", common("A", prompt)), ("common -> B", common("B", prompt))): fields = " , ".join(f"{k}={v}" for k, v in y.items() if k not in ("text", "result", "response")) print(f" {name:<15} {fields}") ``` ``` fields of the common interface : prompt , max_length , temperature , seed , response , status , input , output provider A: 12 fields , carried 8 , lost 4 -> stop , mode , finish , truncated provider B: 10 fields , carried 8 , lost 2 -> schema , reason same prompt , four paths -- fields other than the text: A native + stop state=done , input_tokens=33 , output_tokens=5 , finish=stopped , truncated=False B native status=done , reason=cut , usage={'i': 33, 'o': 14} common -> A status=done , input=33 , output=14 common -> B status=done , input=33 , output=14 ``` The inventory lines give the arithmetic of the abstraction. Provider A offers twelve fields, eight are carried, four are lost. Provider B offers ten fields, eight are carried, two are lost. The reason for the loss is not that the common interface is badly designed: `stop` and `mode` exist only in A, `schema` only in B. **The intersection can carry only what has a counterpart on both sides**, and the more fields a provider offers, the more of its fields it loses in the common interface. The four response formats stand side by side on the same prompt. When A is called with its own interface, the stop sequence does its job: the output is five tokens and the `finish` field says `stopped`. When B is called with its own interface, the output is fourteen tokens and the `reason` field says `cut`, that is, the response was cut. The two rows looking from the common interface are indistinguishable: both say `status: done` and `output: 14`. Whether the response was completed or not **is not written in these two rows**. #### The Cost of the Lost Fields A hundred and twenty fictional requests are passed through five paths. The first two are A's own interface (without and with the stop sequence), the third is B's own interface, the last two are the common interface. Each row prints the input and output tokens, the label accuracy, the second run accuracy, the **actual** number of cuts the simulator recorded and the number of cuts **readable** from that path. ```python PATHS = ("A native", "A native + stop", "B native", "common -> A", "common -> B") def run(path, temperature=0.6, seed=SEED): CALL_COUNT[0], ACTUAL[:] = 0, [] tok_in = tok_out = hits = readable = 0 for x in REQUESTS: prompt = DIRECTIVE + x["text"] if path.startswith("A native"): y = provider_a(prompt, distribution=temperature, seed=seed, stop=("ayric",) if "stop" in path else ()) m, t_in, t_out = y["text"], y["input_tokens"], y["output_tokens"] readable += y["truncated"] # the finish field is readable elif path == "B native": y = provider_b(prompt, variety=2, key=seed) m, t_in, t_out = y["result"], y["usage"]["i"], y["usage"]["o"] readable += y["reason"] == "cut" # the reason field is readable else: y = common(path[-1], prompt, temperature=temperature, seed=seed) m, t_in, t_out = y["response"], y["input"], y["output"] # no finish reason tok_in, tok_out = tok_in + t_in, tok_out + t_out hits += read_label(m) == x["topic"] return tok_in, tok_out, hits / 120, ACTUAL.count("cut"), readable print("path input output accuracy second run actual cuts read") for path in PATHS: a, b = run(path), run(path, seed=SEED + 1) print(f"{path:<15} {a[0]:>6} {a[1]:>6} {a[2]:>8.4f} {b[2]:>13.4f}" f" {a[3]:>15} {a[4]:>7}") print("\nsame common call , two providers:") print("temperature variety sent to B common -> A common -> B difference") for s in (0.0, 0.2, 0.4, 0.6, 0.8, 1.0): a, b = run("common -> A", s)[2], run("common -> B", s)[2] print(f"{s:>11.1f} {round(s * 4):>17} {a:>11.4f} {b:>11.4f} {b - a:>+10.4f}") ``` ``` path input output accuracy second run actual cuts read A native 4054 1638 0.5500 0.5000 117 117 A native + stop 4054 773 0.5500 0.5000 0 0 B native 4054 1638 0.5000 0.3917 117 117 common -> A 4054 1638 0.5500 0.5000 117 0 common -> B 4054 1638 0.5000 0.3917 117 0 same common call , two providers: temperature variety sent to B common -> A common -> B difference 0.0 0 0.8417 0.8417 +0.0000 0.2 1 0.7583 0.7583 +0.0000 0.4 2 0.7583 0.5000 -0.2583 0.6 2 0.5500 0.5000 -0.0500 0.8 3 0.5000 0.3167 -0.1833 1.0 4 0.4500 0.3167 -0.1333 ``` The second row is the cost of the lost `stop` field. When the stop sequence is used the output tokens are **773**, when it is not, **1638**: more than double. The accuracy is 0.5500 in both rows, that is, none of these tokens is useful; the polite closing sentence that comes after the body of the pattern is paid for and thrown away. The common interface **cannot offer** this field, because it has no counterpart in B. The cost of the intersection is written here directly in tokens. The last two columns are the cost of the lost `finish` and `reason` fields, and this is the harshest row of the lesson. Of the hundred and twenty calls, **117** are cut at the length limit. A reports this with the `truncated` field in its own interface, B with the `reason` field in its own interface: both providers are honest. The number of cuts readable from the common interface is **0**. The cutting did not disappear, it **became uncountable**. The rule of the course fits exactly here: a call whose failure is not written down counts as unmeasured. An application using the common interface writes a hundred and seventeen half-finished responses to its record system and not a single counter of it moves. The only thing that can take the place of the lost field is an inference: if `output` equals the limit, the response is counted as cut. This inference also counts as cut the responses that end naturally by landing exactly on the limit, that is, a new class of error takes the place of a measurement field. #### The Same Knob, Two Different Meanings The number of fields is only one face of the abstraction. The assumption that the eight carried fields **express the same thing** is the second and more insidious face. The `temperature` field of the common interface goes to A as it is; on its way to B it is rounded to an integer between zero and four, because that is B's scale. The lower table shows two separate consequences of this. The first: the same setting means two different behaviors. At temperature 0.4, A gives 0.7583 and B gives 0.5000; the difference is **0.2583**. The two numbers in the same row were produced with the same argument of the same function. The second: two different settings mean the same behavior. Temperatures 0.4 and 0.6 are rounded to the same `variety` value on B and both give **0.5000**, while on A the two settings give 0.7583 and 0.5500, a separation of **0.2083**. The continuous-looking knob of the common interface has five positions on B, and the positions in between are silently swallowed. Which of these readings counts as measured comes from the second run column. The difference between the two runs is 0.0500 for A and 0.1083 for B. The provider difference of 0.0500 in the temperature 0.6 row is exactly the run difference and **counts as unmeasured**; at that value no choice can be made between the two providers. The 0.2583 in the temperature 0.4 row, on the other hand, is five times the run difference and is real. When an application using the common interface changes providers, it sees this difference nowhere: the field names stay the same, the call stays the same, only the number changes. #### Summary - The common interface is the intersection of two providers: of A's twelve fields eight are carried and four drop, of B's ten fields eight are carried and two drop. - A field that exists on a single provider cannot enter the intersection; `stop` and `mode` stay in A, `schema` in B. - The loss of the stop sequence is paid for in tokens: the output rises from 773 to 1638 while the accuracy stays fixed at 0.5500, that is, the added tokens are of no use. - Of the hundred and twenty calls, 117 are cut at the length limit; both providers report this in their own field, and the number of cuts readable from the common interface is 0. - A carried field may not mean the same thing either: at temperature 0.4 the two providers separate by 0.2583, and 0.4 and 0.6 collapse to a single value because they are rounded to the same setting on B. - Counting a provider difference as measured depends on its exceeding the run difference: the 0.0500 in the 0.6 row is the run difference itself, the 0.2583 in the 0.4 row is five times it. #### Next Step Up to this lesson every measure was written under the assumption of **one call per request**: a hundred and twenty requests, a hundred and twenty calls. Yet the directive is sent again on every call, and this is why the input tokens stay fixed at 4054 in the table. If the requests had been accumulated and sent in a single call, this repetition would not have been paid for. The next lesson measures the batch call and ties it to a single question: as the batch size grows, the number of calls and tokens fall, but what happens to **the number of requests affected by a single failure**. ### Batch and Asynchronous Calls Source: https://academia.sh/en/courses/model-apis/batch-and-asynchronous-calls Processing a hundred and twenty fictional requests at seven different batch sizes, and batch size moving three separate numbers in opposite directions: input tokens fall from 5854 to 1784, calls from 120 to 10, and rounds at four-way concurrency from 30 to 3. The load other than the repeated directive is exactly 1414 tokens in all seven rows; the whole saving comes from the repeated directive. In return, the number of requests affected by a single failure equals the batch size and rises from 1 to 12; at size 18 the prompt overflows the context window, six of seven calls drop and accuracy collapses to 0.0667. The movement of accuracy with batch size does not exceed two run differences, that is, it counts as unmeasured. If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333. Up to the previous lesson every measure was written under the assumption of **one call per request**: a hundred and twenty requests, a hundred and twenty calls. This assumption has an invisible cost. The same directive is placed at the head of every call and is paid for again each time. When a nightly batch job has thousands of records instead of a hundred and twenty, this repetition alone makes up most of the bill. A batch call removes it: the directive is written once, numbered requests are lined up behind it and the response is read in one go. This lesson treats batch size as a setting and prints three numbers side by side: **calls and tokens**, **waiting**, and **the number of requests affected by a single failure**. The first two fall as batch size grows, the third grows. - **AU63.** The code is a **simulator**; no real endpoint is called. The requests and all the failure rules are **fictional**, seed **20260218**, second run **20260219**. - **AU64. Failure is at call level.** Whatever the reason a call is rejected, **all** the requests inside it are rejected. This is the definition of a batch interface and it is what this lesson actually measures. - **AU65.** There are three sources of rejection: the **context window** (240 tokens), the **rate limit** (one call in thirty-seven) and **malformed output**. The last is a share of one and a half percent **per request**; if one request spoils it, the whole response drops. - **AU66.** That the malformed-output share is **per request** is the key assumption of this lesson. The underlying defect rate is independent of batch size; the only thing that changes is **how many requests one defect takes with it**. - **AU67. Waiting is written in rounds.** Four calls are sent concurrently; the number of rounds is the number of calls divided by four, rounded up. **Raw time is not written**; the round count does not change from run to run. - **AU68.** By default the output limit grows with the batch size (three tokens per request and a margin of two shares). What happens when it is not enlarged is measured separately. - **AU69.** The response is split into rows by a **decoder**: a number, then a label. If a row cannot be read, that request counts as **unanswered**, even if the call succeeded. - **AU70.** The procedure for queues, backoff and asynchronous job management was measured in the Caching, Queues and Asynchronous Processing course of the Backend Development curriculum; it is **not repeated**. This lesson adds only what **batch size** does to the three numbers. #### How Many Requests Fit in One Call The prompt of a batch call is a single piece: the directive once, then the numbered requests. The response is also a single piece, and to be read it has to be split into rows. The block below builds these two ends and shows what happens at three batch sizes. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. The requests and # all the failure rules are fictional. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, CONCURRENCY = 240, 4 # context window , number of concurrent calls def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per piece b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] REQUESTS = [] for i in range(120): # 120 FICTIONAL support requests r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] REQUESTS.append({"id": 4001 + i, "topic": k, "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]}) CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} PREFIX = ("Her talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife ," " kesinti . Her satir icin once numara sonra tek etiket yaz . Talepler: ") PREFIX5 = {k[:5]: k for k in TOPICS} CALL_COUNT = [0] def batch_call(batch, seed=SEED, max_length=None, temperature=0.6): """Sends k requests in a single call. Failure is at call level: whichever rejection occurs, the whole batch is rejected.""" CALL_COUNT[0] += 1 k = len(batch) prompt = PREFIX + " ".join(f"{i + 1} {x['text']} ;" for i, x in enumerate(batch)) toks = tokenize(prompt) max_length = 3 * k + 2 if max_length is None else max_length empty = {"input": len(toks), "output": 0, "response": []} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ hash_text(prompt)) for _ in batch: # each request carries its own malformed-output share if r() < 0.015: # if one request spoils it, the whole response drops return dict(empty, status="malformed") cb = [] for i, x in enumerate(batch): p = {c: 0.15 for c in TOPICS} for t in tokenize(x["text"]): if CLUES.get(t) in p: p[CLUES[t]] += 1.0 names = sorted(p) w = [math.exp(p[a] / temperature) for a in names] cb += [str(i + 1)] + tokenize(names[pick(r(), [q / sum(w) for q in w])]) cut = len(cb) > max_length return {"status": "done", "input": len(toks), "output": min(len(cb), max_length), "response": cb[:max_length], "cut": cut} def decode(cb, k): """Reads one label per row from the batch response. A row that cannot be read stays None.""" out, i = [None] * k, 0 while i < len(cb): if cb[i].isdigit(): n, j = int(cb[i]), i + 1 piece = [] while j < len(cb) and not cb[j].isdigit(): piece.append(cb[j]) j += 1 e = "".join(piece) if 1 <= n <= k and e[:5] in PREFIX5: out[n - 1] = PREFIX5[e[:5]] i = j else: i += 1 return out for k in (1, 6, 18): s = batch_call(REQUESTS[:k]) print(f"batch size {k:>2} : input {s['input']:>3} , output {s['output']:>2} ," f" status {s['status']}") s = batch_call(REQUESTS[:6]) print("\nresponse of a single six-request call:", " ".join(s["response"])) print("decoded:", decode(s["response"], 6)) print("actual :", [x["topic"] for x in REQUESTS[:6]]) d = batch_call(REQUESTS[:6], max_length=8) # if the output limit is not enlarged with the batch size print(f"\nsame call with max_length 8: output {d['output']} , cut {d['cut']} ," f" decoded {sum(e is not None for e in decode(d['response'], 6))} / 6") ``` ``` batch size 1 : input 50 , output 3 , status done batch size 6 : input 107 , output 16 , status done batch size 18 : input 247 , output 0 , status context_overflow response of a single six-request call: 1 kesin ti 2 kesin ti 3 kesin ti 4 ariza 5 sayac 6 fatur a decoded: ['kesinti', 'kesinti', 'kesinti', 'ariza', 'sayac', 'fatura'] actual : ['kesinti', 'fatura', 'ariza', 'ariza', 'sayac', 'tarife'] same call with max_length 8: output 8 , cut True , decoded 3 / 6 ``` The first three lines show the ceiling of batch size. The directive is thirty-seven tokens; a call with a single request comes to fifty tokens, that is, three quarters of the input is directive. In the six-request call the input rises to 107, but the share per request falls to eighteen. At eighteen requests the prompt reaches 247 tokens and exceeds the 240-token window: the call is rejected without ever being made. **The upper limit of batch size is set by the context window, not by the rate limit.** The decoder shows the second face of a batch response. The response is a single sequence of tokens, and the rows can be separated only by their numbers. All six rows are read, three correctly. The last block shows what happens when the output limit is not enlarged with the batch size: a limit of eight tokens cuts the response in the middle, the call **returns successfully** and only three of the six requests are decoded. A cut does not show in the call status; it becomes visible only if the rows the decoder leaves empty are counted. #### The Three Numbers of Batch Size A hundred and twenty requests are processed from start to finish at seven different batch sizes. Each row prints the number of calls, the input and output tokens, the failed calls, the **number of requests affected** by those failures, the delivered accuracy, the second run accuracy and the number of rounds at four-way concurrency. ```python def run(k, seed=SEED, max_length=None): CALL_COUNT[0] = 0 calls = t_in = t_out = failed = affected = correct = unanswered = 0 for i in range(0, 120, k): batch = REQUESTS[i:i + k] s = batch_call(batch, seed, max_length) calls, t_in, t_out = calls + 1, t_in + s["input"], t_out + s["output"] if s["status"] != "done": failed, affected = failed + 1, affected + len(batch) continue for x, e in zip(batch, decode(s["response"], len(batch))): correct, unanswered = correct + (e == x["topic"]), unanswered + (e is None) return calls, t_in, t_out, failed, affected, correct / 120, -(-calls // CONCURRENCY), unanswered print(" size calls input output failed affected accuracy second run rounds") for k in (1, 2, 3, 6, 12, 18, 24): a, b = run(k), run(k, SEED + 1) print(f"{k:>5} {a[0]:>6} {a[1]:>6} {a[2]:>6} {a[3]:>10} {a[4]:>10}" f" {a[5]:>8.4f} {b[5]:>13.4f} {a[6]:>6}") for k in (6, 12): a = run(k, max_length=24) print(f"output limit fixed at 24 , size {k:>2} : unanswered requests {a[7]:>2} ," f" accuracy {a[5]:.4f}") ``` ``` size calls input output failed affected accuracy second run rounds 1 120 5854 300 3 3 0.8250 0.7083 30 2 60 3634 300 3 6 0.7667 0.7250 15 3 40 2894 291 3 9 0.7167 0.6667 10 6 20 2154 299 1 6 0.7833 0.7583 5 12 10 1784 284 1 12 0.7083 0.6833 3 18 7 1673 30 6 108 0.0667 0.0750 2 24 5 1599 0 5 120 0.0000 0.0000 2 output limit fixed at 24 , size 6 : unanswered requests 0 , accuracy 0.7833 output limit fixed at 24 , size 12 : unanswered requests 26 , accuracy 0.5333 ``` The whole drop in the input column comes from a single item, and it can be separated out exactly. The directive is thirty-seven tokens; it is paid for 120 times when sending one by one and 10 times when sending in batches of twelve, that is, it falls from 4440 tokens to 370. The remaining load is **exactly 1414 tokens in all seven rows**: the requests themselves, their numbers and their separators. A batch call removes not the load of the request but **the repetition of the directive**. The output column confirms this reading: three tokens per request and about three hundred in total, independent of batch size. A batch call does not save on the output. The rounds column is the measure of waiting. When a hundred and twenty separate calls are sent four at a time it takes thirty rounds; with batches of twelve, three rounds. The latency class moves **from the single call to the batch call** and the difference between them is a factor of ten. Raw time is not written, but the round count does not change from run to run and is enough for the comparison. #### The Input That Wins, the Radius That Grows The affected requests column is the real finding of this lesson and it reads in a single line: **the number of affected requests is the number of failed calls times the batch size.** When sending one by one, three failures take three requests. In batches of six a single failure takes six requests, in batches of twelve it takes twelve. The underlying defect rate did not change; every request carries the same share of one and a half percent. The only thing that changes is **the radius of one defect**. That the number of failed calls falls as batch size grows can hide this reading. It goes from three to one and on its own sounds like good news. Yet part of this fall is only because **the number of calls has decreased**: the rate limit kicks in once in thirty-seven calls and never kicks in for a setup that makes ten calls. Doing the same work with fewer calls does genuinely dilute the rate limit, but it does not dilute the malformed output; its share is per request, and the number of requests is fixed. The rows for eighteen and twenty-four are beyond a threshold. At eighteen the prompt does not fit the window; six of the seven calls drop without ever being made and 108 requests go unanswered. The one call left standing is the last twelve requests, and accuracy falls to 0.0667. At twenty-four no call is made at all, and accuracy is zero. **The added mechanism making the work worse** is in this table not a curve but a cliff: raising the batch size from twelve to eighteen gains only 111 tokens of input and in return takes the whole job away. Reading the accuracy column calls for a yardstick. In the first five rows accuracy wanders between 0.7083 and 0.8250; the difference between them is 0.1167. The second run column has, in the same rows, a largest difference of 0.1167 (0.8250 and 0.7083 when sending one by one). **The effect of batch size on accuracy does not exceed the run-to-run movement**, that is, it counts as unmeasured. This is not surprising: the label decision is made per request and sending in a batch does not change that decision. Batch size is not a quality setting, it is a **cost and risk** setting. Defending it on quality grounds, or blaming it on quality grounds, goes against the table. The last two rows measure the case where the output limit is forgotten. When the limit is held fixed at twenty-four, batches of six are not affected, because six rows fit into twenty tokens. In batches of twelve, however, the tail of the response is cut and **26 requests go unanswered**; accuracy falls from 0.7083 to 0.5333. This loss does not show in the failed calls column: the calls returned successfully. Enlarging the batch size makes it mandatory to enlarge the output limit with it. #### Summary - A batch call removes the repetition of the directive: input tokens fall from 5854 to 1784, because the 37-token directive is paid for 10 times instead of 120. The remaining load is exactly 1414 tokens in all seven rows and the output tokens are independent of batch size. - Waiting is written in rounds: at four-way concurrency it falls from 30 rounds to 3. - The number of requests affected by a single failure equals the batch size; the same defect rate goes from taking 3 requests to taking 12. - The context window sets the upper limit of batch size: at size 18 the prompt rises to 247 tokens, six of seven calls drop without ever being made and accuracy falls to 0.0667, at size 24 to 0.0000. - The 0.1167 wobble of accuracy in the first five rows stays within two run differences: batch size is not a quality setting but a cost and risk setting. - If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333; this loss never shows in the failed calls column. #### Next Step In eight lessons, what the message-based interface offers was measured one by one: the cost of conversation state, what response streaming changes, the validity rate of tool calling, interface-level schema enforcement, the token bill of the input format, the trade-off of embedding dimension, the fields lost by the common interface and the three numbers of batch size. Under all eight lay the same assumption: **the call succeeds.** In this lesson it was counted that the assumption does not hold, but no answer was given. Rejected calls were not retried, the batch that exceeded the window was not split, the twenty-six unanswered requests were not rescued, the repeated directive was not cached. They were counted and left there. The next topic starts exactly here and in its first lesson builds the second claim of this course: retrying a request can rescue it, but it also multiplies the tokens and strains the rate limit further. Adding resilience is not free, and sometimes it spoils the very number it set out to fix. ## Production Quality Measuring the cost and the failure points of the resilience scaffolding built against calls that are slow, rejected or cut off midway: fixed-interval retry compared with backoff retry under load, and retries multiplying tokens as well; the stale-response share that grows as the cache hit rate rises; the latency-class and accuracy trade-off of the small context, the large context and the two-stage setup; the mean of tokens per request against the share of the most expensive one percent in the total; the accuracy of requests that fall to the fallback model dropping silently and staying invisible when it is not measured; counting the items that stop the same input from giving different outputs. ### Rate Limits and Retries Source: https://academia.sh/en/courses/model-apis/rate-limits-and-retries Running the same 160 fictional tickets under three policies at two load levels: the bare client completes 0.8562 at both loads and never hits the rate limit. At light load retry raises completion to 1.0000, at a cost of 25 extra calls and 1209 tokens. At heavy load the same scaffolding does damage: fixed-interval retry falls to 0.8000 and produces 151 rate limit rejections, while backoff retry stays 0.0188 above the bare client at 0.8750, but the run-to-run difference is also 0.0188. The interactive flow that shares the same limit drops from 0.9500 to 0.8667: retry rescues its own work while breaking someone else's. The previous eight lessons measured what the API offers: carrying conversation state, streaming, tool calls, the structured output mode, multimodal input, the embedding API, the compatible API layer and the batch call. All of them shared one silent assumption: **the call succeeded.** Every token counted was a token that was paid for and received in return. In a production pipeline this assumption does not hold. The call is rejected, cut off midway, or never answers. The remedy for these is a known procedure: retry. Retry was measured in the Observability and Reliability course and in the Resilience and Reliability course; its procedure is not repeated here. The only thing added in this lesson comes from the call being **expensive and uncertain**: every attempt pays for the input once more, every attempt loads one more call onto the same rate limit, and an output cut off midway has spent tokens without being of any use. That is the problem itself — retry feeds the source of what it fixes. - **PQ1.** The code is a **simulator**; no real endpoint is called. The tickets, the templates and the network behavior are **fictional**, the seed is **20260218**. - **PQ2.** Time is counted in **steps**; raw time is not written. One step is one sending round of the client. - **PQ3. The rate limit is read over a sliding window, and rejected calls are counted too:** if the total number of calls made in the last three steps exceeds twenty-four, the call is rejected. On an endpoint that does not count rejections the difference measured below shrinks; this behavior is an assumption and it determines the result. - **PQ4.** A rejected call **spends no tokens**. Every accepted call pays its input in full. - **PQ5.** There are two content failures: the **timeout** (cut off before the output starts, the input is paid) and the **partial output** (the input plus one output token is paid, the response cannot be read). - **PQ6.** The client sends at most twelve calls in a step; a request is tried at most four times. - **PQ7. Two workloads share the same limit:** batch classification of 160 records, and an **interactive** status query that arrives every other step. The interactive query's latency budget is one step. - **PQ8.** Load is the number of requests arriving per step during the burst: **three for light, six for heavy**. #### Three Failures of a Call and Their Cost The simulator carries one more thing in this topic: **network behavior**. The same prompt can be rejected depending on how full the window is; even when it is accepted, the response can be cut off midway. The three classes need to be counted separately, because **their costs differ**. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the network behavior is also # written in the lesson. The tickets and templates are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF QUOTA, WINDOW = 24, 3 # calls counted in the sliding window / window steps CONCURRENCY, HORIZON, MAX_TRIES = 12, 140, 4 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per token b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] EXTRAS = ["ayrica gecen ay tarife degisikligi de yapilmisti", "kayit numarasini ekliyorum ekip hala gelmedi", "odeme gorunmuyor ama dekont elimde duruyor", "duyurulari takip ediyorum yine de bilgi alamadim"] TICKETS = [] for i in range(160): # 160 FICTIONAL support tickets r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = [x for x in TEMPLATES if x[1] == k][int(r() * 3)][0] TICKETS.append({"no": 4001 + i, "topic": k, "text": m + (" " + EXTRAS[int(r() * 4)] if r() < 0.34 else "")}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "dekon": "odeme", "sikay": "sikayet"} INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . ") SERVER = {"history": []} def prompt_of(t): return INSTRUCTION + f"Talep {t['no']} : {t['text']}" def label_of(tok): # most likely choice; what narrows the candidates is the label set candidates = [k for k in CONCEPTS if k[:5] in tok] p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)} for t in tok: if CUES.get(t) in p: p[CUES[t]] += 1.0 return max(sorted(p), key=lambda a: p[a]) def call(prompt, step, attempt=0, seed=SEED): """FICTIONAL endpoint. The rate limit counts in a sliding window, REJECTED CALLS INCLUDED.""" g = SERVER["history"] while len(g) <= step: g.append(0) g[step] += 1 if sum(g[max(0, step - WINDOW + 1):step + 1]) > QUOTA: return {"status": "rate_limit", "input": 0, "output": 0, "reply": ""} tok = tokenize(prompt) r = rng((seed ^ trace_id(prompt)) + 7919 * attempt) if r() < 0.03 + 0.0006 * len(tok): # cut off before the output started return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""} k = label_of(tok) if r() < 0.09: # output cut off midway return {"status": "partial_output", "input": len(tok), "output": 1, "reply": k[:2]} return {"status": "ok", "input": len(tok), "output": len(tokenize(k)), "reply": k} SERVER["history"] = [] print("single call :", call(prompt_of(TICKETS[0]), 0)) SERVER["history"] = [QUOTA] # window already full print("full window :", call(prompt_of(TICKETS[1]), 0)) SERVER["history"] = [] classes = {} for t in TICKETS: SERVER["history"] = [] y = call(prompt_of(t), 0) a, b = classes.get(y["status"], (0, 0)) classes[y["status"]] = (a + 1, b + y["input"] + y["output"]) print(f"\n{'class':<14} {'count':>5} {'tokens':>8} outcome") for d in ("ok", "timeout", "partial_output"): a, b = classes[d] print(f"{d:<14} {a:>5} {b:>8} {'usable' if d == 'ok' else 'discarded'}") ``` ``` single call : {'status': 'ok', 'input': 44, 'output': 2, 'reply': 'kesinti'} full window : {'status': 'rate_limit', 'input': 0, 'output': 0, 'reply': ''} class count tokens outcome ok 137 6523 usable timeout 9 436 discarded partial_output 14 655 discarded ``` In the last table the window was emptied before every call, that is, the rate limit is switched off and only the content failures remain. Twenty-three requests are lost, and **1091 tokens of these requests were paid for and nothing was received in return**. Given that the total is 7614 tokens, the share wasted is **0.1433**. This is the price paid by a single-attempt client, without any retry being made. The two failure classes do not cost the same. The timeout burns 436 tokens over nine requests; the partial output 655 over fourteen. The rejected call, on the other hand, spends zero tokens — in payment terms it is the cheapest failure. This ordering says why a retry policy cannot be set up blindly: sending a rejected call again costs only one call, while sending a cut-off call again makes you pay the whole input a second time. #### Three Policies Under Load Now the same 160 tickets are run along a time axis. During the first hundred steps one request arrives per step; at the thirtieth step a **burst** begins and sixty requests pile up either three at a time or six at a time. The interactive status query coming from the field application every other step also uses the same rate limit. Three policies are compared: no retry, **fixed-interval** (one step after every failure) and **backoff** (after one, two, four steps). ```python def run(policy, load, seed=SEED): """policy: none | fixed | backoff . load: requests per step during the burst.""" SERVER["history"] = [] arrivals = [[] for _ in range(HORIZON)] for i in range(len(TICKETS)): arrivals[i if i < 100 else 30 + (i - 100) // load].append(i) queue, active = [], [] s = dict.fromkeys(("calls", "rate", "content", "tok", "wasted", "done", "correct", "idone", "itotal"), 0) for step in range(HORIZON): if step % 2 == 0 and step < 120: # interactive flow active.append({"step": step, "attempt": 0}) s["itotal"] += 1 for i in arrivals[step]: queue.append({"i": i, "ready": step, "attempt": 0}) for x in list(active): if x["step"] + 1 < step: # latency budget used up active.remove(x) continue y = call(f"Saha uygulamasi sorgusu {x['step']} : acik kayit sayisi", step, x["attempt"], seed) s["tok"] += y["input"] + y["output"] if y["status"] == "ok": s["idone"] += 1 active.remove(x) else: s["wasted"] += y["input"] + y["output"] x["attempt"] += 1 for x in [x for x in queue if x["ready"] <= step][:CONCURRENCY]: queue.remove(x) t = TICKETS[x["i"]] y = call(prompt_of(t), step, x["attempt"], seed) s["calls"] += 1 s["tok"] += y["input"] + y["output"] if y["status"] == "ok": s["done"] += 1 s["correct"] += y["reply"] == t["topic"] continue s["wasted"] += y["input"] + y["output"] s["rate" if y["status"] == "rate_limit" else "content"] += 1 x["attempt"] += 1 if policy == "none" or x["attempt"] >= MAX_TRIES: continue x["ready"] = step + (1 if policy == "fixed" else 2 ** (x["attempt"] - 1)) queue.append(x) n = len(TICKETS) return (s["done"] / n, s["idone"] / s["itotal"], s["calls"], s["rate"], s["content"], s["tok"], s["wasted"], s["correct"] / n) print("load policy completed interactive calls rate limit content error" " tokens wasted correct") for load, name in ((3, "light"), (6, "heavy")): for p in ("none", "fixed", "backoff"): a = run(p, load) print(f"{name} {p:<8} {a[0]:>10.4f} {a[1]:>11.4f} {a[2]:>5} {a[3]:>10}" f" {a[4]:>13} {a[5]:>8} {a[6]:>10} {a[7]:>7.4f}") print("\nsecond run (seed 20260219), completed / interactive:") for load, name in ((3, "light"), (6, "heavy")): b = [run(p, load, SEED + 1) for p in ("none", "fixed", "backoff")] print(f" {name} " + " ".join(f"{p}: {x[0]:.4f} / {x[1]:.4f}" for p, x in zip(("none", "fixed", "backoff"), b))) ``` ``` load policy completed interactive calls rate limit content error tokens wasted correct light none 0.8562 0.9500 160 0 23 8425 1218 0.8438 light fixed 1.0000 0.9500 185 0 25 9634 1311 0.9875 light backoff 1.0000 0.9500 185 0 25 9634 1311 0.9875 heavy none 0.8562 0.9500 160 0 23 8425 1218 0.8438 heavy fixed 0.8000 0.8667 295 151 16 7658 873 0.7937 heavy backoff 0.8750 0.8833 289 129 20 8389 1038 0.8625 second run (seed 20260219), completed / interactive: light none: 0.8750 / 1.0000 fixed: 1.0000 / 1.0000 backoff: 1.0000 / 1.0000 heavy none: 0.8750 / 1.0000 fixed: 0.6813 / 0.8667 backoff: 0.7188 / 0.8833 ``` The first row of the table to be read is that the bare client is **the same at both loads**: 0.8562 completion, **zero rate limit rejections**. The burst rising from three to six per step changes nothing for the bare client, because the incoming load still stays below the limit. The limit will be exceeded not by the workload but **by the scaffolding**. At light load retry works exactly as expected: completion rises from 0.8562 to **1.0000**, and the two policies are **identical to each other** — because when there is no rejection at all, backoff has nothing to delay. The cost is plain: 25 extra calls, 1209 extra tokens, and wasted tokens from 1218 to 1311. That is, for the twenty-three requests rescued, part of the tokens paid went to waste again, because some retries failed too. #### Whose Work Retry Breaks The heavy-load rows are the clearest example of the course's second claim. Fixed-interval retry **lowers completion from 0.8562 to 0.8000**. Scaffolding was added, the number went down. The reason sits in the columns: calls rise from 160 to 295, and **151 of these calls are rejected by the rate limit**. Because rejected calls are also counted in the window, every rejection makes the next rejection easier; the call sent to fix the problem enlarges the problem to be fixed. While content errors fall from 23 to 16, total completion falls too: requests are no longer lost because of their content but **because of the congestion created by their own attempts**. Backoff retry gives 0.8750 under the same load, that is, **0.0188** above the bare client. The second run must be set beside this number: the bare client gives 0.8750 on seed 20260219, and the run-to-run difference is exactly **0.0188**. The gain does not exceed the run-to-run difference, therefore it **counts as unmeasured**. By contrast, the loss of fixed-interval retry is 0.0562, three times the run-to-run difference — the loss is measured. The heavy-load rows of the second run confirm this: there both policies fall below the bare client (0.6813 and 0.7188). The last column is in the place that most measurements never look at. The interactive flow shares the same rate limit and has no retry of its own; its latency budget is one step. With the bare client this flow completes 0.9500. When retry is added to the batch work it drops to **0.8667**. The interactive flow changed nothing, its requests did not increase, its prompt did not grow; only **its neighbor became resilient**. This is the most insidious cost of resilience scaffolding: while rescuing its own measure it consumes a shared resource, and the harm shows up **in another measure**. If a retry policy is evaluated only by its own failure share, this row is never seen. #### Summary - The three failures of a call carry separate costs: a rate limit rejection spends no tokens, a timeout pays the input, a partial output pays the input plus the partial output; the share of tokens wasted in a single-attempt client is 0.1433. - The bare client gives the same result at light and heavy load (0.8562) and never hits the rate limit; the limit is exceeded not by the workload but by the added scaffolding. - At light load retry raises completion to 1.0000 and there is no difference between fixed-interval and backoff; the cost is 25 extra calls and 1209 tokens. - At heavy load fixed-interval retry lowers completion to 0.8000 and produces 151 rate limit rejections: the call sent to fix the problem enlarges the problem to be fixed. - The gain of backoff retry at heavy load is 0.0188, and the run-to-run difference is 0.0188; the gain counts as unmeasured, whereas the 0.0562 loss of fixed-interval retry is measured. - The interactive flow sharing the same limit falls from 0.9500 to 0.8667; the resilience scaffolding leaves the harm in another measure while rescuing its own. #### Next Step In this lesson every call was sent from scratch. Yet all 160 prompts that entered the table begin with the same instruction, and that instruction is tokenized again and paid for again every time. Retry enlarges this repetition by one more layer: at heavy load all 295 calls carry the same shared text. The repetition itself is a cost, and this cost can be lowered by storing the response or the shared context. But a stored response has an age: if the document it rests on has changed, the response is still fast, still cheap and now wrong. The next lesson puts two numbers side by side: how far tokens fall as the cache hit rate rises, and how far the **stale-response share** grows. ### Prompt Caching Source: https://academia.sh/en/courses/model-apis/prompt-caching Measuring the same 240 fictional requests under seven cache setups: 175 of the prompt's 186 tokens are shared by every request. The prefix cache, keyed by content, cuts input tokens from 44367 to 7472 with a 0.9792 hit rate and leaves the correct-response share unchanged. In the response cache, keyed by question, the hit rate rises from 0.4708 to 0.8792 as the lifetime grows and calls fall from 240 to 29, but the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542. Together the two caches cut input to 2946, fifteenfold, and the prefix hit rate falls from 0.9792 to 0.9359 once the response cache is on. In the previous lesson the heavy load made 295 calls, and every one of them began with the same instruction. Retry multiplied that repetition once more: every call sent as a correction tokenized the shared text once again and paid for it once again. The repetition itself had not been measured. In a support assistant this repetition has two layers. The first is the **prefix of the prompt**: the instruction and the field team's fictional document pool are the same in every request and take up most of the prompt. The second is **the question itself**: a few questions produce half of all traffic, and the same question is asked dozens of times a day. Each repetition has a remedy, and **the two remedies are not the same thing**. This lesson builds the two separately and measures them side by side: as the cache hit rate rises, tokens fall, but a stored response acquires an **age**, and when the document it rests on changes, the response is still fast, still cheap and now wrong. - **PQ9.** The code is a **simulator**; no real endpoint is called. The pool, the questions and the update schedule are **fictional**, the seed is **20260218**. - **PQ10.** The pool is **ten fictional documents**; each document has a **version**, and two documents are updated at each of four separate steps. - **PQ11.** The prompt has two parts: the **prefix**, identical in every request (the instruction and the pool), and the question itself. - **PQ12. The prefix cache is keyed by content.** If the prefix is identical to the previous call's, it counts as a hit and **ten percent** of the prefix tokens are paid. If the pool changes, the prefix changes too. - **PQ13. The response cache is keyed by question** and has a **lifetime**. A hit within the lifetime makes no call and spends no tokens. - **PQ14. A stale response** is a response that comes back from the cache and whose underlying document's version is no longer current. - **PQ15. A correct response** is a response whose label is correct **and** that is not stale; this is the product measure of this lesson. - **PQ16.** The workload is 240 requests over 120 steps, two per step; the question distribution is skewed. #### Two Separate Caches, Two Separate Keys First the pool and the questions are built, then one document is updated and the answer to the same question is checked. The distinction becomes visible here: the prefix changes **by itself** when the pool changes; a stored response does not. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called. The document pool, the questions # and the update schedule are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF STEPS, PREFIX_DISCOUNT = 120, 0.10 # run length, share of a hit prefix that is paid def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] DOCUMENTS = [{"code": "FT1", "topic": "fatura", "keyphrase": "fatura tutari", "text": "fatura tutari hesap donemi ve gecikme farki kurallari"}, {"code": "FT2", "topic": "fatura", "keyphrase": "fatura odemesi", "text": "fatura odemesi yontemleri ve borc gorunumu kurallari"}, {"code": "AR1", "topic": "ariza", "keyphrase": "ariza kaydi", "text": "ariza kaydi acma ve ekip yonlendirme yordami"}, {"code": "AR2", "topic": "ariza", "keyphrase": "baglanti bozuklugu", "text": "baglanti bozuk oldugunda uygulanan denetim adimlari"}, {"code": "SY1", "topic": "sayac", "keyphrase": "sayac endeksi", "text": "sayac endeks okumasi ve duzeltme yordami"}, {"code": "SY2", "topic": "sayac", "keyphrase": "sayac degisimi", "text": "sayac degisimi randevu kurallari ve sure taahhudu"}, {"code": "TA1", "topic": "tarife", "keyphrase": "tarife basamagi", "text": "tarife basamaklari ve abone tipine gore birim ucret"}, {"code": "TA2", "topic": "tarife", "keyphrase": "tarife degisikligi", "text": "tarife degisikligi basvurusu ve gecerlilik kosullari"}, {"code": "KS1", "topic": "kesinti", "keyphrase": "planli kesinti", "text": "planli kesinti duyurusu yayimlama ve takip kurallari"}, {"code": "KS2", "topic": "kesinti", "keyphrase": "mahalle kesintisi", "text": "mahallede kesinti bildirimi ve tahmini donus suresi"}] for b in DOCUMENTS: b["version"] = 1 QUESTIONS = [{"doc": i, "topic": b["topic"], "text": tpl.format(a=b["keyphrase"])} for i, b in enumerate(DOCUMENTS) for tpl in ("{a} konusunda ne yapmam gerekiyor", "{a} icin gecerli kural nedir", "{a} ile ilgili son durum ne")] WEIGHTS = [1.0 / (i + 1) ** 1.0 for i in range(len(QUESTIONS))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] UPDATES = {24: [6, 1], 48: [0, 8], 72: [2, 6], 96: [4, 7]} # step -> document CUES = {"fatur": "fatura", "tutar": "fatura", "borc": "fatura", "odeme": "fatura", "ariza": "ariza", "bozuk": "ariza", "bagla": "ariza", "kayit": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} INSTRUCTION = ("Asagidaki dagarcigi kullanarak soruyu yanitla . Yanitta once etiket , " "sonra dayandigin belge kodu ve surumu yaz . Dagarcik: ") COUNTER = [0] def prefix(): return INSTRUCTION + " ; ".join(f"{b['code']} s{b['version']} {b['text']}" for b in DOCUMENTS) + " . Soru: " def call(question, seed=SEED): """FICTIONAL endpoint. Reads the current version of the pool.""" COUNTER[0] += 1 prompt = prefix() + question["text"] tok = tokenize(prompt) p = {k: 0.15 for k in TOPICS} for t in tokenize(question["text"]): if CUES.get(t) in p: p[CUES[t]] += 1.0 r = rng((seed ^ trace_id(question["text"])) + 7919 * COUNTER[0]) e = max(sorted(p), key=lambda a: p[a]) if r() > 0.10 else TOPICS[int(r() * 5)] b = DOCUMENTS[question["doc"]] return {"label": e, "code": b["code"], "version": b["version"], "input": len(tok), "prefix": len(tokenize(prefix())), "output": len(tokenize(f"{e} {b['code']} s{b['version']}"))} def workload(seed=SEED): """Two requests per step; the question distribution is skewed (a few dominate).""" r = rng(seed + 991) return [(a, QUESTIONS[pick(r(), WEIGHTS)]) for a in range(STEPS) for _ in range(2)] for b in DOCUMENTS: b["version"] = 1 first = call(QUESTIONS[0]) print("first call :", {k: first[k] for k in ("label", "code", "version", "input", "prefix")}) DOCUMENTS[0]["version"] += 1 # pool updated second = call(QUESTIONS[0]) print("after the update:", {k: second[k] for k in ("code", "version", "input")}) print("stored response :", first["code"], "s" + str(first["version"]), "-> stale" if first["version"] != DOCUMENTS[0]["version"] else "-> fresh") DOCUMENTS[0]["version"] = 1 reqs = workload() freq = {} for _, q in reqs: freq[q["text"][:26]] = freq.get(q["text"][:26], 0) + 1 print(f"\nrequests {len(reqs)} , unique questions {len(freq)} , prefix {first['prefix']} tokens") print("top three questions:", [(a, n) for a, n in sorted(freq.items(), key=lambda x: -x[1])[:3]]) ``` ``` first call : {'label': 'fatura', 'code': 'FT1', 'version': 1, 'input': 186, 'prefix': 175} after the update: {'code': 'FT1', 'version': 2, 'input': 186} stored response : FT1 s1 -> stale requests 240 , unique questions 29 , prefix 175 tokens top three questions: [('fatura tutari konusunda ne', 56), ('fatura tutari icin gecerli', 41), ('fatura tutari ile ilgili s', 23)] ``` The first line gives the size of the repetition: the prompt is 186 tokens, and **175 of them are prefix**. That is, **0.9409** of the tokens each request pays for are identical to the previous request's. The second line shows that the call after the update reads the same document in its second version. The third line is the distinction itself: the stored response still rests on the first version and has become **stale**. The question distribution matters too. The 240 requests correspond to only 29 distinct questions, and the three most frequent questions alone account for 120 requests. A response cache works only because of this skew; on a workload where every request is different, the response cache's hit rate stays at zero. #### Stale-Response Share as the Hit Rate Rises Seven setups are run on the same workload: the no-cache baseline, the prefix cache only, the response cache with four different lifetimes, and both caches together. ```python def run(prefix_on, lifetime=0, seed=SEED): """prefix_on: is the prefix cache on . lifetime: step lifetime of the response cache.""" for b in DOCUMENTS: b["version"] = 1 COUNTER[0] = 0 cache, last, prev_step = {}, [None], [-1] s = dict.fromkeys(("prefix_hits", "resp_hits", "input", "output", "stale", "correct", "requests"), 0) for step, question in workload(seed): if step != prev_step[0]: prev_step[0] = step for i in UPDATES.get(step, []): DOCUMENTS[i]["version"] += 1 s["requests"] += 1 entry = cache.get(question["text"]) if lifetime and entry and step - entry["step"] <= lifetime: s["resp_hits"] += 1 stale = entry["version"] != DOCUMENTS[question["doc"]]["version"] s["stale"] += stale s["correct"] += entry["label"] == question["topic"] and not stale continue resp = call(question, seed) s["output"] += resp["output"] if prefix_on and last[0] == prefix(): # prefix is keyed by content s["prefix_hits"] += 1 s["input"] += (round(resp["prefix"] * PREFIX_DISCOUNT) + resp["input"] - resp["prefix"]) else: s["input"] += resp["input"] last[0] = prefix() cache[question["text"]] = {"label": resp["label"], "version": resp["version"], "step": step} s["correct"] += resp["label"] == question["topic"] n, c = s["requests"], max(COUNTER[0], 1) return (s["resp_hits"] / n, s["prefix_hits"] / c, s["input"], s["input"] + s["output"], COUNTER[0], s["stale"] / n, s["correct"] / n) SETUPS = [("no cache", 0, 0), ("prefix cache", 1, 0), ("response lifetime 8", 0, 8), ("response lifetime 24", 0, 24), ("response lifetime 64", 0, 64), ("response lifetime unlimited", 0, 10 ** 6), ("prefix + response lifetime 24", 1, 24)] print("setup resp hit prefix hit input total" " calls stale correct") for name, o, m in SETUPS: a = run(o, m) print(f"{name:<29} {a[0]:>8.4f} {a[1]:>10.4f} {a[2]:>6} {a[3]:>6} {a[4]:>5}" f" {a[5]:>6.4f} {a[6]:>7.4f}") print("\nsecond run (seed 20260219), response hit / stale / correct:") for name, o, m in SETUPS: a = run(o, m, SEED + 1) print(f" {name:<29} {a[0]:.4f} / {a[5]:.4f} / {a[6]:.4f}") ``` ``` setup resp hit prefix hit input total calls stale correct no cache 0.0000 0.0000 44367 45263 240 0.0000 0.9333 prefix cache 0.0000 0.9792 7472 8368 240 0.0000 0.9333 response lifetime 8 0.4708 0.0000 23456 23917 127 0.0208 0.9208 response lifetime 24 0.6750 0.0000 14407 14686 78 0.0500 0.9292 response lifetime 64 0.8000 0.0000 8870 9042 48 0.1333 0.8625 response lifetime unlimited 0.8792 0.0000 5357 5461 29 0.4417 0.5542 prefix + response lifetime 24 0.6750 0.9359 2946 3225 78 0.0500 0.9292 second run (seed 20260219), response hit / stale / correct: no cache 0.0000 / 0.0000 / 0.9083 prefix cache 0.0000 / 0.0000 / 0.9083 response lifetime 8 0.5292 / 0.0292 / 0.8667 response lifetime 24 0.6958 / 0.0833 / 0.8250 response lifetime 64 0.8208 / 0.2167 / 0.6833 response lifetime unlimited 0.9000 / 0.5000 / 0.4042 prefix + response lifetime 24 0.6958 / 0.0833 / 0.8250 ``` The second row is the cheapest gain of this lesson and has no catch. The prefix cache, with a 0.9792 hit rate, cuts input tokens from **44367 to 7472**, that is, **5.94 times**. The stale-response share is **0.0000**, and the correct-response share **does not change**. The reason is in the key: the prefix is keyed **by content**. The moment the pool is updated, the prefix text changes too, the match breaks, and the next call is paid in full. A cache goes stale when its key comes apart from the content. The response-cache rows are the opposite. The key is the **question** text, and the question does not change; what changes is the document the question rests on. With a lifetime of eight steps the hit rate is 0.4708 and the stale share 0.0208; with an unlimited lifetime the hit rate rises to 0.8792 and the stale share climbs to **0.4417**. Calls fall from 240 to 29, and input tokens fall to 5357. From the cost side, the last row is the best row. From the product side, the correct-response share has fallen **from 0.9333 to 0.5542**: in about one of every two requests, the user is told a rule that is no longer valid. #### Choosing the Lifetime Is an Accuracy Decision Where the loss becomes measurable is read from the run-to-run difference. The no-cache baseline gives 0.9083 on the second seed, so the run-to-run difference is **0.0250**. At a lifetime of eight the loss is 0.0125, at twenty-four 0.0041 — both below the run-to-run difference and **counted as unmeasured**. At a lifetime of sixty-four the loss is 0.0708 and at unlimited 0.3791; these are about three and fifteen times the run-to-run difference respectively, that is, **measured**. The decision can be read from here: under this update schedule, extending the lifetime up to twenty-four costs nothing, while raising it to sixty-four does not. This reading carries a warning. In the second run the loss at lifetime twenty-four rises to 0.0833, that is, three times the run-to-run difference relative to its own baseline. The same setting is below the noise in one run and measurable in the other. This comes from twenty-four steps being **close to the update interval**: since the pool changes every twenty-four steps, fitting the lifetime exactly to that interval leaves the decision to the timing luck of the document update. The cache lifetime cannot be chosen **independently** of how often the data changes. The last row is the setup in which the two caches work together, and it produces a number that neither gives alone: input tokens of **2946**, that is, a **15.06-times** reduction from the baseline, with a stale share identical to lifetime twenty-four's, 0.0500. The same row holds a subtler effect: the prefix hit rate falls from 0.9792 to **0.9359**. Because the response cache cuts the number of calls from 240 to 78, the matches broken by the four updates are now divided by a much smaller divisor. One cache improves the other's hit rate by **lowering** it; the numbers of the two scaffoldings cannot be measured separately and added up. #### Summary - Of the prompt's 186 tokens, 175 are shared by every request; which cache will pay off cannot be known until the size of the repetition is measured. - The prefix cache is keyed by content, so it does not go stale: with a 0.9792 hit rate it cuts input from 44367 to 7472 and leaves the correct-response share unchanged. - In the response cache, as the lifetime grows the hit rate rises from 0.4708 to 0.8792, while the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542. - The run-to-run difference is 0.0250: the losses at lifetimes eight and twenty-four stay below this difference and count as unmeasured, while the losses at sixty-four and unlimited are measured. - When the lifetime is chosen close to the update interval, the decision becomes unstable; the same setting loses nothing above the noise in one run and three times it in the other. - The two caches together cut input to 2946, but the response cache lowers the prefix hit rate from 0.9792 to 0.9359: the numbers of the scaffoldings cannot be measured separately and added up. #### Next Step What this lesson gained is tokens. A response returned from the cache makes no call, so it also shortens the user's wait, but this lesson did not count that side at all: the table carries only token, call and accuracy columns. Yet in a support assistant the most visible measure is waiting time, and what determines it is not the cache alone. The size of the context, whether the call is made alone or as part of a chain, and whether the expensive step runs on every request or only when needed each change the wait separately. The next lesson compares these three setups: which **latency class** the small context, the large context and the two-stage call sit in, and what each one's accuracy is. ### Latency Optimization Source: https://academia.sh/en/courses/model-apis/latency-optimization Comparing the same 240 fictional requests under five setups by latency class and accuracy: the small context holds the wait at 1.0000 steps in the single-call class but its accuracy is 0.7750, while the large context in the same class makes the user wait 3.0000 steps and gives 0.9292. The two-stage setup moves 0.2500 of the requests to the chain, cuts the wait to 1.7500 and input tokens to 19716, and loses only 0.0125 of accuracy. The parallel setup pulls the wait down to 1.5000 but at 47352 tokens costs more than the large context. Sending the second stage in batches raises the wait to 3.5542 without changing accuracy or tokens: a batch call is not a latency mechanism. The previous lesson counted tokens and showed that the cache could cut them up to fifteenfold. No column of that table measured what the user **waits** for. Yet in a support assistant this is the most visible measure, and the cache is not the only lever that determines it. Wait cannot be written down as raw time: the same code gives a different number on another machine. What can be written down is the **structure**. How many sequential calls does the response of a request wait on, are those calls made at the same time or one after another, and how large is the input token count of each call. This lesson compares three setups — the small context, the large context and the two-stage call — on these two axes and writes each one's **accuracy** beside it. Then it adds two more setups, because there are two more ways to lower latency and neither of them is free. - **PQ17.** The code is a **simulator**; no real endpoint is called. The pool, the questions and the misleading extra sentences are **fictional**, the seed is **20260218**. - **PQ18.** Wait is counted in **steps**. A call takes `1 + input // 60` steps: a longer input makes the user wait longer. - **PQ19. There are four latency classes.** **Single call**: one call is waited on for one request. **Chain**: the second call waits for the response of the first. **Parallel**: two calls are sent at the same time, and the wait is that of the one that finishes later. **Batch**: the call is part of a batch and is waited on until the batch fills up. - **PQ20.** The cheap **prefilter** is outside the model, spends no tokens at all and picks a single document from the pool. Because it counts fragments, it is caught by misleading extra sentences. - **PQ21.** When the whole pool is placed in the prompt, the model can compare the documents, so it matches the keyphrase as a whole; this is the source of the large context's accuracy advantage. - **PQ22.** In the two-stage setup the first stage also reports whether the given document is **sufficient**. This report is wrong **one time in ten**; only the requests it calls insufficient escalate to the second stage. - **PQ23. A correct response** is a response that rests on the right document and whose label is right. - **PQ24.** The workload is 240 requests over 120 steps, two per step; thirty questions are cycled through in order. - **PQ25.** In the batch setup the batch size is eight; the batch left incomplete at the end of the run is sent at the last step. #### Context Size Determines Both Wait and Accuracy First the pool, the questions and the two selection routines are built: the cheap prefilter, and the case where the whole pool is in the prompt. Then the two are placed side by side on one easy and one misleading question. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called. The document pool, the questions # and the misleading extra sentences are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF UNIT = 60 # input tokens that one step carries def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] DOCUMENTS = [("FT1", "fatura tutari", "fatura tutari hesap donemi ve gecikme farki"), ("FT2", "fatura odemesi", "fatura odemesi yontemleri ve borc gorunumu"), ("AR1", "ariza kaydi", "ariza kaydi acma ve ekip yonlendirme yordami"), ("AR2", "baglanti bozuklugu", "baglanti bozuk oldugunda uygulanan denetim"), ("SY1", "sayac endeksi", "sayac endeks okumasi ve duzeltme yordami"), ("SY2", "sayac degisimi", "sayac degisimi randevu kurallari ve sure taahhudu"), ("TA1", "tarife basamagi", "tarife basamaklari ve abone tipine gore ucret"), ("TA2", "tarife degisikligi", "tarife degisikligi basvurusu ve gecerlilik"), ("KS1", "planli kesinti", "planli kesinti duyurusu ve takip kurallari"), ("KS2", "mahalle kesintisi", "mahallede kesinti bildirimi ve donus suresi")] DOCUMENTS = [{"code": k, "topic": TOPICS[i // 2], "keyphrase": a, "text": m} for i, (k, a, m) in enumerate(DOCUMENTS)] TEMPLATES = ["{a} konusunda ne yapmam gerekiyor", "{a} icin gecerli kural nedir", "{a} ile ilgili son durum ne"] QUESTIONS = [] for i, b in enumerate(DOCUMENTS): for j, k in enumerate(TEMPLATES): m = k.format(a=b["keyphrase"]) if (i + j) % 3 == 1: # misleading extra sentence m += " " + DOCUMENTS[(i + 4) % 10]["text"].split(" ve ")[-1] QUESTIONS.append({"doc": i, "topic": b["topic"], "text": m}) FULL = " ; ".join(f"{b['code']} {b['text']}" for b in DOCUMENTS) INSTRUCTION = "Soruyu verilen belgeye dayanarak yanitla . Etiket ve belge kodu yaz . Belge: " def select(question, full): """full=False: cheap prefilter (fragment counting). full=True: the pool is in the prompt, so the model can match the keyphrase as a whole.""" st, p = tokenize(question["text"]), [] for b in DOCUMENTS: x = sum(t in tokenize(b["text"]) for t in st) if full and all(t in st for t in tokenize(b["keyphrase"])): x += 8.0 p.append(x) ranked = sorted(range(10), key=lambda i: (-p[i], i)) return ranked[0], p[ranked[0]] - p[ranked[1]] def call(question, context, seed=SEED, order=0): """FICTIONAL endpoint. Reads the given context; input tokens grow with the context.""" tok = tokenize(INSTRUCTION + context + " . Soru: " + question["text"]) r = rng((seed ^ trace_id(question["text"])) + 7919 * order) return (question["topic"] if r() > 0.08 else TOPICS[int(r() * 5)]), len(tok) def wait_steps(g): return 1 + g // UNIT print("documents", len(DOCUMENTS), ", questions", len(QUESTIONS), ", full pool", len(tokenize(FULL)), "tokens") z = sum(select(s, False)[0] == s["doc"] for s in QUESTIONS) g = sum(select(s, True)[0] == s["doc"] for s in QUESTIONS) print(f"reached the right document: prefilter {z}/{len(QUESTIONS)} , " f"full pool {g}/{len(QUESTIONS)}") for j in (0, 8): s0 = QUESTIONS[j] i0, i1 = select(s0, False)[0], select(s0, True)[0] gs, gl = call(s0, DOCUMENTS[i0]["text"])[1], call(s0, FULL)[1] print(f"\nquestion : {s0['text']}") print(f" correct {DOCUMENTS[s0['doc']]['code']} , prefilter {DOCUMENTS[i0]['code']} , " f"full pool {DOCUMENTS[i1]['code']}") print(f" small context {gs} tokens -> wait {wait_steps(gs)} ; " f"large context {gl} tokens -> wait {wait_steps(gl)}") ``` ``` documents 10 , questions 30 , full pool 121 tokens reached the right document: prefilter 25/30 , full pool 30/30 question : fatura tutari konusunda ne yapmam gerekiyor correct FT1 , prefilter FT1 , full pool FT1 small context 44 tokens -> wait 1 ; large context 154 tokens -> wait 3 question : ariza kaydi ile ilgili son durum ne abone tipine gore ucret correct AR1 , prefilter TA1 , full pool AR1 small context 46 tokens -> wait 1 ; large context 156 tokens -> wait 3 ``` Comparing the two questions shows where the trade-off comes from. On the easy question both paths reach the same document, and the small context does the same job with **one third** of the wait. On the misleading question, the extra sentence carries the words of a tariff document; the prefilter, which counts fragments, says `TA1`, although the right document is `AR1`. With the whole pool in the prompt this mistake does not happen, because the model can match the keyphrase as a whole. Out of thirty questions the prefilter links 25 and the full pool links 30 to the right document. On the wait side the table is just as clear: the small context takes one step with 44 tokens, the large context three steps with 154 tokens. **Both are in the same latency class** — single call — but the wait is three times as long. The class alone is not a measure; within the class the wait grows with the input tokens. #### Class and Accuracy Table of the Five Setups The five setups are run on the same workload. In the two-stage setup the first stage is the cheap call, and it also reports whether the given document is sufficient; those that say insufficient escalate to the second stage. In the parallel setup two calls are sent at the same time. In the last one, the escalating requests are collected in batches of eight. ```python def workload(): """240 requests over 120 steps, two per step; the questions are cycled in order.""" return [(a, QUESTIONS[(2 * a + j) % len(QUESTIONS)]) for a in range(120) for j in (0, 1)] def stage1(question, seed, order): """Cheap stage: the single document the prefilter gives. The model also reports whether the document is sufficient; this report is wrong one time in ten.""" i, _ = select(question, False) e, g = call(question, DOCUMENTS[i]["text"], seed, order) r = rng((seed ^ trace_id(question["text"])) + 31 * order + 5) sufficient = (i == question["doc"]) != (r() < 0.10) return i, e, g, sufficient def stage2(question, seed, order): i, _ = select(question, True) e, g = call(question, FULL, seed, order) return i, e, g def run(setup, seed=SEED): s = dict.fromkeys(("correct", "input", "calls", "wait", "escalated"), 0) classes, pending, n = {}, [], 0 for a, question in workload(): n += 1 i, e, g, sufficient = stage1(question, seed, n) if setup == "small": s["input"] += g s["calls"] += 1 s["wait"] += wait_steps(g) classes["single call"] = classes.get("single call", 0) + 1 s["correct"] += i == question["doc"] and e == question["topic"] continue if setup == "large": i2, e2, g2 = stage2(question, seed, n) s["input"] += g2 s["calls"] += 1 s["wait"] += wait_steps(g2) classes["single call"] = classes.get("single call", 0) + 1 s["correct"] += i2 == question["doc"] and e2 == question["topic"] continue if setup == "parallel": # two calls at the same time i2, e2, g2 = stage2(question, seed, n) s["input"] += g + g2 s["calls"] += 2 if sufficient: s["wait"] += wait_steps(g) classes["parallel-early"] = classes.get("parallel-early", 0) + 1 s["correct"] += i == question["doc"] and e == question["topic"] else: s["wait"] += max(wait_steps(g), wait_steps(g2)) classes["parallel"] = classes.get("parallel", 0) + 1 s["correct"] += i2 == question["doc"] and e2 == question["topic"] continue s["input"] += g # two-stage setups s["calls"] += 1 if sufficient: s["wait"] += wait_steps(g) classes["single call"] = classes.get("single call", 0) + 1 s["correct"] += i == question["doc"] and e == question["topic"] continue s["escalated"] += 1 i2, e2, g2 = stage2(question, seed, n) s["input"] += g2 s["calls"] += 1 s["correct"] += i2 == question["doc"] and e2 == question["topic"] if setup == "two": s["wait"] += wait_steps(g) + wait_steps(g2) classes["chain"] = classes.get("chain", 0) + 1 else: # second stage sent in a batch pending.append((a, wait_steps(g), wait_steps(g2))) if len(pending) == 8: for a0, b1, b2 in pending: s["wait"] += b1 + (a - a0) + b2 classes["batch"] = classes.get("batch", 0) + 8 pending = [] for a0, b1, b2 in pending: # batch left over at the end of the run s["wait"] += b1 + (119 - a0) + b2 classes["batch"] = classes.get("batch", 0) + 1 return (s["correct"] / n, s["wait"] / n, s["input"], s["calls"], s["escalated"] / n, " , ".join(f"{k} {v / n:.4f}" for k, v in sorted(classes.items()))) SETUPS = (("small context", "small"), ("large context", "large"), ("two-stage", "two"), ("parallel", "parallel"), ("two-stage + batch", "batch")) print("setup correct wait input calls escalated") for name, k in SETUPS: a = run(k) print(f"{name:<19} {a[0]:.4f} {a[1]:>7.4f} {a[2]:>5} {a[3]:>5} {a[4]:>9.4f}") print("\nlatency class distribution:") for name, k in SETUPS: print(f" {name:<19} {run(k)[5]}") print("\nsecond run (seed 20260219), correct / wait / escalated:") for name, k in SETUPS: a = run(k, SEED + 1) print(f" {name:<19} {a[0]:.4f} / {a[1]:.4f} / {a[4]:.4f}") ``` ``` setup correct wait input calls escalated small context 0.7750 1.0000 10376 240 0.0000 large context 0.9292 3.0000 36976 240 0.0000 two-stage 0.9167 1.7500 19716 300 0.2500 parallel 0.9167 1.5000 47352 480 0.0000 two-stage + batch 0.9167 3.5542 19716 300 0.2500 latency class distribution: small context single call 1.0000 large context single call 1.0000 two-stage chain 0.2500 , single call 0.7500 parallel parallel 0.2500 , parallel-early 0.7500 two-stage + batch batch 0.2500 , single call 0.7500 second run (seed 20260219), correct / wait / escalated: small context 0.7792 / 1.0000 / 0.0000 large context 0.9250 / 3.0000 / 0.0000 two-stage 0.9083 / 1.6375 / 0.2125 parallel 0.9083 / 1.4250 / 0.0000 two-stage + batch 0.9083 / 3.3583 / 0.2125 ``` The first two rows are the ends of the trade-off. The small context is the fastest, the cheapest and the most wrong: a wait of 1.0000 steps, input of 10376, accuracy **0.7750**. The large context raises accuracy to **0.9292** and in return triples the wait and multiplies input tokens by **3.56**. The run-to-run difference is 0.0042 in both rows; the accuracy gap of 0.1542 between them is more than thirty times this difference, that is, it is measured beyond dispute. The third row is the two-stage setup and the best-balanced row of the table. **0.2500** of the requests escalate to the second stage; the remaining three quarters finish in a single call. The result: a wait of 1.7500, that is, **0.5833** of the large context; input tokens of 19716, that is, **0.5332** of it. The price paid has two items. The first is 60 extra calls (240 up to 300). The second is accuracy: from 0.9292 to 0.9167, that is, **0.0125**. This loss is about three times the run-to-run difference and counts as measured, but it stays small next to the small context's loss of 0.1542. The report on escalation being wrong one time in ten lands exactly here. #### What You Buy Latency With The fourth row reaches the same accuracy with a shorter wait: 1.5000 steps. The reason is written in the class distribution — there is no chain here, the two calls go **at the same time**, so an escalating request does not pay a second time for the one step of the cheap call. The gain is 0.2500 steps. The cost is in the cost column: 480 calls and **47352** input tokens. This is 2.40 times the two-stage setup and **1.28 times even the large context**. The parallel setup, climbing above even the most expensive setup, buys only wait, without gaining any accuracy. There is always a way to shorten latency; the question to ask is at the expense of which column. The last row is the third example in this topic of the course's second claim. Sending the escalating requests in batches of eight does not change accuracy (0.9167) and does not change input tokens (19716) — even the number of calls is the same. The only thing that changes is the wait, and it has gone up **from 1.7500 to 3.5542**. The batch setup is the slowest row of the table; it is slower even than the single call of the large context. The reason is plain: while the batch is waiting to fill, the request that entered first sits there for nothing. A batch call is a throughput mechanism, not a latency mechanism; opened without looking at the latency measure, it doubles that measure. The second-run rows confirm this reading. The escalating share rises from 0.2500 to 0.2125, and the wait numbers rise with it (1.6375 instead of 1.7500, 3.3583 instead of 3.5542). That is, the two-stage setup's wait gain **depends on the escalating share**, and that share moves from run to run. The only constant is the ordering: parallel gives the shortest wait in every run, batch the longest. #### Summary - Wait is written not as raw time but as a latency class and a step count; the small and the large context are in the same class (single call), but their waits are 1.0000 and 3.0000 steps. - The cheap prefilter links 25 of thirty questions to the right document, the full pool links 30; the small context's accuracy is 0.7750, the large context's 0.9292. - The two-stage setup, by moving 0.2500 of the requests to the chain, cuts the wait to 0.5833 of the large context's and input tokens to 0.5332 of it, and gives up 0.0125 of accuracy. - The parallel setup pulls the wait down to 1.5000 but at 47352 input tokens rises to 1.28 times the large context: only wait is bought, without gaining accuracy. - Sending the second stage in batches raises the wait from 1.7500 to 3.5542 without changing accuracy, tokens or the number of calls at all; a batch call is not a latency mechanism. - The two-stage setup's wait gain depends on the escalating share, and that share moves between 0.2500 and 0.2125 from run to run. #### Next Step In three lessons resilience, cache and latency were measured separately, and a token total was written beside each: 10376, 19716, 47352. These numbers are enough to compare one setup with another, but they do not answer a single question — **among whom this total was shared.** Did all two hundred and forty requests spend as much as the mean, or did a few requests swallow a significant part of the total. A budget is not kept by the mean: the most expensive one percent of a line that stays below the mean can exceed the budget by itself, and this does not show up in any table that looks at totals. The next lesson writes cost **per request** and looks at the tail of the distribution. ### Cost Monitoring Source: https://academia.sh/en/courses/model-apis/cost-monitoring Keeping the tokens-per-request ledger for a fictional workload of four hundred requests and counting the tail of the distribution: the mean is 81.6 tokens while the median is 69, the 99th percentile 224 and the largest request 303. The most expensive one percent spends 0.0299 of the total tokens, the most expensive ten percent 0.2124. Across two runs the mean goes from 81.6 to 83.0 while the share of the most expensive one percent goes from 0.0299 to 0.0409, that is, the stable one is the mean and the one that moves is the tail. When the bill is split into items, the instruction takes 0.3921, the conversation state 0.3658 and the ticket text, which carries the new information, only 0.1190. Three policies that shorten the conversation state bring tokens per request from 81.6 down to 50.5, but the difference narrows when read in tokens per correct answer. The previous three lessons measured the network side of the call: the request that hit the rate limit was retried, the repeated context was cached, and the latency class was lowered by changing the context size and the setup. All three made something cheaper, and none of them asked **how much** cheaper. The latency class is the user's measure; it is not the unit of the bill. The unit of this lesson is **the token**. No currency is written, because price depends on the provider and on time; the token count, on the other hand, is a number the setup itself produces and it carries from run to run. Two things are to be measured: how many tokens a request costs, and **how this number is distributed** across requests. Without the second, the first misleads. - **PQ31.** The code is a **simulator**; no real endpoint is called. The tickets, the conversations and the documents are **fictional**, the seed is **20260218**. - **PQ32. The procedure of cost accounting is not repeated.** Metric collection, the dashboard and unit cost allocation were measured in the Observability and Reliability course, in the DevOps and Platform Engineering curriculum and in the Cloud Governance, Security and Cost course. The only thing added here is **the unit**: the cost of a request is written in tokens. - **PQ33. In this lesson the workload is conversations, not single tickets.** The state is carried on the client and resent with every request; the third request of a conversation also contains the text of the first request. This is the source of the tail. - **PQ34.** Conversation lengths are drawn with weights from the set `[1, 2, 3, 5, 8, 12]`. Even the last request of the longest conversation stays below the context window, so the trimming decision does not come into play in this lesson; the cost of trimming was measured in the Introduction to AI Engineering course. - **PQ35.** A request consists of one or several calls. The retry procedure is not repeated; the ledger records only **how many calls were made** and the input and output tokens of each call. - **PQ36.** On retry the simulator's seed is increased by one. Because the same prompt with the same seed gives the same response, a second attempt can give a different result only this way. - **PQ37.** The cost of a request is **input plus output tokens**; the input of a failed call is counted too, because it was sent. #### The Per-Request Ledger Reporting the total tokens as a single number is a sum, not accounting. Accounting requires writing **which request** a cost belongs to. The ledger below keeps one row for every request: how many calls were made, how many tokens went in, how many tokens came out. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations # and documents are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per token b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme", "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"} FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi", "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek", "kaydiniz alindi ilgili birim sizinle iletisime gececek"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the call count return {"status": "rate_limit", "input": len(tok), "output": 0, "reply": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""} candidates = [k for k in CONCEPTS if k[:5] in tok] # the label set narrows the candidates p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)} for t in tok: if CUES.get(t) in p: p[CUES[t]] += 1.0 if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: w = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [x / sum(w) for x in w])] y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k) out = tokenize(y) if len(out) > max_output: # stopping condition out, y = out[:max_output], " ".join(out[:max_output]) return {"status": "ok", "input": len(tok), "output": len(out), "reply": y} def read_label(y): return next((s for s in y.split() if s in TOPICS), None) INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: ") DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir", "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir", "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir", "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir", "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"] CONVERSATIONS, i = [], 0 while sum(len(c["turns"]) for c in CONVERSATIONS) < 400: # FICTIONAL workload: conversations r = rng(SEED + 900 + i) i += 1 k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])] CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]), "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]} for t in range(u)]}) REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]} for c in CONVERSATIONS for t in range(len(c["turns"]))][:400] def context(x, mode="full"): t = x["conv"]["turns"][:x["pos"]] t = [] if mode == "stateless" else (t[-2:] if mode == "two turns" else t) if mode == "summary state": g = "".join(f"Onceki konu: {o['topic']} . " for o in t) else: g = "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . " for o in t) return g + "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . " for j in range(x["conv"]["doc"])) def ledger(mode="full", seed=SEED): """A separate row for every request: calls, input tokens, output tokens.""" COUNTER[0] = 0 d = [] for x in REQUESTS: c = inp = out = 0 prompt = INSTRUCTION + context(x, mode) + x["ticket"]["text"] for a in range(3): s = call(prompt, seed + a) # the seed grows on retry c, inp, out = c + 1, inp + s["input"], out + s["output"] if s["status"] == "ok": break d.append({"total": inp + out, "calls": c, "output": out, "correct": read_label(s["reply"]) == x["ticket"]["topic"]}) return d base = ledger() t = sorted(z["total"] for z in base) total = sum(t) print("conversations", len(CONVERSATIONS), " requests", len(base), " calls", sum(z["calls"] for z in base), " total tokens", total) print("mean %.1f median %d p75 %d p90 %d p99 %d max %d" % (total / 400, t[200], t[300], t[360], t[396], t[-1])) for p, d in ((1, 4), (5, 20), (10, 40)): print("top %-4s %5d tokens, %.4f of the total" % (f"{p}%", sum(t[-d:]), sum(t[-d:]) / total)) base2 = ledger(seed=SEED + 1) t2 = sorted(z["total"] for z in base2) print("run 2 mean %.1f median %d top 1%% share %.4f" % (sum(t2) / 400, t2[200], sum(t2[-4:]) / sum(t2))) print("correct %.4f / %.4f" % (sum(z["correct"] for z in base) / 400, sum(z["correct"] for z in base2) / 400)) ``` ``` conversations 154 requests 400 calls 414 total tokens 32642 mean 81.6 median 69 p75 100 p90 139 p99 224 max 303 top 1% 975 tokens, 0.0299 of the total top 5% 3892 tokens, 0.1192 of the total top 10% 6932 tokens, 0.2124 of the total run 2 mean 83.0 median 69 top 1% share 0.0409 correct 0.9125 / 0.8925 ``` #### The Tail the Mean Hides A report reduced to a single number would have said: four hundred requests, thirty-two thousand six hundred forty-two tokens, **81.6** per request. A plan that looks at this number writes the cost of doubling the request count as double too, and is wrong. The median is **69**, that is, half of the requests are below 69. What pulls the mean up is the upper percentiles: the 75th percentile is 100, the 90th percentile 139, the 99th percentile **224** and the most expensive request **303** tokens. The most expensive request is more than four times the median. This is not a defect; it is the unavoidable result of long conversations. Because the state is carried on the client, the twelfth request of a conversation resends the eleven turns of that conversation too. The weight of the tail is read as a share. The **most expensive one percent**, that is, four of the four hundred requests, spends **0.0299** of the total: three times its own share. The most expensive ten percent spends **0.2124**, more than twice its share. In an investigation of a budget overrun, the place to look is these forty requests; making each of the remaining three hundred sixty cheaper by one token does not give the same gain. The last two lines test the stability of the measure itself. In the second run the mean goes from **81.6 to 83.0**, that is, it moves by seventeen in a thousand; the median is **constant at 69**. By contrast the share of the most expensive one percent goes from **0.0299 to 0.0409**, that is, it moves by more than a third. Because the tail rests on a small number of requests, it is noisy. A direct consequence follows: the tail share is not read from **a single run**, it is accumulated and read over a window. That the mean is stable and the tail volatile is a reason not to trust the mean but to measure the tail for longer. #### The Items of the Bill Where the total tokens go is found by counting the pieces of text inside the request separately. The block below splits the same ledger into items and then runs the three policies that shorten the conversation state on the same workload. ```python items = {"instruction": 400 * len(tokenize(INSTRUCTION)), "conversation state": sum(len(tokenize(context(x))) - len(tokenize(context(x, "stateless"))) for x in REQUESTS), "document": sum(len(tokenize(context(x, "stateless"))) for x in REQUESTS), "ticket text": sum(len(tokenize(x["ticket"]["text"])) for x in REQUESTS), "output": sum(z["output"] for z in base)} items["retries"] = total - sum(items.values()) print(f"{'item':<18} {'tokens':>8} {'share':>7}") for a, v in items.items(): print(f"{a:<18} {v:>8} {v / total:>7.4f}") print(f"\n{'policy':<13} {'total':>7} {'per request':>11} {'p99':>9} {'top 1% share':>13}" f" {'correct':>7} {'per correct':>11}") for mode in ("full", "two turns", "summary state", "stateless"): d = ledger(mode) s = sorted(z["total"] for z in d) tt, n_correct = sum(s), sum(z["correct"] for z in d) print(f"{mode:<13} {tt:>7} {tt / 400:>11.1f} {s[396]:>9} {sum(s[-4:]) / tt:>13.4f}" f" {n_correct / 400:>7.4f} {tt / n_correct:>11.1f}") ``` ``` item tokens share instruction 12800 0.3921 conversation state 11939 0.3658 document 2344 0.0718 ticket text 3885 0.1190 output 625 0.0191 retries 1049 0.0321 policy total per request p99 top 1% share correct per correct full 32642 81.6 224 0.0299 0.9125 89.4 two turns 27401 68.5 198 0.0314 0.9125 75.1 summary state 24355 60.9 120 0.0259 0.9075 67.1 stateless 20198 50.5 95 0.0276 0.7975 63.3 ``` The first reading of the item table is this: **only 0.1190 of this bill carries new information.** The ticket text, that is, the sentence the subscriber wrote in that request, is three thousand eight hundred eighty-five tokens. The two largest of the remaining items are repeated text, the instruction at **0.3921** and the conversation state at **0.3658**, and together they make up nearly three quarters of the total. The output tokens are **0.0191**, that is, negligible; in this setup the model writes a single label, and the cost of what it writes is invisible next to the cost of what it reads. This split also says which scaffolding touches where. The instruction item is identical in every request, and prompt caching targets it; the conversation state item grows from request to request, and shortening targets it. Without counting the two separately, it cannot be known what each scaffolding will gain. #### Which Unit Is Used to Look The policy table compares four setups on the same workload. A reading that looks at the **per request** column is clear: dropping the state entirely takes 81.6 down to **50.5**, that is, it gains more than a third. The accuracy of the same row is **0.7975**, which is 0.1150 lower than the **0.9125** of the full state. Since the run-to-run difference is 0.0200, this drop is real, and it means forty-six wrong labels in four hundred requests. The last column reads the same four rows **per correct answer** and narrows the difference: 89.4 / 75.1 / 67.1 / **63.3**. The stateless setup, although its accuracy is the lowest, is the cheapest per correct answer. The two units point to the same winner but write the difference at a different size, because the first counts the cost of a wrong answer as zero. On a support line a wrong label is not free; forty-six wrongly routed tickets create manual correction work. Which unit is right is a decision of the product, not of the measurement, but **writing which unit was used** is the responsibility of the measurement. The third row is a gain that cannot be measured. The summary state setup lowers the accuracy from 0.9125 to **0.9075**; the difference is 0.0050, a quarter of the run-to-run difference. This row can be read as "it does not lower the accuracy", but the correct reading is "**whether it lowers it cannot be known with this measurement**". The 8287 tokens it gains in return, on the other hand, are measured. The tail column also shows a reverse behavior. Although the two turns setup lowers the total from 32642 to 27401, the share of the most expensive one percent **rises from 0.0299 to 0.0314**. The share grows because shortening cuts the total more than it cuts the tail. Since the tail share is a ratio, it cannot be read alone; the total must be written next to it. #### Summary - The unit of cost is the token; no currency is written, because price depends on the provider, while the token count is a number the setup itself produces. - The mean is 81.6, the median 69, the 99th percentile 224 and the largest request 303 tokens; a plan that looks at the mean cannot see the cost of long conversations. - The most expensive one percent spends 0.0299 of the total, the most expensive ten percent 0.2124: a budget investigation is done by looking at forty requests. - The mean moves from 81.6 to 83.0 across two runs while the tail share goes from 0.0299 to 0.0409; the tail is read not from a single run but from a window. - Of the bill, 0.3921 is the instruction, 0.3658 is the conversation state, and only 0.1190 is the ticket text that carries new information; the output tokens are 0.0191. - The same four policies rank the same in both units, but the difference narrows (50.5 against 63.3); without writing which unit was used, this difference cannot be read. #### Next Step The ledger of this lesson counts every request the same way: a request went, a response came, this many tokens were held. There is one column the ledger cannot see. When part of the four hundred requests pass through another route because the endpoint did not answer, they will write the same row too: the call count, the tokens, even a label. **Where the response came from** is not written to the ledger. The next lesson counts the share of requests that fall onto a second route in the face of an outage and prints the accuracy of those requests separately: once the outage is covered, the error visible to the user drops to almost zero, the total accuracy hardly moves, and the drop shows only when the source of the response is written to the ledger. ### Error and Outage Management Source: https://academia.sh/en/courses/model-apis/error-and-outage-management Measuring what dropping a request to a fallback tier does to quality when the primary endpoint does not respond: in the setup without fallback the outage turns 26 requests into errors, while in the setup with fallback errors fall to 2 and the total accuracy rises from 0.8450 to 0.8850, that is, the fallback looks like an improvement. Read against the no-outage baseline, the same number has fallen from 0.8975 to 0.8850, and the drop is 0.0125, below the run-to-run difference of 0.0200. When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667. When the outage share is raised from 0.00 to 0.35, the drop in total accuracy exceeds the run-to-run difference only at 0.18; the accuracy of the primary tier does not move across the four rows, what moves is the mix. The ledger of the previous lesson wrote one row for every request: how many calls, how many tokens, correct or not. The only thing not in that ledger was **where the response came from**. When an endpoint does not respond, a second route is opened so that the request is not lost, and this route writes a row in the ledger that looks the same as the first. The circuit breaker, graceful degradation and failover were set up in the Resilience and Reliability course: when to flip the switch, how long an open circuit stays open and the procedure of the half-open trial were measured there. They are not repeated here. The only thing added to this lesson is this: for a language model, the fallback is **not another server doing the same job**. It is a smaller, narrower and less accurate component; and it does this without telling anyone. - **PQ38.** The code is a **simulator**; no real endpoint is called. The tickets, the conversations and the outage window are **fictional**, the seed is **20260218**. - **PQ39.** The workload is the same as in the previous lesson: 400 requests from 154 fictional conversations. - **PQ40. The fallback tier differs in three places:** its context window is **120** instead of 240, **eleven** of the twenty-three entries of the cue table are absent from it, and the label set written into the prompt does **not narrow** its candidate set. All three are known limits of a real small model. - **PQ41.** The outage is set up as a **window**: starting from request 160, for a given share of the workload, the primary tier rejects 0.70 of the requests. - **PQ42.** Outside the outage too, the primary tier fails to answer a share of requests: rate limit, timeout and context overflow. These bring the fallback into play **even without an outage**. - **PQ43. Three setups are compared:** one that turns the dropped request into an error, one that routes it to a human line, and one that drops it to the fallback. All three work on the output of **the same primary run**, so the set of dropped requests is exactly the same in all three. - **PQ44.** Accuracy is counted separately in three places: **total** over all requests, **primary** over what the primary tier answered, **fallback** over what the fallback answered. #### The Silent Cost of the Fallback ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations # and the outage window are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per token b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme", "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"} FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi", "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek", "kaydiniz alindi ilgili birim sizinle iletisime gececek"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the call count return {"status": "rate_limit", "input": len(tok), "output": 0, "reply": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""} candidates = [k for k in CONCEPTS if k[:5] in tok] # the label set narrows the candidates p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)} for t in tok: if CUES.get(t) in p: p[CUES[t]] += 1.0 if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: w = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [x / sum(w) for x in w])] y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k) out = tokenize(y) if len(out) > max_output: # stopping condition out, y = out[:max_output], " ".join(out[:max_output]) return {"status": "ok", "input": len(tok), "output": len(out), "reply": y} def read_label(y): return next((s for s in y.split() if s in TOPICS), None) INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: ") DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir", "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir", "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir", "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir", "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"] CONVERSATIONS, i = [], 0 while sum(len(c["turns"]) for c in CONVERSATIONS) < 400: # FICTIONAL workload: conversations r = rng(SEED + 900 + i) i += 1 k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])] CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]), "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]} for t in range(u)]}) REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]} for c in CONVERSATIONS for t in range(len(c["turns"]))][:400] def context(x): return "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . " for o in x["conv"]["turns"][:x["pos"]]) \ + "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . " for j in range(x["conv"]["doc"])) FALLBACK_CUES = {a: b for a, b in CUES.items() if a not in ("tutar", "ayrin", "bozuk", "akmiy", "endek", "okuma", "basam", "abone", "duyur", "mahal", "calis")} FALLBACK_WINDOW = 120 def ask_fallback(prompt, seed=SEED): """Fallback tier: narrow window, few cues, does not narrow the label set.""" COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > FALLBACK_WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""} p = {a: 0.15 for a in CONCEPTS} for t in tok: if FALLBACK_CUES.get(t) in p: p[FALLBACK_CUES[t]] += 1.0 w = [math.exp(p[a] / 1.3) for a in sorted(p)] r = rng((seed + 5) ^ trace_id(prompt)) return {"status": "ok", "input": len(tok), "output": 1, "reply": sorted(p)[pick(r(), [x / sum(w) for x in w])]} def primary(share=0.06, seed=SEED): """Outage window: starting from request 160, the given share of the workload.""" COUNTER[0] = 0 cut, result = int(400 * share), [] for i, x in enumerate(REQUESTS): prompt = INSTRUCTION + context(x) + x["ticket"]["text"] if 160 <= i < 160 + cut and rng(seed ^ trace_id(prompt))() < 0.70: s = {"status": "outage", "input": len(tokenize(prompt)), "output": 0, "reply": ""} COUNTER[0] += 1 else: s = call(prompt, seed) result.append((x, prompt, s)) return result def apply_setup(mode, raw): """The ledger also writes response provenance: primary and fallback are counted apart.""" n = dict(dropped=0, to_fallback=0, error=0, human=0, p_ok=0, p_n=0, f_ok=0, f_n=0, tok=0) for x, prompt, s in raw: n["tok"] += s["input"] + s["output"] if s["status"] == "ok": n["p_n"] += 1 n["p_ok"] += read_label(s["reply"]) == x["ticket"]["topic"] continue n["dropped"] += 1 if mode == "no_fallback": n["error"] += 1 elif mode == "to_human": n["human"] += 1 else: y = ask_fallback(prompt) n["tok"] += y["input"] + y["output"] if y["status"] != "ok": n["error"] += 1 else: n["to_fallback"] += 1 n["f_n"] += 1 n["f_ok"] += read_label(y["reply"]) == x["ticket"]["topic"] n["total"] = (n["p_ok"] + n["f_ok"]) / 400 n["primary_acc"] = n["p_ok"] / max(n["p_n"], 1) n["fallback_acc"] = n["f_ok"] / max(n["f_n"], 1) return n print(f"{'setup':<22} {'dropped':>7} {'error':>5} {'to fallback':>11} {'to human':>8}" f" {'total':>7} {'primary':>7} {'fallback':>8} {'tokens':>7}") for name, mode, raw in (("no outage, no fallback", "no_fallback", primary(0.0)), ("outage, no fallback", "no_fallback", primary()), ("outage, to human", "to_human", primary()), ("outage, fallback", "fallback", primary()), ("no outage, fallback", "fallback", primary(0.0))): a = apply_setup(mode, raw) print(f"{name:<22} {a['dropped']:>7} {a['error']:>5} {a['to_fallback']:>11}" f" {a['human']:>8} {a['total']:>7.4f} {a['primary_acc']:>7.4f}" f" {a['fallback_acc']:>8.4f} {a['tok']:>7}") ``` ``` setup dropped error to fallback to human total primary fallback tokens no outage, no fallback 13 13 0 0 0.8750 0.9044 0.0000 31569 outage, no fallback 26 26 0 0 0.8450 0.9037 0.0000 31548 outage, to human 26 0 0 26 0.8450 0.9037 0.0000 31548 outage, fallback 26 2 24 0 0.8850 0.9037 0.6667 33775 no outage, fallback 13 1 12 0 0.8975 0.9044 0.7500 32726 ``` Read the second and fourth rows side by side, and the fallback looks like a gain. During the outage the setup without fallback turns twenty-six requests into errors and leaves the total accuracy at **0.8450**; in the setup with fallback errors fall to **2** and total accuracy rises to **0.8850**. The difference is **0.0400**, at a cost of **2227 tokens**. On a dashboard this row is read as proof that the scaffolding works. The fifth row sets the same number against another baseline. Had there been no outage at all, total accuracy would have been **0.8975**. The outage happened and the number fell to **0.8850**: a drop of **0.0125**. As will be seen in the next section, the run-to-run difference of this measure is **0.0200**. That is, the drop stayed below the measurement noise. The outage happened, twenty-four requests were answered by another model, and total accuracy did not show it. Only the last two columns say where the drop is. The accuracy of the primary tier is **0.9037**, that of the requests that fell to the fallback **0.6667**. The **0.2371** between them is not a small difference; in one of every three requests the label is wrong. All that was needed for this number to appear was to add one field to the ledger: response provenance, that is, **which tier** it came from. That field holds zero tokens. The third row measures another option. When the dropped request is routed to the human line instead of being given to the fallback, total accuracy stays at **0.8450**, the same as in the setup without fallback; but the error returned to the user is **zero**, and the cost is written to **a countable place** as twenty-six requests' worth of human work. The fallback hides, the human line counts. The choice between these two forms of graceful degradation is about whether making the hidden cost visible is worth it. Finally, the narrow window of the fallback leaves two requests unanswerable. These are the longest conversations: since the window of the fallback tier is 120 tokens, it cannot carry the context the primary tier can carry. The requests that fall to the fallback are not random; **those with the longest context struggle most**. #### As the Outage Share Grows That a drop is not visible does not mean the drop is small. The block below raises the outage share from zero to 0.35 and at every step measures total accuracy with two seeds. ```python baseline = apply_setup("fallback", primary(0.0))["total"] print(f"{'outage':<8} {'fallback share':>14} {'total A':>9} {'total B':>9} {'run-to-run':>10}" f" {'drop':>6} {'visible':<7} {'primary':>7} {'fallback':>8} {'sample':>6}") for share in (0.0, 0.06, 0.18, 0.35): a = apply_setup("fallback", primary(share)) b = apply_setup("fallback", primary(share, SEED + 1)) f, d = abs(a["total"] - b["total"]), baseline - a["total"] print(f"{share:<8.2f} {a['to_fallback'] / 400:>14.4f} {a['total']:>9.4f} {b['total']:>9.4f}" f" {f:>10.4f} {d:>6.4f} {'yes' if d > f else 'no':<7}" f" {a['primary_acc']:>7.4f} {a['fallback_acc']:>8.4f} {a['f_n']:>6}") ``` ``` outage fallback share total A total B run-to-run drop visible primary fallback sample 0.00 0.0300 0.8975 0.8725 0.0250 0.0000 no 0.9044 0.7500 12 0.06 0.0600 0.8850 0.8650 0.0200 0.0125 no 0.9037 0.6667 24 0.18 0.1275 0.8500 0.8325 0.0175 0.0475 yes 0.9032 0.6275 51 0.35 0.2350 0.7850 0.7750 0.0100 0.1125 yes 0.8927 0.5957 94 ``` The visibility column flips between the second and third rows. At outage share 0.06 the drop is 0.0125 and the run-to-run difference 0.0200: an alert threshold that looks at total accuracy **sees nothing**. When the outage share rises to 0.18, the drop is 0.0475 and exceeds the run-to-run difference. That is, total accuracy is not a quality indicator but an **indicator that depends on the outage share**; it reports quality only when the outage becomes large enough. The first row is the extreme case of this. Even when there is no outage at all, twelve requests are answered by the fallback, because the rate limit and the timeout produce a share in every run. The accuracy of these twelve requests is **0.7500**; in normal operation, without any event record being written, **0.0300** of the requests are answered by a worse model. The primary accuracy column stays in place in all four rows: **0.9044 / 0.9037 / 0.9032 / 0.8927**. The primary model did not degrade; what degraded is **the mix**. The accuracy of the fallback tier, on the other hand, falls with the share (from 0.7500 to 0.5957), because as the outage grows, the proportion of long-context requests among those that fall to the fallback rises. The last column also writes the limit of this reading: the 0.6667 in the 0.06 row was computed from only twenty-four requests. It is a number with a small sample and cannot be tied to a threshold on its own; but it is **far more informative** than total accuracy, because it measures what it wants to measure. #### Summary - The fallback tier is not another server doing the same job: its window is narrow, its cue table is incomplete, and the label set does not narrow its candidate set. - During the outage the fallback brings errors down from 26 to 2, raises total accuracy from 0.8450 to 0.8850 and adds 2227 tokens; against the no-outage baseline, it brings accuracy down from 0.8975 to 0.8850. - The drop is 0.0125 and the run-to-run difference is 0.0200: an indicator that looks at total accuracy never sees the outage. - When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667, difference 0.2371; the cost of this field is zero tokens. - Total accuracy is not a quality indicator but an indicator that depends on the outage share: the drop exceeds the run-to-run difference only when the outage share reaches 0.18. - The accuracy of the primary tier does not move across the four shares (between 0.9044 and 0.8927); what degrades is the mix, and even with no outage 0.0300 of the requests are answered by the fallback. #### Next Step In this lesson the run-to-run difference was used as a threshold: every difference below 0.0200 was counted as unmeasured. This threshold itself was never questioned. Why do the same workload, the same prompt and the same code give different numbers in two runs, which item moves when it is released, and does the movement stop if everything is pinned? The next lesson counts the items of a model call one by one, releases each of them alone and measures how many responses change, and separates how many of these items are **in our hands**. ### Nondeterminism and Reproducibility Source: https://academia.sh/en/courses/model-apis/nondeterminism-and-reproducibility Counting the items that determine the output of a model call and releasing each of them alone: 11 of the twelve items move the responses, and the largest movement is in the format word of the instruction (all responses change, accuracy falls from 0.9125 to 0.5525). Whitespace layout, which changes no meaning at all, moves 0.0700 of the responses, and client normalization moves 0.1300. The maximum output length, which alone moves nothing, drops accuracy from 0.5525 to 0.2675 when released together with the instruction word. Nine of the items are on the client and 3 on the provider, and when the three are released together 0.1150 of the responses move and accuracy drops by 0.0200: this is the threshold of the previous two lessons. In the previous two lessons one threshold was used again and again: the difference between two runs. In the cost ledger it came out as 0.0200, in the outage table again 0.0200, and every gain below this number was counted as unmeasured. The threshold itself was never asked about. Why do the same code, the same workload and the same prompt give two different numbers in two runs. Part of the question was closed earlier. The Visualization, Reporting and Analysis at Scale course wrote the reproduction of a report as fifty-nine items: the thirty-nine decisions of the preparation ledger, the nine items of the runtime environment and the eleven items of the report itself. The Deep Learning course did the same count for a training run, added the eight items specific to training and raised it to sixty-seven. That count is not repeated here. The added question is this: what are the items of a **model call**, and how many of them are in our hands. - **PQ45.** The code is a **simulator**; no real endpoint is called. The tickets, the conversations and the documents are **fictional**, the seed is **20260218**. - **PQ46.** The workload is the same as in the previous two lessons: 400 requests from 154 fictional conversations. - **PQ47. An item** is every decision that could determine the output of the call and that has to be carried from run to run. Each item is moved to an **equally defensible** alternative; not to a wrong value. - **PQ48.** Two measures are printed: the **changed-response share** (how many of the 400 responses differ from those of the base run) and **accuracy**. The two are different things. - **PQ49. The items are split into two sides.** Nine are on the client, three on the provider. The provider items can be pinned in the simulator but cannot be pinned on a real endpoint; in this lesson they count as **unpinnable**. - **PQ50.** Inherited items: the **39** decisions of the preparation ledger and the **9** items of the runtime environment. The eight items specific to training are not inherited, because the model is given and is not trained. - **PQ51.** The format word item of the instruction is the extreme of the list: the simulator ties the format of the response to the word `one` in the prompt. This is the modeled form of format compliance depending on the **surface word**. - **PQ52.** The version change of the provider is modeled by adding two entries to the cue table. #### The Items of a Call ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations # and documents are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW = 240 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per token b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme", "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"} FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi", "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek", "kaydiniz alindi ilgili birim sizinle iletisime gececek"] COUNTER = [0] def read_label(y): return next((s for s in y.split() if s in TOPICS), None) DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir", "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir", "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir", "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir", "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"] CONVERSATIONS, i = [], 0 while sum(len(c["turns"]) for c in CONVERSATIONS) < 400: # FICTIONAL workload: conversations r = rng(SEED + 900 + i) i += 1 k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])] CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]), "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]} for t in range(u)]}) REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]} for c in CONVERSATIONS for t in range(len(c["turns"]))][:400] DEFAULTS = dict(seed=SEED, temperature=0.6, max_output=24, word="tek", labels="listed", spacing="single", state="full", piece="document last", normalize="raw", tries=3, cues="version a", threshold=37) CUES_B = dict(CUES, gecen="fatura", ekip="kesinti") # the version change of the provider def call_config(prompt, a): """Call with a settings dictionary: every item is a separate key.""" COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "reply": ""} if COUNTER[0] % a["threshold"] == 0: return {"status": "rate_limit", "reply": ""} r = rng(a["seed"] ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "reply": ""} cues = CUES if a["cues"] == "version a" else CUES_B candidates = [k for k in CONCEPTS if k[:5] in tok] p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)} for t in tok: if cues.get(t) in p: p[cues[t]] += 1.0 w = [math.exp(p[q] / a["temperature"]) for q in sorted(p)] k = sorted(p)[pick(r(), [x / sum(w) for x in w])] y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k) out_tok = tokenize(y) return {"status": "ok", "reply": " ".join(out_tok[:a["max_output"]]) if len(out_tok) > a["max_output"] else y} def build_prompt(x, a): labels = TOPICS if a["labels"] == "listed" else sorted(TOPICS) instruction = ("Talebi su etiketlerden birine ata: " + " , ".join(labels) + " . Yanit olarak " + ("tek" if a["word"] == "tek" else "yalnizca") + " etiket yaz . Talep: ") past = x["conv"]["turns"][:x["pos"]] history = "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . " for o in (past[-2:] if a["state"] == "two turns" else past)) docs = "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . " for j in range(x["conv"]["doc"])) s = instruction + (docs + history if a["piece"] == "document first" else history + docs) \ + x["ticket"]["text"] if a["spacing"] == "double": s = s.replace(" ", " ") if a["normalize"] == "simplified": s = " ".join(s.lower().split()) return s def run(**d): a = dict(DEFAULTS, **d) COUNTER[0] = 0 outputs = [] for x in REQUESTS: prompt = build_prompt(x, a) for i in range(a["tries"]): s = call_config(prompt, dict(a, seed=a["seed"] + i)) if s["status"] == "ok": break outputs.append(s["reply"]) return outputs def accuracy(c): return sum(read_label(y) == x["ticket"]["topic"] for y, x in zip(c, REQUESTS)) / 400 def changed(c, ref): return sum(a != b for a, b in zip(ref, c)) / 400 ITEMS = [("provider", "sampling state", dict(seed=SEED + 1)), ("provider", "model version", dict(cues="version b")), ("provider", "rate limit threshold", dict(threshold=31)), ("client", "distribution temperature", dict(temperature=0.8)), ("client", "maximum output length", dict(max_output=10)), ("client", "instruction format word", dict(word="yalnizca")), ("client", "label set order", dict(labels="alphabetical")), ("client", "whitespace layout", dict(spacing="double")), ("client", "conversation state", dict(state="two turns")), ("client", "context piece order", dict(piece="document first")), ("client", "client normalization", dict(normalize="simplified")), ("client", "retry count", dict(tries=1))] BASE = run() print("base accuracy %.4f , item count %d" % (accuracy(BASE), len(ITEMS))) print("side item changed accuracy diff") for side, name, d in ITEMS: c = run(**d) print(f"{side:<10} {name:<25} {changed(c, BASE):>7.4f} {accuracy(c):>8.4f}" f" {accuracy(c) - accuracy(BASE):>+8.4f}") print("items that move", sum(changed(run(**d), BASE) > 0 for _, _, d in ITEMS), "/", len(ITEMS)) ``` ``` base accuracy 0.9125 , item count 12 side item changed accuracy diff provider sampling state 0.1200 0.8925 -0.0200 provider model version 0.0150 0.9175 +0.0050 provider rate limit threshold 0.0050 0.9075 -0.0050 client distribution temperature 0.0600 0.8575 -0.0550 client maximum output length 0.0000 0.9125 +0.0000 client instruction format word 1.0000 0.5525 -0.3600 client label set order 0.1100 0.9100 -0.0025 client whitespace layout 0.0700 0.9325 +0.0200 client conversation state 0.0050 0.9125 +0.0000 client context piece order 0.0025 0.9150 +0.0025 client client normalization 0.1300 0.8775 -0.0350 client retry count 0.0375 0.8750 -0.0375 items that move 11 / 12 ``` Of the twelve items, **eleven** move the responses when released alone. The most instructive rows of the list are the ones that were not expected to move anything. **Whitespace layout** does not touch the meaning of the prompt at all, it sends the same words in the same order, and it changes **0.0700** of the responses. **Client normalization**, while keeping the meaning in the same way, changes **0.1300** of them. **The order of the label set** writes the same five labels in another order and moves **0.1100**. The text of a prompt is an input not by its meaning but by **its exact self**. A changed response and a broken response are not the same thing, and the table keeps them in separate columns. Whitespace layout changes seven percent of the responses but **raises accuracy from 0.9125 to 0.9325**; the gain is accidental and can reverse in the next run. The label order moves 0.1100 and lowers accuracy by only 0.0025. By contrast, the **distribution temperature** moves only 0.0600 but lowers accuracy by **0.0550**: few responses change, but nearly all of those that change go straight to wrong. The movement share alone is not a measure of harm. Two rows come out smaller than expected. **Conversation state** moves only 0.0050, although the prompt text changed in 113 of the 400 requests; **context piece order** moves 0.0025, although 60 requests had two pieces at once. Long conversations are the most stable requests, because when traces from the same topic accumulate, the choice gathers on a single concept and the draw cannot disturb it. Stability varies by request; a single movement share does not describe the whole workload. The largest row, though, is separated from the others by scale. When the **format word of the instruction** is replaced with its synonym, all of the responses change and accuracy falls from 0.9125 to **0.5525**. This item changes not the draw but the **task**: once the format constraint is lifted, the model starts writing sentences instead of a single label, and a part of the sentences carry no label at all. A synonymous word lowered the measure by 0.3600. #### Items Released Together ```python both = run(max_output=10, word="yalnizca") single = run(word="yalnizca") print("only maximum output length changed %.4f accuracy %.4f" % (changed(run(max_output=10), BASE), accuracy(run(max_output=10)))) print("only instruction format word changed %.4f accuracy %.4f" % (changed(single, BASE), accuracy(single))) print("both together changed %.4f accuracy %.4f" % (changed(both, BASE), accuracy(both))) three = run(seed=SEED + 1, cues="version b", threshold=31) print("nine client items fixed, three provider items released:" " changed %.4f accuracy %.4f" % (changed(three, BASE), accuracy(three))) print("fixable items 39 + 9 + 9 =", 39 + 9 + 9, ", unfixable 3 , total", 39 + 9 + 12) ``` ``` only maximum output length changed 0.0000 accuracy 0.9125 only instruction format word changed 1.0000 accuracy 0.5525 both together changed 1.0000 accuracy 0.2675 nine client items fixed, three provider items released: changed 0.1150 accuracy 0.8925 fixable items 39 + 9 + 9 = 57 , unfixable 3 , total 60 ``` The only zero in the table was **the maximum output length**. When the limit is lowered from twenty-four to ten, no response changes, because single-label responses are no longer than two tokens. When preparing a pinning list, dropping this item looks plausible. The second and third rows show why that is wrong. When the format word is released alone, accuracy falls to 0.5525; when **both** are released together, it falls to **0.2675**. The **0.2850** in between is the account of the item that alone moves nothing: the responses that turned into sentences are cut at ten tokens, and the label cannot be read in the cut text. Two more courses had reached the same result: the Visualization, Reporting and Analysis at Scale course measured that an item that alone does not move the report took the ratio from 87.168 to 94.432 together with another item, and the Deep Learning course measured that a number of rounds that alone moves zero shifted two subscribers together with the batch size. **An item that does not move alone is not an item that need not be pinned.** #### Three Unpinnable Items The last two rows give the closing number of this topic. Reproducing a model call requires pinning **60** items: the 39 decisions of the preparation ledger, the 9 items of the runtime environment and the 12 items counted in this lesson. **57** of these are in our hands. The remaining **three** are the sampling state, the model version and the rate limit threshold; they are in the hands of the provider and cannot be pinned by writing them to any file. When these three are released together while the nine client items are pinned, **0.1150** of the responses move and accuracy falls from 0.9125 to **0.8925**. The drop is **0.0200**. This is exactly the threshold of the previous two lessons: the 0.0200 in the cost ledger and the 0.0200 in the outage table are not noise, they are the **name** of these three items. The threshold is not an arbitrarily chosen number, it is the measured share of the unpinnable items. The conclusion from this is structurally different from that of the previous two courses. A report can be pinned with 59 of its 59 items, a training run with 67 of its 67; both are fully reproducible. A model call cannot be pinned. This is why determinism is not a goal but a **limit**, and the best that can be reached is for the movement to be **measured**. Its counterpart in practice is a single rule: the run-to-run difference is written next to every measurement report, and no gain below that difference counts as a gain. #### Summary - Reproducing a model call requires 60 items: the 39 inherited preparation decisions, the 9 environment items and the 12 items specific to the call. - Of the twelve items, 11 move the responses when released alone; the largest is the format word of the instruction and it drops accuracy from 0.9125 to 0.5525. - Items that change no meaning also move responses: whitespace layout 0.0700, client normalization 0.1300, label set order 0.1100. - A changed response and a broken response are counted separately: whitespace layout moves 0.0700 and raises accuracy, while distribution temperature moves 0.0600 and lowers it by 0.0550. - The maximum output length, which alone moves 0.0000, drops accuracy from 0.5525 to 0.2675 when released together with the format word. - Three items are on the provider and cannot be pinned; when released together they move 0.1150 of the responses and accuracy falls by 0.0200, that is, the threshold of the previous two lessons is measured. #### Next Step In fourteen lessons the call was handled **inside a product**. What the message-based interface offers, what streaming changes, how many of the tool calls are valid, what the rate limit costs, what the cache gains, which class the latency falls to, which items the bill consists of, how the outage silently lowers quality and why the same input gives two outputs were counted. In all of these measurements the one making the call was an application, and the person who wrote that application stayed outside the measurement. Yet the same model also works at the developer's own desk: while completing code, while explaining an existing function, while converting a file, even while taking over a multi-step task. The next topic carries the measure there and asks the same question: in a fictional codebase, in which task class does the bare call collapse, how many tokens does each context cost, and how many of the accepted suggestions are actually correct. ## AI in the Development Workflow Counting from the defensive side what a code assistant gains on a fictional codebase and what it misses: acceptance rate by task class across completion, explanation, and transformation uses, and the point where a bare call collapses without scaffolding; tokens and hit rate as the number of files grows, and the point where irrelevant context lowers the hit rate; how often a multi-step handoff must stop for an error to be caught before it spreads; the defect each of the acceptance check suite, the pattern scan, and the fabricated dependency existence check catches, and the residual none of them catches; the rate at which generated tests catch planted defects and the share of vacuous tests; the number of fields that fall into a secret class in text that enters the context, how many a filter removes, and the share of wrong among accepted suggestions. ### Code Assistants Source: https://academia.sh/en/courses/model-apis/code-assistants Measuring completion, explanation, and transformation uses on a fictional codebase by acceptance rate: the call without context leads only seven of twenty-four tasks to acceptance, and the collapse is in completion (1/8) and transformation (2/8); the only class left standing is explanation (4/8). Open-file context gives 10/24, but the gain is narrow because the run-to-run difference of the same setup is 2/24. The whole codebase raises it to 18/24 at a cost of 1104 tokens rising to 4224 tokens; acceptance per thousand tokens falls from 6.34 to 4.26. In the run-independent expected acceptance column, transformation, even with the whole codebase (0.4675), stays below explanation with no context at all (0.6450). The lessons of this course so far have always treated the call inside a product: a subscriber writes a short support ticket, a client sends it to a model endpoint, and what was measured was the tokens, retries, cache hits, and outage share of that pipeline. That the same call is made in one more place was never counted: **at the developer's own desk**, on code that has not yet reached any subscriber. The measure there is different. The one who reads the response is not a subscriber but the person who writes the code; and that person either takes the response into use as it is, or throws it away and writes their own. This is the measure of this lesson: **acceptance**. The lesson builds a fictional codebase, measures three usage classes on the same set of tasks, and counts **the class in which the bare call collapses**. - **AD1.** The codebase is **fictional**: the measurement network's own helper scripts. It is not any real repository, package, framework, or product. **Five modules, 53 lines.** - **AD2.** The code assistant is a **simulator**; no real assistant, editor, or plugin is called. The seed is **20260218**, and `random` is not used. - **AD3.** The measure is **acceptance**: if the developer takes the suggestion into use it is an acceptance, if the developer rewrites it, a rejection. Whether the suggestion is actually correct is a separate question and is not asked in this lesson. - **AD4.** Three usage classes (`tamamlama` for completion, `aciklama` for explanation, `donusturme` for transformation in the output): **completion** (writing a missing body), **explanation** (saying what an existing piece does), **transformation** (converting an existing piece into another form). Eight fictional tasks per class, **twenty-four** in total. - **AD5.** The base acceptance probability differs by class: explanation **0.72**, completion **0.34**, transformation **0.18**. The reason is structural: the input of an explanation is **inside** the prompt; completion and transformation have to fit the **rest** of the codebase. - **AD6.** Context overlap raises acceptance, **an irrelevant file lowers it**, and task difficulty lowers it. The three are combined into a single score, and the score is clipped between zero and one. - **AD7.** The cost is tokens: base **40**, **26** per file in the context, **18** extra for transformation. The latency class is the same in all three setups: **single call**, one per task. - **AD8.** The run-independent measure is **expected acceptance** (the mean of the scores); the counted acceptance is printed separately for two runs (seed **20260218** and seed **20260219**). #### The Fictional Codebase The measurement network's field crew carries a handful of helper scripts that turn meter readings into a report. These scripts are not a product; they are short files with which a team does its own work, and they are **fictional**. The five modules are held in a dictionary; this way they can be shown in a single block and the context can be selected **by file name**. ```python # FICTIONAL CODEBASE -- not a real repository, package, or product. The measurement # network's own helper scripts: five modules. Seed 20260218. CODE = { "okuma.py": """ def parse_line(line): p = line.split(";") if len(p) != 4: return None if not p[3].lstrip("-").isdigit(): return None return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])} def usage(previous, current): diff = current - previous if diff < 0: diff = diff + 100000 return diff """, "tarife.py": """ TIERS = ((10, 4), (25, 7), (40, 11)) ABOVE = 16 def cost(m3): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper return total + remaining * ABOVE """, "denetim.py": """ VALID_REGIONS = ("east", "west", "north", "south", "central") def is_valid(record): if record is None: return False if record["index"] < 0: return False return record["region"] in VALID_REGIONS def filter_valid(records): return [r for r in records if is_valid(r)] """, "ozet.py": """ def region_total(records, usages): total = {} for r, u in zip(records, usages): total[r["region"]] = total.get(r["region"], 0) + u return total def region_count(records): count = {} for r in records: count[r["region"]] = count.get(r["region"], 0) + 1 return count """, "bicim.py": """ def row(region, m3, amount): return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9) def report(title, rows): return "\\n".join([title] + rows) """, } def load(code): modules = {} for name, source in code.items(): scope = {} exec(source, scope) modules[name] = scope return modules MODULES = load(CODE) SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()} print("module lines") for n in CODE: print(f" {n:<10}{SIZE[n]:>4}") print(f" {'total':<10}{sum(SIZE.values()):>4}") READINGS = [ # FICTIONAL meter readings "A017;east;2026-01;41250", "A017;east;2026-02;41312", "B204;west;2026-02;99800", "B204;west;2026-03;300", "C088;north;2026-02;7130", "C088;north;2026-03;7166", "D311;outside;2026-03;5000", "E402;south;2026-03;corrupt", ] records = [MODULES["okuma.py"]["parse_line"](s) for s in READINGS] clean = MODULES["denetim.py"]["filter_valid"](records) print("\nread", len(records), "valid", len(clean)) pairs = [(clean[i], clean[i + 1]) for i in range(0, len(clean) - 1, 2)] usages = [MODULES["okuma.py"]["usage"](a["index"], b["index"]) for a, b in pairs] latest = [b for _, b in pairs] totals = MODULES["ozet.py"]["region_total"](latest, usages) rows = [MODULES["bicim.py"]["row"](r, m, MODULES["tarife.py"]["cost"](m)) for r, m in sorted(totals.items())] print(MODULES["bicim.py"]["report"]("region m3 amount", rows)) ``` ``` module lines okuma.py 13 tarife.py 13 denetim.py 11 ozet.py 11 bicim.py 5 total 53 read 8 valid 6 region m3 amount east 62 662 north 36 266 west 500 7670 ``` The five modules are interdependent and build a single pipeline: `okuma.py` (reading) turns a line into a record, `denetim.py` (checks) discards the invalid ones, `ozet.py` (summary) totals by region, `tarife.py` (tariff) computes the tiered cost, `bicim.py` (render) writes the aligned row. Two of the eight readings are eliminated: one carries an unrecognized region, and one has an index that is not a number. The 500 cubic meters of the west region is also the result of a rule; the meter wrapped from 99800 to 300, and `usage` compensates for that wrap. **All 53 lines fit inside a single prompt.** Keeping the codebase small is deliberate: here "enlarging the context" is really an option and its cost can be counted. In a large repository this option does not exist anyway, but the direction of the measure is the same. #### Three Usage Classes and Acceptance Twenty-four fictional tasks are divided into three classes. **Completion** asks for writing a body: a new function that takes a region count, a missing branch, an edge case. **Explanation** asks what an existing piece does: why `usage` adds 100000, what the `TIERS` tuple encodes. **Transformation** converts an existing piece into another form: returning a tuple instead of a dictionary, turning a loop into a comprehension, merging two modules under a common signature. The acceptance rate of these three classes is not the same, and the reason is written into the simulator as the base probability. The **input of an explanation is inside the prompt**: the piece to be explained has already been sent, and nothing needs to be known from outside. Completion and transformation, on the other hand, have to fit the **unseen part** of the codebase: which key names are used, what each function returns, where which invalidity rule is applied. ```python # ASSISTANT SIMULATOR -- FICTIONAL. No real code assistant is called. # Acceptance is computed from the task class and the context overlap; random is not used. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py") REQUIRED = { # task number -> required files 0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py"), } TASKS = [ # 24 FICTIONAL tasks: 3 classes x 8 {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8) ] BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def assistant(task, context, seed=SEED): r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context)))) overlap = sum(1 for f in task["required"] if f in context) / len(task["required"]) irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) accepted = r() < score tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0) return {"accepted": accepted, "score": score, "tokens": tokens} SETUPS = (("none", ()), ("open file", ("okuma.py",)), ("whole codebase", FILES)) def measure(context, label, seed=SEED): t = [x for x in TASKS if x["label"] == label] s = [assistant(x, context, seed) for x in t] return (sum(x["accepted"] for x in s), sum(x["score"] for x in s) / len(s), sum(x["tokens"] for x in s)) print("task count", len(TASKS), " example:", ", ".join(t["name"] + " -> " + "+".join(t["required"]) for t in TASKS[:2])) print("\ncontext class expected accept A accept B") for name, ctx in SETUPS: for label in CLASSES: acc_a, p, _ = measure(ctx, label) acc_b, _, _ = measure(ctx, label, SEED + 1) print(f"{name:<18}{label:<12}{p:>8.4f}{acc_a:>7}/8{acc_b:>7}/8") print("\ncontext total A total B tokens accepted/1000 tokens") for name, ctx in SETUPS: total_a = sum(measure(ctx, label)[0] for label in CLASSES) total_b = sum(measure(ctx, label, SEED + 1)[0] for label in CLASSES) tokens = sum(measure(ctx, label)[2] for label in CLASSES) print(f"{name:<18}{total_a:>7}/24{total_b:>8}/24{tokens:>10}{1000 * total_a / tokens:>17.2f}") ``` ``` task count 24 example: tamamlama-1 -> okuma.py, tamamlama-2 -> tarife.py context class expected accept A accept B none tamamlama 0.2650 1/8 1/8 none aciklama 0.6450 4/8 6/8 none donusturme 0.1050 2/8 0/8 open file tamamlama 0.3954 3/8 1/8 open file aciklama 0.7417 5/8 6/8 open file donusturme 0.2429 2/8 1/8 whole codebase tamamlama 0.6275 6/8 7/8 whole codebase aciklama 0.9813 8/8 8/8 whole codebase donusturme 0.4675 4/8 5/8 context total A total B tokens accepted/1000 tokens none 7/24 7/24 1104 6.34 open file 10/24 8/24 1728 5.79 whole codebase 18/24 20/24 4224 4.26 ``` #### Where the Bare Call Collapses The first three rows are the table of the bare call: there is no context, and only the task description is sent. **Seven** of the twenty-four tasks are accepted, and **seven** in the second run as well. Even though the total looks constant, the distribution is not constant, and the real information is in the distribution. The collapse is in two classes. **Completion 1/8**, the same in both runs; expected acceptance **0.2650**. **Transformation 2/8** and **0/8** in the second run; expected acceptance **0.1050**, the lowest number in the table. The only class left standing is **explanation**: 4/8 and 6/8, expected acceptance **0.6450**. A developer can ask an assistant without context "what does this function do" and use more than half of the answers; when asking the same assistant to "complete this function", the developer will throw away seven of eight attempts. The reason is structural, and the number confirms it. In an explanation task, everything that needs to be known has been put into the prompt. In a completion task, however, the keys of the dictionary that the body to be written will return, the name of the function it will call, and the invalidity rule it must follow are **in other files**. Because the assistant cannot see them, it produces code that is plausible but does not fit; plausible code is also rejected code. Within a class there is also a distribution, and a single number hides it. The eight tasks of each class are spread over four distinct difficulty levels, and difficulty is subtracted directly from the acceptance score; the mean of the class lies somewhere between the easiest task and the hardest one. The run-to-run difference measures a separate thing: the same task, the same context, and the same score can reach two distinct decisions under two seeds. This is why the **expected acceptance** column is placed next to the counted acceptance in the table; the counted acceptance depends on the run, the expected acceptance on the setup itself. #### What Context Adds and What It Costs The second setup is what a developer does most readily: also sending the file that is open at that moment. Here that file is `okuma.py`, and it is the same for every task. The total rises **from 7/24 to 10/24**, a gain of **3/24**. The run-to-run difference of the same setup is **2/24** (10 versus 8). The gain is only one and a half times the difference; moreover, in the completion column the direction **reverses** (3/8 and 1/8). A single fixed file is not scaffolding; when it is relevant to the task it helps, and when it is not it counts as irrelevant context. The third setup is not ambiguous. When the whole codebase is sent, the total is **18/24**, and **20/24** in the second run; the gain over the baseline is **11/24**, more than five times the run-to-run difference. All three classes rise: completion from 1/8 to 6/8, explanation from 4/8 to 8/8, transformation from 2/8 to 4/8. The cost column writes the price of this gain. The number of calls is **24** in all three setups and the latency class stays the same as **single call**; only the tokens change: **from 1104 to 4224**, about **3.8 times**. Acceptance per thousand tokens falls **from 6.34 to 5.79, and from there to 4.26**. The scaffolding improved the work and **lowered the yield per unit**; both are true at the same time, and both have to be written down. Finally, the run-independent column fixes an ordering. Transformation stays at **0.4675** even when the **whole codebase** is given; explanation is at **0.6450 with no context at all**. Adding context does not change the places of the classes; it moves all of them up together. Describing a code assistant as "good" or "bad" with a single number is therefore misleading: the number changes with the task class, and the hard class that receives the most context can remain behind the easy class that receives the least. #### Summary - The fictional codebase is five modules and 53 lines; reading, validation, summary, tariff, and format build a single report pipeline, and the whole of it fits inside a prompt. - The measure is acceptance and it is counted separately for the three usage classes; the difference between classes is structural, because the input of an explanation is inside the prompt, while the input of completion and transformation is outside. - The call without context gives 7/24 acceptance; the collapse is in completion (1/8) and transformation (2/8), and the only class left standing is explanation (4/8, expected 0.6450). - Adding a single fixed file gives 10/24, but the run-to-run difference of the same setup is 2/24 and the direction reverses in the completion column: fixed context is not scaffolding. - The whole codebase raises it to 18/24; the cost rises from 1104 tokens to 4224 tokens, and acceptance per thousand tokens falls from 6.34 to 4.26. The number of calls and the latency class do not change. - Context does not change the order of the classes: transformation with full context (0.4675) stays below explanation without context (0.6450). #### Next Step In this lesson the context was given three times as **fixed**: none, always the same file, or all of it. None of the three is a choice. At a real desk, the context is rebuilt for every task, and two questions are asked: which files, and how many. The table above leaves both of these questions unanswered; it shows only the ends. The next lesson fills the gap: as the number of files increases one by one, how the tokens grow, where acceptance peaks, and what **a file irrelevant to the task** does to acceptance. What will be measured there is not the **length** of the context but its **selection**. ### Providing Context Source: https://academia.sh/en/courses/model-apis/providing-context Measuring enlarging the context against selecting the context side by side on the same fictional codebase: sending all the files raises expected acceptance to 0.6921 and spends 4224 tokens and 120 files, while sending only the required files raises it to 0.8233 and spends 2274 tokens and 45 files. Every irrelevant file takes 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file. The token cost of the context provider is independent of its hit rate and is fixed at 2274, but expected acceptance varies between 0.4960 and 0.8233; for the provider to beat sending the whole codebase, it has to find the required files in about three of four tasks. The previous lesson gave the context three times as **fixed**: none, always the same file, or all of it. The distance between the ends was large — from seven of twenty-four tasks to eighteen — but none of the three was a choice. At a developer's desk the context is rebuilt for every task, and two separate questions are asked: **which files** and **how many**. This lesson separates those two questions. First the context is **enlarged**: the same set of files is given to everyone and its size grows one by one. Then the context is **selected**: each task receives only the files that task requires. The two curves are placed side by side on the token axis, and a third number is added on top: **what a file irrelevant to the task does to acceptance**. - **AD9.** The codebase is the same **fictional** codebase built with its bodies in the previous lesson: five modules, **53 lines**. Only its structure is needed here, because the context is selected **by file name**. - **AD10.** The set of tasks and the assistant simulator are also the same: twenty-four fictional tasks, three classes, seed **20260218**, no `random`. - **AD11.** The effect of context on acceptance has two terms: **overlap** (how many of the required files are in the context) raises acceptance, **irrelevance** (how many of the files in the context are unnecessary) lowers it. - **AD12.** The cost is a flat **26 tokens** per file; file size does not enter the cost. In a real context a large file is expensive, so the cost curve here is **optimistic**. - **AD13.** The number of files sent is counted separately; it is a total for the twenty-four tasks and a volume measure independent of tokens. - **AD14. The context provider is fictional** and has a single setting: the **hit rate**, that is, the share of tasks in which it finds the required set correctly. When it misses, it drops one required file and puts an irrelevant file in its place; **the file count does not change**, only which file it is. - **AD15.** The provider's own draw is **fixed by the seed**; in the second run only the uncertainty of the assistant changes. This way the difference between the two runs is attributed only to the model, not to the selection. - **AD16.** The run-independent measure is again **expected acceptance**; the counted acceptance is printed separately for two seeds. The latency class is **single call** in all setups, one per task. #### The Codebase Structure and the Provider For context selection, what is needed is not the bodies of the codebase but its **structure**: which modules exist, how long they are, and which names they expose to the outside. This is also the plane a context provider works on. ```python # THE SAME FICTIONAL CODEBASE -- five modules, 53 lines; the bodies were built in the previous lesson. # Only its structure is needed here: size and exposed names. SIZE = {"okuma.py": 13, "tarife.py": 13, "denetim.py": 11, "ozet.py": 11, "bicim.py": 5} EXPOSED = { "okuma.py": ("parse_line", "usage"), "tarife.py": ("TIERS", "ABOVE", "cost"), "denetim.py": ("VALID_REGIONS", "is_valid", "filter_valid"), "ozet.py": ("region_total", "region_count"), "bicim.py": ("row", "report"), } FILES = tuple(SIZE) SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") REQUIRED = { 0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py"), } TASKS = [ # 24 FICTIONAL tasks: 3 classes x 8 {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8) ] BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def assistant(task, context, seed=SEED): # SIMULATOR -- FICTIONAL r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context)))) overlap = sum(1 for f in task["required"] if f in context) / len(task["required"]) irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) accepted = r() < score tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0) return {"accepted": accepted, "score": score, "tokens": tokens} def provider(task, hit, seed=SEED): """FICTIONAL context provider. When it hits, it finds the required set; when it misses, it drops one required file and puts an irrelevant file in its place. The file count stays the same.""" r = rng(seed ^ mix("saglayici|" + task["name"])) if r() < hit: return tuple(task["required"]) extra = tuple(f for f in FILES if f not in task["required"])[:1] return tuple(task["required"][1:]) + extra print("module lines exposed names") for f in FILES: print(f" {f:<10}{SIZE[f]:>4} {', '.join(EXPOSED[f])}") distribution = {} for t in TASKS: distribution[len(t["required"])] = distribution.get(len(t["required"]), 0) + 1 print(f"\ntasks {len(TASKS)}, required file count distribution {dict(sorted(distribution.items()))}" f", total required files {sum(len(t['required']) for t in TASKS)}") print("example task:", TASKS[6]["name"], "->", "+".join(TASKS[6]["required"]), " tokens", assistant(TASKS[6], TASKS[6]["required"])["tokens"], "/ with the whole codebase", assistant(TASKS[6], FILES)["tokens"]) ``` ``` module lines exposed names okuma.py 13 parse_line, usage tarife.py 13 TIERS, ABOVE, cost denetim.py 11 VALID_REGIONS, is_valid, filter_valid ozet.py 11 region_total, region_count bicim.py 5 row, report tasks 24, required file count distribution {1: 6, 2: 15, 3: 3}, total required files 45 example task: tamamlama-7 -> ozet.py+okuma.py+tarife.py tokens 118 / with the whole codebase 170 ``` More than half of the tasks ask for **two** files, six ask for one, three ask for three. The total need of the twenty-four tasks is **45 files**. Sending the whole codebase to every task, on the other hand, means 120 files, which is **more than twice** what is needed. In the example task the difference looks small on its own: 118 tokens against 170. Over twenty-four tasks the same difference accumulates. #### Enlarging versus Selecting The two curves are printed in the same table. The **batch** rows give everyone the same first `k` files; the **selected** rows give each task only its own required files, and irrelevant files are added on top in turn. ```python def measure(select, seed=SEED): accepted = tokens = files = 0 score = 0.0 for t in TASKS: ctx = select(t) s = assistant(t, ctx, seed) accepted += s["accepted"] score += s["score"] tokens += s["tokens"] files += len(ctx) return accepted, score / len(TASKS), tokens, files def show(name, select): a = measure(select) acc_b = measure(select, SEED + 1)[0] print(f"{name:<26}{a[3]:>7}{a[1]:>11.4f}{a[0]:>8}/24{acc_b:>8}/24{a[2]:>10}") print("setup files expected accept A accept B tokens") for k in range(6): show(f"batch {k}", lambda t, b=FILES[:k]: b) show("selected (required only)", lambda t: t["required"]) for n in (1, 2, 3): def select(t, n=n): extra = tuple(f for f in FILES if f not in t["required"])[:n] return tuple(t["required"]) + extra show(f"selected + {n} irrelevant", select) print("\nprovider hit rate found required share expected accept A accept B tokens") for h in (0.00, 0.25, 0.50, 0.75, 1.00): def select(t, i=h): return provider(t, i) a = measure(select) acc_b = measure(select, SEED + 1)[0] found = sum(sum(1 for f in t["required"] if f in select(t)) for t in TASKS) print(f"{h:>13.2f}{found / 45:>22.4f}{a[1]:>10.4f}{a[0]:>8}/24{acc_b:>8}/24{a[2]:>10}") ``` ``` setup files expected accept A accept B tokens batch 0 0 0.3383 7/24 7/24 1104 batch 1 24 0.4600 10/24 8/24 1728 batch 2 48 0.5402 12/24 13/24 2352 batch 3 72 0.6013 15/24 13/24 2976 batch 4 96 0.6608 15/24 15/24 3600 batch 5 120 0.6921 18/24 20/24 4224 selected (required only) 45 0.8233 19/24 19/24 2274 selected + 1 irrelevant 69 0.7833 19/24 18/24 2898 selected + 2 irrelevant 93 0.7433 20/24 18/24 3522 selected + 3 irrelevant 114 0.7029 17/24 20/24 4068 provider hit rate found required share expected accept A accept B tokens 0.00 0.4667 0.4960 13/24 10/24 2274 0.25 0.6222 0.5796 15/24 12/24 2274 0.50 0.7333 0.6404 15/24 14/24 2274 0.75 0.8222 0.7050 17/24 15/24 2274 1.00 1.0000 0.8233 19/24 19/24 2274 ``` #### Reading the Curve The batch rows confirm intuition: adding files raises expected acceptance **from 0.3383 to 0.6921** and never lowers it at any step. The cost is also regular — every step is exactly **624 tokens**, because one more file enters each of the twenty-four tasks. The curve is the number behind the claim that enlarging the context works, and the claim is true. The curve has a weakness that does not show in the table: **order**. The batch rows add the files in the order of the module tuple, so which file the second row carries is not a decision but an accident. With a different order, a different curve would emerge between the same two endpoints. The selected setup has no such dependence, because it never asks the question of order. If the number of a setup is sensitive to an arbitrary decision that is not listed, that number should not be read on its own. The second weakness is on the cost side and it is **in favor of selection**. In this measurement every file costs 26 tokens regardless of its size; yet the longest module of the codebase is more than two and a half times longer than the shortest. In a real context, sending in batch also pays for the cost of the long files, while selected sending often does not. So the difference below is a **lower bound**. The selected row displaces the same claim. When only the required files are sent, expected acceptance is **0.8233**; **0.1312** higher than the **0.6921** of the whole codebase. It does this with **45 files** and **2274 tokens**: **a little more than a third** of the file volume, **54%** of the tokens. A setup that wins on both axes at once is rarely seen, and the reason here is clear: sending the whole codebase also sends the required files, but alongside them it carries **on average three irrelevant files**. The comparison at equal cost makes this sharper. The **batch 2** setup spends 2352 tokens and gives **0.5402**; the selected setup gives **0.8233** with 2274 tokens, that is, slightly fewer. A difference of **0.2831** at almost the same budget. If the token budget is a design constraint, where you spend that budget determines more than how much you spend. The counted acceptance columns carry a warning. The selected row is **19/24** and the whole codebase is **18/24**; the difference is a single task. But in the second run the ranking reverses: 19/24 against **20/24**. By counted acceptance these two setups are **indistinguishable**, because the difference stays within the difference between two runs. What makes the distinction is the run-independent column. This is a direct application of the course's measure rule: a number that moves between two runs does not support a decision when it is read from a single run. #### An Irrelevant File Has No Threshold The last three selected rows answer a single question: after the right files have already been sent, what does **one more file** do. The answer is regular and one-directional. Expected acceptance falls **from 0.8233 to 0.7833, to 0.7433, and to 0.7029**; every irrelevant file takes exactly **0.0400**. At the same time the cost rises **from 2274 to 4068**; 624, 624, and 546 tokens per step. The "point where the hit rate falls" being looked for here is not a threshold: it **starts with the first irrelevant file**. There is no free irrelevant file, because the irrelevance term is linear and the overlap term is already saturated. Adding three irrelevant files increases the tokens by **79%** and takes **0.1204** from expected acceptance — this is the plainest row in which the added machinery makes the work worse. Counted acceptance is misleading here too, and it can be seen why: it goes 19, 19, 20, 17, and in the second run 19, 18, 18, 20. Someone looking at the first run could say "two irrelevant files raised acceptance". The expected acceptance column closes that reading. In a set of twenty-four tasks the weight of a single acceptance is 0.0417, that is, larger than the real effect of an irrelevant file; counted acceptance does not have the resolution to show this detail and requires a larger set of tasks. #### The Provider's Hit Rate Is Independent of Cost The second table turns context selection into a capability. At every hit rate the provider sends the **same number of files**, so the token column is **2274** in all five rows. The only thing that changes is **which** files are sent, and expected acceptance moves **between 0.4960 and 0.8233**. Context quality is not bought with tokens here; the measured difference comes entirely from the selection itself. The last row of this table is the same as the selected row of the previous table, and it has to be: a **hit rate of 1.00** is a provider that knows the required set in every task. Such a provider is a measurement fiction, not a setup. Knowing which files are required is information very close to the task having been solved; a real provider **guesses** it from name similarity, call relations, or past changes. This is why 0.8233 is not a target but an **upper bound**, and the table shows not the cost of getting close to that bound but the cost of failing to get close. Even at a hit rate of zero, the share of required files found is **0.4667**, because when one of the two files of a two-file task is dropped, the other stays in place. So a bad provider is not zero, only weak — and an expected acceptance of 0.4960 is above sending no context at all (0.3383). The real decision point is in the middle. Sending the whole codebase gives **0.6921**. The provider cannot beat it at a 0.50 hit rate (**0.6404**) and beats it at a 0.75 hit rate (**0.7050**). So a context provider that can find the required files in **about three of four tasks** is better than sending everything; if it cannot, it is worse, and because it is cheap, this is hard to notice. Since the cost column is fixed, the provider's badness does not show on the invoice, only in acceptance. #### Summary - Enlarging the context raises expected acceptance from 0.3383 to 0.6921 and every step costs exactly 624 tokens; the curve is regular and never falls at any step. - Selecting the context gives 0.8233 on the same set of tasks; it does this with 45 files and 2274 tokens, that is, 0.1312 higher than the whole codebase and at about half the cost. - At almost equal budget the difference is even larger: batch of two files 0.5402 with 2352 tokens, selected context 0.8233 with 2274 tokens. - Every irrelevant file takes exactly 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file and loses 0.1204 at three files. - Counted acceptance does not distinguish in either of these decisions (19/24 against 18/24, reversed in the second run); what makes the distinction is the run-independent expected acceptance column. - The token cost of the context provider is independent of its hit rate and fixed at 2274; for it to beat sending the whole codebase, it has to find the required files in about three of four tasks. #### Next Step In these two lessons every task was **a single call**: the context went, the suggestion came, the developer accepted it or did not. The decision point was single and right there. A development job often does not stand like that — reading a file, changing a function, updating the callers, and reformatting the report, when delegated with a single request, produce a **multi-step** job, and the developer no longer looks at every step but at the place where they stop. The next lesson measures this stopping place: when stopping every how many steps catches an error **before it propagates**, how many steps are wasted, and how many calls the stopping itself costs. ### Agentic Coding Source: https://academia.sh/en/courses/model-apis/agentic-coding Measuring the checkpoint interval when a multi-step job is handed off: when the 480 steps of sixty fictional jobs are run at six different intervals, stopping at every step gives 478/480 correct steps but asks for 479 stops, stopping every eight steps drops to 60 stops and falls to 355/480. Wasted steps per caught error rise from 1.00 to 5.64; missed errors from 1 to 21. The number of calls is almost constant across the six intervals (between 549 and 598), so the cost of stopping is not calls but human attention. Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9), so the most accurate interval and the cheapest interval are not the same. In the previous two lessons every task was **a single call**: the context went, the suggestion came, the developer accepted it or did not. The decision point was single and in the same place as the suggestion. When reading a file, changing a function, updating the callers, and reformatting the report are delegated with a single request, this arrangement breaks: the tool takes steps one after another, each step is built **on top of** the previous one, and the developer no longer looks at every step but **at the place where they stop**. This lesson measures that stopping place and nothing else. How the tool plans, what it remembers, and how it closes its loop are not covered here; those are the subject of the AI Agents course of this curriculum. The only question here is this: **when stopping every how many steps, is the error caught before it propagates, and what does that cost.** - **AD17.** The setup is the same as in the previous two lessons: the **fictional** codebase (five modules, 53 lines), twenty-four fictional tasks, and the assistant **simulator**. No real agentic coding tool is run. - **AD18.** The multi-step job: **60 fictional jobs, 8 steps each**, **480 steps** in total. The context of each step is **selected**, that is, only the files that step requires are sent. - **AD19. Propagation rule:** if a rejected step goes unnoticed, the later steps are built on top of it and are counted as **contaminated**. This is the only structural fact this lesson measures. - **AD20.** A **checkpoint** is placed every `d` steps and at the end of the job; at the stop, the human looks at that window. - **AD21. The detection share falls as the window grows:** `0.95 - 0.07 x (d - 1)`, lower bound `0.40`. Reading an eight-step change in one go is harder than reading a single step. - **AD22.** A caught error is **rerun**: the steps from the faulty step to the stop are called again and assumed to be fixed. These are **wasted steps**. - **AD23.** An error that is not caught **enters the baseline**: the remaining steps of the job are contaminated and later stops cannot see it, because what is looked at is not the accepted baseline but the new work. - **AD24.** The cost has three separate items: **stop count** (the number of times the human looks), **calls**, and **tokens**; each stop also adds a **50-token** status summary. - **AD25.** The delivery measure is the **correct step**: in a job with a missed error, the number of correct steps extends only up to the place where the error is. - **AD26.** Step results are produced with the same seed independently of the interval; the only thing that changes between intervals is **where it stops**. The second run uses seed **20260219**. #### The Trace of a Job First a single job is run with two different intervals. The same eight steps, the same results; only the place of the stops changes. ```python # THE SAME FICTIONAL SETUP -- codebase of five modules (53 lines), 24 fictional tasks, # assistant simulator. No real agentic coding tool is run. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py") REQUIRED = { 0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py"), } TASKS = [ {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8) ] BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def assistant(task, context, seed=SEED): # SIMULATOR -- FICTIONAL r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context)))) overlap = sum(1 for f in task["required"] if f in context) / len(task["required"]) irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) accepted = r() < score tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0) return {"accepted": accepted, "score": score, "tokens": tokens} JOB_COUNT, STEPS, STATUS_TOKENS = 60, 8, 50 def detection(d): """The human's share of seeing the error falls as the checkpoint window grows.""" return max(0.95 - 0.07 * (d - 1), 0.40) def handoff(j, d, seed=SEED, verbose=False): """Running a single multi-step job with a stop every d steps.""" count = {"calls": 0, "tokens": 0, "stops": 0, "caught": 0, "missed": 0, "wasted": 0, "correct": 0} faulty, escaped = None, False for k in range(STEPS): task = TASKS[(j * STEPS + k) % len(TASKS)] result = assistant(task, task["required"], seed ^ mix("adim|" + str(j) + "|" + str(k))) count["calls"] += 1 count["tokens"] += result["tokens"] if escaped: if verbose: print(f" step {k} {task['name']:<14} contaminated (on top of a missed error)") continue if faulty is None and not result["accepted"]: faulty = k if verbose: label = "accept" if result["accepted"] else "REJECT" if faulty is not None and faulty < k: label = "contaminated" print(f" step {k} {task['name']:<14} {label}") if (k + 1) % d == 0 or k == STEPS - 1: count["stops"] += 1 count["tokens"] += STATUS_TOKENS if faulty is None: if verbose: print(f" STOP step {k}: clean") continue r = rng(seed ^ mix("denetim|" + str(j) + "|" + str(k))) if r() < detection(d): redo = k - faulty + 1 count["caught"] += 1 count["wasted"] += redo count["calls"] += redo count["tokens"] += redo * result["tokens"] if verbose: print(f" STOP step {k}: error at step {faulty} was seen," f" {redo} steps rerun") faulty = None else: count["missed"] += 1 escaped = True if verbose: print(f" STOP step {k}: not seen, the error entered the baseline") count["correct"] = faulty if escaped else STEPS return count print("job 0, step interval 4:") a = handoff(0, 4, verbose=True) print(" ->", {x: a[x] for x in ("correct", "wasted", "stops", "calls", "tokens")}) print("\njob 0, step interval 8:") b = handoff(0, 8, verbose=True) print(" ->", {x: b[x] for x in ("correct", "wasted", "stops", "calls", "tokens")}) ``` ``` job 0, step interval 4: step 0 tamamlama-1 accept step 1 tamamlama-2 accept step 2 tamamlama-3 REJECT step 3 tamamlama-4 contaminated STOP step 3: error at step 2 was seen, 2 steps rerun step 4 tamamlama-5 accept step 5 tamamlama-6 accept step 6 tamamlama-7 REJECT step 7 tamamlama-8 contaminated STOP step 7: error at step 6 was seen, 2 steps rerun -> {'correct': 8, 'wasted': 4, 'stops': 2, 'calls': 12, 'tokens': 1178} job 0, step interval 8: step 0 tamamlama-1 accept step 1 tamamlama-2 accept step 2 tamamlama-3 REJECT step 3 tamamlama-4 contaminated step 4 tamamlama-5 contaminated step 5 tamamlama-6 contaminated step 6 tamamlama-7 contaminated step 7 tamamlama-8 contaminated STOP step 7: error at step 2 was seen, 6 steps rerun -> {'correct': 8, 'wasted': 6, 'stops': 1, 'calls': 14, 'tokens': 1312} ``` The two traces carry the same eight steps and the underlying results are also the same: steps `2` and `6` are rejected. The only thing that changes is the place of the stops, but the trace is not the same. In the four-step interval the first error is caught after spreading two steps, and so is the second; four steps are rerun in total. In the eight-step interval the first error spreads **six steps** and the second error is **never seen** — step `6` is already contaminated at that point, so it is not counted as a separate error. Both deliver eight correct steps in the end, but one spends 12 calls and 1178 tokens, the other 14 calls and 1312 tokens. This single job is a good example but it is not a number: here the check saw the error in both. What happens when it does not see can only be read over enough jobs. #### Sweeping the Interval Sixty jobs, four hundred and eighty steps, six different checkpoint intervals. Step results are independent of the interval; the only thing that changes is the place of the stops. ```python def run_all(d, seed=SEED): total = {"calls": 0, "tokens": 0, "stops": 0, "caught": 0, "missed": 0, "wasted": 0, "correct": 0} for j in range(JOB_COUNT): for key, v in handoff(j, d, seed).items(): total[key] += v return total TOTAL_STEPS = JOB_COUNT * STEPS HEADER = (" every detect stops caught missed waste right step calls tokens tok/right") def show(d, seed): a = run_all(d, seed) print(f"{d:>6}{detection(d):>7.2f}{a['stops']:>7}{a['caught']:>11}{a['missed']:>7}" f"{a['wasted']:>6}{a['correct']:>8}/{TOTAL_STEPS}{a['calls']:>7}{a['tokens']:>10}" f"{a['tokens'] / a['correct']:>11.1f}") print("run A (seed 20260218)") print(HEADER) for d in (1, 2, 3, 4, 6, 8): show(d, SEED) print("\nrun B (seed 20260219)") print(HEADER) for d in (1, 2, 3, 4, 6, 8): show(d, SEED + 1) ``` ``` run A (seed 20260218) every detect stops caught missed waste right step calls tokens tok/right 1 0.95 479 90 1 90 478/480 570 78648 164.5 2 0.88 233 69 8 109 455/480 589 67790 149.0 3 0.81 172 52 11 104 441/480 584 64746 146.8 4 0.74 117 45 11 118 430/480 598 63320 147.3 6 0.60 104 23 23 69 367/480 549 57784 157.4 8 0.46 60 14 21 79 355/480 559 56414 158.9 run B (seed 20260219) every detect stops caught missed waste right step calls tokens tok/right 1 0.95 467 71 4 71 463/480 551 76070 164.3 2 0.88 224 52 11 79 430/480 559 64318 149.6 3 0.81 168 44 12 78 428/480 558 62010 144.9 4 0.74 107 29 14 85 392/480 565 59550 151.9 6 0.60 105 24 17 79 393/480 559 58790 149.6 8 0.46 60 18 16 101 393/480 581 59032 150.2 ``` #### The Measure of Propagation The first half of the question — is the error caught before it propagates — is the quotient of two columns in the table. Wasted steps divided by the number of caught errors give **how many steps the error spread before it was caught**. In the first run this number is, in order, **1.00 / 1.58 / 2.00 / 2.62 / 3.00 / 5.64**; in the second run **1.00 / 1.52 / 1.77 / 2.93 / 3.29 / 5.61**. The two runs say the same thing separately: when stopping at every step the error **does not spread**; when stopping every eight steps it is caught on average after spreading **five and a half steps**. The missed-error column gives the second half. Stopping at every step misses **1** error in the first run and **4** in the second; stopping every eight steps misses **21** and **16**. The reason is written in the table: the detection share falls from 0.95 to 0.46, because reading an eight-step change in one go is hard. One column is misleading here and should not be read: the **number of caught errors**. As the interval grows it falls from 90 to 14, as if the errors had decreased. What decreases is not the errors but **the work in which we can count errors**; since a missed error contaminates the rest of the job, no other error is counted in that job. The quality of delegated work cannot be measured by "how many errors were found". The only thing that can be measured is the **correct steps delivered**: from 478/480 to 355/480, and in the second run from 463/480 to 393/480. The same dependence shows in the stop column. In the run that stops at every step the number of stops is not 480 but **479**; in the second run **467**. The missing stops are the remainder of jobs whose error was missed: because the baseline is already contaminated, there is no clean work left to look at in those stops and they are not counted. The columns of the table are not independent of each other, because the delegated job is not made of independent steps either. #### The Cost of Stopping The cost has three items and the three do not go in the same direction. The **number of calls** stays between 549 and 598 across the six intervals, and between 551 and 581 in the second run — that is, almost constant. Stopping often creates reruns, stopping rarely creates longer reruns; the two cancel each other out. **Stopping is not a matter of calls.** **Tokens** fall steadily: from 78648 to 56414, that is, stopping rarely saves about **twenty-eight percent** of the total tokens. Most of this comes from the 50-token status summary added per stop; the difference between 479 stops and 60 stops alone is 20950 tokens. The real cost is in the third item and it is not paid in tokens: the **stop count**. Stopping at every step calls the human to the desk 479 times, stopping every eight steps 60 times. The difference is **eightfold**, and what is gained in return is 123 correct steps. For every correct step saved, the human has looked about **3.4 times** more. This trade-off is an engineering decision and it is written in the table; which side to choose depends on how reversible a step is. The last column shows a third optimization point. Tokens per correct step are lowest at the **three-step interval** in both runs: **146.8** and **144.9**. This is not the interval that gives the most correct steps (one); nor is it the interval that asks for the fewest stops (eight). Three separate criteria, three separate winners. Choosing a checkpoint interval is therefore not an optimization but a **declaration**: an interval chosen without saying which cost it lowers is not counted as measured. #### Summary - In a multi-step handoff, an error that is not caught spreads to later steps; the checkpoint is where this contamination is cut, and the only thing it measures is **where it stops**. - The number of steps spread before being caught is 1.00 when stopping at every step and 5.64 when stopping every eight steps; 1.00 and 5.61 in the second run. - Missed errors rise from 1 to 21 (from 4 to 16 in the second run), because the detection share falls from 0.95 to 0.46 as the window grows. - The number of caught errors is not a quality measure: it falls as the interval grows, because a missed error makes the rest of the job uncountable. The only thing that can be measured is the correct steps delivered (478/480 against 355/480). - The three items of cost separate: the number of calls is almost constant (between 549 and 598), tokens fall by twenty-eight percent, the stop count falls eightfold. The real price of frequent stopping is human attention. - Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9); the most accurate interval, the cheapest interval, and the interval with the fewest stops are three separate rows. #### Next Step In this lesson the checkpoint was treated as a **place**, and the detection share was squeezed into a single number: the human who stops sees the error with a probability, and if they do not, the error enters the baseline. What was inside that number was never opened. What exactly is looked at at the stop — is the code running, is it giving the right result, is it calling a dependency that does not exist, what was carried in the text that entered the context. The checkpoint was placed, but **what will be checked** was not said. The next lesson fills this gap and turns the detection share from an assumption into a measured number: independent checks are run on the same generated code, the defect each one catches is counted separately, and the **residual** that none of them catches is written by name. ### Reviewing Generated Code Source: https://academia.sh/en/courses/model-apis/reviewing-generated-code Passing twenty-four generated patches on a fictional codebase through three defensive checks: the acceptance check suite fails five of the six behavioral defect classes, the pattern scan eight of eight findings, and the dependency existence check nine of nine fabricated names. Twenty-six of twenty-eight defects are caught, two slip through as format errors, and one of the missed ones enters the codebase without receiving any warning. The previous lesson measured where to put the **checkpoint** in a multi-step handoff: how many steps apart the stops must be for the error to be caught before it spreads, and what stopping costs. A checkpoint is a place. **What to do** there was not said. The person who stops has a patch in front of them, and looking at that patch has its own scope, its own cost, and its own blind spot. This lesson splits that look into three separate checks and counts the **defect class each one reaches**: the codebase's own **check run**, the **pattern scan** of the generated text, and the **existence check** of the dependencies taken in. The general procedure of code review was measured in the Software Quality and Testing curriculum, and the procedure of security review in the Cybersecurity and the Application Security and DevSecOps curricula; it is not repeated here. The only thing added is **the identity of the producer**: the patch under review comes not from a human but from an uncertain component, so the **distribution** of defects is different, and the boundary of the area the checks cover has to be counted. - **AD31.** The codebase is **fictional**: the measurement network's own helper scripts. It is not any real repository, package, or product. **Five modules, 53 lines.** Seed **20260218**. - **AD32.** This lesson is not an attack lesson. **The code that runs is always on the defense side**: checking, scanning, counting, verifying. No exploit code, payload, or evasion procedure is written. - **AD33.** The generated patch has two parts: a changed **module** and a new **helper file**. No check calls the helper file; it has not yet been wired into the codebase. - **AD34.** The six **behavioral defects** placed on the codebase are single string substitutions in the source text; the sixth is a **format** defect and breaks the alignment of the output. - **AD35.** The acceptance check suite has **ten items** and is written inside the lesson. It is the codebase's own check suite, not a generated one. - **AD36.** The pattern scan looks for four strings. A finding is not proof but a **flag**; it points to the line a human should look at. - **AD37.** The existence check: if the name on an `import` line is neither among the codebase's modules nor among the allowed standard library names, it is **fabricated**. The fabricated names are fictional. - **AD38.** The counted unit is the **defect, not the candidate**. A candidate can carry more than one defect; a **flagged candidate** is one that receives a warning from at least one check. #### The Fictional Codebase and the Subject of Review A review needs a body to stand on. The body is this: five small modules of the measurement network, written for reading, tariff calculation, record validation, region summary, and output formatting. The modules are held as text and loaded into a separate namespace when they are to be run; this makes it cheap to run a **corrupted version** of the same codebase side by side. ```python # FICTIONAL codebase: the measurement network's own helper scripts. Not a real repository, # package, or product. Five modules, 53 lines. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CODE = { "okuma.py": """ def parse_line(line): p = line.split(";") if len(p) != 4: return None if not p[3].lstrip("-").isdigit(): return None return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])} def usage(previous, current): diff = current - previous if diff < 0: diff = diff + 100000 return diff """, "tarife.py": """ TIERS = ((10, 4), (25, 7), (40, 11)) ABOVE = 16 def cost(m3): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper return total + remaining * ABOVE """, "denetim.py": """ VALID_REGIONS = ("east", "west", "north", "south", "central") def is_valid(record): if record is None: return False if record["index"] < 0: return False return record["region"] in VALID_REGIONS def filter_valid(records): return [r for r in records if is_valid(r)] """, "ozet.py": """ def region_total(records, usages): total = {} for r, u in zip(records, usages): total[r["region"]] = total.get(r["region"], 0) + u return total def region_count(records): count = {} for r in records: count[r["region"]] = count.get(r["region"], 0) + 1 return count """, "bicim.py": """ def row(region, m3, amount): return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9) def report(title, rows): return "\\n".join([title] + rows) """, } def load(code): modules = {} for name, source in code.items(): scope = {} exec(source, scope) modules[name] = scope return modules CLASSES = ("tamamlama", "aciklama", "donusturme") FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py") REQUIRED = {0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py")} TASKS = [{"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8)] SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()} MODULES = load(CODE) print("codebase:", SIZE, "total", sum(SIZE.values()), "lines") print("task count:", len(TASKS), "-", " , ".join(CLASSES)) print("cost(0,10,25,40,60):", [MODULES["tarife.py"]["cost"](m) for m in (0, 10, 25, 40, 60)]) print("usage(99800,300):", MODULES["okuma.py"]["usage"](99800, 300)) ``` ``` codebase: {'okuma.py': 13, 'tarife.py': 13, 'denetim.py': 11, 'ozet.py': 11, 'bicim.py': 5} total 53 lines task count: 24 - tamamlama , aciklama , donusturme cost(0,10,25,40,60): [0, 40, 145, 310, 630] usage(99800,300): 500 ``` The last two lines fix the **correct behavior** of the codebase. In the tiered tariff, sixty cubic meters comes to **630**; when the meter wraps from ninety-nine thousand to three hundred, the usage is **500**. These two numbers will be the yardstick from now on: if a patch changes them, the patch is wrong. This is the first rule of the review procedure: correctness is not an impression but **an expectation written in advance**. The twenty-four fictional tasks are divided into three classes, and each task concerns certain modules. The patch that arrives as the answer to a task changes one of those modules and leaves a helper file next to it. The reach of the checks will be divided between these two parts. #### The Reach of Three Checks Three checks look at three separate places. The **check run** sees only the changed module and only the behavior that the checks measure. The **pattern scan** looks at the text; it does not run it. The **existence check** looks only at `import` lines. The union of the three is not the whole codebase, and what this lesson measures is exactly the gap in between. ```python DEFECTS = [("H1", "tarife.py", "ABOVE = 16", "ABOVE = 11"), ("H2", "okuma.py", "if diff < 0:", "if diff < -100000:"), ("H3", "denetim.py", 'record["index"] < 0', 'record["index"] < -1'), ("H4", "ozet.py", 'total.get(r["region"], 0) + u', 'total.get(r["region"], 0) + 1'), ("H5", "okuma.py", "len(p) != 4", "len(p) < 4"), ("H6", "bicim.py", "rjust(9)", "rjust(7)")] FABRICATED = ("measurement_tools", "tariff_helper", "region_summary", "meter_kit") PATTERNS = ("eval(", "except:", "== None", "open(") EXTRA = {"eval(": "def evaluate(expression):\n return eval(expression)\n", "except:": "def to_int(s):\n try:\n return int(s)\n except:\n return 0\n", "== None": "def is_empty(record):\n return record == None\n", "open(": "def backup(path):\n return open(path).read()\n"} STANDARD = ("math", "json") def mutate(code, defect): _, file, old, new = defect new_code = dict(code) new_code[file] = code[file].replace(old, new, 1) return new_code def generate(task, seed=SEED): r = rng(seed ^ mix(task["name"])) u = [r() for _ in range(6)] defect = DEFECTS[int(u[1] * 6)] if u[0] < 0.46 else None imported = FABRICATED[int(u[3] * 4)] if u[2] < 0.30 else "math" pattern = PATTERNS[int(u[5] * 4)] if u[4] < 0.34 else None return {"code": mutate(CODE, defect) if defect else dict(CODE), "defect": defect[0] if defect else None, "imported": imported, "pattern": pattern, "helper": "import " + imported + "\n\n" + (EXTRA[pattern] if pattern else "")} RECORDS = [{"region": "east", "index": 120}, {"region": "star", "index": 40}, {"region": "west", "index": 90}] def run_checks(code): a = load(code) o, t, d, z, b = (a["okuma.py"], a["tarife.py"], a["denetim.py"], a["ozet.py"], a["bicim.py"]) failed = [] for name, result in ( ("parse-fields", o["parse_line"]("A017;east;2026-01;41250") == {"subscriber": "A017", "region": "east", "period": "2026-01", "index": 41250}), ("parse-extra", o["parse_line"]("A017;east;2026-01;41250;extra") is None), ("usage-wrap", o["usage"](99800, 300) == 500), ("cost-tiers", [t["cost"](m) for m in (0, 10, 25, 40, 60)] == [0, 40, 145, 310, 630]), ("valid-negative", d["is_valid"]({"region": "east", "index": -1}) is False), ("filter-region", len(d["filter_valid"](RECORDS)) == 2), ("total-region", z["region_total"](RECORDS, [12, 5, 8]) == {"east": 12, "star": 5, "west": 8}), ("count-region", z["region_count"](RECORDS) == {"east": 1, "star": 1, "west": 1}), ("format-row", "east" in b["row"]("east", 12, 48)), ("format-report", b["report"]("head", ["a", "b"]).count("\n") == 2)): if not result: failed.append(name) return failed def scan_patterns(text): return [p for p in PATTERNS if p in text] def check_dependencies(text, code): missing = [] for line in text.split("\n"): if line.startswith("import "): a = line[7:].strip() if a + ".py" not in code and a not in STANDARD: missing.append(a) return missing def measure(seed=SEED): defects = {"behavior": 0, "pattern": 0, "fabricated": 0} caught = {"checks": 0, "pattern": 0, "dependency": 0} missed, silent, flagged = [], 0, 0 for task in TASKS: a = generate(task, seed) flag = 0 if a["pattern"]: defects["pattern"] += 1 if scan_patterns(a["helper"]): caught["pattern"] += 1 flag += 1 if a["imported"] in FABRICATED: defects["fabricated"] += 1 if check_dependencies(a["helper"], a["code"]): caught["dependency"] += 1 flag += 1 if a["defect"]: defects["behavior"] += 1 if run_checks(a["code"]): caught["checks"] += 1 flag += 1 else: missed.append((task["name"], a["defect"], "flagged" if flag else "clean")) silent += flag == 0 flagged += flag > 0 return defects, caught, missed, silent, flagged print("baseline pattern findings:", scan_patterns("".join(CODE.values())) or "none") print("defect file failed checks") for h in DEFECTS: print("%-6s %-12s %s" % (h[0], h[1], run_checks(mutate(CODE, h)) or "none")) print() print("seed | behavior pattern fabricated | checks pattern dependency" " | flagged missed silent") result = {} for seed in (SEED, SEED + 1): d, c, ms, sl, fl = measure(seed) result[seed] = ms print("%d | %8d %7d %10d | %6d %7d %10d | %7d %6d %6d" % (seed, d["behavior"], d["pattern"], d["fabricated"], c["checks"], c["pattern"], c["dependency"], fl, len(ms), sl)) print("missed defects, seed 20260218:", result[SEED]) print("missed defects, seed 20260219:", result[SEED + 1] or "none") ``` ``` baseline pattern findings: none defect file failed checks H1 tarife.py ['cost-tiers'] H2 okuma.py ['usage-wrap'] H3 denetim.py ['valid-negative'] H4 ozet.py ['total-region'] H5 okuma.py ['parse-extra'] H6 bicim.py none seed | behavior pattern fabricated | checks pattern dependency | flagged missed silent 20260218 | 11 8 9 | 9 8 9 | 16 2 1 20260219 | 6 9 12 | 6 9 12 | 17 0 0 missed defects, seed 20260218: [('tamamlama-4', 'H6', 'flagged'), ('aciklama-7', 'H6', 'clean')] missed defects, seed 20260219: none ``` The first line says the scan ran over a **clean baseline**: none of the four searched strings is in the codebase itself. If a finding comes out, it means it came from the generated code; this is the precondition that makes the scan results interpretable. The second table is this lesson's **run-independent** measure. The six behavioral defects are placed on the codebase one by one and the acceptance check suite is run. Five of them each fail a single check: the tiered cost, the counter wrap, the negative index check, the region total, and the row with extra fields. The sixth **fails no check at all**. The reason is in the check suite itself: the two items that measure the format module look at the region name appearing in the output and at the line count, and **do not look at the column width**. The defect is there and the check does not look there. This result is independent of the seed, the task distribution, and the generation. #### The Residual and How a Flag Misleads The third table puts the two runs side by side. In the first run, the twenty-four patches contain **11 behavioral defects, 8 pattern findings, and 9 fabricated dependencies**; **28 defects** in total. The pattern scan catches eight of eight, the existence check nine of nine. The check run catches **nine** of eleven. Of the 28 defects, 26 are seen, **2 slip through**, and both are format defects. In the second run nothing slips through, and this does not mean the check has improved. In that run no format defect was generated; all six of the behavioral defects that were generated are in places the check looks. The number missed depends on **the distribution of defects**, not on the coverage of the check. The number that reads the coverage is in the second table, and there one defect in six stands uncovered. The strength of a check is measured not by the zero it gives on a good run but by **the name of the class it cannot reach**. The last line questions the flag itself. One of the two missed patches (`tamamlama-4`) is in fact **flagged**: it took a fabricated dependency and its helper file has a pattern finding. A human who looks at this patch sees two warnings, fixes both, and accepts the patch. What is fixed is a defect, but **not the whole of the defects**; the format error enters the codebase together with the acceptance. The second (`aciklama-7`) receives no warning at all and enters directly. The cost column is here too: in the first run **16 candidates are flagged**, that is, the number of patches a human has to look at is sixteen of twenty-four; **one** of the eight patches that pass unflagged **is defective**. In the second run the flagged count rises to **17** and the gain is zero, because nothing slipped through in that run anyway. The review load grows not with the defect count but with **the flag count**. #### Summary - The fictional codebase is five modules and 53 lines; its correct behavior is fixed by two numbers, 630 in the tiered cost and 500 in the counter wrap. - Three checks look at three separate places: the check run at the behavior of the changed module, the pattern scan at the text, the existence check only at the import lines. - The run-independent measure: five of the six behavioral defects fail the acceptance check suite, the format defect fails none, because the check suite does not look at the column width. - In the first run 26 of 28 defects are caught and 2 are missed; in the second run nothing is missed, but the reason is that the distribution of defects changed, not the coverage. - The number of flagged candidates is not the number of caught defects: one of the two missed patches is flagged because of another defect, and it is accepted once its warnings are fixed. - The review cost is measured by the flag count; in the first run 16 of 24 patches go to a human, and one of the eight that pass unflagged carries a defect. #### Next Step In this lesson the acceptance check suite was **written by hand** and its coverage was a known quantity: ten items, five modules, one known blind spot. Writing the check suite is also a job, and it is the most often suggested of the tasks that can be handed to the same helper. The next lesson moves the check suite to the generated side and asks the same question of the check suite itself: how many of the six defects placed on the codebase do the generated checks catch, how many checks give the same result on all versions and distinguish nothing, and how many claims in the generated documentation can be verified by machine. ### Documentation and Test Generation Source: https://academia.sh/en/courses/model-apis/documentation-and-test-generation Measuring the discriminating power of a generated test suite: of fourteen generated tests run on the six mutated versions of the fictional codebase, two fail on clean code, nine of the remaining twelve give the same result on all seven versions, and only three distinguish a defect. Detection is 3/6, the ceiling reachable with the whole pool is 6/6, and a single hand-added alignment item makes it 4/6. In the generated documentation ten claims can be checked by machine, six are right, and one describes a function that does not exist. In the previous lesson the acceptance check suite was **written by hand**: ten items, a known scope, a known blind spot. The test itself is also something to be written, and it is the task most often handed to a code assistant. Documentation is the same. What the two have in common is that the output is **not run directly**: unless the generated test is run and the generated document is read, its wrongness is not visible. This lesson makes the two outputs measurable. For the test the measure is **discriminating power**: can it tell the mutated versions of the codebase from the clean version. For the documentation the measure is **checkability**: can the truth of a claim be tested by machine by looking at the code. Both measures look not at the generated text itself but at **the information it gives**. - **AD39.** The codebase is **fictional** and is five modules and **53 lines**. It is not any real repository, package, or product. Seed **20260218**. - **AD40.** The six defects are single string substitutions in the source text; each one produces a separate **version**. Together with the clean one there are **seven versions**. - **AD41.** A **test** consists of an **observation point** and an **expected value**. The observation pool of the suite is twenty-one points. - **AD42.** The generated suite selects **fourteen points out of twenty**, leaving out the alignment point, and reads the expected value from the code. The reading is not always right: a share carries a **wrong expectation**. - **AD43.** A **test with a wrong expectation** is a test that fails on the clean version. This is the validity check, and it is done before the discrimination measurement. - **AD44.** A **vacuous test** is a test that gives the same result on all seven versions. This does not mean that the test is worthless; it means that **it carries no information against these six defects**. - **AD45.** The generated suite measures the output format **by content and line count**, not **by column width**. This was also the blind spot of the test written by hand in the previous lesson. - **AD46.** The generated documentation has two parts: **machine-checkable claims** carrying a function name and a parameter count, and **one free sentence** per module. A free sentence cannot be tested by machine. - **AD47.** This lesson is not an attack lesson; the code that runs is testing, counting, and verifying. #### Seven Versions and Where a Defect Shows To measure discriminating power there must be something to discriminate. The six defects placed on the codebase provide it: each one changes a single string in the source text and moves an observable result. The block below builds the six versions and prints **at which observation** each defect shows. ```python # FICTIONAL codebase: the measurement network's own helper scripts. Not a real repository, # package, or product. Five modules, 53 lines. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CODE = { "okuma.py": """ def parse_line(line): p = line.split(";") if len(p) != 4: return None if not p[3].lstrip("-").isdigit(): return None return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])} def usage(previous, current): diff = current - previous if diff < 0: diff = diff + 100000 return diff """, "tarife.py": """ TIERS = ((10, 4), (25, 7), (40, 11)) ABOVE = 16 def cost(m3): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper return total + remaining * ABOVE """, "denetim.py": """ VALID_REGIONS = ("east", "west", "north", "south", "central") def is_valid(record): if record is None: return False if record["index"] < 0: return False return record["region"] in VALID_REGIONS def filter_valid(records): return [r for r in records if is_valid(r)] """, "ozet.py": """ def region_total(records, usages): total = {} for r, u in zip(records, usages): total[r["region"]] = total.get(r["region"], 0) + u return total def region_count(records): count = {} for r in records: count[r["region"]] = count.get(r["region"], 0) + 1 return count """, "bicim.py": """ def row(region, m3, amount): return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9) def report(title, rows): return "\\n".join([title] + rows) """, } DEFECTS = [("H1", "tarife.py", "ABOVE = 16", "ABOVE = 11"), ("H2", "okuma.py", "if diff < 0:", "if diff < -100000:"), ("H3", "denetim.py", 'record["index"] < 0', 'record["index"] < -1'), ("H4", "ozet.py", 'total.get(r["region"], 0) + u', 'total.get(r["region"], 0) + 1'), ("H5", "okuma.py", "len(p) != 4", "len(p) < 4"), ("H6", "bicim.py", "rjust(9)", "rjust(7)")] RECORDS = [{"region": "east", "index": 120}, {"region": "star", "index": 40}, {"region": "west", "index": 90}] def load(code): modules = {} for name, source in code.items(): scope = {} exec(source, scope) modules[name] = scope return modules def mutate(code, defect): _, file, old, new = defect new_code = dict(code) new_code[file] = code[file].replace(old, new, 1) return new_code def observe(code): a = load(code) return {"cost60": a["tarife.py"]["cost"](60), "wrap": a["okuma.py"]["usage"](99800, 300), "negative": a["denetim.py"]["is_valid"]({"region": "east", "index": -1}), "total": a["ozet.py"]["region_total"](RECORDS, [12, 5, 8])["east"], "extra": a["okuma.py"]["parse_line"]("A017;east;2026-01;41250;extra") is None, "length": len(a["bicim.py"]["row"]("east", 12, 48))} SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()} CLEAN = observe(CODE) print("codebase:", SIZE, "total", sum(SIZE.values()), "lines") print("clean observation:", CLEAN) for h in DEFECTS: b = observe(mutate(CODE, h)) print(h[0], h[1], {k: (CLEAN[k], b[k]) for k in CLEAN if CLEAN[k] != b[k]}) ``` ``` codebase: {'okuma.py': 13, 'tarife.py': 13, 'denetim.py': 11, 'ozet.py': 11, 'bicim.py': 5} total 53 lines clean observation: {'cost60': 630, 'wrap': 500, 'negative': False, 'total': 12, 'extra': True, 'length': 24} H1 tarife.py {'cost60': (630, 530)} H2 okuma.py {'wrap': (500, -99500)} H3 denetim.py {'negative': (False, True)} H4 ozet.py {'total': (12, 1)} H5 okuma.py {'extra': (True, False)} H6 bicim.py {'length': (24, 22)} ``` All six of the six defects are **observable**. None is hidden, none throws an error at run time; each one moves a number or a truth value. This keeps the measurement fair: if a defect is not caught, the reason is not that the defect is invisible but that **the test does not look there**. The last row is the clearest example of this. The format defect shows only in **the line length**: the twenty-four-character line drops to twenty-two. The region name is in place, the numbers are right, the line count is the same. The only observation point that catches this defect is the one that measures the length of the line. #### The Discriminating Power of a Generated Test When a test is asked of the assistant, what comes is this: fourteen of the twenty observation points and, for each, an expected value read from the code. The reading is mostly right, a share of it is not. The generated suite is first put through **validity**, then through **discriminating power**. ```python def observe_all(code): a = load(code) o, t, d, z, b = (a["okuma.py"], a["tarife.py"], a["denetim.py"], a["ozet.py"], a["bicim.py"]) return {"parse-full": o["parse_line"]("A017;east;2026-01;41250")["index"], "parse-empty": o["parse_line"]("") is None, "parse-extra": o["parse_line"]("A017;east;2026-01;41250;extra") is None, "parse-letters": o["parse_line"]("A017;east;2026-01;xx") is None, "usage-plain": o["usage"](100, 300), "usage-wrap": o["usage"](99800, 300), "cost-zero": t["cost"](0), "cost-first": t["cost"](10), "cost-mid": t["cost"](25), "cost-top": t["cost"](60), "valid-none": d["is_valid"](None), "valid-negative": d["is_valid"]({"region": "east", "index": -1}), "valid-region": d["is_valid"]({"region": "star", "index": 5}), "filter-empty": d["filter_valid"]([]), "filter-three": len(d["filter_valid"](RECORDS)), "total-region": z["region_total"](RECORDS, [12, 5, 8]), "count-empty": z["region_count"]([]), "count-three": z["region_count"](RECORDS), "row-content": "east" in b["row"]("east", 12, 48), "report-lines": b["report"]("head", ["a", "b"]).count("\n"), "row-length": len(b["row"]("east", 12, 48))} POOL = [a for a in observe_all(CODE) if a != "row-length"] # the generated suite does not look at alignment VERSIONS = [("clean", CODE)] + [(h[0], mutate(CODE, h)) for h in DEFECTS] TABLE = {name: observe_all(c) for name, c in VERSIONS} def flip(v): if isinstance(v, bool): return not v if isinstance(v, int): return v + 30 if isinstance(v, dict): return {} return [0] def generate_tests(seed=SEED, n=14): r = rng(seed) remaining, selected = list(POOL), [] for _ in range(n): selected.append(remaining.pop(int(r() * len(remaining)))) return [(name, flip(TABLE["clean"][name]) if r() < 0.14 else TABLE["clean"][name]) for name in selected] def run(suite, version): return [name for name, expected in suite if TABLE[version][name] != expected] def evaluate(suite): wrong = run(suite, "clean") sound = [(name, e) for name, e in suite if name not in wrong] caught = [h[0] for h in DEFECTS if run(sound, h[0])] always_pass = [name for name, e in sound if not any(name in run(sound, h[0]) for h in DEFECTS)] return len(suite), len(wrong), len(sound), caught, always_pass print("version count:", len(VERSIONS), "| pool:", len(POOL), "observations") print("suite | tests wrong sound | caught | always pass") for label, suite in (("seed 218", generate_tests(SEED)), ("seed 219", generate_tests(SEED + 1)), ("whole pool", [(a, TABLE["clean"][a]) for a in observe_all(CODE)])): n, w, s_, cg, ap = evaluate(suite) print("%-12s | %6d %6d %6d | %d/6 %-15s | %d" % (label, n, w, s_, len(cg), ",".join(cg), len(ap))) suite = generate_tests(SEED) print("wrong expectation:", run(suite, "clean")) print("always pass:", sorted(evaluate(suite)[4])) by_hand = suite + [("row-length", TABLE["clean"]["row-length"])] print("after adding one item by hand:", evaluate(by_hand)[3]) def generate_doc(seed=SEED): a = load(CODE) claims = [] for file in CODE: r = rng(seed ^ mix(file)) for f in sorted(x for x in a[file] if not x.startswith("__") and callable(a[file][x])): n = a[file][f].__code__.co_argcount claims.append((file, f, n + 1 if r() < 0.22 else n)) if r() < 0.40: claims.append((file, "verify", 1)) return claims def check_doc(claims): a = load(CODE) missing = [c[0] + ":" + c[1] for c in claims if c[1] not in a[c[0]]] wrong = [c[0] + ":" + c[1] for c in claims if c[1] in a[c[0]] and a[c[0]][c[1]].__code__.co_argcount != c[2]] return missing, wrong print() print("seed | checkable free | missing function | wrong count | right") for seed in (SEED, SEED + 1): claims = generate_doc(seed) missing, wrong = check_doc(claims) print("%d | %10d %7d | %16d | %11d | %5d" % (seed, len(claims), len(CODE), len(missing), len(wrong), len(claims) - len(missing) - len(wrong))) claims = generate_doc(SEED) missing, wrong = check_doc(claims) print("missing function claim:", missing, "| wrong parameter count:", wrong) ``` ``` version count: 7 | pool: 20 observations suite | tests wrong sound | caught | always pass seed 218 | 14 2 12 | 3/6 H1,H2,H3 | 9 seed 219 | 14 1 13 | 3/6 H1,H2,H4 | 10 whole pool | 21 0 21 | 6/6 H1,H2,H3,H4,H5,H6 | 15 wrong expectation: ['cost-zero', 'filter-empty'] always pass: ['cost-mid', 'filter-three', 'parse-empty', 'parse-letters', 'report-lines', 'row-content', 'usage-plain', 'valid-none', 'valid-region'] after adding one item by hand: ['H1', 'H2', 'H3', 'H6'] seed | checkable free | missing function | wrong count | right 20260218 | 10 5 | 1 | 3 | 6 20260219 | 11 5 | 2 | 0 | 9 missing function claim: ['bicim.py:verify'] | wrong parameter count: ['denetim.py:filter_valid', 'bicim.py:report', 'bicim.py:row'] ``` The first row says **two of the fourteen tests fail on clean code**: the zero input of the cost calculation and filtering an empty list. These do not find defects, they **invent** defects. This is where the real danger lies for a developer who meets a red test: fitting the code to a wrong test means breaking working code. Weeding out these two items is a job that comes before the measurement and is paid for in human time. Of the remaining twelve tests, **nine** give the same result on all seven versions. They are not faulty; filtering an empty list, rejecting an invalid region, and counting the report lines are plausible items. But they carry no information against any of these six defects. The number of tests that carry information is **three** and detection is **3/6**. The discriminating share of a fourteen-item suite is **0.2143**, that is, one and a half in every seven tests. The second run shows that the number is stable: fourteen tests, one wrong expectation, thirteen sound, again **3/6** detection and ten vacuous. The **identity** of the caught defects changes (H3 in the first run, H4 in the second); the total does not. The third row gives the ceiling. When all twenty-one points of the pool are turned into tests, detection becomes **6/6** and the vacuous count rises to **15**. This is the proof that always passing is not a defect: even in a suite of full coverage, five of every seven items are silent against these six defects. Discriminating power comes **not with the number of items but with the right point**. The last row shows this in a single move. **One** item is added to the generated suite by hand, the item that measures the length of the line, and detection goes **from 3/6 to 4/6**. One test adds more than fourteen tests added, because that test looks where none of the others look. #### The Checkable Part of the Generated Documentation On the documentation side the measure is different: not truth but **verifiability** comes first. The generated documentation contains one free sentence per module and one structural claim per function. Free sentences cannot be tested by looking at the code; structural claims can, because whether a function exists and how many parameters it takes is read from the loaded namespace. In the first run there are **ten** machine-checkable claims and **five** free sentences: **0.6667** of the fifteen statements are in the area the machine can reach. **Six** of the ten that are checked are right. Three give the parameter count of the function as one too many, and one describes a verification function that does not exist. In the second run nine of eleven claims are right and two are functions that do not exist. In both runs the share of wrong claims is not zero, and the wrong comes in **two separate kinds**: describing wrongly something that exists, and **describing something that does not exist**. The second is more dangerous, because the reader goes on to call the function he sees in the documentation. #### Summary - All six of the six defects are observable; the format defect shows only in the line length and drops the twenty-four-character line to twenty-two. - Two of the fourteen generated tests fail on clean code: they do not find defects, they invent them, and weeding them out is a job that comes before the measurement. - Nine of the remaining twelve tests give the same result on all seven versions; with the three tests that carry information, detection is 3/6, and it stays at 3/6 in the second run too. - The whole pool gives 6/6 but the vacuous count rises to 15: discriminating power comes not from the number of items but from the point looked at. - A single hand-added alignment item raises detection to 4/6; one correct test adds more than fourteen generated tests. - In the generated documentation ten of fifteen statements can be checked by machine and six are right; one kind of wrong is describing an existing function wrongly, the other is describing a function that does not exist. #### Next Step In both lessons what was measured was **the quality of the output**: the correctness of the patch, the discriminating power of the test, the checkability of the documentation. All of these measurements are made after the output is obtained, and all of them assume that the defect is **in the codebase**. Yet there is still something to count at the two ends of the handoff. At the input end, what is inside the context sent to the model was never asked: which fields fall into a secret class, and how many a filter removes. At the output end, the acceptance decision itself was never questioned: how often is a human right when accepting a suggestion. The next lesson counts the three risks one by one and closes the topic. ### Limits and Risks Source: https://academia.sh/en/courses/model-apis/limits-and-risks Counting the three risks of AI-assisted development: of the 456 context lines in twenty-four request bundles, 168 fall into a secret class; a filter that looks at field names removes all of them but cannot reach the 12 traces left in free text, so its coverage stays at 0.9333; nine of the generated pieces of code carry a dependency name that does not exist; and ten of the nineteen accepted suggestions are defective, so the share of wrong among the accepted is 0.5263, above the baseline share of 0.4583. The previous two lessons measured the quality of the generated output: the correctness of the patch, the discriminating power of the test, the checkability of the documentation. All three assume that the defect is **in the codebase** and look for it after the output is obtained. At the two ends of the handoff, though, there is something that has not been counted. At the input end it is **what was sent** to the model, and at the output end it is **the acceptance decision itself**. This lesson counts the three risks one by one: how many fields in the text that enters the context fall into a secret class and how many a filter removes, how many dependency names in the generated code point to something that does not exist, and what share of the accepted suggestions is wrong. The procedure of secret management was covered in the Application Security and DevSecOps curriculum, and dependency policy and component analysis were measured in the same curriculum; neither is repeated here. The only thing added is **the count of these three risks**. - **AD48.** The codebase is **fictional** and is five modules and **53 lines**. The subscriber records, the working settings, and the support requests are fictional too. Seed **20260218**. - **AD49.** This is not an attack lesson. **The code that runs is always on the defending side**: counting, classification, filter, existence check. No exploit code, payload, evasion, or publishing procedure is written. - **AD50.** A **request bundle** is the whole of the context sent with a task: the task line, the file list, one to three subscriber records, the working settings, and a support request. - **AD51.** Five field names map to four **secret classes**: identity, contact, location, and access. The values are fictional and none of them is real data. - **AD52.** **The filter looks at the field name**: if the key of the line appears in the secret-class table, the line is removed. That is the whole procedure; key management, masking, and storage are not covered. - **AD53.** A **trace word** is the first part of a secret-class field name. A trace that appears in free text is a leak the filter cannot see, and it is counted separately. - **AD54.** A **fabricated dependency** is a name in an `import` line that is found neither among the modules of the codebase nor among the allowed standard library names. The fabricated names are fictional and no real package, repository, or publishing path is mentioned. - **AD55.** A **defective suggestion** is a suggestion that changes the behavior of the codebase. The defect is the change of a single comparison boundary or a single constant. - **AD56.** **Acceptance** is the developer's decision to take the suggestion as it is. The decision depends on context overlap, task class, and difficulty; it **does not look** at whether the suggestion has a defect, because these defects are single-character changes. The truth of this assumption is tested separately in the measurement. - **AD57.** The numbers are printed on two seeds; the second run's seed is **20260219**. #### Secret-Class Fields That Enter the Context What is sent when a task is given to a code assistant is not the task sentence. Along with it go the file list, sample records, working settings, and often a user text that is the source of the problem. This bundle leaves over a network, and what is in it is not known **until it is counted**. ```python # FICTIONAL: the helper codebase of a measurement network, five modules and 53 lines, with # subscriber records, working settings, and support requests. None of it is real. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py") REQUIRED = {0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py")} TASKS = [{"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8)] SECRET_FIELDS = {"subscriber_id": "identity", "contact": "contact", "address": "location", "access_string": "access", "session_key": "access"} TRACE_WORDS = tuple(sorted({a.split("_")[0] for a in SECRET_FIELDS})) SUBSCRIBERS = [{"subscriber_id": "A0417", "contact": "line-3", "address": "east-12", "region": "east", "period": "2026-01", "index": "41250"}, {"subscriber_id": "A0912", "address": "west-4", "region": "west", "period": "2026-01", "index": "38104"}, {"subscriber_id": "A1330", "contact": "line-1", "region": "north", "period": "2026-01", "index": "50077"}] SETTINGS = {"measurement_network": "east", "timeout": "20", "log_path": "data/measurement", "access_string": "SAMPLE", "session_key": "SAMPLE"} REQUESTS = ["my address changed please register me for the new period", "my contact information looks wrong in the system", "the meter reading was entered incomplete please correct it", "my tariff tier was calculated wrongly", "my subscriber record was opened twice please merge them", "the reading was not taken after the outage"] def build_context(task, i): s = ["task: " + task["name"], "file: " + ",".join(task["required"])] for rec in SUBSCRIBERS[:1 + i % 3]: s += [a + ": " + d for a, d in rec.items()] s += [a + ": " + d for a, d in SETTINGS.items()] s.append("request: " + REQUESTS[i % len(REQUESTS)]) return s def count_secrets(lines): count = {} for s in lines: label = SECRET_FIELDS.get(s.split(":")[0]) if label: count[label] = count.get(label, 0) + 1 return count def apply_filter(lines): return [s for s in lines if s.split(":")[0] not in SECRET_FIELDS] def free_text_traces(lines): return sum(1 for s in lines if s.startswith("request:") for w in TRACE_WORDS if w in s) totals = {"lines": 0, "secret": 0, "kept": 0, "trace": 0} class_count = {} for i, task in enumerate(TASKS): bundle = build_context(task, i) kept = apply_filter(bundle) totals["lines"] += len(bundle) totals["secret"] += len(bundle) - len(kept) totals["kept"] += len(kept) totals["trace"] += free_text_traces(kept) for label, n in count_secrets(bundle).items(): class_count[label] = class_count.get(label, 0) + n print("trace words:", TRACE_WORDS) print("context lines:", totals["lines"], "| after filtering:", totals["kept"]) print("secret class fields:", dict(sorted(class_count.items())), "total", totals["secret"]) exposed = totals["secret"] + totals["trace"] print("removed by filter:", totals["secret"], "| traces left in free text:", totals["trace"], "| coverage: %.4f" % (totals["secret"] / exposed)) print("sample context (task 1), filtered:") for s in apply_filter(build_context(TASKS[0], 0)): print(" " + s) ``` ``` trace words: ('access', 'address', 'contact', 'session', 'subscriber') context lines: 456 | after filtering: 288 secret class fields: {'access': 48, 'contact': 32, 'identity': 48, 'location': 40} total 168 removed by filter: 168 | traces left in free text: 12 | coverage: 0.9333 sample context (task 1), filtered: task: tamamlama-1 file: okuma.py region: east period: 2026-01 index: 41250 measurement_network: east timeout: 20 log_path: data/measurement request: my address changed please register me for the new period ``` In the twenty-four request bundles there are **456 context lines**, and **168** of them fall into one of the four secret classes: identity 48, access 48, location 40, contact 32. That is, a little more than one in every three lines sent is a field that contributes nothing to the task but still leaves. This ratio is **0.3684** and it is written nowhere; it is not known without counting. The filter removes **all** of the structured fields: 168 of 168. By its own criterion this is a perfect result. But the criterion is wrong. What the filter looks at is the field name, and secret-class information does not live only in the field name. The **12 traces** left in free text are the proof; the real coverage of the filter is **0.9333**. In the sample bundle this is plain to see: after the `subscriber_id`, `contact`, and `address` lines have been cleaned, the request line at the bottom still talks about the subscriber's address. Deleting a structured field does not delete the sentence that carries the same information. The cost of the filter can be read in the same output. In the remaining record there are region, period, and index and **no subscriber field**. That is the first of the four columns of the parser, and the task concerns the reading module. Removing the secret-class fields also takes away one column of the example the task needs. #### Fabricated Dependency and the Acceptance Decision The second and third risks are on the output side. Generated code can carry a dependency name, and that name may correspond to nothing. The way to see this is not to **try** the name but to look at whether the name exists: the modules of the codebase and the allowed standard library names are a known set, and every name outside it is suspect. The third risk is the hardest to measure, because what is measured is not the model's output but **the human's decision**. ```python FABRICATED = ("measurement_tools", "tariff_helper", "region_summary", "meter_kit") PATTERNS = ("eval(", "except:", "== None", "open(") STANDARD = ("math", "json") BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def generate_code(task, seed=SEED): r = rng(seed ^ mix(task["name"])) u = [r() for _ in range(6)] return {"defective": u[0] < 0.46, "imported": FABRICATED[int(u[3] * 4)] if u[2] < 0.30 else "math", "pattern": PATTERNS[int(u[5] * 4)] if u[4] < 0.34 else None} def existence_check(name): return name + ".py" not in FILES and name not in STANDARD def assistant(task, selection, seed=SEED): r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(selection)))) overlap = sum(1 for f in task["required"] if f in selection) / len(task["required"]) irrelevant = sum(1 for f in selection if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) return {"accepted": r() < score, "score": score, "tokens": 40 + 26 * len(selection) + (18 if task["label"] == "donusturme" else 0)} def measure(seed=SEED): d = {"fabricated": 0, "distinct": set(), "pattern": 0, "defective": 0, "accepted": 0, "accepted_wrong": 0, "rejected_right": 0, "clean_accepted": 0, "tokens": 0} for task in TASKS: code = generate_code(task, seed) y = assistant(task, task["required"], seed) if existence_check(code["imported"]): d["fabricated"] += 1 d["distinct"].add(code["imported"]) d["pattern"] += code["pattern"] is not None d["defective"] += code["defective"] d["accepted"] += y["accepted"] d["tokens"] += y["tokens"] d["accepted_wrong"] += y["accepted"] and code["defective"] d["rejected_right"] += (not y["accepted"]) and (not code["defective"]) d["clean_accepted"] += y["accepted"] and not code["defective"] return d print("seed | fabricated distinct pattern | defective accepted accepted-wrong" " | baseline share accepted share | rejected-right") for seed in (SEED, SEED + 1): d = measure(seed) print("%d | %10d %8d %7d | %9d %8d %14d | %14.4f %14.4f | %14d" % (seed, d["fabricated"], len(d["distinct"]), d["pattern"], d["defective"], d["accepted"], d["accepted_wrong"], d["defective"] / 24, d["accepted_wrong"] / d["accepted"], d["rejected_right"])) print("acceptance rate | on defective suggestions | on clean suggestions") for seed in (SEED, SEED + 1): d = measure(seed) de, cl = d["defective"], 24 - d["defective"] print("%d | %-24s | %s" % (seed, "%d/%d = %.4f" % (d["accepted_wrong"], de, d["accepted_wrong"] / de), "%d/%d = %.4f" % (d["clean_accepted"], cl, d["clean_accepted"] / cl))) d = measure(SEED) print("fabricated names:", sorted(d["distinct"])) print("context tokens per request:", d["tokens"] // 24) ``` ``` seed | fabricated distinct pattern | defective accepted accepted-wrong | baseline share accepted share | rejected-right 20260218 | 9 4 8 | 11 19 10 | 0.4583 0.5263 | 4 20260219 | 12 4 9 | 6 19 5 | 0.2500 0.2632 | 4 acceptance rate | on defective suggestions | on clean suggestions 20260218 | 10/11 = 0.9091 | 9/13 = 0.6923 20260219 | 5/6 = 0.8333 | 14/18 = 0.7778 fabricated names: ['measurement_tools', 'meter_kit', 'region_summary', 'tariff_helper'] context tokens per request: 94 ``` The number for the second risk is in the first columns. **Nine** of the twenty-four generated pieces refer to a name that does not exist, and they are spread over only **four distinct names**. In the second run the number is **12**, and the distinct-name count is again four. A fabricated dependency is not scattered noise: the same small set of names is produced again and again, because the name is derived from the codebase's own naming habit and looks plausible. This is also why the existence check is cheap: the set to compare against is known, and the check is nothing more than a string test. #### The Number for Overconfidence The third risk is read in the last columns, and it is the most uncomfortable number of this lesson. In the first run **11** of the twenty-four suggestions are defective; the baseline share is **0.4583**. The developer accepts **19** of these suggestions. **10 of the accepted ones are defective**, that is, the share of wrong among the accepted is **0.5263**. The acceptance decision **did not lower** the share of wrong relative to the baseline share; in this run it even raised it slightly. The second run repeats the same structure with a lower baseline share: defective **6**, baseline share **0.2500**, acceptance again **19**, the share of wrong among the accepted **0.2632**. In both runs the acceptance share is above the baseline and the two numbers are close to each other. What is to be read from this is the following: **the acceptance decision carries no information about defects.** The proportion of wrong among the accepted falls in the same place as the proportion of wrong in the pool of suggestions. The last table tests this directly. The acceptance rate on defective suggestions is **0.9091** in the first run and on clean suggestions **0.6923**; in the second run **0.8333** and **0.7778**. All four numbers are close to each other, and the defective side is not lower in either run. Assumption AD56 is thus tested: a single-character change is not seen in a quick reading. **Four** of the five rejected suggestions, meanwhile, are clean; the rejection decision carries no information either, it only produces work. The cost is collected in three places. On the context side **94 tokens** go out per request, and part of that is secret-class fields. On the dependency side nine of the twenty-four pieces fall to review. On the acceptance side the cost is written directly into the codebase: in the first run **10 defects** were accepted, that is, they entered the patches that go into review through the developer's own decision. The three checks built in the previous lesson are therefore not a decoration; they are the second line **behind** the acceptance decision. #### Summary - In the twenty-four request bundles, 168 of the 456 context lines fall into four secret classes: identity 48, access 48, location 40, contact 32. - A filter that looks at field names removes all of the structured fields but cannot reach the 12 traces in free text; the real coverage is 0.9333. - The cost of the filter is visible: when the subscriber field is removed, one of the parser's four columns drops out of the example. - Nine of the twenty-four generated pieces carry a dependency name that does not exist, and they are spread over only four distinct names; in the second run the number is 12 and the distinct-name count does not change. - Ten of the nineteen accepted suggestions are defective; the share of wrong among the accepted is 0.5263 and the baseline share is 0.4583, and in the second run 0.2632 and 0.2500. - The acceptance rates on defective and clean suggestions are close to each other and the defective side is not lower in either run: the acceptance decision carries no information about defects. #### Next Step Over six lessons the assistant itself was measured: in which task class it is accepted, what context adds and where it drags down, where the checkpoint stops the error, what the three checks catch, what the generated test discriminates, and that the acceptance decision discriminates nothing. This topic closes here. One thing stayed constant throughout the course's twenty lessons and was never questioned: the model was always an endpoint that stood **far away**. The call went out over the network, queued, hit the rate limit, was cut off midway, fell into the cache, and was paid for in tokens. The context bundle left by the same road, and the fields in it had to be counted. Taking the endpoint onto **your own hardware** moves all these numbers to another plane: network latency and rate limit go away, and in their place come a memory budget, the number of concurrent requests, and the hardware itself. The next topic measures this trade-off and asks the first question like this: when you host a model yourself, what exactly is held in memory, and how many concurrent requests does that budget cover. ## Local Models and Adaptation Counting the tradeoffs of running a model on your own hardware and of adapting it: the number of values that parameter count and context length hold in memory and the concurrent requests a budget affords, the memory that falls as the number of digits drops and the point where accuracy falls and which task it falls on first, the prompt-side mechanism and fine-tuning measured side by side on the same task and the case where fine-tuning does not win, the updated-parameter share of full update, layer freezing and low-rank update with its accuracy and the count of stored values, the gap a small model trained on a large model's output closes and the gap it leaves open, and the deployment options that four classes of usage conditions close in one scenario. ### Self-Hosted Models Source: https://academia.sh/en/courses/model-apis/self-hosted-models Running a model on your own hardware makes the unit memory instead of tokens: the weight memory of three fictional scales is 70.5 million, 367.5 million and 2.14 billion values, the intermediate state per request in a 960-token context is 11.8 million, 47.2 million and 157.3 million values, the concurrent requests a three-billion-value budget affords are 248, 55 and 5, and the label accuracy on the same 120 fictional tickets is 0.6333, 0.8167 and 0.8667. The product of concurrent requests and accuracy falls from 157.1 to 4.3, so the largest scale yields one thirty-sixth of the correct-response capacity. With the distribution temperature at zero both runs give the same 0.8667, and at 0.02 28 of the 120 tickets change from run to run. In the previous twenty lessons the model was always an endpoint standing **far away**. Everything measured was read from the surface of that endpoint: how many calls went out, how many tokens were spent, how many requests were rejected, how many responses were cut short. All of those numbers shared one assumption: the model itself sits on someone else's machine, and how many requests are processed there at the same time is **not your problem**. When the model moves to your own hardware, that assumption falls and the unit changes. What is spent is no longer tokens but **memory**, and memory is not a growing bill but a **fixed ceiling**. This lesson counts that ceiling: how many values the weights of a scale hold in memory, how many values the intermediate state of one request adds, and how many **concurrent requests** a given budget affords. The memory accounting on the training side was measured in the Training Practice topic of the Deep Learning course; that procedure is not repeated here, only its counterpart on the inference side is built. - **LA1.** The code is a **simulator**; no real model, runtime or hardware is called. The tickets and the scales are **fictional**, the seed is **20260218**. - **LA2.** Memory is counted as a **count of values**. How many digits a value is stored with is the subject of the next lesson and is held fixed here. - **LA3.** A scale is given by three structural quantities: the **number of layers**, the **hidden width** and the vocabulary size. The parameter count is computed from them. - **LA4.** The simulator's own weight table is small and exists only to produce **accuracy**. The memory accounting is done from the structural description of the scales, not from the simulator's table. - **LA5.** Weight memory is **independent** of the requests and is a single copy. The **intermediate state** is separate for each request and is directly proportional to the context length. - **LA6.** The intermediate state keeps two arrays of width size in every layer for every token in the context; that is, per request two times layers times width times context values. - **LA7.** As the scale grows, the number of **distinguishing rows** of the simulator grows (12, 26, 121). This models capability growing with scale and carries no other claim. - **LA8.** With the distribution temperature at zero the **most likely choice** is made and the output is the same from run to run; above zero, seed-dependent sampling is performed. - **LA9.** The budget is three billion values; weights and intermediate state are paid for from the **same** budget. - **LA10.** The units measured are the count of values and the number of concurrent requests; raw time is not written. #### Three Scales, Three Capabilities The simulator splits a text into pieces of five characters and keeps a **weight row** for each piece. The row has a base shared by all five topics; what distinguishes is the **small difference** added on top of the base for each topic. While reading a ticket, the model takes the highest value it sees for each topic and picks the topic with the highest score. Scale changes only one thing here: how many pieces have a distinguishing row. ```python # SIMULATOR -- FICTIONAL. There is no real model, runtime or hardware; # memory is counted as a count of values. import math SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def text_hash(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek", "odedim ama sistemde borc gorunuyor", "ayrintili dokumu nereden gorebilirim"], "breakdown": ["evde su akmiyor bir sorun var", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "reading": ["okuma yanlis girilmis endeks tutmuyor", "cihaz degisimi icin randevu istiyorum", "endeksi ile fatura ayni degil"], "tariff": ["hangi basamaktayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmaliyim", "birim ucret abone tipine gore degisir mi"], "outage": ["mahallede elektrik yok ne zaman gelecek", "planli duyuru nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi", "ekip bilgi verecekti", "degisiklik talebim de var"] TICKETS = [] # 120 FICTIONAL support tickets for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "topic": k, "text": m}) PIECE_TOPICS = {} # which topic's template each piece occurs in for k, items in TEMPLATES.items(): for m in items: for t in set(tokenize(m)): PIECE_TOPICS.setdefault(t, {})[k] = 1 for e in EXTRAS: for t in set(tokenize(e)): PIECE_TOPICS.setdefault(t, {}) BASE = 0.50 # weight of a piece shared by all topics WEIGHTS = {} for i, t in enumerate(sorted(PIECE_TOPICS)): r = rng(SEED + 7919 * (i + 1)) row = {k: BASE for k in TOPICS} if PIECE_TOPICS[t]: for k in PIECE_TOPICS[t]: row[k] = BASE + 0.05 + 0.40 * r() # small distinguishing difference else: row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r() # misleading link WEIGHTS[t] = row ORDER = sorted(WEIGHTS, key=lambda t: (BASE - max(WEIGHTS[t].values()), t)) SCALES = {"small": (12, 512, 12), "medium": (24, 1024, 26), "large": (40, 2048, len(WEIGHTS))} # layers, width, distinguishing rows def score(text, rows): learned = set(ORDER[:rows]) p = {k: BASE for k in TOPICS} for t in tokenize(text): if t in learned: for k in TOPICS: p[k] = max(p[k], WEIGHTS[t][k]) return p def respond(text, scale, temperature=0.0, seed=SEED): p = score(text, SCALES[scale][2]) if temperature <= 0.0: return max(sorted(p), key=lambda a: p[a]) # most likely choice u = [math.exp((p[k] - max(p.values())) / temperature) for k in TOPICS] return TOPICS[pick(rng(seed ^ text_hash(text))(), [x / sum(u) for x in u])] print("vocabulary", len(WEIGHTS), "pieces ; sample ticket:", TICKETS[0]["text"]) print("scale accuracy") for name in SCALES: d = sum(respond(x["text"], name) == x["topic"] for x in TICKETS) print(f"{name:<7} {d / 120:.4f}") print("\ntemperature seed 20260218 seed 20260219") for s in (0.0, 0.02): a, b = (sum(respond(x["text"], "large", s, t) == x["topic"] for x in TICKETS) / 120 for t in (SEED, SEED + 1)) print(f"{s:>11.2f} {a:>13.4f} {b:>13.4f}") ``` ``` vocabulary 121 pieces ; sample ticket: planli duyuru nereden takip edilir scale accuracy small 0.6333 medium 0.8167 large 0.8667 temperature seed 20260218 seed 20260219 0.00 0.8667 0.8667 0.02 0.8167 0.8583 ``` The first table says what the scale buys. The small scale assigns **0.6333** of the hundred and twenty tickets to the right topic, the medium **0.8167**, the large **0.8667**. The difference between them is not capability itself but how many distinctions have been learned: the small scale knows which topic only twelve pieces point to, and the remaining one hundred and nine pieces are indistinguishable from one another for it. The second table shows a side of self-hosting that an endpoint far away does not have. With the distribution temperature at zero, two different seeds give the same **0.8667**; the response to no ticket changes. When the temperature rises to 0.02, the same two runs give **0.8167** and **0.8583**. The sampling setting is not in the provider's hands but in yours; determinism becomes not a request but a **configuration**. This is also the reason the numbers in the following sections stay fixed from run to run. #### Memory Budget and Concurrent Requests Memory consists of two separate items, and the two behave very differently. **Weight memory** holds the parameters of the model; it is a single copy no matter how many requests arrive, and it does not change. The **intermediate state** is kept for the context of each request; it grows with the number of requests and with the context length. The budget has to cover both, so the number of concurrent requests is determined by what remains of the budget after the weights are subtracted. ```python BUDGET, VOCAB = 3_000_000_000, 32000 # memory budget and FICTIONAL structural vocabulary CONTEXTS = (240, 960, 3840) def parameter_count(L, H): return 12 * L * H * H + 2 * VOCAB * H def intermediate_state(L, H, context): return 2 * L * H * context print("scale weights(count) state/request concurrent accuracy capacity") for name, (L, H, rows) in SCALES.items(): a, d = parameter_count(L, H), intermediate_state(L, H, 960) i = sum(respond(x["text"], name) == x["topic"] for x in TICKETS) / 120 e = max(0, (BUDGET - a) // d) print(f"{name:<7} {a:>14} {d:>14} {e:>11} {i:>9.4f} {e * i:>9.1f}") print("\nscale " + " ".join(f"context {b:>4}" for b in CONTEXTS)) for name, (L, H, rows) in SCALES.items(): a = parameter_count(L, H) e = [max(0, (BUDGET - a) // intermediate_state(L, H, b)) for b in CONTEXTS] print(f"{name:<7} " + " ".join(f"{x:>12}" for x in e)) print("\nrounds for 120 requests (context 960):") for name, (L, H, rows) in SCALES.items(): e = max(0, (BUDGET - parameter_count(L, H)) // intermediate_state(L, H, 960)) print(f" {name:<7} concurrent {e:>4} round {-(-120 // e):>3}") print("if the budget drops to 2000000000 values, the large scale cannot be loaded:", parameter_count(40, 2048) > 2_000_000_000) ``` ``` scale weights(count) state/request concurrent accuracy capacity small 70516736 11796480 248 0.6333 157.1 medium 367525888 47185920 55 0.8167 44.9 large 2144337920 157286400 5 0.8667 4.3 scale context 240 context 960 context 3840 small 993 248 62 medium 223 55 13 large 21 5 1 rounds for 120 requests (context 960): small concurrent 248 round 1 medium concurrent 55 round 3 large concurrent 5 round 24 if the budget drops to 2000000000 values, the large scale cannot be loaded: True ``` The first column gives the distance between the scales: weight memory is **70.5 million**, **367.5 million** and **2.14 billion** values. When the width doubles, the layer parameters quadruple, because the parameter count of each layer grows with the square of the width. The effect of the number of layers, on the other hand, is linear. The difference between these two growth rates explains why the choice of scale is not gradual but jumps. The second column is less intuitive. The intermediate state grows with the width itself, **not its square**, but it is directly proportional to the context length. In the large scale, a context of 960 tokens asks for **157.3 million** values per request; that is one thirteenth of the weights of the same scale. Thirteen concurrent requests consume the same memory as loading a second model. #### The Choice the Budget Imposes The concurrent request column turns the table into a choice. The three-billion-value budget can carry **248** requests at the same time in the small scale, **55** in the medium scale and **5** in the large scale, with a context of 960 tokens. Moving to the large scale raises accuracy from 0.8167 to 0.8667; the gain is **0.0500**. The cost of the same step is that concurrent requests drop to **one tenth**. The last column reduces these two numbers to a single measure. The product of concurrent requests and accuracy is the **correct-response capacity** the budget can produce at the same time: **157.1** in the small scale, **44.9** in the medium, **4.3** in the large. The largest scale gives the highest accuracy and produces **one thirty-sixth** of the correct-response capacity. Which column wins depends on the workload: if the number of incoming tickets is below what the budget can carry, the accuracy column decides; if it is above, the capacity column decides. This is a question never asked at a distant endpoint, because there concurrency is paid for from the provider's budget and the bill shows up only as tokens. The second table shows the same tradeoff on the context axis. When the context goes from 240 to 3840, concurrent requests fall to one sixteenth in every scale; in the large scale to **1**, that is, to queueing. Context length looks like a quality setting but in a local setup it is **directly a capacity setting**. The last line writes the hard side of the ceiling. If the budget drops to two billion values, the large scale cannot be loaded; the number of concurrent requests does not fall, it becomes **zero**. At a distant endpoint, overload produces a rate limit response and is met with a retry; in a local setup the counterpart is **never starting**. The two failures do not have the same form, and the second has no retry. #### Concurrency Is a Latency Class The number of concurrent requests alone is a capacity measure; applied to a workload it turns into a **latency class**. When all one hundred and twenty fictional tickets are queued, the small scale serves them all in **one round**, the medium scale needs **three**, the large scale **twenty-four** rounds. The round count is not raw time and does not vary with hardware; it is a number, fixed from run to run, that comes out of the ratio of the budget to the intermediate state. This is exactly what is needed to measure how long the last ticket in line waits: a queue of twenty-four rounds is not in the same latency class as a queue of one round. The question of which column a runtime choice touches in this table and which it cannot touch is answered from here as well. The weight column comes from the structural description of the scale and does not depend on which runtime is chosen. The only item a runtime can change is **how the intermediate state is held**: whether the part of the context shared between requests is shared, whether the place of a finished request is freed immediately, how many requests are processed together in a round. All of these move the second column, and therefore the number of concurrent requests and the round count. The measurable way to compare one runtime with another is also to put these two numbers side by side at the same budget and the same context length; the column is compared, not the name. #### Summary - In self-hosting the unit is the **count of values**, not tokens; memory is not a growing bill but a fixed ceiling. - Weight memory is a single copy and independent of the requests; the intermediate state is separate for each request and grows in direct proportion to the context length. - The weights of the three fictional scales are 70.5 million, 367.5 million and 2.14 billion values; when the width doubles, the layer parameters quadruple. - A three-billion-value budget affords 248, 55 and 5 concurrent requests in a 960-token context; accuracy is 0.6333, 0.8167 and 0.8667. - The product of concurrent requests and accuracy falls from 157.1 to 4.3: the scale with the highest accuracy yields one thirty-sixth of the correct-response capacity. - With the distribution temperature at zero both runs give the same 0.8667; in a local setup determinism is a configuration decision. #### Next Step All of this lesson's accounting rested on a single assumption: memory was counted as a **count of values**, and how much room a value takes in memory was assumed fixed. That assumption can be lifted. If a value is stored with fewer digits, the same parameter count takes less room, and a larger scale or more concurrent requests fit into the same budget. In return, the **small differences** the model rests on become coarser, and what distinguished the weight rows of this lesson was exactly those small differences. The next lesson lowers the number of digits at three levels and prints two numbers side by side: how much the memory falls, at what point accuracy falls, and **on which task it falls first**. ### Quantization Source: https://academia.sh/en/courses/model-apis/quantization Measuring on the same 120 fictional tickets how the number of digits a weight is stored with affects memory and accuracy: at 16 digits weight memory is 34.3 billion digits and concurrent requests are 5, at 8 digits 17.2 billion and 24, at 4 digits 8.6 billion and 62. Five-class topic assignment is 0.8667 at 16 digits, stays at 0.8750 at 8 digits, and falls to 0.8000 at 4 digits, 0.6333 at 3 digits and 0.2083 at 2 digits; the two-class urgency distinction at the last four levels gives 0.9417, 0.9333, 0.8583 and 0.3583. Because the mean decision margin is 0.1597 in five classes and 0.2665 in two classes, the fine distinction goes first. The distinguishing difference of the 121 rows is erased completely in 24 rows at 4 digits and in 53 rows at 3 digits. In the previous lesson memory was counted as a **count of values**, and how much room a value takes in memory was held fixed. That assumption is a choice, not a necessity. The same weight can be stored with fewer digits; then memory shrinks while the parameter count stays the same, more concurrent requests fit into the same budget, and the ceiling in the last table of the previous lesson rises. The price is in the same place. What distinguished the weight rows of the previous lesson was not the shared base but the **small difference** added to the base for each topic. As the number of digits falls, the representable values grow sparse, and past a certain point those small differences are rounded to the same value. This lesson places three digit levels side by side and answers two questions separately: how much the memory falls, **at what point accuracy** falls and **on which task first**. - **LA11.** The code is a **simulator**; the simulator of the previous lesson is rebuilt in shortened form. The tickets are **fictional**, the seed is **20260218**. - **LA12. Quantization here is a single operation:** the range from zero to two is divided into the equal step given by the number of digits, and each weight is rounded to the nearest step. The only quantity measured is the **number of digits**. - **LA13.** Memory is counted in **digits**: weight memory is the parameter count times the number of digits. The intermediate state is held with the same number of digits. - **LA14.** The budget is 48 billion digits; this is the 16-digit counterpart of the previous lesson's three-billion-value budget. The scale is fixed and is the large scale of the previous lesson. - **LA15.** The same response is read at **three resolutions**: five-class topic assignment, three-class team routing (billing, field, meter) and two-class urgency distinction. The mappings are **fictional**. - **LA16. Margin** is the robustness of a decision: the distance between the score of the chosen topic and the score of the nearest rival. The rival is defined differently in the three tasks, so the margin is different too. - **LA17.** The distribution temperature is zero; all numbers are independent of the run. In place of uncertainty, the **margin distribution** is measured. - **LA18.** Ties are resolved alphabetically. When all weights are rounded to the same value, the model collapses to a single label. - **LA19.** The number of digits affects only the weights; tokenization, the vocabulary and the task definitions do not change. - **LA20.** The concurrent request count is the same as the previous lesson's computation; the only input that changes is how many digits a value holds. #### What the Number of Digits Frees in Memory The memory side of quantization is linear and beyond dispute: if the digits are halved, weight memory is halved. The interesting part is the second effect. Because the intermediate state is held with the same number of digits, the budget is freed in two places at once, and the concurrent request count grows **faster than the ratio of the digits** as the number of digits falls: the weights are subtracted from the budget once, and all of the remainder goes to requests. ```python # SIMULATOR -- FICTIONAL. The simulator of the previous lesson, built in shortened form. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek", "odedim ama sistemde borc gorunuyor", "ayrintili dokumu nereden gorebilirim"], "breakdown": ["evde su akmiyor bir sorun var", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "reading": ["okuma yanlis girilmis endeks tutmuyor", "cihaz degisimi icin randevu istiyorum", "endeksi ile fatura ayni degil"], "tariff": ["hangi basamaktayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmaliyim", "birim ucret abone tipine gore degisir mi"], "outage": ["mahallede elektrik yok ne zaman gelecek", "planli duyuru nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi", "ekip bilgi verecekti", "degisiklik talebim de var"] TICKETS = [] # 120 FICTIONAL support tickets for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "topic": k, "text": m}) PIECE_TOPICS = {} for k, items in TEMPLATES.items(): for m in items: for t in set(tokenize(m)): PIECE_TOPICS.setdefault(t, {})[k] = 1 for e in EXTRAS: for t in set(tokenize(e)): PIECE_TOPICS.setdefault(t, {}) BASE, WEIGHTS = 0.50, {} for i, t in enumerate(sorted(PIECE_TOPICS)): r = rng(SEED + 7919 * (i + 1)) row = {k: BASE for k in TOPICS} if PIECE_TOPICS[t]: for k in PIECE_TOPICS[t]: row[k] = BASE + 0.05 + 0.40 * r() else: row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r() WEIGHTS[t] = row GROUP = {"invoice": "billing", "tariff": "billing", "breakdown": "field", # team routing "outage": "field", "reading": "meter"} URGENCY = {"invoice": "normal", "tariff": "normal", "reading": "normal", # urgency distinction "breakdown": "urgent", "outage": "urgent"} def quantize(x, bits): """The range from zero to two is divided into 2**bits - 1 equal steps; rounded to the nearest step.""" a = 2.0 / (2 ** bits - 1) return round(x / a) * a def score(text, bits): p = {k: quantize(BASE, bits) for k in TOPICS} for t in tokenize(text): if t in WEIGHTS: for k in TOPICS: p[k] = max(p[k], quantize(WEIGHTS[t][k], bits)) return p def choose(p): return max(sorted(p), key=lambda a: p[a]) PARAMETERS, STATE, BUDGET = 2144337920, 157286400, 48_000_000_000 print("digits step weights(digits) concurrent five-class three-class two-class") for bits in (16, 12, 8, 6, 5, 4, 3, 2): a5 = a3 = a2 = 0 for x in TICKETS: s = choose(score(x["text"], bits)) a5 += s == x["topic"] a3 += GROUP[s] == GROUP[x["topic"]] a2 += URGENCY[s] == URGENCY[x["topic"]] e = max(0, (BUDGET // bits - PARAMETERS) // STATE) print(f"{bits:>6} {2.0 / (2 ** bits - 1):>7.5f} {PARAMETERS * bits:>16} {e:>11}" f" {a5 / 120:>11.4f} {a3 / 120:>12.4f} {a2 / 120:>10.4f}") ``` ``` digits step weights(digits) concurrent five-class three-class two-class 16 0.00003 34309406720 5 0.8667 0.8667 0.9417 12 0.00049 25732055040 11 0.8667 0.8667 0.9417 8 0.00784 17154703360 24 0.8750 0.8750 0.9417 6 0.03175 12866027520 37 0.8500 0.8750 0.9417 5 0.06452 10721689600 47 0.8417 0.8667 0.9333 4 0.13333 8577351680 62 0.8000 0.9083 0.9333 3 0.28571 6433013760 88 0.6333 0.7583 0.8583 2 0.66667 4288675840 138 0.2083 0.3583 0.3583 ``` The three main rows of the table are 16, 8 and 4 digits. Memory goes from **34.3 billion** to 17.2 billion and from there to 8.6 billion digits; exactly a quarter. Concurrent requests, on the other hand, go from 5 to 24 and from there to **62**: twelve times. This disproportion comes from LA13. The weights are subtracted from the budget once and take almost the whole budget at 16 digits; as the digits decrease, the share of the weights shrinks and **all** of the remainder goes to requests. The accuracy columns do not move at the same speed. Five-class topic assignment is **0.8667** at 16 digits and **0.8750** at eight digits, that is, the first halving leaves no measurable trace in accuracy. At six and five digits it slips slightly, to 0.8500 and 0.8417. At four digits **0.8000**, at three digits **0.6333**, at two digits **0.2083**. The last row is not an accuracy but a collapse: 0.2083 is the rate at which the simulator falls on the alphabetically first label. #### The Margin Falling Below the Step The reason for this fall is not something to guess but something to compute. The robustness of a decision is the **margin**: the distance between the score of the chosen topic and the score of its nearest rival. Rounding can move this distance only by about a step; if the margin is larger than the step the decision stays in place, if it is smaller it may vanish. Both measures below are independent of the run and can be looked at before accuracy is measured. ```python def margins(bits): f = {"five-class": [], "three-class": [], "two-class": []} for x in TICKETS: p = score(x["text"], bits) v = sorted(p.values(), reverse=True) s = choose(p) f["five-class"].append(v[0] - v[1]) f["three-class"].append(p[s] - max(p[k] for k in TOPICS if GROUP[k] != GROUP[s])) f["two-class"].append(p[s] - max(p[k] for k in TOPICS if URGENCY[k] != URGENCY[s])) return f print("task mean margin share with margin below the step (digits 8 / 4 / 3)") for name, f in margins(16).items(): o = [sum(1 for x in f if x < 2.0 / (2 ** b - 1)) / 120 for b in (8, 4, 3)] print(f"{name:<11} {sum(f) / 120:>11.4f} " + " ".join(f"{x:.4f}" for x in o)) print("\ndigits distinct values rows with distinction erased (of 121 rows)") for b in (16, 8, 4, 3, 2): d = {quantize(w, b) for s in WEIGHTS.values() for w in s.values()} z = sum(1 for s in WEIGHTS.values() if len({quantize(w, b) for w in s.values()}) == 1) print(f"{b:>6} {len(d):>15} {z:>27}") print("\nthe 'endek' row at the digit levels:") for b in (16, 8, 4, 3): print(f" {b:>2} " + " ".join(f"{k}={quantize(WEIGHTS['endek'][k], b):.4f}" for k in TOPICS)) ``` ``` task mean margin share with margin below the step (digits 8 / 4 / 3) five-class 0.1597 0.0167 0.4167 0.7250 three-class 0.1995 0.0167 0.2750 0.6667 two-class 0.2665 0.0083 0.2000 0.4500 digits distinct values rows with distinction erased (of 121 rows) 16 133 0 8 55 0 4 4 24 3 2 53 2 1 121 the 'endek' row at the digit levels: 16 invoice=0.5000 breakdown=0.5000 reading=0.7095 tariff=0.5000 outage=0.5000 8 invoice=0.5020 breakdown=0.5020 reading=0.7059 tariff=0.5020 outage=0.5020 4 invoice=0.5333 breakdown=0.5333 reading=0.6667 tariff=0.5333 outage=0.5333 3 invoice=0.5714 breakdown=0.5714 reading=0.5714 tariff=0.5714 outage=0.5714 ``` The first table explains the order of the accuracy table **without looking at accuracy**. The mean margin is **0.1597** in the five-class task, 0.1995 in the three-class task and **0.2665** in the two-class task. As the task gets coarser the rival moves away, because confusion between invoice and tariff does not break the team routing, and confusion between breakdown and outage does not break the urgency. The same response, read coarsely, is a sturdier decision. The three columns on the right count this level by level. At eight digits the step is 0.00784 and the margin is below this step in only **0.0167** of the records; that accuracy moves so little at this level is therefore an expected result. At four digits the step rises to 0.13333 and **0.4167** of the records become fragile in the five-class task, and **0.2000** in the two-class task. At three digits the shares are 0.7250 and 0.4500. The order of the fragility share is the same as the order in which accuracy falls, and this order can be computed before the digits are lowered. The second table shows the mechanism. The 605 weights of the 121 rows carry **133** distinct values at 16 digits; at eight digits this falls to 55 but no row loses its distinction. At four digits the number of representable values falls to **4**, and the values that **24 rows** hold for the five topics are made equal to one another; those rows no longer distinguish anything. At three digits this number is 53, at two digits 121, that is, the whole table. Quantization does not shrink the weights, it **makes them indistinguishable from one another**; what is erased is not magnitude but distinction. The last block shows the same thing in a single row. The `endek` piece, which points to a meter reading, carries 0.7095 for the meter and 0.5000 for the other four topics at 16 digits; the distinction is 0.2095. At eight digits it is 0.7059 against 0.5020, a distinction of 0.2039, almost the same. At four digits the values settle at 0.6667 and 0.5333 and the distinction falls to 0.1334. At three digits all five values are **0.5714**; the row no longer points to the meter. None of the weights was zeroed and none shrank meaningfully; only the distance between them became unrepresentable. #### The Point Where Accuracy Falls The place to read in the table is between four and three digits. Going from eight to four, five-class accuracy slips 0.0750 and memory is halved; going from four to three, accuracy slips **0.1667** and memory falls by only another quarter. The curve of cost and the curve of gain cross here: below four digits, each digit gained comes at a higher price than the one before. The answer to which task falls first is also in the table. At four digits the five-class distinction falls from 0.8667 to **0.8000**, while the three-class team routing stays inside the band between 0.8667 and 0.9083 and the two-class urgency distinction slips from 0.9417 to only **0.9333**. Two digits lower, at three digits, the coarse tasks fall too, but far behind the five-class task: 0.7583 and 0.8583. The rule is this: **the fine distinction goes first.** The same quantized model has become unable to tell which topic a ticket belongs to while still knowing which team to send it to. The counterpart in design is direct. Whether lowering the number of digits is acceptable in a setup depends not on the model but on the **task**. If two jobs run on the same weights, running the coarse one with four digits and running the fine one with four digits are not the same decision. What has to be measured is not a single accuracy but a **separate accuracy for each task**. #### Summary - Quantization here is the division of a range into equal steps; the only quantity measured is the number of digits, and memory is written as parameter count times digits. - Weight memory is 34.3 billion at 16 digits, 17.2 billion at 8 digits, 8.6 billion at 4 digits; the concurrent requests in the same budget rise from 5 to 24 and to 62, because the weights are subtracted from the budget once. - Five-class accuracy stays in the 0.8667 band down to 8 digits, falls to 0.8000 at 4 digits, 0.6333 at 3 digits and 0.2083 at 2 digits; the last row is an alphabetical collapse, not an accuracy. - The mean decision margin is 0.1597 in five classes and 0.2665 in two classes; the share with the margin below the step is 0.4167 and 0.2000 at 4 digits, and this order gives the order in which accuracy falls before the digits are lowered. - 605 weights fall to 4 distinct values at 4 digits and 24 rows lose all of their distinction; quantization erases distinction, not magnitude. - The point where accuracy falls is between four and three digits, and the task that falls first is five-class topic assignment; the coarse tasks hold two digits longer. #### Next Step In these two lessons the weights were always taken as **given**. A scale was chosen, the number of digits was lowered, memory and accuracy were written side by side; but the weights themselves were never touched. Yet this is the real possibility of a local setup that a distant endpoint does not have: because the weights are in your hands, they can be **changed**. If training is continued on a few hundred examples of your own data, the model's accuracy on this task may rise. Whether it rises, by how much, and whether the same gain could already be had by adding a few sentences to the prompt were not measured. The next lesson places the **prompt-side mechanism** and **fine-tuning** side by side on the same task, writes the cost in example count and run count, and shows a point where fine-tuning **does not win**. ### Fine-Tuning Source: https://academia.sh/en/courses/model-apis/fine-tuning Measuring side by side on the same 60 fictional test records what happens when the same labeled examples are put into the prompt and when they are written into the weights: the baseline without scaffolding is 0.6000, fine-tuning on four examples moves no weight and stays at 0.6000, at eight examples it reaches 0.7333 and passes the 0.6500 of prompt scaffolding, at sixteen examples 0.8833. At thirty-two examples the two paths overlap in the band 0.9000 to 0.9500, at sixty examples prompt scaffolding gives 0.9667 and fine-tuning stays at 0.9500: fine-tuning does not win. The cost is in two separate units; prompt scaffolding raises the context per request from 28.48 to 623.48 tokens, that is 21.9 times, while fine-tuning is a one-time 180 update steps and moves a 0.2992 share of the 605 weights. In the previous two lessons the weights were always taken as given: a scale was chosen, the number of digits was lowered, and memory and accuracy were written side by side. The weights themselves were never touched. Yet this is the real possibility of a local setup that a distant endpoint does not have; because the weights are in your hands, they can be changed. The table at the end of the second lesson imposed a choice: when the budget forces the small scale, accuracy on this task stays low. There are two ways to raise it, and both need **the same thing**, labeled examples. The first way is to **put the examples in the prompt**; with every request the model is shown a few solved tickets. The second way is to **write the examples into the weights**; the model is trained once and the prompt stays short. This lesson measures both at the same example count and shows the point where fine-tuning **does not win**. The two-stage structure of pretraining and fine-tuning was covered in the architectures topic of the Introduction to Machine Learning course and in the language models topic of the Introduction to AI Engineering course; that distinction is not repeated here, only the numbers of the two paths are compared. - **LA21.** The code is a **simulator**; the tickets are **fictional**, the seed is **20260218**. The scale is the **small** scale of the previous lessons, because the budget forced it. - **LA22.** The first 60 of the 120 fictional tickets are the **training pool**, the last 60 are the **test set**. No scaffolding touches the test set. - **LA23. Prompt scaffolding:** the chosen examples are put into the prompt. An example in the context gives its own pieces a **temporary** support for its own topic; the support is not carried from request to request. - **LA24. Fine-tuning:** an **error-driven** update on the same examples. Weights move only on examples the model knows wrongly, and the weights that move are **permanent**. - **LA25.** The number of epochs is how many times the training set is passed through from start to end; the number of update steps is the example count times the number of epochs. - **LA26.** The two paths use **the same examples**. The comparison is at equal labeled example counts; the only thing that changes is where the example is put. - **LA27.** The two runs are **two separate draws** of the examples (seed 20260218 and 20260219). At sixty examples the two draws give the same set, so the difference there is zero. - **LA28. Gain rule:** the superiority of a path does not count as measured unless it points the same way in **both** draws. - **LA29.** The cost is in two separate units: prompt scaffolding is **tokens per request**, fine-tuning is **a one-time example count and epoch count**. - **LA30.** The weight table is 121 rows times five topics, that is **605 weights**; the number of records fine-tuning moves is counted out of this. #### The Same Examples, Two Different Places The simulator is the same as in the previous lessons; the only change is that a **correction table** can be added on top of the score. If the correction comes from the prompt it is temporary and goes away with the context of that request; if it comes from training it is permanent and the prompt stays short. The input of both is the same labeled examples. ```python # SIMULATOR -- FICTIONAL. The simulator of the previous lessons, built in shortened form. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek", "odedim ama sistemde borc gorunuyor", "ayrintili dokumu nereden gorebilirim"], "breakdown": ["evde su akmiyor bir sorun var", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "reading": ["okuma yanlis girilmis endeks tutmuyor", "cihaz degisimi icin randevu istiyorum", "endeksi ile fatura ayni degil"], "tariff": ["hangi basamaktayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmaliyim", "birim ucret abone tipine gore degisir mi"], "outage": ["mahallede elektrik yok ne zaman gelecek", "planli duyuru nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi", "ekip bilgi verecekti", "degisiklik talebim de var"] TICKETS = [] # 120 FICTIONAL support tickets for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "topic": k, "text": m}) PIECE_TOPICS = {} for k, items in TEMPLATES.items(): for m in items: for t in set(tokenize(m)): PIECE_TOPICS.setdefault(t, {})[k] = 1 for e in EXTRAS: for t in set(tokenize(e)): PIECE_TOPICS.setdefault(t, {}) BASE, WEIGHTS = 0.50, {} for i, t in enumerate(sorted(PIECE_TOPICS)): r = rng(SEED + 7919 * (i + 1)) row = {k: BASE for k in TOPICS} if PIECE_TOPICS[t]: for k in PIECE_TOPICS[t]: row[k] = BASE + 0.05 + 0.40 * r() else: row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r() WEIGHTS[t] = row ROWS = 12 # small scale: the scale the budget chose LEARNED = set(sorted(WEIGHTS, key=lambda t: (BASE - max(WEIGHTS[t].values()), t))[:ROWS]) TRAIN, TEST = TICKETS[:60], TICKETS[60:] INSTRUCTION = "Label this ticket as one of: invoice breakdown reading tariff outage . " def score(text, correction=None): """The base score is the highest value of the learned rows; the correction is added on top.""" pieces = tokenize(text) p = {k: BASE for k in TOPICS} for t in pieces: if t in LEARNED: for k in TOPICS: p[k] = max(p[k], WEIGHTS[t][k]) for t in pieces: for k in TOPICS: p[k] += (correction or {}).get((t, k), 0.0) return max(sorted(p), key=lambda a: p[a]) def measure(correction=None): return sum(score(x["text"], correction) == x["topic"] for x in TEST) / 60 print("training pool", len(TRAIN), "records ; test", len(TEST), "records") print("baseline accuracy (small scale without scaffolding):", f"{measure():.4f}") print("instruction tokens:", len(tokenize(INSTRUCTION)), "; mean ticket tokens:", round(sum(len(tokenize(x["text"])) for x in TEST) / 60, 2)) ``` ``` training pool 60 records ; test 60 records baseline accuracy (small scale without scaffolding): 0.6000 instruction tokens: 18 ; mean ticket tokens: 10.48 ``` The baseline is **0.6000**. This is the state, on this task, of the scale the budget imposed at the end of the second lesson, and it is the number both paths have to beat. An average ticket is 10.48 tokens and the instruction is 18 tokens; that is, a request without scaffolding is about 28 tokens. This number will be the yardstick when reading the cost of prompt scaffolding. #### As the Example Count Grows The code below puts the same example set in both places and measures only on the test set. The examples are chosen from the training pool with two separate draws; the second draw is there to decide whether a gain counts as measured. ```python EXAMPLE, STEP = 0.30, 0.05 # support from the context, and the fine-tuning step def pick_examples(n, seed): r, h = rng(seed), list(range(60)) for i in range(59, 0, -1): j = int(r() * (i + 1)) h[i], h[j] = h[j], h[i] return [TRAIN[i] for i in sorted(h[:n])] def from_context(examples): # examples put into the prompt: temporary support return {(t, o["topic"]): EXAMPLE for o in examples for t in tokenize(o["text"])} def train_model(examples, epochs): # error-driven update; the weights are permanent D = {} for _ in range(epochs): for o in examples: s = score(o["text"], D) if s != o["topic"]: for t in tokenize(o["text"]): D[(t, o["topic"])] = D.get((t, o["topic"]), 0.0) + STEP D[(t, s)] = D.get((t, s), 0.0) - STEP return D print(" n prompt A prompt B tune k=1 tune k=3 tuneB k=3 prompt tok updated") for n in (4, 8, 16, 32, 60): a, b = pick_examples(n, SEED), pick_examples(n, SEED + 1) ia, ib = measure(from_context(a)), measure(from_context(b)) y1, y3, yb = measure(train_model(a, 1)), measure(train_model(a, 3)), measure(train_model(b, 3)) tok = sum(len(tokenize(o["text"])) for o in a) print(f"{n:>2} {ia:>8.4f} {ib:>8.4f} {y1:>8.4f} {y3:>8.4f} {yb:>9.4f}" f" {tok:>10} {len(train_model(a, 3)):>7}") print("\nn=60 fixed , number of epochs varies:") print(" k accuracy update steps updated records") for k in (1, 2, 3, 6, 10): D = train_model(pick_examples(60, SEED), k) print(f" {k:>2} {measure(D):>8.4f} {60 * k:>12} {len(D):>15}") mean = sum(len(tokenize(x["text"])) for x in TEST) / 60 tok60 = sum(len(tokenize(o["text"])) for o in pick_examples(60, SEED)) with_scaffolding = len(tokenize(INSTRUCTION)) + tok60 + mean tuned = len(tokenize(INSTRUCTION)) + mean D60 = train_model(pick_examples(60, SEED), 3) print(f"\ncontext per request: prompt scaffolding {with_scaffolding:.2f} , fine-tuning {tuned:.2f} tokens" f" ({with_scaffolding / tuned:.1f} times)") print(f"cost of fine-tuning n=60 , k=3: {60 * 3} update steps , {len(D60)} records ," f" {len(D60) / 605:.4f} share of the 605 weights") ``` ``` n prompt A prompt B tune k=1 tune k=3 tuneB k=3 prompt tok updated 4 0.5667 0.6500 0.6000 0.6000 0.7333 38 0 8 0.6500 0.6667 0.7333 0.7333 0.8000 73 35 16 0.8333 0.7500 0.8833 0.8833 0.8667 138 49 32 0.9000 0.9500 0.8500 0.9167 0.9000 292 101 60 0.9667 0.9667 0.9333 0.9500 0.9500 595 181 n=60 fixed , number of epochs varies: k accuracy update steps updated records 1 0.9333 60 121 2 0.9333 120 162 3 0.9500 180 181 6 0.9500 360 181 10 0.9500 600 181 context per request: prompt scaffolding 623.48 , fine-tuning 28.48 tokens (21.9 times) cost of fine-tuning n=60 , k=3: 180 update steps , 181 records , 0.2992 share of the 605 weights ``` The top row is where neither path works. With four examples, fine-tuning **moves no weight**: the number of updated records is **0**, because the model already knew those four examples correctly and an error is needed to trigger an error-driven update. Accuracy stays at the baseline, at 0.6000. When the same four examples are put into the prompt, on the other hand, accuracy **falls** to **0.5667**; the four examples in the context pull a part of the test records toward their own topics. In the second draw the same two numbers are 0.6500 and 0.7333, that is, not even the direction is the same. With four examples nothing has been measured. At eight and sixteen examples fine-tuning is clearly ahead. At eight it is **0.7333** and **0.8000**, above 0.6667, the best of prompt scaffolding in the two draws; at sixteen it is **0.8833** and 0.8667, above 0.8333 and 0.7500 of prompt scaffolding. Because it points the same way in both draws, this superiority counts as measured. The reason is structural: an error-driven update uses an example **several times** and corrects only the places that are known wrongly, while an example put into the context is read once and also moves the places that were already known correctly. #### The Point Where Fine-Tuning Does Not Win At thirty-two examples the table turns. Fine-tuning with three epochs gives **0.9167** and **0.9000**, prompt scaffolding gives **0.9000** and **0.9500**. The bands overlap and the superiority does not point the same way in the two draws; according to LA28 there is no measured gain here. In the same row, fine-tuning with a single epoch is **0.8500**, that is, below both values of prompt scaffolding. Raising the epochs to three gains 0.0667, and this is a gain that comes not from adding examples but from **passing over the same examples again**. At sixty examples no uncertainty is left. Because both draws select the whole pool, the numbers are the same: prompt scaffolding **0.9667**, fine-tuning **0.9500**. The difference is **0.0167** and the run-to-run variation is **zero**. This is the point where fine-tuning **does not win**, and it is not chance. If all sixty examples sit in the context, the model reads every request together with a solved set; fine-tuning, on the other hand, has used from the same examples only the ones that **gave errors**, has moved 181 of the 605 weights and has left the rest untouched. Squeezing all the labeled data you have into a single training run is a loss on this task. More epochs cannot close this gap. When sixty examples are held fixed and the epochs are increased, accuracy is **0.9333** at one epoch, **0.9500** at three, and stays there; six and ten epochs give the same 0.9500. Because after the third epoch no errors are left to correct in the training set, the number of updated records also settles at **181**. The update steps go from 180 to 600 and nothing is received in return. The ceiling of fine-tuning on this task is 0.9500, and that ceiling rises not by adding epochs but only by another way. What decides is not the accuracy column but the last two rows. With sixty examples, prompt scaffolding raises the context of the request from 28.48 tokens to **623.48** tokens, that is, to **21.9 times**, and this cost is paid again on every request. According to the computation of the first lesson, context length is proportional to the intermediate state per request; twenty-one times the context means twenty-one times fewer concurrent requests in the same budget. The cost of fine-tuning, on the other hand, is **one-time**: sixty labeled examples, three epochs, 180 update steps and a 0.2992 share of weights that move. Every later request is still 28.48 tokens. The comparison therefore does not close with a single number. When labeled examples are few, fine-tuning is both ahead and cheaper. As examples multiply, prompt scaffolding catches up and passes in accuracy, but it does so by enlarging the context; if how many requests are processed at the same time matters, an accuracy difference of 0.0167 is not worth twenty-one times the capacity. What has to be measured is both, and these two numbers are not in the same unit. #### Summary - Both paths ask for the same input: labeled examples. The only thing that changes is whether the example is put into the prompt or into the weights; so the comparison is made at equal example counts. - With four examples fine-tuning moves no weight and the baseline stays at 0.6000; when the same four examples are put into the prompt, accuracy falls to 0.5667. The direction of the two draws is not even the same. - At eight and sixteen examples fine-tuning is ahead in both draws: 0.7333 and 0.8833 against 0.6500 and 0.8333. This superiority counts as measured. - At thirty-two examples the bands overlap (0.9000 and 0.9500 against 0.9000 and 0.9167); there is no measured gain, and single-epoch fine-tuning falls behind with 0.8500. - At sixty examples prompt scaffolding is 0.9667 and fine-tuning 0.9500; because the run-to-run variation is zero, the loss of fine-tuning is real. - The costs are not in the same unit: prompt scaffolding raises the context 21.9 times per request, fine-tuning moves a one-time 180 update steps and a 0.2992 share of the 605 weights. #### Next Step The 0.2992 share in the last row was passed over without being asked about. Fine-tuning did not **decide** which weights would move; the error-driven update touched whichever piece passed through an example known wrongly, and 181 records came out that way. This number is a by-product, not a design decision. Yet the number of weights that move is directly a cost: it is the thing that will be stored, carried, rolled back and kept separately for every task. Could the same accuracy be obtained by moving fewer weights, what would be lost if it were decided beforehand which weights to freeze, and how many numbers would be stored if the update itself were written onto the table as an **add-on**: none of these was tried. The next lesson measures side by side the full update, layer freezing and a low-rank correction added to the table, and prints two numbers: the updated-parameter share and accuracy. ### Adaptation Methods Source: https://academia.sh/en/courses/model-apis/adaptation-methods Measuring side by side, in the same field adaptation, the updated-parameter share and accuracy of full update, layer freezing and a low-rank update: on field spelling a fictional local model of 2292 numbers gives 0.2500 and 0.0833, full update 1.0000 and 0.9861, layer freezing that updates only the head 0.6944 and 0.7500, a low-rank update that updates 0.2094 of the parameters 0.9583 and 0.9306. Raising the rank from 3 to 4 makes 210 more numbers stored per adaptation and its mean gain is 0.0070, below the two-run gap. For adaptation to eight domains the stored count is 6132 against 18336, and as the hidden width grows the share of the low-rank update falls from 0.2094 to 0.0284. Fine-tuning carries a model into a new domain, and it does this by **moving all the weights**. The previous lesson wrote the cost of this as example count and epoch count. There is one more cost that was not written: for every domain adapted, **a full copy** of the model is stored. Eight domains means eight copies. This lesson tries moving less. Three setups are placed side by side on the same adaptation task, **full update**, **layer freezing** and a **low-rank update**, and the **updated-parameter share** and the **accuracy** of each are printed together. The number of stored numbers is written next to them, because the real cost of adaptation is paid not during training but after it. - **LA31.** The code is a **simulator**. The local model, vocabulary, records and labels are **fictional**; no real endpoint is called. Seed **20260218**. - **LA32.** The model is three layers: an **input lookup table**, a **body matrix** and a **head**. All of the weights are in our hands; this is the distinguishing side of a model you host yourself. - **LA33.** The two domains are separate: the model is pretrained on **general-spelling** fictional tickets, then adapted to the field team's **abbreviated** fictional records. A share of the records get a misleading suffix carrying the abbreviation of another topic. - **LA34. The low-rank update** does not change the weight matrix in place; next to it, it adds the product of two thin matrices. Because the second starts from zero, the base model is not disturbed at first. The singular value decomposition lesson in the Unsupervised Learning course had measured that a matrix can be written approximately as a low-rank product; the structure here is the **update** side of the same idea. - **LA35.** **The head is updated in every setup**; what separates the setups is how much of the body and the lookup table moves. The two runs are taken with the **pretraining seed** (20260218 and 20260219), the sets do not change. #### Layers and the Weight Budget On a distant endpoint the only thing measurable was the surface. On a model you host yourself, what can be measured is **the numbers themselves**: how many there are, which is in which layer, how many are saved. ```python # SIMULATOR -- FICTIONAL. The local model, vocabulary, records and labels are written # inside the lesson; no real model endpoint is called. Seed 20260218. from math import exp SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] GENERAL = [("bu ayki fatura tutari gecen aya gore yuksek geldi", "invoice"), ("faturami odedim ama sistemde borc gorunuyor", "invoice"), ("evde su akmiyor bir ariza var galiba", "breakdown"), ("baglanti bozuk oldu ve hic calismiyor", "breakdown"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "reading"), ("sayac degisimi icin randevu istiyorum", "reading"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tariff"), ("birim ucret abone tipine gore degisiyor mu", "tariff"), ("mahallede kesinti var ne zaman gelecek", "outage"), ("planli kesinti duyurusu nereden takip edilir", "outage")] FIELD = [("fat sapma itiraz dosyasi acildi", "invoice"), ("tahak farki cikti fat duzelt", "invoice"), ("fat kalem hatasi bildirildi", "invoice"), ("fat iade sureci baslatildi", "invoice"), ("arz hat kesik ekip yonlendirildi", "breakdown"), ("pano arz kodu iki bir dort", "breakdown"), ("arz tekrar etti kalici cozum", "breakdown"), ("arz gecici onarim yapildi", "breakdown"), ("syc endeks sapmasi tespit", "reading"), ("syc degisim isi planlandi", "reading"), ("syc muhur kontrolu yapildi", "reading"), ("syc seri numarasi eslesmedi", "reading"), ("trf basamak gecisi talebi", "tariff"), ("trf birim degisimi islendi", "tariff"), ("trf sozlesme guncellemesi bekliyor", "tariff"), ("trf indirim kosulu incelendi", "tariff"), ("kes duyurusu bolgeye islendi", "outage"), ("kes suresi uzadi bilgilendirme", "outage"), ("kes sonrasi enerji verildi", "outage"), ("kes plani ertelendi yeni saat", "outage")] MISLEADING = [" fat kaydi da acildi", " arz notu eklendi", " syc formu istendi", " trf grubu kontrol edildi"] def make_set(pool, n, seed, decoy_share=0.0): r, k = rng(seed), [] for i in range(n): c = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] candidates = [x for x in pool if x[1] == c] m = candidates[int(r() * len(candidates))][0] if r() < decoy_share: # a suffix carrying the abbreviation of another topic m += MISLEADING[int(r() * 4)] k.append((m, c)) return k PRETRAIN = make_set(GENERAL, 150, SEED) # pretraining domain ADAPT = make_set(FIELD, 48, SEED + 7, 0.30) # field spelling, adaptation examples TEST = make_set(FIELD, 72, SEED + 11, 0.30) # field spelling, test VOCAB = {} for m in [x[0] for x in GENERAL + FIELD] + MISLEADING: for t in tokenize(m): VOCAB.setdefault(t, len(VOCAB)) V, H, C = len(VOCAB), 12, len(TOPICS) def encode(k): return [(sorted({VOCAB[t] for t in tokenize(m)}), TOPICS.index(c)) for m, c in k] PRE_E, ADAPT_E, TEST_E = encode(PRETRAIN), encode(ADAPT), encode(TEST) def build(seed): """Input lookup table, body matrix, head. The body starts around the identity matrix.""" r = rng(seed) return [[[(r() - 0.5) * 0.4 for _ in range(H)] for _ in range(V)], [[1.0 * (k == j) + (r() - 0.5) * 0.2 for j in range(H)] for k in range(H)], [[(r() - 0.5) * 0.4 for _ in range(C)] for _ in range(H)]] def multiply(inputs, W, n): """inputs: (row, value) pairs; sparse multiplication.""" h = [0.0] * n for i, v in inputs: Wi = W[i] for j in range(n): h[j] += v * Wi[j] return h def forward(p, tok, addon=None): """If addon is given, a low-rank update is added to every layer (second section).""" layer, mid = [(t, 1.0) for t in tok], [] for n in (0, 1): h, z = multiply(layer, p[n], H), None if addon: U, Vt = addon[n] z = multiply(layer, U, len(Vt)) for j, x in enumerate(multiply(list(enumerate(z)), Vt, H)): h[j] += x mid.append((layer, h, z)) layer = [(j, x) for j, x in enumerate(h) if x > 0.0] # rectification, sparse return mid, layer, multiply(layer, p[2], C) def backward(p, tok, y, addon, step, which): """which: full | head | add_on. The head is updated in every setup, the others depend on the choice.""" mid, layer, o = forward(p, tok, addon) mx = max(o) e = [exp(x - mx) for x in o] s = sum(e) d = [x / s for x in e] d[y] -= 1.0 B, da = p[2], [0.0] * H for k, v in layer: da[k] = sum(d[j] * B[k][j] for j in range(C)) for j in range(C): B[k][j] -= step * d[j] * v for n in (1, 0): inputs, h, z = mid[n] W, U, Vt = p[n], *(addon[n] if addon else (None, None)) dh = [da[j] if h[j] > 0.0 else 0.0 for j in range(H)] if n == 1: da = [sum(dh[j] * W[k][j] for j in range(H)) for k in range(H)] if addon: dz = [sum(dh[j] * Vt[q][j] for j in range(H)) for q in range(len(Vt))] for k in range(H): da[k] += sum(U[k][q] * dz[q] for q in range(len(Vt))) if which == "full": for i, v in inputs: for j in range(H): W[i][j] -= step * dh[j] * v elif which == "add_on": dz = [sum(dh[j] * Vt[q][j] for j in range(H)) for q in range(len(Vt))] for q in range(len(Vt)): for j in range(H): Vt[q][j] -= step * dh[j] * z[q] for i, v in inputs: for q in range(len(Vt)): U[i][q] -= step * dz[q] * v def accuracy(p, k, addon=None): return sum(max(range(C), key=lambda j: forward(p, tok, addon)[2][j]) == y for tok, y in k) / len(k) def pretrain(seed): p = build(seed) for e in range(30): for tok, y in PRE_E: backward(p, tok, y, None, 0.10, "full") return p BASE_A, BASE_B = pretrain(SEED), pretrain(SEED + 1) print(f"vocabulary {V} pieces , hidden width {H} , classes {C}") print(f"input {V * H} + body {H * H} + head {H * C} = {V * H + H * H + H * C} numbers") print("run pretraining domain field domain") for name, p in (("A", BASE_A), ("B", BASE_B)): print(f" {name} {accuracy(p, PRE_E):>18.4f} {accuracy(p, TEST_E):>12.4f}") ``` ``` vocabulary 174 pieces , hidden width 12 , classes 5 input 2088 + body 144 + head 60 = 2292 numbers run pretraining domain field domain A 1.0000 0.2500 B 1.0000 0.0833 ``` On the pretraining domain accuracy is **1.0000** in both runs; the model has learned its own domain. When it moves to field spelling, the same model gives **0.2500** and **0.0833**. In a five-class task random choice is around 0.2000, so the base model **knows nothing** on field records; the 0.1667 gap between the two runs is the proof of this, what is measured is not knowledge but noise. Adaptation is unavoidable and the question is **which numbers** will move. #### Updated-Parameter Share and Accuracy The setups are run with the same forty-eight adaptation records, the same twenty-five loops and the same step size; the only thing that changes is **which share** backpropagation touches. ```python from copy import deepcopy def build_addon(rank, seed): """Each low-rank update is a pair of U (rows x rank) and Vt (rank x H). Vt starts from zero, so at the first moment of adaptation the update adds nothing and the base model is not disturbed.""" r = rng(seed) return [([[(r() - 0.5) * 0.3 for _ in range(rank)] for _ in range(n)], [[0.0] * H for _ in range(rank)]) for n in (V, H)] def adapt(base, which, rank, seed, loops=25, step=0.08): p = deepcopy(base) addon = build_addon(rank, seed + 3) if rank else None for e in range(loops): for tok, y in ADAPT_E: backward(p, tok, y, addon, step, which) return accuracy(p, TEST_E, addon), accuracy(p, PRE_E, addon) TOTAL = V * H + H * H + H * C SETUPS = [("full update", "full", 0), ("layer freezing", "head", 0)] + \ [(f"low-rank update r={r}", "add_on", r) for r in (1, 2, 3, 4)] print("setup updated share field A field B general A general B") print(f"{'no adaptation':<20} {0:>7} {0.0:.4f} {accuracy(BASE_A, TEST_E):>7.4f}" f" {accuracy(BASE_B, TEST_E):>7.4f} {accuracy(BASE_A, PRE_E):>9.4f}" f" {accuracy(BASE_B, PRE_E):>9.4f}") for name, which, rank in SETUPS: n = TOTAL if which == "full" else (H * C if which == "head" else rank * (V + H) + rank * 2 * H + H * C) a, b = adapt(BASE_A, which, rank, SEED), adapt(BASE_B, which, rank, SEED + 1) print(f"{name:<20} {n:>7} {n / TOTAL:.4f} {a[0]:>7.4f} {b[0]:>7.4f}" f" {a[1]:>9.4f} {b[1]:>9.4f}") print("\nadaptation to eight separate domains, number of stored numbers:") for name, n in (("full update", TOTAL), ("layer freezing", H * C), ("low-rank update r=2", 2 * (V + H) + 2 * 2 * H + H * C)): shared = 0 if n == TOTAL else TOTAL print(f" {name:<20} {shared} + 8 x {n} = {shared + 8 * n}") print("\nas the hidden width grows (independent of the run, count only):") print(" H full freezing r=2 update update share") for h in (H, 64, 256): full = V * h + h * h + h * C addon2 = 2 * (V + h) + 2 * 2 * h + h * C print(f" {h:<5} {full:>7} {h * C:>8} {addon2:>10} {addon2 / full:>12.4f}") ``` ``` setup updated share field A field B general A general B no adaptation 0 0.0000 0.2500 0.0833 1.0000 1.0000 full update 2292 1.0000 1.0000 0.9861 1.0000 1.0000 layer freezing 60 0.0262 0.6944 0.7500 0.5133 0.1733 low-rank update r=1 270 0.1178 0.7639 0.7222 0.5133 0.2733 low-rank update r=2 480 0.2094 0.9583 0.9306 1.0000 0.5533 low-rank update r=3 690 0.3010 1.0000 0.9722 1.0000 1.0000 low-rank update r=4 900 0.3927 0.9861 1.0000 1.0000 1.0000 adaptation to eight separate domains, number of stored numbers: full update 0 + 8 x 2292 = 18336 layer freezing 2292 + 8 x 60 = 2772 low-rank update r=2 2292 + 8 x 480 = 6132 as the hidden width grows (independent of the run, count only): H full freezing r=2 update update share 12 2292 60 480 0.2094 64 15552 320 1052 0.0676 256 111360 1280 3164 0.0284 ``` Full update moves all 2292 numbers and gives **1.0000 / 0.9861**. Layer freezing works with **0.0262** of that, that is, sixty numbers, and stays at **0.6944 / 0.7500**: forty-eight examples are enough to teach the head a new mapping, but because the body still does not recognize the field abbreviations, accuracy stays half-way. The low-rank update at rank two climbs to **0.9583 / 0.9306** with a share of **0.2094**; its distance from full update is between 0.0417 and 0.0555, and its distance from layer freezing is between 0.1806 and 0.2639. Most of the gap is closed with **two tenths** of the updated share. The rank ladder says where to stop. Rank one is not enough (**0.7639 / 0.7222**): a single direction cannot carry the shift between the two spellings. Rank three gives **1.0000 / 0.9722** and rank four **0.9861 / 1.0000**; the means are 0.9861 and 0.9931, so the mean gain of raising the rank from three to four is **0.0070**. The two-run gaps of the same rows are 0.0278 and 0.0139. The gain is below the gap, and in return **210 more numbers** are stored per adaptation: the cost of the added structure is certain, its gain is unmeasurable. The last two columns show the unexpected. Layer freezing, although it protects the body, drops accuracy on the previous domain to **0.5133 / 0.1733**; the protected body does not help, because the head the two domains **share** was rewritten. Full update, on the other hand, keeps the previous domain at **1.0000**: in the lookup table only the rows that appear in field records moved, and the rows of general spelling were not touched. Freezing a layer does not mean preserving that layer's ability; what is preserved is the **weights**, what is lost is the **behavior**. #### Stored Count and Scale The row of adaptation to eight domains shows where the decision is paid for. In full update the base cannot be shared; eight copies come to **18336** numbers. In layer freezing the base is stored once and each domain adds only its own head: **2772**. In the low-rank update the base is again stored once and each domain adds 480 numbers: **6132**, that is, a third of full update. The scale table says this ratio is not constant. When the rank is held fixed, the cost of the low-rank update grows **linearly** with the hidden width, and the cost of full update grows **quadratically** in the body. At width 12 the share of the low-rank update is **0.2094**, at 64 **0.0676**, at 256 **0.0284**. This count is independent of the run, and it explains why the discussion of adaptation methods starts with **large** models. #### Summary - The fictional local model is 2292 numbers (input 2088, body 144, head 60), and its unadapted state stays at 0.2500 and 0.0833 on field spelling, that is, at the level of random choice. - Full update moves all of it and gives 1.0000 / 0.9861; its cost is a full copy for each domain. - Layer freezing stays at 0.6944 / 0.7500 with a 0.0262 share and drops accuracy on the previous domain to 0.5133 / 0.1733: the frozen weights are preserved, the behavior is not. - The low-rank update gives 0.9583 / 0.9306 with a 0.2094 share; its distance from full update is between 0.0417 and 0.0555. - The mean gain of raising the rank from three to four is 0.0070, the two-run gaps of the same rows are 0.0278 and 0.0139: the gain counts as unmeasured, the 210 extra numbers are certain. - The stored count over eight domains is 6132 against 18336, and when the hidden width goes from 12 to 256 the share of the low-rank update falls from 0.2094 to 0.0284. #### Next Step In this lesson the weights of the large model were in our hands. Weights are not always in hand; most of the time the only thing in hand is the **output** of a large model. That output can be used like a label set and made the input for training a **small model**. The next lesson measures this: what does a small model trained with the labels a large model produced gain compared with a model of the same size trained only with human labels, how much of the teacher's accuracy does it capture, and in **which task** does the gap not close. ### Distillation Source: https://academia.sh/en/courses/model-apis/distillation Measuring, on two tasks, what a small model trained on 240 fictional records labeled by the large model gains over the same model trained on only six human labels: on the topic task it rises from 0.5000 to 0.6333 and captures the whole teacher accuracy, but it closes only a third of the gap to the 0.9000 given by 240 human labels. On the separate topic-count task distillation moves backward, falling from 0.5833 to 0.3667 and 0.2417, because the teacher does not count. Of the distilled model's errors, 0.9545 are on records where the teacher also errs; for the model trained on six labels this share is 0.3000. In the previous lesson the weights of the large model were in our hands and we moved some of them. In most cases there are no weights in hand; the only thing in hand is the **responses** the large model gives. These responses can be used like a label set and made the input to the training of **a small model**. What is being done is **distillation**: the behavior of the large model is carried over to a model that holds a thousandth as many numbers. The appeal of distillation is plain. A human label is expensive, while a model label is a call. So what has to be measured is not a single number: **how much of the teacher accuracy** the small model captures, where we would have arrived had we spent the same effort on human labels, and **on which task** this transfer does not work at all. - **LA36.** The code is a **simulator**. The large model, the small model and all of the records are **fictional**; no real endpoint is called. Seed **20260218**. - **LA37.** A record is a body sentence and at most two **short suffixes**. One of the labels is the **topic** (the topic of the body, five labels), the other is the **distinct topic count** (the number of distinct topics that appear in the text, four labels). In a share of the records the body does not give the topic away. - **LA38. The large model scores the topic from a key table and does not count the distinct topic count**; it predicts it from the length class. The second behavior is the fictional counterpart of the weakness of language models on counting tasks, and it had been measured in the Introduction to AI Engineering course. - **LA39. The small model** is a single matrix going directly from the token bag to the label; it carries none of the structure the teacher carries. Its training is written inside the lesson. - **LA40.** Three sources are compared: **six human labels**, **two hundred forty teacher labels** and **two hundred forty human labels**. The last is an **upper bound**, not a real option. - **LA41.** The two runs are taken by changing the **labeling seed of the teacher** (20260218 and 20260219); the records and the test set are the same in both runs. #### The Teacher and Two Tasks The limit of distillation is the teacher itself, so the teacher is measured first. The two tasks were chosen to differ on purpose: one is a classification that can be solved by keyword mapping, the other is **counting** the distinct topics that appear in the text. ```python # SIMULATOR -- FICTIONAL. The large model, the small model and all of the records are # written inside the lesson; no real endpoint is called. Seed 20260218. from math import exp SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] BODY = [("fatura tutari gecen aya gore yuksek geldi", "invoice"), ("faturami odedim ama sistemde borc gorunuyor", "invoice"), ("evde su akmiyor bir ariza var galiba", "breakdown"), ("baglanti bozuk oldu ve hic calismiyor", "breakdown"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "reading"), ("sayac degisimi icin randevu istiyorum", "reading"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tariff"), ("birim ucret abone tipine gore degisiyor mu", "tariff"), ("mahallede kesinti var ne zaman gelecek", "outage"), ("planli kesinti duyurusu nereden takip edilir", "outage")] SHORT = {"invoice": "fatura notu", "breakdown": "ariza kaydi", "reading": "sayac formu", "tariff": "tarife grubu", "outage": "kesinti plani"} VAGUE = ["gecen hafta bir sorun yasadim yardim gerekiyor", "bu durumu kime iletmem gerektigini bilmiyorum"] def records(n, seed): """A body sentence and at most two short suffixes. The topic is the body's; count is the distinct topic count.""" r, k = rng(seed), [] for i in range(n): c = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] if r() < 0.10: # the body does not give the topic away m, seen = VAGUE[int(r() * 2)], [] else: candidates = [x for x in BODY if x[1] == c] m, seen = candidates[int(r() * len(candidates))][0], [c] for threshold in (0.62, 0.45): if r() < threshold: b = TOPICS[int(r() * 5)] if b not in seen: m += " ayrica " + SHORT[b] seen.append(b) k.append({"text": m, "topic": c, "count": len(seen)}) return k LABELED = records(6, SEED + 1) # a human label is expensive: six records UNLABELED = records(240, SEED + 2) # the unlabeled pile TEST = records(120, SEED + 3) TRACE = {"fatur": "invoice", "tutar": "invoice", "borc": "invoice", "odedi": "invoice", "ariza": "breakdown", "bozuk": "breakdown", "calis": "breakdown", "akmiy": "breakdown", "sayac": "reading", "endek": "reading", "okuma": "reading", "randev": "reading", "tarif": "tariff", "birim": "tariff", "abone": "tariff", "basam": "tariff", "kesin": "outage", "duyur": "outage", "mahal": "outage", "planl": "outage"} TRAP = {"notu": "invoice", "kaydi": "breakdown", "formu": "reading", "grubu": "tariff", "plani": "outage"} # the short suffixes sit in the table at full weight CALLS = [0, 0] def large_model(text, seed, temperature=0.50): """FICTIONAL large model. Scores the topic from the key table; does NOT count for the count task, it predicts from the length class. The second behavior matches the counting weakness of the Introduction to AI Engineering course.""" CALLS[0] += 1 tok = tokenize(text) CALLS[1] += len(tok) p = {k: 0.12 for k in TOPICS} for t in tok: for table, w in ((TRACE, 1.0), (TRAP, 0.70)): if table.get(t) in p: p[table[t]] += w r = rng(seed ^ sum((i + 1) * ord(c) for i, c in enumerate(text))) names = sorted(p) e = [exp(p[a] / temperature) for a in names] s = sum(e) topic = names[pick(r(), [x / s for x in e])] length = 0 if len(tok) <= 7 else (1 if len(tok) <= 10 else 2) count = pick(r(), ([0.10, 0.56, 0.24, 0.10], [0.06, 0.32, 0.40, 0.22], [0.04, 0.20, 0.36, 0.40])[length]) return topic, count def measure(dataset, seed): dt = dc = 0 for x in dataset: k, s = large_model(x["text"], seed) dt += k == x["topic"] dc += s == x["count"] return dt / len(dataset), dc / len(dataset) dist = {} for x in TEST: dist[x["count"]] = dist.get(x["count"], 0) + 1 print("test records", len(TEST), ", distinct topic count distribution", dict(sorted(dist.items()))) print("example record:", TEST[1]["text"]) print(" topic", TEST[1]["topic"], ", distinct topic count", TEST[1]["count"]) print("\nlarge model topic accuracy count accuracy") for name, sd in (("run A (20260218)", SEED), ("run B (20260219)", SEED + 1)): a, b = measure(TEST, sd) print(f"{name:<24} {a:>14.4f} {b:>14.4f}") print(f"cost of the test measurement: calls {CALLS[0]} , input tokens {CALLS[1]}") ``` ``` test records 120 , distinct topic count distribution {0: 4, 1: 38, 2: 63, 3: 15} example record: evde su akmiyor bir ariza var galiba topic breakdown , distinct topic count 1 large model topic accuracy count accuracy run A (20260218) 0.6333 0.3583 run B (20260219) 0.7167 0.3000 cost of the test measurement: calls 240 , input tokens 3214 ``` The large model gives **0.6333 / 0.7167** on the topic task and **0.3583 / 0.3000** on the counting task. The counting task has four labels and the most crowded label holds 63 records of the test set; so a routine that gives every record that label would score 0.5250. The teacher is below it. The difference between the two runs is also 0.0834 on the topic task and 0.0583 on the counting task: the teacher is an uncertain component, and its labels are uncertain too. #### The Share the Small Model Captures The small model is trained on three separate sources in both tasks. The labels of the teacher are produced by making one call for each of the two hundred forty unlabeled records. ```python VOCAB = {} for m in [x[0] for x in BODY] + VAGUE + ["ayrica " + v for v in SHORT.values()]: for t in tokenize(m): VOCAB.setdefault(t, len(VOCAB)) V = len(VOCAB) def split_pieces(text): return sorted({VOCAB[t] for t in tokenize(text) if t in VOCAB}) def target(topic, count, task): return TOPICS.index(topic) if task == "topic" else count def train_model(sample, classes, seed, loops=40, step=0.12): """SMALL model: a single matrix from the token bag directly to the label, V x number of labels.""" r = rng(seed) W = [[(r() - 0.5) * 0.2 for _ in range(classes)] for _ in range(V)] for e in range(loops): for tok, y in sample: o = [sum(W[i][j] for i in tok) for j in range(classes)] mx = max(o) d = [exp(x - mx) for x in o] s = sum(d) d = [x / s for x in d] d[y] -= 1.0 for i in tok: for j in range(classes): W[i][j] -= step * d[j] return W def predict(W, tok, classes): o = [sum(W[i][j] for i in tok) for j in range(classes)] return max(range(classes), key=lambda j: o[j]) def small_accuracy(W, classes, task): return sum(predict(W, split_pieces(x["text"]), classes) == target(x["topic"], x["count"], task) for x in TEST) / len(TEST) CALLS[0] = CALLS[1] = 0 DISTILL = {t: [(x, *large_model(x["text"], t)) for x in UNLABELED] for t in (SEED, SEED + 1)} # the unlabeled pile is being labeled by the teacher LABELING = (CALLS[0], CALLS[1]) SOURCES = [("human labels 6", lambda a, t: [(split_pieces(x["text"]), target(x["topic"], x["count"], a)) for x in LABELED]), ("teacher labels 240", lambda a, t: [(split_pieces(x["text"]), target(k, s, a)) for x, k, s in DISTILL[t]]), ("human labels 240", lambda a, t: [(split_pieces(x["text"]), target(x["topic"], x["count"], a)) for x in UNLABELED])] TEACHER = (measure(TEST, SEED), measure(TEST, SEED + 1)) WEIGHTS_BY = {} print("task small model source accuracy A accuracy B teacher A share captured") for task, classes in (("topic", 5), ("count", 4)): og = TEACHER[0][0 if task == "topic" else 1] for name, build_data in SOURCES: W = train_model(build_data(task, SEED), classes, SEED) a = small_accuracy(W, classes, task) b = small_accuracy(train_model(build_data(task, SEED + 1), classes, SEED + 1), classes, task) WEIGHTS_BY[(task, name)] = (W, classes) print(f"{task:<6} {name:<22} {a:>10.4f} {b:>10.4f} {og:>10.4f} {a / og:>14.4f}") print(f"{task:<6} {'large model (teacher)':<22} {og:>10.4f}" f" {TEACHER[1][0 if task == 'topic' else 1]:>10.4f}") print("\nhow much the small model's error overlaps the teacher's (run A):") for task in ("topic", "count"): for name in ("human labels 6", "teacher labels 240"): W, classes = WEIGHTS_BY[(task, name)] shared = errors = 0 for x in TEST: correct = target(x["topic"], x["count"], task) k, s = large_model(x["text"], SEED) se = predict(W, split_pieces(x["text"]), classes) != correct errors += se shared += se and target(k, s, task) != correct print(f" {task:<6} {name:<22} small errors {errors:>3} , shared {shared:>3}" f" , share {shared / errors:.4f}") print(f"\nsmall model {V} x classes ; human labels 6 or 240 ;" f" teacher calls {LABELING[0] // 2} , input tokens {LABELING[1] // 2}") ``` ``` task small model source accuracy A accuracy B teacher A share captured topic human labels 6 0.5000 0.5583 0.6333 0.7895 topic teacher labels 240 0.6333 0.6917 0.6333 1.0000 topic human labels 240 0.9000 0.9000 0.6333 1.4211 topic large model (teacher) 0.6333 0.7167 count human labels 6 0.5833 0.5833 0.3583 1.6279 count teacher labels 240 0.3667 0.2417 0.3583 1.0233 count human labels 240 0.9417 0.9417 0.3583 2.6279 count large model (teacher) 0.3583 0.3000 how much the small model's error overlaps the teacher's (run A): topic human labels 6 small errors 60 , shared 18 , share 0.3000 topic teacher labels 240 small errors 44 , shared 42 , share 0.9545 count human labels 6 small errors 50 , shared 32 , share 0.6400 count teacher labels 240 small errors 76 , shared 58 , share 0.7632 small model 103 x classes ; human labels 6 or 240 ; teacher calls 240 , input tokens 3256 ``` Distillation works on the topic task. The small model trained on six human labels gives **0.5000 / 0.5583**; the same model trained on two hundred forty teacher labels rises to **0.6333 / 0.6917**. The gain is 0.1333 and the **captured share is 1.0000**: the small model catches up with the accuracy of the teacher exactly. Up to here distillation keeps its promise. The third row shows the limit of the promise. Had the same two hundred forty records been given with **human** labels, the small model would have given **0.9000**. The gap the distilled model closed is only **a third** of the distance between 0.5000 and 0.9000. It is not that the small model cannot carry something; it is the teacher that cannot carry it. Distillation copies not the model but **as much as the teacher saw**. The counting task turns this around and gives the real number of the lesson. The small model trained on six human labels scores **0.5833**. The same model trained on two hundred forty teacher labels **falls** to **0.3667 / 0.2417**. Forty times as many labels, two hundred forty calls and 3256 input tokens were spent, and the result is worse than the model trained on six labels. The teacher does not count on that task, it predicts from the length; distillation faithfully transfers this habit of predicting. The upper bound is **0.9417**: the task is not hard for the small model, it is hard for the teacher. The error overlap column says how faithful the transfer is. On the topic task, **42** of the 44 errors of the distilled model are on records where the teacher also errs; the share is **0.9545**. For the model trained on six human labels the same share is **0.3000**. Both make errors, but the distilled one makes the **same** errors. If two components of a system err in the same place, putting the two side by side catches nothing. Finally, the two run columns show a fragility particular to distillation. When the labeling seed of the teacher changes, the counting accuracy of the distilled model falls **from 0.3667 to 0.2417**; a swing of 0.1250. The rows trained on human labels give the same number in both runs. The uncertainty of the teacher was written into the training set and became permanent there. #### Summary - The large model gives 0.6333 / 0.7167 on the topic task and 0.3583 / 0.3000 on the distinct topic count task; on the counting task it stays below the 0.5250 of a routine that chooses the most crowded label. - On the topic task distillation raises accuracy from 0.5000 to 0.6333 relative to six human labels and captures the whole accuracy of the teacher. - The human labels of the same two hundred forty records give 0.9000; distillation closes only a third of the gap, because the ceiling is the teacher, not the small model. - On the counting task distillation moves backward: from 0.5833 to 0.3667 and 0.2417. The upper bound is 0.9417, so the task is hard not for the small model but for the teacher. - Of the 44 topic errors of the distilled model, 42 are on records where the teacher also errs (share 0.9545); for the model trained on six human labels this share is 0.3000. - The cost is two hundred forty teacher calls and 3256 input tokens; in return, two hundred forty model labels are obtained in place of six human labels, and the run uncertainty of the teacher is inherited. #### Next Step In this lesson the small model was born from the output of the large model. The resulting weights are numbers that no human wrote, derived entirely from the responses of another model. The same holds for the adaptation differences of the previous lesson: in hand is an object **derived** from someone else's model. Running that object on your own server, giving it to subscribers inside a package, selling it to someone else, or serving only its output are not the same thing. The next lesson counts this distinction: when four **license condition classes** are applied to a use scenario, how many distribution options close, which condition alone closes the most options, and how many options remain. ### Model Repositories and Terms of Use Source: https://academia.sh/en/courses/model-apis/model-repositories-and-terms-of-use Counting how many distribution options four license condition classes close in a use scenario: of sixteen options built from four asset types and four channels, the usage limit alone closes 6, the redistribution right 3, the derivative-work condition 2 and the commercial-use record 2, a total of 13; together they close 10, because three options are closed by more than one condition. Four of the six options left open are in a single channel. Removing the usage limit opens three options, not six. When the monthly request volume triples, the open options fall from six to four, and at ten times they fall to one. This lesson is the close of the Building Applications with Model APIs course. In the previous lesson the small model was born from the output of the large model. In the lesson before that, adaptation differences were left in hand. Both are objects of the same kind: piles of numbers **derived** from someone else's model, with no meaning on their own. Running these objects on your own server, putting them in a shared environment you rent, giving them to subscribers inside a package or leaving them open to everyone are not the same thing. What this lesson measures is not what a text says, but **how many options close**. Distribution options are a countable set; license conditions are filters over this set. When the filters are applied, how many options are left, how much each filter closes **on its own** and how much it **opens back** when removed can be counted separately. The count is entirely determined; there is no seed-dependent draw, so the measure is independent of the run. - **LA42.** The asset types, channels, volumes and condition classes are **fictional** and are defined inside the lesson. No repository, license, organization, country or regulation name appears; the conditions are referred to only by their **classes**. - **LA43.** Four **asset types** can be distributed: output only, adaptation difference, full weight copy and distilled small model. The adaptation difference and the distilled small model are the objects produced by the previous two lessons. - **LA44.** There are four **channels**: in-house, rented shared environment, paid subscriber package and open release. For each channel the monthly request volume, whether it goes beyond the organization boundary, whether it is commercial and whether its recipient is known are written. - **LA45.** The **usage limit** says that in an option where the base model runs, the monthly request threshold cannot be exceeded. The **redistribution right** says that a copy of the base weight cannot leave the organization boundary. The **derivative-work condition** says that a derived asset can be given only in a channel where the conditions can be carried together with the recipient. The **commercial-use record** asks that, in a commercial channel, the base the use rests on be put into the record, and this record is produced from the call log of the base model. - **LA46.** This is not a law lesson. What is counted is not the meaning of texts but **the number of options closed by particular condition predicates**; the predicates are written above and appear in the code word for word. #### Four Condition Classes and Sixteen Options All of the conditions are applied at the same time, and an option closes if it is caught by at least one condition. So the condition that closes the most on its own does not have to be the condition that opens the most when removed. ```python # FICTIONAL. Asset types, channels, volumes and four condition classes are defined inside # the lesson; no repository, license, organization or regulation name appears. ASSETS = [ # name, does the base model run, is it a copy of the base weight, is it derived ("output only", True, False, False), ("adaptation difference", True, False, True), ("full weight copy", True, True, False), ("distilled small model", False, False, True)] CHANNELS = [ # name, monthly requests, leaves the organization, commercial, recipient known ("in-house", 40000, False, False, True), ("rented shared environment", 120000, True, True, True), ("paid subscriber package", 900000, True, True, True), ("open release", 3000000, True, False, False)] OPTIONS = [(v, k) for v in ASSETS for k in CHANNELS] def c1(v, k, threshold): """Usage limit: in an option where the base model runs, the monthly request threshold cannot be exceeded.""" return v[1] and k[1] > threshold def c2(v, k, threshold): """Redistribution right: a copy of the base weight cannot leave the organization boundary.""" return v[2] and k[2] def c3(v, k, threshold): """Derivative-work condition: a derived asset can be given only in a channel where the conditions can be carried together with the recipient.""" return v[3] and not k[4] def c4(v, k, threshold): """Commercial-use record: in a commercial channel, the base the use rests on goes into the record; the record is produced from the call log of the base model.""" return k[3] and not v[1] CONDITIONS = [("usage limit", c1), ("redistribution right", c2), ("derivative-work condition", c3), ("commercial-use record", c4)] def closed_by(option, conditions, threshold): return [name for name, f in conditions if f(*option, threshold)] def count_closed(conditions, threshold, volume=1): channels = [(a, h * volume, d, t, b) for a, h, d, t, b in CHANNELS] options = [(v, k) for v in ASSETS for k in channels] return sum(1 for s in options if closed_by(s, conditions, threshold)), len(options) THRESHOLD = 250000 print(f"options {len(OPTIONS)} = {len(ASSETS)} assets x {len(CHANNELS)} channels ," f" monthly request threshold {THRESHOLD}") print("\ncondition on its own opened on removal") closed_total = 0 for name, f in CONDITIONS: alone = sum(1 for s in OPTIONS if f(*s, THRESHOLD)) without = count_closed([x for x in CONDITIONS if x[0] != name], THRESHOLD)[0] total_closed = count_closed(CONDITIONS, THRESHOLD)[0] closed_total += alone print(f"{name:<26} {alone:>10} {total_closed - without:>17}") total_closed = count_closed(CONDITIONS, THRESHOLD)[0] print(f"{'total':<26} {closed_total:>10} {'':>17}") print(f"\nthe four conditions together close {total_closed} options , remaining open {len(OPTIONS) - total_closed}") print("\nasset open channels") for v in ASSETS: open_channels = [k[0] for k in CHANNELS if not closed_by((v, k), CONDITIONS, THRESHOLD)] print(f"{v[0]:<23} {' , '.join(open_channels) if open_channels else 'none'}") ``` ``` options 16 = 4 assets x 4 channels , monthly request threshold 250000 condition on its own opened on removal usage limit 6 3 redistribution right 3 1 derivative-work condition 2 1 commercial-use record 2 2 total 13 the four conditions together close 10 options , remaining open 6 asset open channels output only in-house , rented shared environment adaptation difference in-house , rented shared environment full weight copy in-house distilled small model in-house ``` The four conditions close **6, 3, 2 and 2** options **on their own**; the total is **13**. Together, though, they close **10**. The three options in between are caught by more than one condition, that is, they have been counted twice. Adding up the closing counts exaggerates the effect of the conditions by thirty percent. The opened-on-removal column is the counterpart of this. The usage limit closes six options on its own, but when removed it opens only **three**; the other three were already caught by the redistribution right or the derivative-work condition. The commercial-use record closes only two options on its own and when removed opens **both** back; it overlaps with no condition. The weight of a condition lies not in the number it closes but in the number it **closes on its own**. The last table shows which options are left standing. For the full weight copy and the distilled small model a single channel is left: in-house. The two objects this course produced are the adaptation difference and the distilled small model, and they do not close the same number of options: the adaptation difference is caught by the usage limit because it runs the base model, and the distilled model cannot produce the commercial-use record because it does not. Both narrow, but from **separate conditions**. #### When the Threshold and Volume Change The same conditions do not give a fixed number; the number depends on the size of the scenario. ```python multi = sum(1 for s in OPTIONS if len(closed_by(s, CONDITIONS, THRESHOLD)) > 1) print(f"options closed by more than one condition {multi} ; closed by one" f" condition {total_closed - multi} ; total closed {total_closed}") print("\nthreshold and volume sensitivity (count independent of the run):") print(f"{'threshold':>10} {'volume':>6} {'closed':>7} {'open':>4} closed by the usage limit alone") for threshold, h in ((100000, 1), (250000, 1), (1000000, 1), (5000000, 1), (250000, 3), (250000, 10)): closed, total = count_closed(CONDITIONS, threshold, h) alone = sum(1 for v in ASSETS for k in CHANNELS if c1(v, (k[0], k[1] * h), threshold)) print(f"{threshold:>10} {str(h) + 'x':>6} {closed:>7} {total - closed:>4} {alone:>31}") ``` ``` options closed by more than one condition 3 ; closed by one condition 7 ; total closed 10 threshold and volume sensitivity (count independent of the run): threshold volume closed open closed by the usage limit alone 100000 1x 12 4 9 250000 1x 10 6 6 1000000 1x 8 8 3 5000000 1x 7 9 0 250000 3x 12 4 9 250000 10x 15 1 12 ``` When the threshold rises fourfold the open options go from six to eight, and when the threshold is fully relaxed to nine. In the opposite direction the change is sharper: when the monthly volume triples the open options fall to **four**, and when it rises tenfold they fall to **one**. At ten times the volume even the in-house channel exceeds the threshold, and all that remains is running the distilled model in-house. The conditions did not change; what grew was the workload. A distribution decision closes not on the day the condition text is read, but on **the day the volume crosses the threshold**. #### Summary - Sixteen distribution options are built from four asset types and four channels; the conditions are predicates over this set, and the count is independent of the run. - The four conditions close a total of 13 on their own and 10 together; three options are caught by more than one condition. - The usage limit closes 6 options on its own but opens 3 when removed; the commercial-use record closes 2 and opens 2, that is, it overlaps with no condition. - For the full weight copy and the distilled small model a single channel is left, but one narrows from the redistribution right and the other from the commercial-use record. - When the monthly volume triples the open options fall from 6 to 4, and at ten times to 1: what closes the decision is not the condition text but the day the threshold is crossed. #### Course Wrap-Up | Lesson | Without scaffolding | With scaffolding | Cost | |---|---|---|---| | `api-usage/01` — conversation state | state not sent 0.2000 | full transcript 0.8250 (**+0.6250**) | input tokens 8346 → 31160 | | `api-usage/02` — first token | non-streaming, label at the end 16.03 steps | streaming + label first 4.96 steps (**−11.07**) | tokens the same; truncated responses recovered 0 → 31 | | `api-usage/03` — valid call rate | no catalog 0.0500 | name + parameter 0.6917 (**+0.6417**) | input 3694 → 11614; residue 14 wrong tools | | `api-usage/04` — schema conformance | schema written in the prompt 0.7583 | interface field 0.9000 (**+0.1417**) | input tokens 7654 → 4294; per correct record 125.8 → 65.7 | | `api-usage/05` — usable record share | fine grid 0.8167 (183 tokens) | cropped grid 0.7000 (**−0.1167**) | per record 224.1 → 60.0 tokens | | `api-usage/06` — share of three neighbors from the same topic | 144 dimensions 0.9259 | 72 dimensions 0.9028 (**−0.0231**) | stored numbers 10368 → half | | `api-usage/07` — number of fields carried | the provider's own interface 12 | common interface 8 (**−4**) | output tokens 773 → 1638; readable truncation 0 | | `api-usage/08` — number of calls / affected requests | batch size 1: 120 calls, 1 request | batch size 12: 10 calls (**−110**), 12 requests | input 5854 → 1784; accuracy 0.0667 at size 18 | | `ai-production-quality/01` — completion share under heavy load | without scaffolding 0.8562 | retry with backoff 0.8750 (**+0.0188**) | run difference also 0.0188; neighboring stream 0.9500 → 0.8667 | | `ai-production-quality/02` — correct response share | no cache 0.9333 | long-lived response cache 0.5542 (**−0.3791**) | calls 240 → 29; stale share 0.0208 → 0.4417 | | `ai-production-quality/03` — wait step / accuracy | large context 3.0000 steps, 0.9292 | two-stage 1.7500 steps (**−1.2500**), **−0.0125** | input tokens 19716 | | `ai-production-quality/04` — tokens per request | without abbreviation 81.6 | three abbreviation policies 50.5 (**−31.1**) | read per correct label, the gap narrows | | `ai-production-quality/05` — total accuracy | without fallback 0.8450 | with fallback 0.8850 (**+0.0400**) | against the uninterrupted base 0.8975 → 0.8850; fallback tier 0.6667 | | `ai-production-quality/06` — accuracy / share of drifting responses | items fixed 0.9125 | format word free 0.5525 (**−0.3600**) | when three items at the provider drift, share 0.1150 | | `ai-assisted-development/01` — accepted task (24) | no context 7 | whole codebase 18 (**+11**) | 1104 → 4224 tokens; acceptance per thousand tokens 6.34 → 4.26 | | `ai-assisted-development/02` — expected acceptance | all files 0.6921 | selected files 0.8233 (**+0.1312**) | 4224 → 2274 tokens; 120 → 45 files | | `ai-assisted-development/03` — correct step (480) | stopping once in eight steps 355 | stopping at every step 478 (**+123**) | stops 60 → 479; missed errors 21 → 1 | | `ai-assisted-development/04` — caught defect (28) | no review 0 | three reviews 26 (**+26**) | missed 2; one gets in without receiving any warning | | `ai-assisted-development/05` — distinguished broken version (6) | generated test set 3 | single manual alignment item 4 (**+1**) | 2 of 14 tests fail on clean code, 9 distinguish nothing | | `ai-assisted-development/06` — secret class coverage | without filter 0.0000 | domain name filter 0.9333 (**+0.9333**) | 12 traces remain in free text; false share among those accepted 0.5263 | | `local-models-and-adaptation/01` — label accuracy / concurrent request | small scale 0.6333, 248 requests | large scale 0.8667 (**+0.2334**), 5 requests | accuracy times concurrent requests 157.1 → 4.3 | | `local-models-and-adaptation/02` — five-label topic accuracy | 16 digits 0.8667 | 4 digits 0.8000 (**−0.0667**) | memory 34.3 → 8.6 billion digits; concurrent requests 5 → 62 | | `local-models-and-adaptation/03` — accuracy (60 records) | base 0.6000 | prompt with 60 examples 0.9667, fine-tuning 0.9500 (**−0.0167**) | context per request 28.48 → 623.48 tokens; 180 steps, 0.2992 of the weights | | `local-models-and-adaptation/04` — field accuracy | no adaptation 0.2500 | low-rank update 0.9583 (**+0.7083**) | updated share 0.2094; in eight fields 18336 → 6132 numbers | | `local-models-and-adaptation/05` — topic accuracy | six human labels 0.5000 | 240 teacher labels 0.6333 (**+0.1333**) | 240 calls, 3256 tokens; counting task 0.5833 → 0.3667 | | `local-models-and-adaptation/06` — open distribution option (16) | without reading the conditions 16 | four license condition classes 6 (**−10**) | when the volume rises tenfold 1 | The rule of the course is written in the table row by row: **a model call is not a function call but a network operation; a call whose failure is not written down is counted as unmeasured.** None of the numbers in the without-scaffolding column is low because of bad code. All of them are low because it was not taken into account that the call will be delayed, will be queued, will be cut off midway, will be rejected or will use up someone else's limit. **The second claim is read from the same table too: the resilience work added can make things worse.** The rows that say so by name are these. In `ai-production-quality/01`, fixed-interval retry lowers completion under heavy load below that without scaffolding and drops the interactive stream sharing the same limit from 0.9500 to 0.8667. In `ai-production-quality/02`, the response cache cuts the calls from 240 to 29 while taking the correct response share from 0.9333 to 0.5542. In `ai-production-quality/05`, the fallback tier makes total accuracy look higher, but when read against the uninterrupted base there is a drop, and the accuracy of the requests that fall to the fallback is 0.6667. In `ai-assisted-development/02`, enlarging the context lowers the expected acceptance from 0.8233 to 0.6921. In `api-usage/07`, the common interface doubles the output tokens and makes none of the 117 truncations readable. In `api-usage/08`, raising the batch size cuts the calls from 120 to 10 while dropping accuracy to 0.0667 at the point of overrunning the context window. In `local-models-and-adaptation/03`, fine-tuning with sixty examples does not beat the prompt scaffolding. In `local-models-and-adaptation/05`, distillation takes the counting task from 0.5833 to 0.3667. In eight separate places the scaffolding paid for being added. In twenty-six lessons the call was treated as a network operation and the cost of every piece of scaffolding was written down. One thing was never asked: the **information** given to the model was always put into the prompt **by hand**. The document corpus existed as fiction, but which **three** of a corpus of a thousand documents would go into that prompt, against **which base** that selection would be measured and what the cost of choosing the wrong three was were not counted even once. The next course turns this selection into an object of measurement.