--- title: 'Latency Optimization' source: 'https://academia.sh/en/courses/model-apis/latency-optimization' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines latency as a latency class and a step count rather than raw time, and tells the four classes apart.' - 'Compares the small context, the large context and the two-stage setup on the same workload by accuracy and wait.' - 'Calculates the share of requests that escalate in the two-stage setup, the wait it saves and the accuracy it costs.' - 'Shows with numbers the price at which the parallel and batch setups change latency.' --- # Latency Optimization Comparing the same 240 fictional requests under five setups by latency class and accuracy: the small context holds the wait at 1.0000 steps in the single-call class but its accuracy is 0.7750, while the large context in the same class makes the user wait 3.0000 steps and gives 0.9292. The two-stage setup moves 0.2500 of the requests to the chain, cuts the wait to 1.7500 and input tokens to 19716, and loses only 0.0125 of accuracy. The parallel setup pulls the wait down to 1.5000 but at 47352 tokens costs more than the large context. Sending the second stage in batches raises the wait to 3.5542 without changing accuracy or tokens: a batch call is not a latency mechanism. The previous lesson counted tokens and showed that the cache could cut them up to fifteenfold. No column of that table measured what the user **waits** for. Yet in a support assistant this is the most visible measure, and the cache is not the only lever that determines it. Wait cannot be written down as raw time: the same code gives a different number on another machine. What can be written down is the **structure**. How many sequential calls does the response of a request wait on, are those calls made at the same time or one after another, and how large is the input token count of each call. This lesson compares three setups — the small context, the large context and the two-stage call — on these two axes and writes each one's **accuracy** beside it. Then it adds two more setups, because there are two more ways to lower latency and neither of them is free. - **PQ17.** The code is a **simulator**; no real endpoint is called. The pool, the questions and the misleading extra sentences are **fictional**, the seed is **20260218**. - **PQ18.** Wait is counted in **steps**. A call takes `1 + input // 60` steps: a longer input makes the user wait longer. - **PQ19. There are four latency classes.** **Single call**: one call is waited on for one request. **Chain**: the second call waits for the response of the first. **Parallel**: two calls are sent at the same time, and the wait is that of the one that finishes later. **Batch**: the call is part of a batch and is waited on until the batch fills up. - **PQ20.** The cheap **prefilter** is outside the model, spends no tokens at all and picks a single document from the pool. Because it counts fragments, it is caught by misleading extra sentences. - **PQ21.** When the whole pool is placed in the prompt, the model can compare the documents, so it matches the keyphrase as a whole; this is the source of the large context's accuracy advantage. - **PQ22.** In the two-stage setup the first stage also reports whether the given document is **sufficient**. This report is wrong **one time in ten**; only the requests it calls insufficient escalate to the second stage. - **PQ23. A correct response** is a response that rests on the right document and whose label is right. - **PQ24.** The workload is 240 requests over 120 steps, two per step; thirty questions are cycled through in order. - **PQ25.** In the batch setup the batch size is eight; the batch left incomplete at the end of the run is sent at the last step. ## Context Size Determines Both Wait and Accuracy First the pool, the questions and the two selection routines are built: the cheap prefilter, and the case where the whole pool is in the prompt. Then the two are placed side by side on one easy and one misleading question. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called. The document pool, the questions # and the misleading extra sentences are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF UNIT = 60 # input tokens that one step carries def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] DOCUMENTS = [("FT1", "fatura tutari", "fatura tutari hesap donemi ve gecikme farki"), ("FT2", "fatura odemesi", "fatura odemesi yontemleri ve borc gorunumu"), ("AR1", "ariza kaydi", "ariza kaydi acma ve ekip yonlendirme yordami"), ("AR2", "baglanti bozuklugu", "baglanti bozuk oldugunda uygulanan denetim"), ("SY1", "sayac endeksi", "sayac endeks okumasi ve duzeltme yordami"), ("SY2", "sayac degisimi", "sayac degisimi randevu kurallari ve sure taahhudu"), ("TA1", "tarife basamagi", "tarife basamaklari ve abone tipine gore ucret"), ("TA2", "tarife degisikligi", "tarife degisikligi basvurusu ve gecerlilik"), ("KS1", "planli kesinti", "planli kesinti duyurusu ve takip kurallari"), ("KS2", "mahalle kesintisi", "mahallede kesinti bildirimi ve donus suresi")] DOCUMENTS = [{"code": k, "topic": TOPICS[i // 2], "keyphrase": a, "text": m} for i, (k, a, m) in enumerate(DOCUMENTS)] TEMPLATES = ["{a} konusunda ne yapmam gerekiyor", "{a} icin gecerli kural nedir", "{a} ile ilgili son durum ne"] QUESTIONS = [] for i, b in enumerate(DOCUMENTS): for j, k in enumerate(TEMPLATES): m = k.format(a=b["keyphrase"]) if (i + j) % 3 == 1: # misleading extra sentence m += " " + DOCUMENTS[(i + 4) % 10]["text"].split(" ve ")[-1] QUESTIONS.append({"doc": i, "topic": b["topic"], "text": m}) FULL = " ; ".join(f"{b['code']} {b['text']}" for b in DOCUMENTS) INSTRUCTION = "Soruyu verilen belgeye dayanarak yanitla . Etiket ve belge kodu yaz . Belge: " def select(question, full): """full=False: cheap prefilter (fragment counting). full=True: the pool is in the prompt, so the model can match the keyphrase as a whole.""" st, p = tokenize(question["text"]), [] for b in DOCUMENTS: x = sum(t in tokenize(b["text"]) for t in st) if full and all(t in st for t in tokenize(b["keyphrase"])): x += 8.0 p.append(x) ranked = sorted(range(10), key=lambda i: (-p[i], i)) return ranked[0], p[ranked[0]] - p[ranked[1]] def call(question, context, seed=SEED, order=0): """FICTIONAL endpoint. Reads the given context; input tokens grow with the context.""" tok = tokenize(INSTRUCTION + context + " . Soru: " + question["text"]) r = rng((seed ^ trace_id(question["text"])) + 7919 * order) return (question["topic"] if r() > 0.08 else TOPICS[int(r() * 5)]), len(tok) def wait_steps(g): return 1 + g // UNIT print("documents", len(DOCUMENTS), ", questions", len(QUESTIONS), ", full pool", len(tokenize(FULL)), "tokens") z = sum(select(s, False)[0] == s["doc"] for s in QUESTIONS) g = sum(select(s, True)[0] == s["doc"] for s in QUESTIONS) print(f"reached the right document: prefilter {z}/{len(QUESTIONS)} , " f"full pool {g}/{len(QUESTIONS)}") for j in (0, 8): s0 = QUESTIONS[j] i0, i1 = select(s0, False)[0], select(s0, True)[0] gs, gl = call(s0, DOCUMENTS[i0]["text"])[1], call(s0, FULL)[1] print(f"\nquestion : {s0['text']}") print(f" correct {DOCUMENTS[s0['doc']]['code']} , prefilter {DOCUMENTS[i0]['code']} , " f"full pool {DOCUMENTS[i1]['code']}") print(f" small context {gs} tokens -> wait {wait_steps(gs)} ; " f"large context {gl} tokens -> wait {wait_steps(gl)}") ``` ``` documents 10 , questions 30 , full pool 121 tokens reached the right document: prefilter 25/30 , full pool 30/30 question : fatura tutari konusunda ne yapmam gerekiyor correct FT1 , prefilter FT1 , full pool FT1 small context 44 tokens -> wait 1 ; large context 154 tokens -> wait 3 question : ariza kaydi ile ilgili son durum ne abone tipine gore ucret correct AR1 , prefilter TA1 , full pool AR1 small context 46 tokens -> wait 1 ; large context 156 tokens -> wait 3 ``` Comparing the two questions shows where the trade-off comes from. On the easy question both paths reach the same document, and the small context does the same job with **one third** of the wait. On the misleading question, the extra sentence carries the words of a tariff document; the prefilter, which counts fragments, says `TA1`, although the right document is `AR1`. With the whole pool in the prompt this mistake does not happen, because the model can match the keyphrase as a whole. Out of thirty questions the prefilter links 25 and the full pool links 30 to the right document. On the wait side the table is just as clear: the small context takes one step with 44 tokens, the large context three steps with 154 tokens. **Both are in the same latency class** — single call — but the wait is three times as long. The class alone is not a measure; within the class the wait grows with the input tokens. ## Class and Accuracy Table of the Five Setups The five setups are run on the same workload. In the two-stage setup the first stage is the cheap call, and it also reports whether the given document is sufficient; those that say insufficient escalate to the second stage. In the parallel setup two calls are sent at the same time. In the last one, the escalating requests are collected in batches of eight. ```python def workload(): """240 requests over 120 steps, two per step; the questions are cycled in order.""" return [(a, QUESTIONS[(2 * a + j) % len(QUESTIONS)]) for a in range(120) for j in (0, 1)] def stage1(question, seed, order): """Cheap stage: the single document the prefilter gives. The model also reports whether the document is sufficient; this report is wrong one time in ten.""" i, _ = select(question, False) e, g = call(question, DOCUMENTS[i]["text"], seed, order) r = rng((seed ^ trace_id(question["text"])) + 31 * order + 5) sufficient = (i == question["doc"]) != (r() < 0.10) return i, e, g, sufficient def stage2(question, seed, order): i, _ = select(question, True) e, g = call(question, FULL, seed, order) return i, e, g def run(setup, seed=SEED): s = dict.fromkeys(("correct", "input", "calls", "wait", "escalated"), 0) classes, pending, n = {}, [], 0 for a, question in workload(): n += 1 i, e, g, sufficient = stage1(question, seed, n) if setup == "small": s["input"] += g s["calls"] += 1 s["wait"] += wait_steps(g) classes["single call"] = classes.get("single call", 0) + 1 s["correct"] += i == question["doc"] and e == question["topic"] continue if setup == "large": i2, e2, g2 = stage2(question, seed, n) s["input"] += g2 s["calls"] += 1 s["wait"] += wait_steps(g2) classes["single call"] = classes.get("single call", 0) + 1 s["correct"] += i2 == question["doc"] and e2 == question["topic"] continue if setup == "parallel": # two calls at the same time i2, e2, g2 = stage2(question, seed, n) s["input"] += g + g2 s["calls"] += 2 if sufficient: s["wait"] += wait_steps(g) classes["parallel-early"] = classes.get("parallel-early", 0) + 1 s["correct"] += i == question["doc"] and e == question["topic"] else: s["wait"] += max(wait_steps(g), wait_steps(g2)) classes["parallel"] = classes.get("parallel", 0) + 1 s["correct"] += i2 == question["doc"] and e2 == question["topic"] continue s["input"] += g # two-stage setups s["calls"] += 1 if sufficient: s["wait"] += wait_steps(g) classes["single call"] = classes.get("single call", 0) + 1 s["correct"] += i == question["doc"] and e == question["topic"] continue s["escalated"] += 1 i2, e2, g2 = stage2(question, seed, n) s["input"] += g2 s["calls"] += 1 s["correct"] += i2 == question["doc"] and e2 == question["topic"] if setup == "two": s["wait"] += wait_steps(g) + wait_steps(g2) classes["chain"] = classes.get("chain", 0) + 1 else: # second stage sent in a batch pending.append((a, wait_steps(g), wait_steps(g2))) if len(pending) == 8: for a0, b1, b2 in pending: s["wait"] += b1 + (a - a0) + b2 classes["batch"] = classes.get("batch", 0) + 8 pending = [] for a0, b1, b2 in pending: # batch left over at the end of the run s["wait"] += b1 + (119 - a0) + b2 classes["batch"] = classes.get("batch", 0) + 1 return (s["correct"] / n, s["wait"] / n, s["input"], s["calls"], s["escalated"] / n, " , ".join(f"{k} {v / n:.4f}" for k, v in sorted(classes.items()))) SETUPS = (("small context", "small"), ("large context", "large"), ("two-stage", "two"), ("parallel", "parallel"), ("two-stage + batch", "batch")) print("setup correct wait input calls escalated") for name, k in SETUPS: a = run(k) print(f"{name:<19} {a[0]:.4f} {a[1]:>7.4f} {a[2]:>5} {a[3]:>5} {a[4]:>9.4f}") print("\nlatency class distribution:") for name, k in SETUPS: print(f" {name:<19} {run(k)[5]}") print("\nsecond run (seed 20260219), correct / wait / escalated:") for name, k in SETUPS: a = run(k, SEED + 1) print(f" {name:<19} {a[0]:.4f} / {a[1]:.4f} / {a[4]:.4f}") ``` ``` setup correct wait input calls escalated small context 0.7750 1.0000 10376 240 0.0000 large context 0.9292 3.0000 36976 240 0.0000 two-stage 0.9167 1.7500 19716 300 0.2500 parallel 0.9167 1.5000 47352 480 0.0000 two-stage + batch 0.9167 3.5542 19716 300 0.2500 latency class distribution: small context single call 1.0000 large context single call 1.0000 two-stage chain 0.2500 , single call 0.7500 parallel parallel 0.2500 , parallel-early 0.7500 two-stage + batch batch 0.2500 , single call 0.7500 second run (seed 20260219), correct / wait / escalated: small context 0.7792 / 1.0000 / 0.0000 large context 0.9250 / 3.0000 / 0.0000 two-stage 0.9083 / 1.6375 / 0.2125 parallel 0.9083 / 1.4250 / 0.0000 two-stage + batch 0.9083 / 3.3583 / 0.2125 ``` The first two rows are the ends of the trade-off. The small context is the fastest, the cheapest and the most wrong: a wait of 1.0000 steps, input of 10376, accuracy **0.7750**. The large context raises accuracy to **0.9292** and in return triples the wait and multiplies input tokens by **3.56**. The run-to-run difference is 0.0042 in both rows; the accuracy gap of 0.1542 between them is more than thirty times this difference, that is, it is measured beyond dispute. The third row is the two-stage setup and the best-balanced row of the table. **0.2500** of the requests escalate to the second stage; the remaining three quarters finish in a single call. The result: a wait of 1.7500, that is, **0.5833** of the large context; input tokens of 19716, that is, **0.5332** of it. The price paid has two items. The first is 60 extra calls (240 up to 300). The second is accuracy: from 0.9292 to 0.9167, that is, **0.0125**. This loss is about three times the run-to-run difference and counts as measured, but it stays small next to the small context's loss of 0.1542. The report on escalation being wrong one time in ten lands exactly here. ## What You Buy Latency With The fourth row reaches the same accuracy with a shorter wait: 1.5000 steps. The reason is written in the class distribution — there is no chain here, the two calls go **at the same time**, so an escalating request does not pay a second time for the one step of the cheap call. The gain is 0.2500 steps. The cost is in the cost column: 480 calls and **47352** input tokens. This is 2.40 times the two-stage setup and **1.28 times even the large context**. The parallel setup, climbing above even the most expensive setup, buys only wait, without gaining any accuracy. There is always a way to shorten latency; the question to ask is at the expense of which column. The last row is the third example in this topic of the course's second claim. Sending the escalating requests in batches of eight does not change accuracy (0.9167) and does not change input tokens (19716) — even the number of calls is the same. The only thing that changes is the wait, and it has gone up **from 1.7500 to 3.5542**. The batch setup is the slowest row of the table; it is slower even than the single call of the large context. The reason is plain: while the batch is waiting to fill, the request that entered first sits there for nothing. A batch call is a throughput mechanism, not a latency mechanism; opened without looking at the latency measure, it doubles that measure. The second-run rows confirm this reading. The escalating share rises from 0.2500 to 0.2125, and the wait numbers rise with it (1.6375 instead of 1.7500, 3.3583 instead of 3.5542). That is, the two-stage setup's wait gain **depends on the escalating share**, and that share moves from run to run. The only constant is the ordering: parallel gives the shortest wait in every run, batch the longest. ## Summary - Wait is written not as raw time but as a latency class and a step count; the small and the large context are in the same class (single call), but their waits are 1.0000 and 3.0000 steps. - The cheap prefilter links 25 of thirty questions to the right document, the full pool links 30; the small context's accuracy is 0.7750, the large context's 0.9292. - The two-stage setup, by moving 0.2500 of the requests to the chain, cuts the wait to 0.5833 of the large context's and input tokens to 0.5332 of it, and gives up 0.0125 of accuracy. - The parallel setup pulls the wait down to 1.5000 but at 47352 input tokens rises to 1.28 times the large context: only wait is bought, without gaining accuracy. - Sending the second stage in batches raises the wait from 1.7500 to 3.5542 without changing accuracy, tokens or the number of calls at all; a batch call is not a latency mechanism. - The two-stage setup's wait gain depends on the escalating share, and that share moves between 0.2500 and 0.2125 from run to run. ## Next Step In three lessons resilience, cache and latency were measured separately, and a token total was written beside each: 10376, 19716, 47352. These numbers are enough to compare one setup with another, but they do not answer a single question — **among whom this total was shared.** Did all two hundred and forty requests spend as much as the mean, or did a few requests swallow a significant part of the total. A budget is not kept by the mean: the most expensive one percent of a line that stays below the mean can exceed the budget by itself, and this does not show up in any table that looks at totals. The next lesson writes cost **per request** and looks at the tail of the distribution.