Latency Optimization

The previous lesson counted tokens and showed that the cache could cut them up to fifteenfold. No column of that table measured what the user waits for. Yet in a support assistant this is the most visible measure, and the cache is not the only lever that determines it.

Wait cannot be written down as raw time: the same code gives a different number on another machine. What can be written down is the structure. How many sequential calls does the response of a request wait on, are those calls made at the same time or one after another, and how large is the input token count of each call. This lesson compares three setups — the small context, the large context and the two-stage call — on these two axes and writes each one’s accuracy beside it. Then it adds two more setups, because there are two more ways to lower latency and neither of them is free.

• PQ17. The code is a simulator; no real endpoint is called. The pool, the questions and the misleading extra sentences are fictional, the seed is 20260218.
• PQ18. Wait is counted in steps. A call takes 1 + input // 60 steps: a longer input makes the user wait longer.
• PQ19. There are four latency classes. Single call: one call is waited on for one request. Chain: the second call waits for the response of the first. Parallel: two calls are sent at the same time, and the wait is that of the one that finishes later. Batch: the call is part of a batch and is waited on until the batch fills up.
• PQ20. The cheap prefilter is outside the model, spends no tokens at all and picks a single document from the pool. Because it counts fragments, it is caught by misleading extra sentences.
• PQ21. When the whole pool is placed in the prompt, the model can compare the documents, so it matches the keyphrase as a whole; this is the source of the large context’s accuracy advantage.
• PQ22. In the two-stage setup the first stage also reports whether the given document is sufficient. This report is wrong one time in ten; only the requests it calls insufficient escalate to the second stage.
• PQ23. A correct response is a response that rests on the right document and whose label is right.
• PQ24. The workload is 240 requests over 120 steps, two per step; thirty questions are cycled through in order.
• PQ25. In the batch setup the batch size is eight; the batch left incomplete at the end of the run is sent at the last step.

Context Size Determines Both Wait and Accuracy

First the pool, the questions and the two selection routines are built: the cheap prefilter, and the case where the whole pool is in the prompt. Then the two are placed side by side on one easy and one misleading question.

# SIMULATOR -- FICTIONAL. No real endpoint is called. The document pool, the questions
# and the misleading extra sentences are fictional, seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
UNIT = 60                        # input tokens that one step carries


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
DOCUMENTS = [("FT1", "fatura tutari", "fatura tutari hesap donemi ve gecikme farki"),
             ("FT2", "fatura odemesi", "fatura odemesi yontemleri ve borc gorunumu"),
             ("AR1", "ariza kaydi", "ariza kaydi acma ve ekip yonlendirme yordami"),
             ("AR2", "baglanti bozuklugu", "baglanti bozuk oldugunda uygulanan denetim"),
             ("SY1", "sayac endeksi", "sayac endeks okumasi ve duzeltme yordami"),
             ("SY2", "sayac degisimi", "sayac degisimi randevu kurallari ve sure taahhudu"),
             ("TA1", "tarife basamagi", "tarife basamaklari ve abone tipine gore ucret"),
             ("TA2", "tarife degisikligi", "tarife degisikligi basvurusu ve gecerlilik"),
             ("KS1", "planli kesinti", "planli kesinti duyurusu ve takip kurallari"),
             ("KS2", "mahalle kesintisi", "mahallede kesinti bildirimi ve donus suresi")]
DOCUMENTS = [{"code": k, "topic": TOPICS[i // 2], "keyphrase": a, "text": m}
             for i, (k, a, m) in enumerate(DOCUMENTS)]
TEMPLATES = ["{a} konusunda ne yapmam gerekiyor",
             "{a} icin gecerli kural nedir",
             "{a} ile ilgili son durum ne"]
QUESTIONS = []
for i, b in enumerate(DOCUMENTS):
    for j, k in enumerate(TEMPLATES):
        m = k.format(a=b["keyphrase"])
        if (i + j) % 3 == 1:                       # misleading extra sentence
            m += " " + DOCUMENTS[(i + 4) % 10]["text"].split(" ve ")[-1]
        QUESTIONS.append({"doc": i, "topic": b["topic"], "text": m})
FULL = " ; ".join(f"{b['code']} {b['text']}" for b in DOCUMENTS)
INSTRUCTION = "Soruyu verilen belgeye dayanarak yanitla . Etiket ve belge kodu yaz . Belge: "


def select(question, full):
    """full=False: cheap prefilter (fragment counting). full=True: the pool is in the
    prompt, so the model can match the keyphrase as a whole."""
    st, p = tokenize(question["text"]), []
    for b in DOCUMENTS:
        x = sum(t in tokenize(b["text"]) for t in st)
        if full and all(t in st for t in tokenize(b["keyphrase"])):
            x += 8.0
        p.append(x)
    ranked = sorted(range(10), key=lambda i: (-p[i], i))
    return ranked[0], p[ranked[0]] - p[ranked[1]]


def call(question, context, seed=SEED, order=0):
    """FICTIONAL endpoint. Reads the given context; input tokens grow with the context."""
    tok = tokenize(INSTRUCTION + context + " . Soru: " + question["text"])
    r = rng((seed ^ trace_id(question["text"])) + 7919 * order)
    return (question["topic"] if r() > 0.08 else TOPICS[int(r() * 5)]), len(tok)


def wait_steps(g):
    return 1 + g // UNIT


print("documents", len(DOCUMENTS), ", questions", len(QUESTIONS), ", full pool",
      len(tokenize(FULL)), "tokens")
z = sum(select(s, False)[0] == s["doc"] for s in QUESTIONS)
g = sum(select(s, True)[0] == s["doc"] for s in QUESTIONS)
print(f"reached the right document: prefilter {z}/{len(QUESTIONS)} , "
      f"full pool {g}/{len(QUESTIONS)}")
for j in (0, 8):
    s0 = QUESTIONS[j]
    i0, i1 = select(s0, False)[0], select(s0, True)[0]
    gs, gl = call(s0, DOCUMENTS[i0]["text"])[1], call(s0, FULL)[1]
    print(f"\nquestion : {s0['text']}")
    print(f"  correct {DOCUMENTS[s0['doc']]['code']} , prefilter {DOCUMENTS[i0]['code']} , "
          f"full pool {DOCUMENTS[i1]['code']}")
    print(f"  small context {gs} tokens -> wait {wait_steps(gs)} ; "
          f"large context {gl} tokens -> wait {wait_steps(gl)}")

documents 10 , questions 30 , full pool 121 tokens
reached the right document: prefilter 25/30 , full pool 30/30

question : fatura tutari konusunda ne yapmam gerekiyor
  correct FT1 , prefilter FT1 , full pool FT1
  small context 44 tokens -> wait 1 ; large context 154 tokens -> wait 3

question : ariza kaydi ile ilgili son durum ne abone tipine gore ucret
  correct AR1 , prefilter TA1 , full pool AR1
  small context 46 tokens -> wait 1 ; large context 156 tokens -> wait 3

Comparing the two questions shows where the trade-off comes from. On the easy question both paths reach the same document, and the small context does the same job with one third of the wait. On the misleading question, the extra sentence carries the words of a tariff document; the prefilter, which counts fragments, says TA1, although the right document is AR1. With the whole pool in the prompt this mistake does not happen, because the model can match the keyphrase as a whole. Out of thirty questions the prefilter links 25 and the full pool links 30 to the right document.

On the wait side the table is just as clear: the small context takes one step with 44 tokens, the large context three steps with 154 tokens. Both are in the same latency class — single call — but the wait is three times as long. The class alone is not a measure; within the class the wait grows with the input tokens.

Class and Accuracy Table of the Five Setups

The five setups are run on the same workload. In the two-stage setup the first stage is the cheap call, and it also reports whether the given document is sufficient; those that say insufficient escalate to the second stage. In the parallel setup two calls are sent at the same time. In the last one, the escalating requests are collected in batches of eight.

def workload():
    """240 requests over 120 steps, two per step; the questions are cycled in order."""
    return [(a, QUESTIONS[(2 * a + j) % len(QUESTIONS)]) for a in range(120) for j in (0, 1)]


def stage1(question, seed, order):
    """Cheap stage: the single document the prefilter gives. The model also reports
    whether the document is sufficient; this report is wrong one time in ten."""
    i, _ = select(question, False)
    e, g = call(question, DOCUMENTS[i]["text"], seed, order)
    r = rng((seed ^ trace_id(question["text"])) + 31 * order + 5)
    sufficient = (i == question["doc"]) != (r() < 0.10)
    return i, e, g, sufficient


def stage2(question, seed, order):
    i, _ = select(question, True)
    e, g = call(question, FULL, seed, order)
    return i, e, g


def run(setup, seed=SEED):
    s = dict.fromkeys(("correct", "input", "calls", "wait", "escalated"), 0)
    classes, pending, n = {}, [], 0
    for a, question in workload():
        n += 1
        i, e, g, sufficient = stage1(question, seed, n)
        if setup == "small":
            s["input"] += g
            s["calls"] += 1
            s["wait"] += wait_steps(g)
            classes["single call"] = classes.get("single call", 0) + 1
            s["correct"] += i == question["doc"] and e == question["topic"]
            continue
        if setup == "large":
            i2, e2, g2 = stage2(question, seed, n)
            s["input"] += g2
            s["calls"] += 1
            s["wait"] += wait_steps(g2)
            classes["single call"] = classes.get("single call", 0) + 1
            s["correct"] += i2 == question["doc"] and e2 == question["topic"]
            continue
        if setup == "parallel":                        # two calls at the same time
            i2, e2, g2 = stage2(question, seed, n)
            s["input"] += g + g2
            s["calls"] += 2
            if sufficient:
                s["wait"] += wait_steps(g)
                classes["parallel-early"] = classes.get("parallel-early", 0) + 1
                s["correct"] += i == question["doc"] and e == question["topic"]
            else:
                s["wait"] += max(wait_steps(g), wait_steps(g2))
                classes["parallel"] = classes.get("parallel", 0) + 1
                s["correct"] += i2 == question["doc"] and e2 == question["topic"]
            continue
        s["input"] += g                                # two-stage setups
        s["calls"] += 1
        if sufficient:
            s["wait"] += wait_steps(g)
            classes["single call"] = classes.get("single call", 0) + 1
            s["correct"] += i == question["doc"] and e == question["topic"]
            continue
        s["escalated"] += 1
        i2, e2, g2 = stage2(question, seed, n)
        s["input"] += g2
        s["calls"] += 1
        s["correct"] += i2 == question["doc"] and e2 == question["topic"]
        if setup == "two":
            s["wait"] += wait_steps(g) + wait_steps(g2)
            classes["chain"] = classes.get("chain", 0) + 1
        else:                                          # second stage sent in a batch
            pending.append((a, wait_steps(g), wait_steps(g2)))
            if len(pending) == 8:
                for a0, b1, b2 in pending:
                    s["wait"] += b1 + (a - a0) + b2
                classes["batch"] = classes.get("batch", 0) + 8
                pending = []
    for a0, b1, b2 in pending:                         # batch left over at the end of the run
        s["wait"] += b1 + (119 - a0) + b2
        classes["batch"] = classes.get("batch", 0) + 1
    return (s["correct"] / n, s["wait"] / n, s["input"], s["calls"], s["escalated"] / n,
            " , ".join(f"{k} {v / n:.4f}" for k, v in sorted(classes.items())))


SETUPS = (("small context", "small"), ("large context", "large"),
          ("two-stage", "two"), ("parallel", "parallel"),
          ("two-stage + batch", "batch"))
print("setup              correct     wait  input  calls  escalated")
for name, k in SETUPS:
    a = run(k)
    print(f"{name:<19} {a[0]:.4f}  {a[1]:>7.4f}  {a[2]:>5}  {a[3]:>5}  {a[4]:>9.4f}")
print("\nlatency class distribution:")
for name, k in SETUPS:
    print(f"  {name:<19} {run(k)[5]}")
print("\nsecond run (seed 20260219), correct / wait / escalated:")
for name, k in SETUPS:
    a = run(k, SEED + 1)
    print(f"  {name:<19} {a[0]:.4f} / {a[1]:.4f} / {a[4]:.4f}")

setup              correct     wait  input  calls  escalated
small context       0.7750   1.0000  10376    240     0.0000
large context       0.9292   3.0000  36976    240     0.0000
two-stage           0.9167   1.7500  19716    300     0.2500
parallel            0.9167   1.5000  47352    480     0.0000
two-stage + batch   0.9167   3.5542  19716    300     0.2500

latency class distribution:
  small context       single call 1.0000
  large context       single call 1.0000
  two-stage           chain 0.2500 , single call 0.7500
  parallel            parallel 0.2500 , parallel-early 0.7500
  two-stage + batch   batch 0.2500 , single call 0.7500

second run (seed 20260219), correct / wait / escalated:
  small context       0.7792 / 1.0000 / 0.0000
  large context       0.9250 / 3.0000 / 0.0000
  two-stage           0.9083 / 1.6375 / 0.2125
  parallel            0.9083 / 1.4250 / 0.0000
  two-stage + batch   0.9083 / 3.3583 / 0.2125

The first two rows are the ends of the trade-off. The small context is the fastest, the cheapest and the most wrong: a wait of 1.0000 steps, input of 10376, accuracy 0.7750. The large context raises accuracy to 0.9292 and in return triples the wait and multiplies input tokens by 3.56. The run-to-run difference is 0.0042 in both rows; the accuracy gap of 0.1542 between them is more than thirty times this difference, that is, it is measured beyond dispute.

The third row is the two-stage setup and the best-balanced row of the table. 0.2500 of the requests escalate to the second stage; the remaining three quarters finish in a single call. The result: a wait of 1.7500, that is, 0.5833 of the large context; input tokens of 19716, that is, 0.5332 of it. The price paid has two items. The first is 60 extra calls (240 up to 300). The second is accuracy: from 0.9292 to 0.9167, that is, 0.0125. This loss is about three times the run-to-run difference and counts as measured, but it stays small next to the small context’s loss of 0.1542. The report on escalation being wrong one time in ten lands exactly here.

What You Buy Latency With

The fourth row reaches the same accuracy with a shorter wait: 1.5000 steps. The reason is written in the class distribution — there is no chain here, the two calls go at the same time, so an escalating request does not pay a second time for the one step of the cheap call. The gain is 0.2500 steps. The cost is in the cost column: 480 calls and 47352 input tokens. This is 2.40 times the two-stage setup and 1.28 times even the large context. The parallel setup, climbing above even the most expensive setup, buys only wait, without gaining any accuracy. There is always a way to shorten latency; the question to ask is at the expense of which column.

The last row is the third example in this topic of the course’s second claim. Sending the escalating requests in batches of eight does not change accuracy (0.9167) and does not change input tokens (19716) — even the number of calls is the same. The only thing that changes is the wait, and it has gone up from 1.7500 to 3.5542. The batch setup is the slowest row of the table; it is slower even than the single call of the large context. The reason is plain: while the batch is waiting to fill, the request that entered first sits there for nothing. A batch call is a throughput mechanism, not a latency mechanism; opened without looking at the latency measure, it doubles that measure.

The second-run rows confirm this reading. The escalating share rises from 0.2500 to 0.2125, and the wait numbers rise with it (1.6375 instead of 1.7500, 3.3583 instead of 3.5542). That is, the two-stage setup’s wait gain depends on the escalating share, and that share moves from run to run. The only constant is the ordering: parallel gives the shortest wait in every run, batch the longest.

Summary

• Wait is written not as raw time but as a latency class and a step count; the small and the large context are in the same class (single call), but their waits are 1.0000 and 3.0000 steps.
• The cheap prefilter links 25 of thirty questions to the right document, the full pool links 30; the small context’s accuracy is 0.7750, the large context’s 0.9292.
• The two-stage setup, by moving 0.2500 of the requests to the chain, cuts the wait to 0.5833 of the large context’s and input tokens to 0.5332 of it, and gives up 0.0125 of accuracy.
• The parallel setup pulls the wait down to 1.5000 but at 47352 input tokens rises to 1.28 times the large context: only wait is bought, without gaining accuracy.
• Sending the second stage in batches raises the wait from 1.7500 to 3.5542 without changing accuracy, tokens or the number of calls at all; a batch call is not a latency mechanism.
• The two-stage setup’s wait gain depends on the escalating share, and that share moves between 0.2500 and 0.2125 from run to run.

Next Step

In three lessons resilience, cache and latency were measured separately, and a token total was written beside each: 10376, 19716, 47352. These numbers are enough to compare one setup with another, but they do not answer a single question — among whom this total was shared. Did all two hundred and forty requests spend as much as the mean, or did a few requests swallow a significant part of the total. A budget is not kept by the mean: the most expensive one percent of a line that stays below the mean can exceed the budget by itself, and this does not show up in any table that looks at totals. The next lesson writes cost per request and looks at the tail of the distribution.