Prompt Caching

In the previous lesson the heavy load made 295 calls, and every one of them began with the same instruction. Retry multiplied that repetition once more: every call sent as a correction tokenized the shared text once again and paid for it once again. The repetition itself had not been measured.

In a support assistant this repetition has two layers. The first is the prefix of the prompt: the instruction and the field team’s fictional document pool are the same in every request and take up most of the prompt. The second is the question itself: a few questions produce half of all traffic, and the same question is asked dozens of times a day. Each repetition has a remedy, and the two remedies are not the same thing. This lesson builds the two separately and measures them side by side: as the cache hit rate rises, tokens fall, but a stored response acquires an age, and when the document it rests on changes, the response is still fast, still cheap and now wrong.

• PQ9. The code is a simulator; no real endpoint is called. The pool, the questions and the update schedule are fictional, the seed is 20260218.
• PQ10. The pool is ten fictional documents; each document has a version, and two documents are updated at each of four separate steps.
• PQ11. The prompt has two parts: the prefix, identical in every request (the instruction and the pool), and the question itself.
• PQ12. The prefix cache is keyed by content. If the prefix is identical to the previous call’s, it counts as a hit and ten percent of the prefix tokens are paid. If the pool changes, the prefix changes too.
• PQ13. The response cache is keyed by question and has a lifetime. A hit within the lifetime makes no call and spends no tokens.
• PQ14. A stale response is a response that comes back from the cache and whose underlying document’s version is no longer current.
• PQ15. A correct response is a response whose label is correct and that is not stale; this is the product measure of this lesson.
• PQ16. The workload is 240 requests over 120 steps, two per step; the question distribution is skewed.

Two Separate Caches, Two Separate Keys

First the pool and the questions are built, then one document is updated and the answer to the same question is checked. The distinction becomes visible here: the prefix changes by itself when the pool changes; a stored response does not.

# SIMULATOR -- FICTIONAL. No real endpoint is called. The document pool, the questions
# and the update schedule are fictional, seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
STEPS, PREFIX_DISCOUNT = 120, 0.10     # run length, share of a hit prefix that is paid


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
DOCUMENTS = [{"code": "FT1", "topic": "fatura", "keyphrase": "fatura tutari",
              "text": "fatura tutari hesap donemi ve gecikme farki kurallari"},
             {"code": "FT2", "topic": "fatura", "keyphrase": "fatura odemesi",
              "text": "fatura odemesi yontemleri ve borc gorunumu kurallari"},
             {"code": "AR1", "topic": "ariza", "keyphrase": "ariza kaydi",
              "text": "ariza kaydi acma ve ekip yonlendirme yordami"},
             {"code": "AR2", "topic": "ariza", "keyphrase": "baglanti bozuklugu",
              "text": "baglanti bozuk oldugunda uygulanan denetim adimlari"},
             {"code": "SY1", "topic": "sayac", "keyphrase": "sayac endeksi",
              "text": "sayac endeks okumasi ve duzeltme yordami"},
             {"code": "SY2", "topic": "sayac", "keyphrase": "sayac degisimi",
              "text": "sayac degisimi randevu kurallari ve sure taahhudu"},
             {"code": "TA1", "topic": "tarife", "keyphrase": "tarife basamagi",
              "text": "tarife basamaklari ve abone tipine gore birim ucret"},
             {"code": "TA2", "topic": "tarife", "keyphrase": "tarife degisikligi",
              "text": "tarife degisikligi basvurusu ve gecerlilik kosullari"},
             {"code": "KS1", "topic": "kesinti", "keyphrase": "planli kesinti",
              "text": "planli kesinti duyurusu yayimlama ve takip kurallari"},
             {"code": "KS2", "topic": "kesinti", "keyphrase": "mahalle kesintisi",
              "text": "mahallede kesinti bildirimi ve tahmini donus suresi"}]
for b in DOCUMENTS:
    b["version"] = 1
QUESTIONS = [{"doc": i, "topic": b["topic"], "text": tpl.format(a=b["keyphrase"])}
             for i, b in enumerate(DOCUMENTS)
             for tpl in ("{a} konusunda ne yapmam gerekiyor",
                         "{a} icin gecerli kural nedir",
                         "{a} ile ilgili son durum ne")]
WEIGHTS = [1.0 / (i + 1) ** 1.0 for i in range(len(QUESTIONS))]
WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS]
UPDATES = {24: [6, 1], 48: [0, 8], 72: [2, 6], 96: [4, 7]}   # step -> document
CUES = {"fatur": "fatura", "tutar": "fatura", "borc": "fatura", "odeme": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "bagla": "ariza", "kayit": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife",
        "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti",
        "duyur": "kesinti", "mahal": "kesinti"}
INSTRUCTION = ("Asagidaki dagarcigi kullanarak soruyu yanitla . Yanitta once etiket , "
               "sonra dayandigin belge kodu ve surumu yaz . Dagarcik: ")
COUNTER = [0]


def prefix():
    return INSTRUCTION + " ; ".join(f"{b['code']} s{b['version']} {b['text']}"
                                    for b in DOCUMENTS) + " . Soru: "


def call(question, seed=SEED):
    """FICTIONAL endpoint. Reads the current version of the pool."""
    COUNTER[0] += 1
    prompt = prefix() + question["text"]
    tok = tokenize(prompt)
    p = {k: 0.15 for k in TOPICS}
    for t in tokenize(question["text"]):
        if CUES.get(t) in p:
            p[CUES[t]] += 1.0
    r = rng((seed ^ trace_id(question["text"])) + 7919 * COUNTER[0])
    e = max(sorted(p), key=lambda a: p[a]) if r() > 0.10 else TOPICS[int(r() * 5)]
    b = DOCUMENTS[question["doc"]]
    return {"label": e, "code": b["code"], "version": b["version"],
            "input": len(tok), "prefix": len(tokenize(prefix())),
            "output": len(tokenize(f"{e} {b['code']} s{b['version']}"))}


def workload(seed=SEED):
    """Two requests per step; the question distribution is skewed (a few dominate)."""
    r = rng(seed + 991)
    return [(a, QUESTIONS[pick(r(), WEIGHTS)]) for a in range(STEPS) for _ in range(2)]


for b in DOCUMENTS:
    b["version"] = 1
first = call(QUESTIONS[0])
print("first call :", {k: first[k] for k in ("label", "code", "version", "input", "prefix")})
DOCUMENTS[0]["version"] += 1                      # pool updated
second = call(QUESTIONS[0])
print("after the update:", {k: second[k] for k in ("code", "version", "input")})
print("stored response  :", first["code"], "s" + str(first["version"]),
      "-> stale" if first["version"] != DOCUMENTS[0]["version"] else "-> fresh")
DOCUMENTS[0]["version"] = 1
reqs = workload()
freq = {}
for _, q in reqs:
    freq[q["text"][:26]] = freq.get(q["text"][:26], 0) + 1
print(f"\nrequests {len(reqs)} , unique questions {len(freq)} , prefix {first['prefix']} tokens")
print("top three questions:", [(a, n) for a, n in sorted(freq.items(), key=lambda x: -x[1])[:3]])

first call : {'label': 'fatura', 'code': 'FT1', 'version': 1, 'input': 186, 'prefix': 175}
after the update: {'code': 'FT1', 'version': 2, 'input': 186}
stored response  : FT1 s1 -> stale

requests 240 , unique questions 29 , prefix 175 tokens
top three questions: [('fatura tutari konusunda ne', 56), ('fatura tutari icin gecerli', 41), ('fatura tutari ile ilgili s', 23)]

The first line gives the size of the repetition: the prompt is 186 tokens, and 175 of them are prefix. That is, 0.9409 of the tokens each request pays for are identical to the previous request’s. The second line shows that the call after the update reads the same document in its second version. The third line is the distinction itself: the stored response still rests on the first version and has become stale.

The question distribution matters too. The 240 requests correspond to only 29 distinct questions, and the three most frequent questions alone account for 120 requests. A response cache works only because of this skew; on a workload where every request is different, the response cache’s hit rate stays at zero.

Stale-Response Share as the Hit Rate Rises

Seven setups are run on the same workload: the no-cache baseline, the prefix cache only, the response cache with four different lifetimes, and both caches together.

def run(prefix_on, lifetime=0, seed=SEED):
    """prefix_on: is the prefix cache on . lifetime: step lifetime of the response cache."""
    for b in DOCUMENTS:
        b["version"] = 1
    COUNTER[0] = 0
    cache, last, prev_step = {}, [None], [-1]
    s = dict.fromkeys(("prefix_hits", "resp_hits", "input", "output", "stale",
                       "correct", "requests"), 0)
    for step, question in workload(seed):
        if step != prev_step[0]:
            prev_step[0] = step
            for i in UPDATES.get(step, []):
                DOCUMENTS[i]["version"] += 1
        s["requests"] += 1
        entry = cache.get(question["text"])
        if lifetime and entry and step - entry["step"] <= lifetime:
            s["resp_hits"] += 1
            stale = entry["version"] != DOCUMENTS[question["doc"]]["version"]
            s["stale"] += stale
            s["correct"] += entry["label"] == question["topic"] and not stale
            continue
        resp = call(question, seed)
        s["output"] += resp["output"]
        if prefix_on and last[0] == prefix():           # prefix is keyed by content
            s["prefix_hits"] += 1
            s["input"] += (round(resp["prefix"] * PREFIX_DISCOUNT)
                           + resp["input"] - resp["prefix"])
        else:
            s["input"] += resp["input"]
        last[0] = prefix()
        cache[question["text"]] = {"label": resp["label"], "version": resp["version"],
                                   "step": step}
        s["correct"] += resp["label"] == question["topic"]
    n, c = s["requests"], max(COUNTER[0], 1)
    return (s["resp_hits"] / n, s["prefix_hits"] / c, s["input"], s["input"] + s["output"],
            COUNTER[0], s["stale"] / n, s["correct"] / n)


SETUPS = [("no cache", 0, 0), ("prefix cache", 1, 0), ("response lifetime 8", 0, 8),
          ("response lifetime 24", 0, 24), ("response lifetime 64", 0, 64),
          ("response lifetime unlimited", 0, 10 ** 6),
          ("prefix + response lifetime 24", 1, 24)]
print("setup                         resp hit  prefix hit   input   total"
      "  calls   stale  correct")
for name, o, m in SETUPS:
    a = run(o, m)
    print(f"{name:<29} {a[0]:>8.4f}  {a[1]:>10.4f}  {a[2]:>6}  {a[3]:>6}  {a[4]:>5}"
          f"  {a[5]:>6.4f}  {a[6]:>7.4f}")
print("\nsecond run (seed 20260219), response hit / stale / correct:")
for name, o, m in SETUPS:
    a = run(o, m, SEED + 1)
    print(f"  {name:<29} {a[0]:.4f} / {a[5]:.4f} / {a[6]:.4f}")

setup                         resp hit  prefix hit   input   total  calls   stale  correct
no cache                        0.0000      0.0000   44367   45263    240  0.0000   0.9333
prefix cache                    0.0000      0.9792    7472    8368    240  0.0000   0.9333
response lifetime 8             0.4708      0.0000   23456   23917    127  0.0208   0.9208
response lifetime 24            0.6750      0.0000   14407   14686     78  0.0500   0.9292
response lifetime 64            0.8000      0.0000    8870    9042     48  0.1333   0.8625
response lifetime unlimited     0.8792      0.0000    5357    5461     29  0.4417   0.5542
prefix + response lifetime 24   0.6750      0.9359    2946    3225     78  0.0500   0.9292

second run (seed 20260219), response hit / stale / correct:
  no cache                      0.0000 / 0.0000 / 0.9083
  prefix cache                  0.0000 / 0.0000 / 0.9083
  response lifetime 8           0.5292 / 0.0292 / 0.8667
  response lifetime 24          0.6958 / 0.0833 / 0.8250
  response lifetime 64          0.8208 / 0.2167 / 0.6833
  response lifetime unlimited   0.9000 / 0.5000 / 0.4042
  prefix + response lifetime 24 0.6958 / 0.0833 / 0.8250

The second row is the cheapest gain of this lesson and has no catch. The prefix cache, with a 0.9792 hit rate, cuts input tokens from 44367 to 7472, that is, 5.94 times. The stale-response share is 0.0000, and the correct-response share does not change. The reason is in the key: the prefix is keyed by content. The moment the pool is updated, the prefix text changes too, the match breaks, and the next call is paid in full. A cache goes stale when its key comes apart from the content.

The response-cache rows are the opposite. The key is the question text, and the question does not change; what changes is the document the question rests on. With a lifetime of eight steps the hit rate is 0.4708 and the stale share 0.0208; with an unlimited lifetime the hit rate rises to 0.8792 and the stale share climbs to 0.4417. Calls fall from 240 to 29, and input tokens fall to 5357. From the cost side, the last row is the best row. From the product side, the correct-response share has fallen from 0.9333 to 0.5542: in about one of every two requests, the user is told a rule that is no longer valid.

Choosing the Lifetime Is an Accuracy Decision

Where the loss becomes measurable is read from the run-to-run difference. The no-cache baseline gives 0.9083 on the second seed, so the run-to-run difference is 0.0250. At a lifetime of eight the loss is 0.0125, at twenty-four 0.0041 — both below the run-to-run difference and counted as unmeasured. At a lifetime of sixty-four the loss is 0.0708 and at unlimited 0.3791; these are about three and fifteen times the run-to-run difference respectively, that is, measured. The decision can be read from here: under this update schedule, extending the lifetime up to twenty-four costs nothing, while raising it to sixty-four does not.

This reading carries a warning. In the second run the loss at lifetime twenty-four rises to 0.0833, that is, three times the run-to-run difference relative to its own baseline. The same setting is below the noise in one run and measurable in the other. This comes from twenty-four steps being close to the update interval: since the pool changes every twenty-four steps, fitting the lifetime exactly to that interval leaves the decision to the timing luck of the document update. The cache lifetime cannot be chosen independently of how often the data changes.

The last row is the setup in which the two caches work together, and it produces a number that neither gives alone: input tokens of 2946, that is, a 15.06-times reduction from the baseline, with a stale share identical to lifetime twenty-four’s, 0.0500. The same row holds a subtler effect: the prefix hit rate falls from 0.9792 to 0.9359. Because the response cache cuts the number of calls from 240 to 78, the matches broken by the four updates are now divided by a much smaller divisor. One cache improves the other’s hit rate by lowering it; the numbers of the two scaffoldings cannot be measured separately and added up.

Summary

• Of the prompt’s 186 tokens, 175 are shared by every request; which cache will pay off cannot be known until the size of the repetition is measured.
• The prefix cache is keyed by content, so it does not go stale: with a 0.9792 hit rate it cuts input from 44367 to 7472 and leaves the correct-response share unchanged.
• In the response cache, as the lifetime grows the hit rate rises from 0.4708 to 0.8792, while the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542.
• The run-to-run difference is 0.0250: the losses at lifetimes eight and twenty-four stay below this difference and count as unmeasured, while the losses at sixty-four and unlimited are measured.
• When the lifetime is chosen close to the update interval, the decision becomes unstable; the same setting loses nothing above the noise in one run and three times it in the other.
• The two caches together cut input to 2946, but the response cache lowers the prefix hit rate from 0.9792 to 0.9359: the numbers of the scaffoldings cannot be measured separately and added up.

Next Step

What this lesson gained is tokens. A response returned from the cache makes no call, so it also shortens the user’s wait, but this lesson did not count that side at all: the table carries only token, call and accuracy columns. Yet in a support assistant the most visible measure is waiting time, and what determines it is not the cache alone. The size of the context, whether the call is made alone or as part of a chain, and whether the expensive step runs on every request or only when needed each change the wait separately. The next lesson compares these three setups: which latency class the small context, the large context and the two-stage call sit in, and what each one’s accuracy is.